The Eclipse Metrics program relies on the excellent GrimoireLab toolchain for its data retrieval process.
(From GrimoireLab’s official website)
The aim of this project is to provide an open source toolkit for:
- Automatic and incremental data gathering from almost any tool (data source) related with contributing to Open Source development (source code management, issue tracking systems, forums, etc.)
- Automatic gathered data enrichment, merging duplicated identities, adding additional information about contributors affiliation, calculation delays, geographical data, etc.
- Data consumption and visualization, allowing filtering by time range, project, repository, contributor, etc.
The overall architecture is described below:
This website is regenerated every night from the OpenSearch database.
It should be noted however that the database itself may take up to a couple of days for e.g. a new commit to actually show up in the dashboard. Although GrimoireLab crawlers are executed daily, tasks take a substantial time, considering the huge amount of data to fetch from the various data sources. The data is then enriched, and identity management resolves and associates the various identities (across the data sources) of individuals and organisations.
Then, depending on the modification that was made (e.g. in the case of a metadata change), indices may need to be regenerated - which can take another few days.