Documentation - Known Limitations

This section describes known limitations of the process, and reasons why metrics should always be taken with a pinch of salt.

About metrics

Important

Metrics are a double-edged sword. They can be as useful as they can harm, depending on how they are used. Be cautious when presenting or using them.

Always think twice with metrics. As useful as they can be, they’re also the best way to kill your team’s motivation, go into a feedback loop, or otherwise entirely miss the point of an initiative. A very good example is the number of source lines of code (SLOC) – one of the most used, and correlated software metrics around. On the one hand, SLOC correlates with the number of bugs, the efficiency of a team, and maintainability. On the other hand, if used without any context, it can also counter basic best practices like refactoring, execution and resources efficiency, or even code readability and maintainability. Same applies for number of issues - do they reflect the activity of the project (nobody uses it so nobody submits issues - and more people using it implies getting more issues) or its quality (the code is so great that it contains almost no bug)?

We recommend having some good reading and look at the established best practices about metrics programs before trying to leverage these numbers and charts:

The Goal-Question-Metric approach is a method developed by Victor R. Basili and the NASA in 1984, to collect valid software engineering data.
A view on developer productivity: introducing SPACE Metrics.
Martin Fowler’s website offers some good insights and advice about various software metrics.

Identity management and Affiliations

Identity management is a tough topic. Users tend to use different identities and handles across the various forges and data sources, and reconciliating them reliably is a hazardous and error-prone process. Sorting Hat, the tool responsible for identity management within the GrimoireLab framework, provides several features and heuristics to correctly identify individuals across forges, and to establish their affiliation (i.e. what organisation they are working for). Although it does a pretty good job at it, there is no garantee on the outcome. Some of the situations we identified are:

Individuals working for several organisations at the same time: how do we assign contributions to organisations? Similarly there are individuals working on some project for an organisation, and on different projects on their personal time.
Common names: some first- and last- names can be very common in specific countries, times, cultures. In the most difficult cases, there can be different people with identical names, thus having similar email addresses, and possibly working on the same project.

Important

If you identify a case of wrong affiliation or identity, please let us know! We will fix it promptly.

Numbers

While we try hard to get the numbers rights, there can be errors or inconsistencies.