LFX Insights review for projects

[jmertic]

Please share any additional details on this topic

LFX Insights has come a long way, and we'd like to start having projects review the data by each project lead.

https://insights.lfx.linuxfoundation.org/foundation/aswf

Detail what actions or feedback you would like from the TAC

Each TSC lead review, and continue the discussion in the Project Leads Office Hours starting in January.

How much time do you need for this topic?

None

[lgritz]

Is this where we're supposed to report what's weird about these reports?

[jmertic]

Yep!

[lgritz]

I know somebody worked hard on this, and honestly I feel like the biggest jerk in the world just laying it all out as bluntly as I'm about to. So, sorry for this.

Is there documentation somewhere? An explanation pop-up upon hover over almost any word or widget would be incredibly helpful. I found myself not quite sure what almost everything really meant, what computation it represented.

For example, What are "commit activities"? I'm not aware that it's not a term GitHub uses or that we use. Is that just counting the individual commits? Does the credit go to the author? The person who did the merge? Both? If a PR is merged and then trivially backported by cherry-picking to a release branch, is that one "commit activity" or two?

It would be helpful if the pull-down for the date range that's being analyzed had "lifetime of the project" as one choice. You're a little stuck if the project has existed for more than 10 years, the longest menu choice they have. You can pick "custom range" and VERY TEDIOUSLY scroll back month by month to before the project was started (if you even know that).

Analyzing OIIO, it says I'm the number one contributor, with 20.61K "activities" (whatever that means -- it doesn't explain) and 70% of contributions over the life of the project. But in the next panel to the right of that, it says Sony Pictures Entertainment is the number one contributing organization (ok so far) with 1.3K "activities" accounting for 30%. Huh? How can SPE be a lower percentage than a single person who works for SPE? Hovering or clicking on my name in the person leaderboard doesn't reveal where it thinks I work, and hovering or clicking on SPE in the org leaderboard doesn't reveal which people it has associated with that company or what methodology it's using to pair people with companies. No explanation at all.

(Aside: Let's check the math. If 20.6K is 70% of something, the total of that something must be around 29.4. The 1.3K activities attributed to SPE is 1.3/29.4 = 4.4%. So where does the 30% figure attributed to SPE come from, exactly? It's stuff like this all the way down.)

If you select just the last year, and just "commit activities", it's even more fun. I'm number one again (fine) with 229 "commit activities". Sony Pictures Entertainment is number 10 with a total of 14 "commit activities." That definitively proves that it's not associating me with Sony, right? Who else is wrong?

For the last 10 years... ugh, sorry, I wanted to go back to the whole history of the project again, but after 3 frustrating tries, I simply could not force the calendar widget to do the right thing again to get the full history starting in 2008.

Where was I? Yeah, for the last 10 years, it says the project has 727 contributors. What does that mean? Is that code contributors (the single most important number I'd be curious about)? Or anybody who has ever interacted with the project including commenting on an issue or whatever? It doesn't say. I don't think it's committers, because I try to keep track of that and I think there are about 230-ish over the whole history of the project.

Geographic regions are fun, too. Can you find out how it associates a person with a region, or which people are associated with a region? You cannot. But it confidently tells us that 35% of ... what exactly, it doesn't say; this one doesn't reveal or let you choose the metric such as "activities", whatever that is. Anyway, 35% of something came from the US. And 12% of that something came from Canada. I live in Canada, and it thinks I account for 70% of "activities", so, um, what is it measuring and how?

Does it respect the .mailmap file in the project that documents the equivalencies between the multiple emails that some people have used over the history of the project? No idea.

Does it respect the .git-blame-ignore-revs file in the project that documents which individual commits are meaningless bulk-reformatting tasks that should not really be credited as meaningful work to an person or organization? No idea.

If an engineer works for Pixar, and then moves to Autodesk, which org gets the credit for their activity? The place they originally worked? The place they happen to work on the day that the analysis is performed (even if they've contributed to the project for 10 years at Pixar and moved to Apple just last week)? Or does it somehow try to figure out who they worked for when? (I'm betting no.) Merely posing this question and thinking about it for a minute sure seems to make the org stats appear meaningless, right?

Everything on the page is like this. Nothing with a number gives any UI affordance that leads to an explanation of what what the number is actually supposed to represent, how it was computed, or which (things, people, orgs, commits) that number includes. The math doesn't makes sense half the time. I have no idea what any of it really means, or which parts can be trusted to be accurate.