Closed Bug 1133543 Opened 11 years ago Closed 11 years ago

Application chart in stats dashboard shows significant dips for backfilled data after bug 1122110

Categories

(addons.mozilla.org Graveyard :: Statistics, defect)

Product:
addons.mozilla.org Graveyard
Component:
Statistics
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED
Milestone:
2015-03

People

(Reporter: jorgev, Assigned: jason)

References

Details

(Whiteboard: [qa-])

After fixing bug 1122110, the main stats appear to have normalized. However, there appears to be a bug in the "by application" section of the dashboard: https://addons.mozilla.org/addon/mouse-gestures-suite/statistics/usage/applications/?last=30 https://addons.mozilla.org/addon/adblock-plus/statistics/usage/applications/?last=30 They take a while to load. After loading, you'll see big dips during weekdays, which are the days with the backfilled data. It remains to be seen if this week will be affected by the same bug or if this only has to do with the data we backfilled.
To me this looks like the wrong data appear for point releases in the "by application" section. The point releases are not taken into account in the table and they are combined into the major version - or at least they are supposed to. The dips and missing counts appear for column "Firefox 35.0" while recently most people were upgraded to 35.0.1. Now that people move to Fx 36 the stats appear to be correct for "36.0". When/if 36.0.1 is released we may see the problem again, I suppose. I notice the same kind of incorrect stats for "SeaMonkey 2.32" column - while most SeaMonkey users have switched to 2.32.1 and "2.32" doesn't properly include 2.32.1 users.
Lemon_juice, could you please tell me more about this "inclusion"? Are the stats supposed to be grouped by major version? From what I can see, they aren't at the moment, not in the graphs, and neither in the table.
Correct, the stats are not grouped by major version now but they definitely should be because minor versions don't show in the stats at all so if they are not included in the major versions they are not there at all. For example, there is a column for "Firefox 35.0" but there is no column for "Firefox 35.0.1" so where are 35.0.1 users reported? Currently, they do not seem to be included in "Firefox 35.0", which means they are missing altogether from the stats. Before this bug appeared, minor version users were grouped under major version - so that Firefox 35.0.1 users would be included in the "Firefox 35.0" stats.
Do you know if there were grouped by major version before? I guess when we say "major" we mean the two first numbers, and "minor" the third one, right? This is maybe not linked to the fix in the bug 1122110 after all. I'm trying to understand if we need to fix something, or improve something that was always like this, to get a feel of the urgency/priority.
(In reply to Mathieu Agopian [:magopian] from comment #4) > Do you know if there were grouped by major version before? I can't be sure 100% from a programmatic point of view but from my observations I can say yes - they were always grouped by major version (yes, by "major" we mean the two first numbers, and "minor" the third one). Minor versions have never appeared in the stats at all but when a minor version was released there were no anomalies in the stats, the numbers for the major version clearly included the minor version(s) as well. At this moment something is clearly incorrect about minor version users - to me it looks like they are missing from the stats at all. > This is maybe not linked to the fix in the bug 1122110 after all. Possible, but before bug 1122110 appeared there were also no problems with minor versions as far as I can tell. > I'm trying to understand if we need to fix something, or improve something > that was always like this, to get a feel of the urgency/priority. Including minor versions users in the major version seemed to always work before. It is only recently (last month or so) after bug 1122110 was fixed that the weird numbers for minor version users started to appear.
Thanks a lot for this answer, I think I understand the context a bit better now. I think I now understand the issue: fixing bug 1122110 means that the data is now validated (meaning that a bogus application version, outright wrong, or non existing, is ignored). And to know which application versions are correct, we use the same data that is used in this page: https://addons.mozilla.org/en-US/firefox/pages/appversions/ As you can see, for some reason I don't know yet, the version 35.0.1 isn't in there. So the data is absent from the database I guess, ignored. I need to investigate why this version (and maybe others?) are missing from the "product details". If they're missing on purpose, I'll need to change the code to validate that the version number requested is prefixed by a valid version number from the "product details", and we'll need some backfilling (again). Jorge, do you have more information on the version 35.0.1 missing from the product details?
Flags: needinfo?(jorge)
(In reply to Mathieu Agopian [:magopian] from comment #6) > I think I now understand the issue: fixing bug 1122110 means that the data > is now validated (meaning that a bogus application version, outright wrong, > or non existing, is ignored). And to know which application versions are > correct, we use the same data that is used in this page: > https://addons.mozilla.org/en-US/firefox/pages/appversions/ I see, that would explain it. But I think from a users (add-on developers) perspective this is a change for the worse because it is still good to see stats for bogus application versions. For example, I used to be able to see the number of Pale Moon users because they were identified as Firefox 24.9 - now I get no stats about them. I think the best solution is how it worked before without validation (or at least without such strict validation). At worst, the validation process should take into account wildcard versions - so if https://addons.mozilla.org/en-US/firefox/pages/appversions/ lists 24.* and 35.* as valid versions then 24.9 and 35.0.1 should automatically be allowed, too. Anyway, ignoring numbers is the worst solution and is misleading because the numbers simply don't add up. If the validator took wildcards into account and listed other (more extremely) bogus versions in a separate column like "others" - I think that would be good enough.
We don't include minor versions in /appversions/ because it's not really necessary. If an add-on is compatible with 35.0.1, it should also be compatible with 35.0, so we only need the latter in the list of valid versions. I think we should allow X\.[0-9]{1,3}(\.[0-9]{1,3})? where X is an integer between 1 and the latest major version in /appversions/. As for folding minor version numbers into major ones, I would prefer that we don't do that. If that's what we're doing currently, it's no big deal.
Flags: needinfo?(jorge)
Fixed in https://github.com/mozilla/olympia/commit/cb08099589d4e7b4d353e91066e8709c79674ea6 Let's see how this behaves in production (push is next thursday) for a few days, and if it's ok, it'll need backfilling (again :'(
Whiteboard: [qa-]
The fix appears to work - the stats from yesterday (Thursday) again show proper numbers for minor versions - folded into major versions, just like it worked before - especially visible now with Firefox 36.0 that has been upgraded to 36.0.1. Thanks.
excellent news, let's see if we can backfill the data now. Assigning this bug to Jason.
Assignee: mathieu → jthomas
How far back do we need to backfill? Do we need to start the backfill from Jan 11th 2015?
I'm afraid we do. Please don't hate me.
(In reply to Mathieu Agopian [:magopian] from comment #14) > I'm afraid we do. Please don't hate me. No worries, just wondering. I began the backfill process. This may take a few days to complete. I will update the bug once completed.
01-11-2015 - 01-31-2015 done. Currently processing February. How does it look so far?
Backfill done. Please verify and let me know if there are any issues.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Product: addons.mozilla.org → addons.mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.