Closed Bug 469376 Opened 16 years ago Closed 16 years ago

AMO update pings broken for 2 weeks

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Type:
task
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: fligtar, Assigned: fligtar)

References

Details

Update pings were again not counted for versioncheck.addons.mozilla.org this week.

This is misleading developers in both the statistics dashboard and the new compatibility dashboard.

Please make sure the versioncheck.amo logs are in the right place and re-run the update ping counting scripts for 2008-12-03 and 2008-12-10.
Assignee: server-ops → thardcastle
This issue is probably caused by bug 469128, which is now fixed.
Running 2008-12-03 right now, and it's picking up the missing access_2008-12-03*.z*.gz files.
2008-12-03 completed, and 2008-12-10 had already parsed all logs, but both gave this:

updatepings counted: 0
        Skipped because of unknown_guid: 0
        Skipped because of SJ: 0
        Skipped because of NL: 0
        Skipped because of CN: 0
I looked at the script and it appears it won't handle that extra space I mentioned in Comment 1.  Going forward it is fixed, but the old logs still have a space.
Ok, I'll take a look at modifying the regex to support a leading space in each logfile line and we can re-run the scripts again. Re-running the scripts today didn't really make a difference in the missing stats from what I can tell.
Blocks: 468570
I added support for there being an extra space in front of the date and updated prodexternal for bin to pickup that change. Please svn up dm-stats01 to get that change, and then to rerun the scripts, let's do this:

1) Run SQL to mark logfiles unparsed for 12/3 and 12/10:
UPDATE logs_parsed SET updatepings_done=0 WHERE name LIKE 'access_2008-12-03%' OR name LIKE 'access_2008-12-10%';

2) Run SQL to erase miscounted update pings:
DELETE FROM update_counts WHERE date='2008-12-03' OR date='2008-12-10';

3) Re-run update ping counter for all datacenters and logs for 2008-12-03 and 2008-12-10

This should fix update ping counts. I need to assess download counts, because presumably the space messed those up as well. Jeremy, what is the date range for that extra space being present?
(In reply to comment #6)
> I need to assess download counts, because
> presumably the space messed those up as well.

I was about to ask about that myself. If you look at https://addons.mozilla.org/en-US/firefox/statistics/addon/1865, Adblock Plus 1.0 release on first of December had no influence on download counts whatsoever. However, the downloads more than tripled on December 11 which makes no sense.
Wladimir, do you think the ABP 1.0 release would have made much difference to the numbers given that the download stats don't include (so far as I know) any updates, but only new installs? I saw the 1.0 release mentioned in a few blogs, but unless it got covered in like, Techcrunch or the New York Times, I wouldn't expect a *major* bump on top of your standard huge number of daily installs. However, you do seem to have had at least a small uptick from 2-8dec, tho nothing like the one you had from 29oct->14thNov. From the 9th Dec, however, things go completely insane. 

Looking at the AMO stats for Interclue (addon 4999), and comparing them with the hitcounts for our onInstall page, I would say that the AMO stats for

16Nov-8Dec: have roughly twice the expected number. 
Dec 9: has approx 0.5x the expected number. 
Dec 10: has 2-3x the expected number.
Dec 11-13: have 4-5x the expected number.

Sorry I can't be more accurate, but I don't by any means take our hit-count on the oninstall page as gospel, given that it won't count anyone using noscript, and sometimes the page gets hit from sources other than an actual install, but up until a little while ago it was always *reasonably* close to the AMO count, and since then it's been way out. 

Also, our total download count and weekly download count are currently way higher than they ought to be, and, as mentioned in the title of this bug, the last two weekly update-ping counts are broken as well, and conversely are on the very low side. 

I'm pretty sure that the AMO numbers made sense a week ago, and that the doubling of the 16Nov-8Dec download counts must have happened in the last few days, possibly due to reprocessing of the logfiles? 

The pre-Nov-16th counts I'm not sure about, but they seem close enough. We got a big boost Nov 16 from being in Fashion your Firefox, then a bigger one Dec 1st as we went into the featured list, but not as big a boost as seen in the current numbers! 

Anyway, hope that helps! Good luck getting things sorted out. 

Hmmm, I was just trying to zoom out another month on our stats and it says "Failed to load data xml from /en-US/statistics/csv/4999/downloads
Unauthorized" so I presume someone is doing a bit of surgery at the moment!
I had over 50,000 visits on adblockplus.org on December 2nd. Given that people seem to more often click "Install" than "Check for updates" and that the "Install" link on my site goes to AMO - yes, I would expect it to make an impact. But of course I cannot be sure where the majority of downloads on AMO comes from so maybe releases are indeed almost irrelevant.
Assignee: thardcastle → oremj
Still have 3 more directories to go, but results of the first script:

updatepings counted: 28594320
        Skipped because of unknown_guid: 16320416
        Skipped because of SJ: 0
        Skipped because of NL: 2928
        Skipped because of CN: 1

Execution time: 9248.2748830318
It counted 28.5m, but skipped 16.3m due to unknown GUIDs? Does that happen often? I presume this is the GUID of the addon in question, so would that be because there are a bunch of addons out there that are checking for updates at AMO but are not currently hosted there? Or is it a sign that something went wrong with the count process?
I have both an addon with a numerical GUID and one with an alphanumerical GUID and they are both NOT counted. One addon is relatively old, one is relatively new. Both addons are not for Firefox, but for Thunderbird. Both addons are in the Sandbox atm... Both now have 0 (zero!) active users.
Find the difference and solve it please. And get them out of the sandbox please :-)
https://addons.mozilla.org/en-US/thunderbird/statistics/addon/2299
https://addons.mozilla.org/en-US/thunderbird/statistics/addon/7026
12/3 and 12/10 are done processing.
Status: NEW → RESOLVED
Closed: 16 years ago
Resolution: --- → FIXED
Are you kidding. This isn't fixed!
Haven't you read comment #12 ?
> https://addons.mozilla.org/en-US/thunderbird/statistics/addon/7026

That add-on is still in the sandbox :-(
Assignee: oremj → fligtar
Also total download count on main page for other add-on is 0 (zero) :-(
https://addons.mozilla.org/en-US/thunderbird/addon/2299
Those look fine to me.  Probably needed to let the caches update themselves.
Onno, what exactly are you complaining about? I see "Active daily users: 3,692"
for your add-on, this matches the download count AFAICT. Download count is also fine, at least on the statistics page. Note that all pages are cached, check again in an hour.

The statistics for Adblock Plus are still strange - 3/12 has about half the
expected number and there is an additional data point at 11/12 with 464 active
daily users but those are minor glitches now.
Right, the download counters are fixed now.
I was "complaining" about my extensions (still) being in the sandbox and then I noticed the wrong download counter. It needed a zeropoint correction. Or cache refreshment if you prefer.
Both my extensions are still in the sandbox and I was trying some humor in here to get them out of there. This bug is verified fixed!
Status: RESOLVED → VERIFIED
This bug isn't going to get your extension out of the sandbox, talk to amo reviewers on IRC or through the editors mailing list if the developer tools nomination isn't working for you. This is strictly a server bug and isn't being handled by the folks who do policy/content-reviews.
  https://addons.mozilla.org/en-US/firefox/pages/policy
  http://shawnwilsher.com/archives/108 (more concise, less official)
The adblock plus stats look odd, Active daily users 464, down 99.99%?
https://addons.mozilla.org/en-US/firefox/statistics/addon/1865

Ditto Firebug: https://addons.mozilla.org/en-US/firefox/statistics/addon/1843

For any reasonably popular addon the Active Daily Users number is a huge drop, like the last numbers we have are for Dec 10 and the number used is from Dec 11. For some of the "long-tail" addons the last Active Daily Users number is more in line with the average -- the red summary line doesn't take a huge nose-dive to zero but remains as a floating point that just ends at the cut-off date.
I guess the script counted a few seconds into December 11 and made it a separate data point - if the add-on had a ping in that time it got a 99.99% drop.
Hmmm. Apart from the ADU, the daily download counts for ABP, and Interclue as I mentioned in Comment #8 - and probably many other addons - are still displaying wrong from 9 Dec (very low) through 10 Dec (high) and 11,12,13,14,15 Dec (stupidly high) 

The incorrect daily download counts have also been passed into the numeric weekly and all time totals. I'd like to *think* Interclue had been downloaded a million times, in fact I'd throw a party, but I'm pretty sure it's closer to ~830,000 in actuality. 

But this bug pertains to the update pings. Is there another bug about the incorrect download counts? I sortof assumed the problems were related.
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.