Closed Bug 552162 Opened 14 years ago Closed 14 years ago

Daily download counts broken for add-ons updated since 2010-03-10

Categories

(addons.mozilla.org Graveyard :: Statistics, defect)

Product:
addons.mozilla.org Graveyard
Component:
Statistics
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: ma1, Assigned: dre)

References

(
URL
)

Details

I submitted FlashGot 1.2.1.17 (id=220, version_id=99099) and had it approved on March 10 2010.

The download counts split by source for the same day show all the downloads counted in the "null" source group, but this weirdness was probably a general and isolated maintenance issue, since it appears in other add-ons such as Adblock Plus and NoScript.

However those other extensions recovered to their "normal" numbers the day after, while FlashGot's dropped down and appear to be locked steadily under the hundred (81 on March 11 and 59 on March 12).

This is a CVS extraction from March 9, showing both the "null" weirdness and the breakage which followed:

#    addons.mozilla.org Statistics for add-on 220
#
#    Generated Sat, 13 Mar 2010 01:53:54 -0800
#    from https://addons.mozilla.org/en-US/firefox/statistics/csv/220/downloadsource?start=2010-03-09&end=2010-03-13
#
#    This data is provided "AS IS" and is subject to Mozilla's Legal Disclaimers
#    and Limitations policy, available at http://www.mozilla.com/en-US/about/legal.html.
#
# Fields: [date;count;null;addondetail;api;search;sharingapi;category;recommended;homepagebrowse;homepagepromo;collection;userprofile;oftenusedwith]
2010-03-09,21269,400,10182,5052,2924,822,658,647,364,125,72,18,5
2010-03-10,16734,16734,0,0,0,0,0,0,0,0,0,0,0
2010-03-11,81,77,1,2,1,0,0,0,0,0,0,0,0
2010-03-12,59,58,0,0,1,0,0,0,0,0,0,0,0
This seems to affect all the add-ons with updates submitted on March 10 or after.

Some more data:

Dafizilla ViewSourceWith 0.6.1, updated on March 10
https://addons.mozilla.org/en-US/statistics/addon/394

Clean And Close 2.5.1, updated on March 11
https://addons.mozilla.org/en-US/statistics/addon/2608

Google Shortcuts 1.9.0, updated on March 11
https://addons.mozilla.org/en-US/statistics/addon/3576

Firebug 1.5.3, updated on March 12
https://addons.mozilla.org/en-US/statistics/addon/1843

StockFox 1.2, updated on March 13
https://addons.mozilla.org/en-US/statistics/addon/77707


Add-ons updated on March 9 or before are unaffected:

Gmail Watcher 1.00, updated on March 9
https://addons.mozilla.org/en-US/statistics/addon/60148
URL: https://addons.mozilla.org/en-US/fire... → https://addons.mozilla.org/en-US/stat...
Severity: normal → major
Summary: Daily download counts broken since latest extension version update → Daily download counts broken for add-ons updated since 2010-04-10
CCing metrics.  Is the stats script broken?  For Giorgio's first example in comment 1 the sources are messed up from the 10th on:

mysql> select * from download_counts where addon_id=394 order by date desc limit 50;
+----------+----------+-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+
| id       | addon_id | count | src                                                                                                                                                                                        | date       |
+----------+----------+-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+
| 24154206 |      394 |     8 | a:1:{s:4:"null";i:8;}                                                                                                                                                                      | 2010-03-13 | 
| 24147201 |      394 |     1 | a:1:{s:4:"null";i:1;}                                                                                                                                                                      | 2010-03-12 | 
| 24140287 |      394 |    11 | a:1:{s:4:"null";i:11;}                                                                                                                                                                     | 2010-03-11 | 
| 24132832 |      394 |   234 | a:1:{s:4:"null";i:234;}                                                                                                                                                                    | 2010-03-10 | 
| 24125151 |      394 |   231 | a:6:{s:10:"collection";i:3;s:10:"developers";i:75;s:8:"category";i:2;s:3:"api";i:59;s:6:"search";i:66;s:4:"null";i:26;}                                                                    | 2010-03-09 | 
| 24118251 |      394 |   200 | a:6:{s:10:"collection";i:4;s:10:"developers";i:71;s:8:"category";i:4;s:3:"api";i:58;s:6:"search";i:54;s:4:"null";i:9;}                                                                     | 2010-03-08 | 
| 24111077 |      394 |   155 | a:8:{s:10:"collection";i:2;s:10:"developers";i:56;s:8:"category";i:7;s:3:"api";i:40;s:11:"addondetail";i:1;s:6:"search";i:37;s:4:"null";i:11;s:10:"sharingapi";i:1;}                       | 2010-03-07 | 
| 24103997 |      394 |   156 | a:5:{s:10:"developers";i:54;s:8:"category";i:8;s:3:"api";i:29;s:6:"search";i:45;s:4:"null";i:20;}                                                                                          | 2010-03-06 | 
| 24096755 |      394 |   201 | a:6:{s:10:"collection";i:5;s:10:"developers";i:73;s:8:"category";i:2;s:3:"api";i:58;s:6:"search";i:54;s:4:"null";i:9;}                                                                     | 2010-03-05 | 
| 24089760 |      394 |   184 | a:6:{s:10:"collection";i:3;s:10:"developers";i:72;s:8:"category";i:1;s:3:"api";i:54;s:6:"search";i:47;s:4:"null";i:7;}                                                                     | 2010-03-04 |
Summary: Daily download counts broken for add-ons updated since 2010-04-10 → Daily download counts broken for add-ons updated since 2010-03-10
Looking at Firebug's data, seems that *only* downloads with a null source are counted after the update (March 12, in this case):

#    addons.mozilla.org Statistics for add-on 1843
#
#    Generated Tue, 16 Mar 2010 05:28:26 -0700
#    from https://addons.mozilla.org/en-US/firefox/statistics/csv/1843/downloadsource?start=2010-03-12&end=2010-03-16
#
#    This data is provided "AS IS" and is subject to Mozilla's Legal Disclaimers
#    and Limitations policy, available at http://www.mozilla.com/en-US/about/legal.html.
#
# Fields: [date;count;null;addondetail;api;search;homepagepromo;collection;category;oftenusedwith;userprofile;sharingapi;fxcustomization;homepagebrowse;external-firstrun-3.6c]
2010-03-12,27068,8177,7701,6827,3163,612,266,237,29,24,12,11,8,1
2010-03-13,972,970,1,1,0,0,0,0,0,0,0,0,0,0
2010-03-14,799,799,0,0,0,0,0,0,0,0,0,0,0,0
2010-03-15,1497,1497,0,0,0,0,0,0,0,0,0,0,0,0
Assignee: nobody → deinspanjer
I just landed in China, but I'll take a look at this today (my time) and post an update to the bug in less than 12 hours.
(In reply to comment #5)
> I just landed in China, but I'll take a look at this today (my time) and post
> an update to the bug in less than 12 hours.

We're getting more complaints about this.  Update?
Upping to critical- we're one week into this issue and this is an issue we haven't seen before.
Severity: major → critical
Sorry for the missed update before.  Adjusting to the time is screwy, but I missed this.
Whatever happened on March 10th affected the matching of downloads to addon_ids.  I need to investigate more, but it would be very useful to hear from you guys if anything changed in the format of add-on download requests on that date.

What I look for is a request to download a particular file:
/en-US/firefox/downloads/file/55367/xpi/<add-on name>
I then match up that file_id with the corresponding addon_id.

I'll be looking at that logic right now.
Okay.  I've got it.  The module that pulls down new add-on information from amo stopped running on the 10th due to a code update I pushed then.  I'll start backprocessing this data immediately.

It would be a much faster backprocess if I could skip the versioncheck data, but that would mean that any completely new addons (as defined by the addon_guid) would be missing from the stats as well.  Should I run this all at once or in two batches with the versioncheck data taking a day or two more to catch up?
I'd run in two batches, as completely new add-ons are a comparatively small set.  This will also test your hypothesis more quickly, no?
Good.  I'm running the backprocessing on downloads only right now.  We'll get them fixed then I'll start backprocessing updates when downloads is finished.
It is currently taking about 10 minutes per hour of data and we have 24 hours * 7 days = 168 hours of data to backprocess.  

Rough estimate is that download backprocessing should be done in just over one day.

Note that the problem has been corrected in the current processing and the data starting 2010-03-18 should be correct.
Still broken, sorry.

(In reply to comment #11)

> Rough estimate is that download backprocessing should be done in just over one
> day.

Does it mean that starting with 2010-03-20 we should have seen also the "old" data restored? If it does, I'm afraid that doesn't work, see https://addons.mozilla.org/en-US/statistics/addon/1843

> Note that the problem has been corrected in the current processing and the data
> starting 2010-03-18 should be correct.

This is true for extensions updated before 2010-03-18 (including those broken after 2010-03-10 like FlashGot and Firebug), but those updated on 2010-03-18 are broken again, see https://addons.mozilla.org/en-US/statistics/addon/8442

So, to recap:
1) Backprocessing doesn't seem to have worked (or is not done yet?)
2) "live" counts for extensions updated between 2010-03-10 and 2010-03-17 are back, even though the breakage "hole" remains (see #1)
3) counts for extensions updated on 2010-03-18 (and on?) are broken again
1. backprocessing started out at the described speed, but it slowed down considerably after a few hours.  It has errored out twice due to intermittant database connectivity.  I've been keeping track of it over the weekend.  It has currently processed through the 13th.  I'll post updates as it progresses.  I can't push the updates until it is finished because I have to have a maintenance window on the metrics data warehouse to correct the bad data all at once.

2 & 3. What do you mean by "live" counts? ADU?  What we should see is that download numbers are still missing for any add-ons that had updates between the 10th and the 18th pending completion of item #1.  Download numbers for all add-ons should be visible after the 18th.
(In reply to comment #13)
> 1. backprocessing started out at the described speed, but it slowed down
> considerably after a few hours.

OK, thanks for the update.

> 2 & 3. What do you mean by "live" counts?

I'm talking about *daily counts* on dates following your intervention.
I mean that, "old data" aside (i.e. starting with March 18th), daily counts are now working (and are visible in the dashboard) for any add-on *updated before March the 18th* (including FlashGot, Firebug and the other add-ons referenced as an example in Comment #1.

*But* they're not working anymore for add-ons posted on March 18th (I couldn't find public stats for add-ons posted on March 19th, so I can't tell whether they're affected as well or not):

#    addons.mozilla.org Statistics for add-on 8442
#
#    Generated Sun, 21 Mar 2010 03:29:03 -0700
#    from https://addons.mozilla.org/en-US/firefox/statistics/csv/8442/downloads?start=2010-03-17&end=2010-03-21
#
# Fields: [date;count]
2010-03-17,1002
2010-03-18,935
2010-03-19,109
2010-03-20,20

#    addons.mozilla.org Statistics for add-on 722
#
#    Generated Sun, 21 Mar 2010 03:30:44 -0700
#    from https://addons.mozilla.org/en-US/firefox/statistics/csv/722/downloads?start=2010-03-17&end=2010-03-21

# Fields: [date;count]
2010-03-17,27076
2010-03-18,26326
2010-03-19,16974
2010-03-20,71


Looks like daily counts for these two add-ons are severely broken on 2010-03-19 and 2010-03-20. Am I missing something?
Scripts are still collecting broken data:

#    addons.mozilla.org Statistics for add-on 8442
#
#    Generated Mon, 22 Mar 2010 01:44:33 -0700
#    from https://addons.mozilla.org/en-US/firefox/statistics/csv/8442/downloads?start=2010-03-17&end=2010-03-22
#
#    This data is provided "AS IS" and is subject to Mozilla's Legal Disclaimers
#    and Limitations policy, available at http://www.mozilla.com/en-US/about/legal.html.
#
# Fields: [date;count]
2010-03-17,1002
2010-03-18,935
2010-03-19,109
2010-03-20,20
2010-03-21,27


#    addons.mozilla.org Statistics for add-on 722
#
#    Generated Mon, 22 Mar 2010 01:45:45 -0700
#    from https://addons.mozilla.org/en-US/firefox/statistics/csv/722/downloads?start=2010-03-17&end=2010-03-22
#
#    This data is provided "AS IS" and is subject to Mozilla's Legal Disclaimers
#    and Limitations policy, available at http://www.mozilla.com/en-US/about/legal.html.
#
# Fields: [date;count]
2010-03-17,27076
2010-03-18,26326
2010-03-19,16974
2010-03-20,71
2010-03-21,34
Alright.  The fix was deployed to the server running the backprocessing, but not to the one running the current processing.  It is in place now, and I'll make sure that the backprocessing picks up all the days for which we have bad data.
The backprocessing has gotten up to 2010-03-16 so far.  I've had to stop it a few times because it is putting too much load on the database servers and causing problems with the regular processing.
The export for the 22nd will go out in a few hours.  I would appreciate hearing if the numbers look any better for that.  I'll still need to reprocess the 22nd because there were a few hours of data processed that did not have the fix.
(In reply to comment #17)
> The export for the 22nd will go out in a few hours.  I would appreciate hearing
> if the numbers look any better for that.

Yes, they definitely look better:

#    addons.mozilla.org Statistics for add-on 8442
#
#    Generated Tue, 23 Mar 2010 06:56:25 -0700
#    from https://addons.mozilla.org/en-US/firefox/statistics/csv/8442/downloads?start=2010-03-17&end=2010-03-22
#
#    This data is provided "AS IS" and is subject to Mozilla's Legal Disclaimers
#    and Limitations policy, available at http://www.mozilla.com/en-US/about/legal.html.
#
# Fields: [date;count]
2010-03-17,1002
2010-03-18,935
2010-03-19,109
2010-03-20,20
2010-03-21,27
2010-03-22,840
::grumble grumble::  The 17th didn't run while I was on the plane back from China.  I typoed the command and didn't realize it before I disconnected.  It is running now.  Another update tomorrow morning.
Okay, everything finished running through the 22nd and the data was exported into AMO this morning.  When the AMO cron job runs that makes this available, the statistics mentioned above should all be corrected.  Please check this and verify the bug if it is okay.
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
Thank you, data for the extensions mentioned in this bug looks normal.

I also checked an extension updated on March 25, to stay on the safe side, and its data for March 26 looks normal as well:
https://addons.mozilla.org/en-US/statistics/addon/11950
Status: RESOLVED → VERIFIED
Me and other theme developers are noticing a very weird decrease on downloads since 1st April. It seems this issue affects only themes, not extensions.
(In reply to comment #22)
> Me and other theme developers are noticing a very weird decrease on downloads
> since 1st April. It seems this issue affects only themes, not extensions.

Please file a separate bug with any details that we can use to investigate this. Keep in mind that as Personas become more popular, less people will install full themes.
(In reply to comment #23)
> (In reply to comment #22)
> > Me and other theme developers are noticing a very weird decrease on downloads
> > since 1st April. It seems this issue affects only themes, not extensions.
> 
> Please file a separate bug with any details that we can use to investigate
> this. Keep in mind that as Personas become more popular, less people will
> install full themes.

Ah... I understand. Actually the fact Personas becoming more popular should cause exactly the opposite, arousing the interest of users for this type of customization.

I believe this is happening more because themes are, today, completely hidden from users; that is the result of all measures taken to discriminate themes.
Product: addons.mozilla.org → addons.mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.