Last Comment Bug 552162 - Daily download counts broken for add-ons updated since 2010-03-10
: Daily download counts broken for add-ons updated since 2010-03-10
Status: VERIFIED FIXED
:
Product: addons.mozilla.org Graveyard
Classification: Graveyard
Component: Statistics (show other bugs)
: unspecified
: All All
: -- critical
: ---
Assigned To: Daniel Einspanjer [:dre] [:deinspanjer]
:
:
Mentors:
https://addons.mozilla.org/en-US/stat...
: 552633 (view as bug list)
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2010-03-13 02:06 PST by Giorgio Maone [:mao]
Modified: 2016-02-04 14:54 PST (History)
6 users (show)
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments

Description Giorgio Maone [:mao] 2010-03-13 02:06:34 PST
I submitted FlashGot 1.2.1.17 (id=220, version_id=99099) and had it approved on March 10 2010.

The download counts split by source for the same day show all the downloads counted in the "null" source group, but this weirdness was probably a general and isolated maintenance issue, since it appears in other add-ons such as Adblock Plus and NoScript.

However those other extensions recovered to their "normal" numbers the day after, while FlashGot's dropped down and appear to be locked steadily under the hundred (81 on March 11 and 59 on March 12).

This is a CVS extraction from March 9, showing both the "null" weirdness and the breakage which followed:

#    addons.mozilla.org Statistics for add-on 220
#
#    Generated Sat, 13 Mar 2010 01:53:54 -0800
#    from https://addons.mozilla.org/en-US/firefox/statistics/csv/220/downloadsource?start=2010-03-09&end=2010-03-13
#
#    This data is provided "AS IS" and is subject to Mozilla's Legal Disclaimers
#    and Limitations policy, available at http://www.mozilla.com/en-US/about/legal.html.
#
# Fields: [date;count;null;addondetail;api;search;sharingapi;category;recommended;homepagebrowse;homepagepromo;collection;userprofile;oftenusedwith]
2010-03-09,21269,400,10182,5052,2924,822,658,647,364,125,72,18,5
2010-03-10,16734,16734,0,0,0,0,0,0,0,0,0,0,0
2010-03-11,81,77,1,2,1,0,0,0,0,0,0,0,0
2010-03-12,59,58,0,0,1,0,0,0,0,0,0,0,0
Comment 1 Giorgio Maone [:mao] 2010-03-15 03:37:11 PDT
This seems to affect all the add-ons with updates submitted on March 10 or after.

Some more data:

Dafizilla ViewSourceWith 0.6.1, updated on March 10
https://addons.mozilla.org/en-US/statistics/addon/394

Clean And Close 2.5.1, updated on March 11
https://addons.mozilla.org/en-US/statistics/addon/2608

Google Shortcuts 1.9.0, updated on March 11
https://addons.mozilla.org/en-US/statistics/addon/3576

Firebug 1.5.3, updated on March 12
https://addons.mozilla.org/en-US/statistics/addon/1843

StockFox 1.2, updated on March 13
https://addons.mozilla.org/en-US/statistics/addon/77707


Add-ons updated on March 9 or before are unaffected:

Gmail Watcher 1.00, updated on March 9
https://addons.mozilla.org/en-US/statistics/addon/60148
Comment 2 Wil Clouser [:clouserw] 2010-03-15 09:49:27 PDT
CCing metrics.  Is the stats script broken?  For Giorgio's first example in comment 1 the sources are messed up from the 10th on:

mysql> select * from download_counts where addon_id=394 order by date desc limit 50;
+----------+----------+-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+
| id       | addon_id | count | src                                                                                                                                                                                        | date       |
+----------+----------+-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+
| 24154206 |      394 |     8 | a:1:{s:4:"null";i:8;}                                                                                                                                                                      | 2010-03-13 | 
| 24147201 |      394 |     1 | a:1:{s:4:"null";i:1;}                                                                                                                                                                      | 2010-03-12 | 
| 24140287 |      394 |    11 | a:1:{s:4:"null";i:11;}                                                                                                                                                                     | 2010-03-11 | 
| 24132832 |      394 |   234 | a:1:{s:4:"null";i:234;}                                                                                                                                                                    | 2010-03-10 | 
| 24125151 |      394 |   231 | a:6:{s:10:"collection";i:3;s:10:"developers";i:75;s:8:"category";i:2;s:3:"api";i:59;s:6:"search";i:66;s:4:"null";i:26;}                                                                    | 2010-03-09 | 
| 24118251 |      394 |   200 | a:6:{s:10:"collection";i:4;s:10:"developers";i:71;s:8:"category";i:4;s:3:"api";i:58;s:6:"search";i:54;s:4:"null";i:9;}                                                                     | 2010-03-08 | 
| 24111077 |      394 |   155 | a:8:{s:10:"collection";i:2;s:10:"developers";i:56;s:8:"category";i:7;s:3:"api";i:40;s:11:"addondetail";i:1;s:6:"search";i:37;s:4:"null";i:11;s:10:"sharingapi";i:1;}                       | 2010-03-07 | 
| 24103997 |      394 |   156 | a:5:{s:10:"developers";i:54;s:8:"category";i:8;s:3:"api";i:29;s:6:"search";i:45;s:4:"null";i:20;}                                                                                          | 2010-03-06 | 
| 24096755 |      394 |   201 | a:6:{s:10:"collection";i:5;s:10:"developers";i:73;s:8:"category";i:2;s:3:"api";i:58;s:6:"search";i:54;s:4:"null";i:9;}                                                                     | 2010-03-05 | 
| 24089760 |      394 |   184 | a:6:{s:10:"collection";i:3;s:10:"developers";i:72;s:8:"category";i:1;s:3:"api";i:54;s:6:"search";i:47;s:4:"null";i:7;}                                                                     | 2010-03-04 |
Comment 3 Giorgio Maone [:mao] 2010-03-16 05:29:25 PDT
Looking at Firebug's data, seems that *only* downloads with a null source are counted after the update (March 12, in this case):

#    addons.mozilla.org Statistics for add-on 1843
#
#    Generated Tue, 16 Mar 2010 05:28:26 -0700
#    from https://addons.mozilla.org/en-US/firefox/statistics/csv/1843/downloadsource?start=2010-03-12&end=2010-03-16
#
#    This data is provided "AS IS" and is subject to Mozilla's Legal Disclaimers
#    and Limitations policy, available at http://www.mozilla.com/en-US/about/legal.html.
#
# Fields: [date;count;null;addondetail;api;search;homepagepromo;collection;category;oftenusedwith;userprofile;sharingapi;fxcustomization;homepagebrowse;external-firstrun-3.6c]
2010-03-12,27068,8177,7701,6827,3163,612,266,237,29,24,12,11,8,1
2010-03-13,972,970,1,1,0,0,0,0,0,0,0,0,0,0
2010-03-14,799,799,0,0,0,0,0,0,0,0,0,0,0,0
2010-03-15,1497,1497,0,0,0,0,0,0,0,0,0,0,0,0
Comment 4 Wil Clouser [:clouserw] 2010-03-16 08:37:02 PDT
*** Bug 552633 has been marked as a duplicate of this bug. ***
Comment 5 Daniel Einspanjer [:dre] [:deinspanjer] 2010-03-16 16:13:49 PDT
I just landed in China, but I'll take a look at this today (my time) and post an update to the bug in less than 12 hours.
Comment 6 Wil Clouser [:clouserw] 2010-03-17 14:26:30 PDT
(In reply to comment #5)
> I just landed in China, but I'll take a look at this today (my time) and post
> an update to the bug in less than 12 hours.

We're getting more complaints about this.  Update?
Comment 7 Nick Nguyen [:osunick] 2010-03-17 15:33:51 PDT
Upping to critical- we're one week into this issue and this is an issue we haven't seen before.
Comment 8 Daniel Einspanjer [:dre] [:deinspanjer] 2010-03-17 16:54:20 PDT
Sorry for the missed update before.  Adjusting to the time is screwy, but I missed this.
Whatever happened on March 10th affected the matching of downloads to addon_ids.  I need to investigate more, but it would be very useful to hear from you guys if anything changed in the format of add-on download requests on that date.

What I look for is a request to download a particular file:
/en-US/firefox/downloads/file/55367/xpi/<add-on name>
I then match up that file_id with the corresponding addon_id.

I'll be looking at that logic right now.
Comment 9 Daniel Einspanjer [:dre] [:deinspanjer] 2010-03-17 17:48:35 PDT
Okay.  I've got it.  The module that pulls down new add-on information from amo stopped running on the 10th due to a code update I pushed then.  I'll start backprocessing this data immediately.

It would be a much faster backprocess if I could skip the versioncheck data, but that would mean that any completely new addons (as defined by the addon_guid) would be missing from the stats as well.  Should I run this all at once or in two batches with the versioncheck data taking a day or two more to catch up?
Comment 10 Nick Nguyen [:osunick] 2010-03-17 22:17:17 PDT
I'd run in two batches, as completely new add-ons are a comparatively small set.  This will also test your hypothesis more quickly, no?
Comment 11 Daniel Einspanjer [:dre] [:deinspanjer] 2010-03-18 01:53:03 PDT
Good.  I'm running the backprocessing on downloads only right now.  We'll get them fixed then I'll start backprocessing updates when downloads is finished.
It is currently taking about 10 minutes per hour of data and we have 24 hours * 7 days = 168 hours of data to backprocess.  

Rough estimate is that download backprocessing should be done in just over one day.

Note that the problem has been corrected in the current processing and the data starting 2010-03-18 should be correct.
Comment 12 Giorgio Maone [:mao] 2010-03-21 01:55:42 PDT
Still broken, sorry.

(In reply to comment #11)

> Rough estimate is that download backprocessing should be done in just over one
> day.

Does it mean that starting with 2010-03-20 we should have seen also the "old" data restored? If it does, I'm afraid that doesn't work, see https://addons.mozilla.org/en-US/statistics/addon/1843

> Note that the problem has been corrected in the current processing and the data
> starting 2010-03-18 should be correct.

This is true for extensions updated before 2010-03-18 (including those broken after 2010-03-10 like FlashGot and Firebug), but those updated on 2010-03-18 are broken again, see https://addons.mozilla.org/en-US/statistics/addon/8442

So, to recap:
1) Backprocessing doesn't seem to have worked (or is not done yet?)
2) "live" counts for extensions updated between 2010-03-10 and 2010-03-17 are back, even though the breakage "hole" remains (see #1)
3) counts for extensions updated on 2010-03-18 (and on?) are broken again
Comment 13 Daniel Einspanjer [:dre] [:deinspanjer] 2010-03-21 03:12:30 PDT
1. backprocessing started out at the described speed, but it slowed down considerably after a few hours.  It has errored out twice due to intermittant database connectivity.  I've been keeping track of it over the weekend.  It has currently processed through the 13th.  I'll post updates as it progresses.  I can't push the updates until it is finished because I have to have a maintenance window on the metrics data warehouse to correct the bad data all at once.

2 & 3. What do you mean by "live" counts? ADU?  What we should see is that download numbers are still missing for any add-ons that had updates between the 10th and the 18th pending completion of item #1.  Download numbers for all add-ons should be visible after the 18th.
Comment 14 Giorgio Maone [:mao] 2010-03-21 03:36:15 PDT
(In reply to comment #13)
> 1. backprocessing started out at the described speed, but it slowed down
> considerably after a few hours.

OK, thanks for the update.

> 2 & 3. What do you mean by "live" counts?

I'm talking about *daily counts* on dates following your intervention.
I mean that, "old data" aside (i.e. starting with March 18th), daily counts are now working (and are visible in the dashboard) for any add-on *updated before March the 18th* (including FlashGot, Firebug and the other add-ons referenced as an example in Comment #1.

*But* they're not working anymore for add-ons posted on March 18th (I couldn't find public stats for add-ons posted on March 19th, so I can't tell whether they're affected as well or not):

#    addons.mozilla.org Statistics for add-on 8442
#
#    Generated Sun, 21 Mar 2010 03:29:03 -0700
#    from https://addons.mozilla.org/en-US/firefox/statistics/csv/8442/downloads?start=2010-03-17&end=2010-03-21
#
# Fields: [date;count]
2010-03-17,1002
2010-03-18,935
2010-03-19,109
2010-03-20,20

#    addons.mozilla.org Statistics for add-on 722
#
#    Generated Sun, 21 Mar 2010 03:30:44 -0700
#    from https://addons.mozilla.org/en-US/firefox/statistics/csv/722/downloads?start=2010-03-17&end=2010-03-21

# Fields: [date;count]
2010-03-17,27076
2010-03-18,26326
2010-03-19,16974
2010-03-20,71


Looks like daily counts for these two add-ons are severely broken on 2010-03-19 and 2010-03-20. Am I missing something?
Comment 15 Giorgio Maone [:mao] 2010-03-22 01:48:01 PDT
Scripts are still collecting broken data:

#    addons.mozilla.org Statistics for add-on 8442
#
#    Generated Mon, 22 Mar 2010 01:44:33 -0700
#    from https://addons.mozilla.org/en-US/firefox/statistics/csv/8442/downloads?start=2010-03-17&end=2010-03-22
#
#    This data is provided "AS IS" and is subject to Mozilla's Legal Disclaimers
#    and Limitations policy, available at http://www.mozilla.com/en-US/about/legal.html.
#
# Fields: [date;count]
2010-03-17,1002
2010-03-18,935
2010-03-19,109
2010-03-20,20
2010-03-21,27


#    addons.mozilla.org Statistics for add-on 722
#
#    Generated Mon, 22 Mar 2010 01:45:45 -0700
#    from https://addons.mozilla.org/en-US/firefox/statistics/csv/722/downloads?start=2010-03-17&end=2010-03-22
#
#    This data is provided "AS IS" and is subject to Mozilla's Legal Disclaimers
#    and Limitations policy, available at http://www.mozilla.com/en-US/about/legal.html.
#
# Fields: [date;count]
2010-03-17,27076
2010-03-18,26326
2010-03-19,16974
2010-03-20,71
2010-03-21,34
Comment 16 Daniel Einspanjer [:dre] [:deinspanjer] 2010-03-22 03:09:35 PDT
Alright.  The fix was deployed to the server running the backprocessing, but not to the one running the current processing.  It is in place now, and I'll make sure that the backprocessing picks up all the days for which we have bad data.
Comment 17 Daniel Einspanjer [:dre] [:deinspanjer] 2010-03-22 22:15:39 PDT
The backprocessing has gotten up to 2010-03-16 so far.  I've had to stop it a few times because it is putting too much load on the database servers and causing problems with the regular processing.
The export for the 22nd will go out in a few hours.  I would appreciate hearing if the numbers look any better for that.  I'll still need to reprocess the 22nd because there were a few hours of data processed that did not have the fix.
Comment 18 Giorgio Maone [:mao] 2010-03-23 06:59:35 PDT
(In reply to comment #17)
> The export for the 22nd will go out in a few hours.  I would appreciate hearing
> if the numbers look any better for that.

Yes, they definitely look better:

#    addons.mozilla.org Statistics for add-on 8442
#
#    Generated Tue, 23 Mar 2010 06:56:25 -0700
#    from https://addons.mozilla.org/en-US/firefox/statistics/csv/8442/downloads?start=2010-03-17&end=2010-03-22
#
#    This data is provided "AS IS" and is subject to Mozilla's Legal Disclaimers
#    and Limitations policy, available at http://www.mozilla.com/en-US/about/legal.html.
#
# Fields: [date;count]
2010-03-17,1002
2010-03-18,935
2010-03-19,109
2010-03-20,20
2010-03-21,27
2010-03-22,840
Comment 19 Daniel Einspanjer [:dre] [:deinspanjer] 2010-03-23 19:14:53 PDT
::grumble grumble::  The 17th didn't run while I was on the plane back from China.  I typoed the command and didn't realize it before I disconnected.  It is running now.  Another update tomorrow morning.
Comment 20 Daniel Einspanjer [:dre] [:deinspanjer] 2010-03-26 09:32:45 PDT
Okay, everything finished running through the 22nd and the data was exported into AMO this morning.  When the AMO cron job runs that makes this available, the statistics mentioned above should all be corrected.  Please check this and verify the bug if it is okay.
Comment 21 Giorgio Maone [:mao] 2010-03-27 09:59:36 PDT
Thank you, data for the extensions mentioned in this bug looks normal.

I also checked an extension updated on March 25, to stay on the safe side, and its data for March 26 looks normal as well:
https://addons.mozilla.org/en-US/statistics/addon/11950
Comment 22 Pardal Freudenthal (:ShareBird) 2010-05-09 07:33:43 PDT
Me and other theme developers are noticing a very weird decrease on downloads since 1st April. It seems this issue affects only themes, not extensions.
Comment 23 Justin Scott [:fligtar] 2010-05-09 23:59:28 PDT
(In reply to comment #22)
> Me and other theme developers are noticing a very weird decrease on downloads
> since 1st April. It seems this issue affects only themes, not extensions.

Please file a separate bug with any details that we can use to investigate this. Keep in mind that as Personas become more popular, less people will install full themes.
Comment 24 Pardal Freudenthal (:ShareBird) 2010-05-10 11:49:20 PDT
(In reply to comment #23)
> (In reply to comment #22)
> > Me and other theme developers are noticing a very weird decrease on downloads
> > since 1st April. It seems this issue affects only themes, not extensions.
> 
> Please file a separate bug with any details that we can use to investigate
> this. Keep in mind that as Personas become more popular, less people will
> install full themes.

Ah... I understand. Actually the fact Personas becoming more popular should cause exactly the opposite, arousing the interest of users for this type of customization.

I believe this is happening more because themes are, today, completely hidden from users; that is the result of all measures taken to discriminate themes.

Note You need to log in before you can comment on or make changes to this bug.