404079 - Download numbers don't add up...

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: Maintenance Scripts, defect)

Description

[Stuart Parmenter]

On my developer page it shows:

Downloads:
2296 (week)
1783 (total)

Not sure how I can have more weekly downloads than total downloads...

Comment 1

[Justin Scott [:fligtar]]

mysql> select * from download_counts where addon_id=5972;
+--------+----------+-------+------------+
| id     | addon_id | count | date       |
+--------+----------+-------+------------+
| 156387 |     5972 |    45 | 2007-11-07 |
| 157793 |     5972 |    93 | 2007-11-08 |
| 161593 |     5972 |    66 | 2007-11-09 |
| 163797 |     5972 |    90 | 2007-11-10 |
| 166155 |     5972 |   480 | 2007-11-11 |
| 168618 |     5972 |   446 | 2007-11-12 |
| 171577 |     5972 |   314 | 2007-11-13 |
| 173497 |     5972 |   249 | 2007-11-14 |
+--------+----------+-------+------------+
8 rows in set (0.42 sec)

mysql> select sum(count) from download_counts where addon_id=5972;
+------------+
| sum(count) |
+------------+
|       1783 |
+------------+
1 row in set (0.03 sec)

mysql> select totaldownloads from addons where id=5972;
+----------------+
| totaldownloads |
+----------------+
|           1783 |
+----------------+
1 row in set (0.12 sec)

The weekly and total counts are done by cron at different times every day, so I suspect it will look better in a few days. You're only noticing this because the extension is new, but after the first two weeks it shouldn't be noticable.

Let us know how it looks in a few days.

Comment 2

[lazierthanthou]

Extension id = 5817
total download has remained static since the last 4 days. But the weekly download fluctuates between nearly 600 and 5000. e.g., it showed 5000 something a couple of minutes back, but a refresh of page has brought it to 694. This has already been happening since the last few days. Ironically, total downloads = 4877. So, I can understand that total downloads might not increase and that weekly downloads number might decline, but how can the fluctuation be explained? That too, more than 20 times in the last 3 days.

Comment 4

[Justin Scott [:fligtar]]

There have been enough reports of issues that this is apparently an issue with the new download counter script that went live Tuesday.

Looking into it.

Comment 5

[Justin Scott [:fligtar]]

Found a problem.

Comment 6

[Justin Scott [:fligtar]]

Attachment: counter patch, v1

This patch doesn't actually fix this bug, but I started out with it trying to so I'm still putting it here!

The actual source of the problem was a bad database slave that hasn't been updating since Thursday or Friday. Really bad timing with launching the new download script.

So, we're currently working with IT on the issue and it may not be fixed until mid-week.

As for this patch, the main issue it addresses is the database class not working with mysql_num_rows() causing a new row to be inserted in download_counts for every download. This didn't actually cause any effect on the stats, just made the table huge. This patch fixes it, and the next patch is a script to repair the damage it did.

Comment 7

[Justin Scott [:fligtar]]

Attachment: repair script, v1

Makes sure that each add-on can only have one record for a single day. Mainly to repair the damaged for the last few days, but there were a few days in the past that seemed to have more than one.

This script will take 2-3 hours to run - I just tested it on khan. Note that in production we need to have the stats user granted DELETE on download_counts for it to work.

Comment 8

[Michael Morgan [:morgamic]]

Comment on attachment 289426 [details] [diff] [review]
repair script, v1

Looks like this should fix the extra rows problem.

Comment 9

[Justin Scott [:fligtar]]

The bad slave was fixed last night and put back into the rotation. It turns out that the big problem with millions of extra rows was actually only a problem on the bad slave (which is the one that takes the database dumps) so the main thing this patch fixed wasn't that important.

I'm currently copying the dump from this morning to my laptop as I'm going to be busy and have limited access the rest of the week. But I'm going to confirm what I said above and see if we need to do anything else to fix this, like re-run certain day counts or whatnot. This morning's dump is only 580megs, as opposed to 598 the day before, so that's a good sign.

Comment 10

[Simon Paquet [:sipaq]]

Justin, does this (comment 9) mean that this will be fixed tomorrow or will this take longer?

Comment 11

[Justin Scott [:fligtar]]

Attachment: oops

So uh. I'm not very smart.

The read and write databases were swapped, so we've been writing to the slave database. Oops.

Can try to get this pushed live today and stuff should be fixed after the normal download script runs tonight.

The run-once script isn't needed now, but I wanted to fix a problem with it before I forget.

Comment 12

[Michael Morgan [:morgamic]]

Comment on attachment 289682 [details] [diff] [review]
oops

My fault too, didn't catch this myself.

Comment 13

[Simon Paquet [:sipaq]]

Justin, the total download number for Lightning (ID: 2313) was updated tonight. Thanks for that. 

However, I think that some downloads were lost during the lifetime of this bug. For the 7 days that this bug lasted, I only get about 3300 downloads of Lightning (470 downloads a days). Normal download volume for Lightning was 1500-2500 downloads a days during the last 3 months. 

Any chance to fix that, too?

Comment 14

[Stuart Parmenter]

yeah, my totals updated as well but they're way way lower than the number of downloads I had last week (I had like 6000 downloads last week -- my new total is 2100).  Is there anyway to recover that data?

Comment 15

[Simon Paquet [:sipaq]]

Ok, so today my total downloads went up by nearly 5000, which is the second biggest increase of one day ever. Interestingly the downloads per week went back to zero (0).

I'm not saying these numbers are false, but they surely are doubtful.

Comment 16

[Simon Paquet [:sipaq]]

Justin, it seems that the weekly download numbers are now totally going berserk.

Yesterday (comment 15) they were at 0, today they are at 2,685,652. That's over three times as many weekly downloads than total downloads, which are at 891,254.

Comment 17

[Chuck Baker]

Moreover, the numbers are updating in real time instead of once per day.

Comment 18

[Simon Paquet [:sipaq]]

(In reply to comment #17)
> Moreover, the numbers are updating in real time instead of once per day.

Which is really great.

Comment 19

[Justin Scott [:fligtar]]

It looks like the counter is parsing logs more than once for some reason, so I'll try to look into that today - but it might have to wait until Tuesday when I can sit down with a sysadmin and work all of these issues out.

If the numbers are updating in real time, it just means you were looking at the site as the download counter was running.

Comment 20

[Justin Scott [:fligtar]]

Looked into it.

The logs weren't being parsed twice - they were catching up from the lost days last week.

So, now everything looks good except for two days - 11/20 and 11/21. Those were the days before the script was fixed that a bunch of extra rows were made. I haven't figured out yet why the extra rows have the values they do, so I'm working on that and will decide whether we can fix them or need to delete them and reparse those 2 days.

The update ping script ran for 11/21 as well but had the same multi-rows problem, so I'm currently testing it to see if we just need to run it again now that it's fixed, or see if the fix didn't actually work. We didn't get cron output email so it's hard to tell.

Comment 21

[Stuart Parmenter]

fwiw, i'm now showing:
25741 (week)
4137 (total)

neither of which seem right.

Comment 22

[Michel Gutierrez]

My data (extension {b9db16a4-6edc-47ec-a1f4-b86292ed211d}):
38678698 (week)
2127010 (total)

The total might be correct, still have some doubts on the week count ;-)

Comment 23

[Justin Scott [:fligtar]]

This has been fixed for awhile.