Closed Bug 489905 Opened 16 years ago Closed 16 years ago

Firefox Downloads appear to have dropped by factor 10!

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: ottodv, Unassigned)

References

Details

User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032713 Fedora/3.0.8-1.fc9 Firefox/3.0.8 Build Identifier: I keep track of the number of Firefox downloads and downloads per second. On April 22nd the number of Firefox downloads appears to have dropped to 1/10th of it's previous totals. Example April 22/23: 889,625,189 downloads at 0.9 per second on 2009-04-23 22:30 UTC 889,528,437 downloads at 1 per second on 2009-04-22 22:11 UTC = 96,752 downloads over a period of 24 hours and 19 minutes. Example April 21/22: 889,365,938 downloads at 20.7 per second on 2009-04-22 15:37 UTC 888,128,932 downloads at 17.2 per second on 2009-04-21 15:32 UTC = 1,237,006 downloads over a period of 24 hours and 5 minutes. Source: http://twitter.com/FirefoxCounter Also surprising is that the drop occurred at a specific time: 889,510,689 downloads at 1.5 per second on 2009-04-22 18:21 UTC 889,509,982 downloads at 15.4 per second on 2009-04-22 18:13 UTC One moment the download rate is 15.4 per second and just 7 minutes later it's only 1.5 per second. You can see from the twitter feed that download rates prior to that moment are mostly in the 10-20 range while those after this specific moment are all aound 1 download per second. I can only conclude from this that something changed in those 7 minutes on April 22nd between 18:13 UTC and 18:21 UTC. Reproducible: Always The download counter is becoming more and more visible in the run up to the 1 billionth download, so I think it's important to get this right.
We disabled bouncer logging, so that's why... Guess we could probably re-enable it now.
When bouncer logging is disabled, are those downloads lost forever to it? Would we be able to take some "known good" number and use an alternate mechanism for counting, or would gaps from bouncer logging being intermittently disabled permanently skew the count?
(In reply to comment #2) > When bouncer logging is disabled, are those downloads lost forever to it? Well, we still have apache logs of the downloads, but bouncer's mysql db won't have any of the downloads made while logging was off. When I say "bouncer logging", I'm only referring to the internal database counts bouncer keeps of a download and what mirror it was downloaded from. Because this counting can really cause the database issues, we routinely turn it off during a major release in order to ease pressure on the bouncer database servers. As far as I know, only sfx uses this number, so it's not a real big deal. Metrics parses the actual apache logs, correct? > Would we be able to take some "known good" number and use an alternate > mechanism for counting, or would gaps from bouncer logging being intermittently > disabled permanently skew the count? We could take the apache logs and repopulate the bouncer mysql db with them, but I'm not sure it's worth our time... As I said, we pretty routinely disable logging, and, afaik, only sfx download counter is affected.
"We could take the apache logs and repopulate the bouncer mysql db with them, but I'm not sure it's worth our time..." I would say it is definitely worth the spreadfirefox communities time. I think we should make a post on spreadfirefox to let everyone know what is happening with the firefox download counter Does all this mean that it is possible we have already passed the billon download? Best, Paul
That is what I was getting at. We could look at trying to switch over to metrics for counting downloads, but without a starting point for the cumulative number, it would be extremely difficult to come up with the right number.
Perhaps we should switch focus to how many firefox downloads there were last month. The figure being accurately determined by parsing apache logs at the end of the month. If the firefox download figure is not accurate but instead at best a lower estimate with no real idea of the margin off error involved i think now would be a good time to reveal this and let our community now that we have probably passed the billion mark already than we leave it and approach the billion download mark in several month time. Paul
I've turned bouncer logging back on for now. Can I resolve this bug as fixed now and let the topic of whether we should use something else for sfx download counts go into some other bug somewhere?
"I would say it is definitely worth the spreadfirefox communities time." Totally agree with Paul, the SFx community uses this metric as a marketing tool. It's highly visible, the counter being inconsistent makes us look bad. If we change to a new and more reliable counting mechanism, it's probably best to continue from where the counter is right now.
@metrics folk - could we invent some new process than uses apache logs and does nightly/hourly inserts into the database, not unlike the stats processing you do for AMO? It's going to be nearly impossible to continue to scale bouncer logging this current configuration.
Think we'll need webdev to lead with the process and dev, but can move to post-processing with metrics handling it. This might piss a bunch of people off as it won't be real-time, but I think we need to move this direction to scale.
Seems like oremj had been repopulating the database from the logs after logging got re-enabled the previous times we've disabled it, but maybe I'm remembering wrong.
The SeaMonkey Download stats at http://dev.seamonkey.at/download-stats are populated daily by reading a feed from a PHP on download.m.o - I guess those numbers are from the DB as well and we also get missing counts there during major Firefox releases, right? In that case, this is suboptimal as well, of course :(
Depends on: 464778
(In reply to comment #10) > Think we'll need webdev to lead with the process and dev, but can move to > post-processing with metrics handling it. Bug 464778 comment #1 seems to suggest that there's not code change needed - we'd just leave logging off and process access_logs.
Hi guys: Catching up here. I reading through the comments and see that Paul noted that we might have already passed 1 billion. Is this the case?
(In reply to comment #14) > Hi guys: Catching up here. I reading through the comments and see that Paul > noted that we might have already passed 1 billion. Is this the case? Eh, it's possible, maybe, but we don't know for sure. Pretty sure none of the download day numbers from Firefox 3 were included in SFx's count, as we disabled bouncer logging then, and then we've disabled it during a lot of our past Firefox releases due to the load it causes on the database servers. I don't know if we're off by a 100 million, though -- that's a lot of downloads, and people usually complain within a week if bouncer logging gets turned off and not turned back on after whatever release has finished.
Just to be clear are we saying that spreadfirefox has a counter for the number of downloadeds that can be turned off at any time without anyone at spreadfirefox be notified and that afterwards no effort is then made to recalculate the numbers from the apache logs for missing day(s). If that's the case i think we should try to do better for the spreadfirefox commmunity Best, Paul
Yes, and we should certainly work on building a more reliable method for supplying this metric. Keep in mind though, we don't have any ability to track downloads from mirror sites like download.com. That alone is easily tens of millions of downloads we haven't "seen". Also, even within the downloads we do see, we can't say with certainty which downloads are "countable" or not. Bots and spiders download from our site, people sometimes might download three or four times in a row depending on trouble they are having with their machine or their internet. It would be extremely difficult to have a metric for downloads that would not run into all these fuzzy spots.
True and add to that that Linux users usually get Firefox along with their distribution (my guess is that is 2-4% of total Firefox downloads). But that's still no reason not to give a more or less accurate count of downloads directly from mozilla. I would like to know what the plan is for when 3.5 get's released, I think a lot of people will love to see the download rates shoot up (or the FirefoxTweets on Twitter tweet multiple times per day). It's all part of creating more buzz around the event.
Just adding to what Daniel said in comment #17... I'm not entirely sure what one's definition is for a "download". Our download numbers are based on bouncer, which tells us "download requests". Past analysis has shown that perhaps 20% of download request don't result in the user receiving the full data. Both accuracy and the 1 Billion milestone are critical for the entire community (as Paul points out). Before we get too far down the path of correcting for the particular issue highlighted by this bug, it seems like we should have a bit of understanding and consensus around what we define as a "download" and the imperfections that come along with that particular methodology/definition.
I just disabled logging again in bouncer. DBs were backing up under the 3.0.10 load.
Going to close this. The original comment #0 has been answered (we disabled logging, and do that often during a release). Long term we'll re-do bouncer logging with Metrics. That discussion is over in bug 464778.
Status: UNCONFIRMED → RESOLVED
Closed: 16 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.