308691 - Addon listings do not show most recent versions

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task, P1)

Description

[Giuliano Masseroni [:jooliaan]]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; it; rv:1.8b4) Gecko/20050914 Firefox/1.4
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; it; rv:1.8b4) Gecko/20050914 Firefox/1.4

On the web page indicated above, the extension BBCodeXtra is still listed as
version 0.2.3 and not as 0.2.5 which is the latest version.
Instead by looking at the extension details on the web page
https://addons.mozilla.org/extensions/moreinfo.php?id=491&os=Windows&application=firefox&version=1.4&go=Go
BBCodeXtra is correctly listed as 0.2.5 that's the latest version

Reproducible: Always

Steps to Reproduce:
1.Go here
https://addons.mozilla.org/extensions/showlist.php?application=firefox&category=Miscellaneous&numpg=10&pageid=2
and the version is 0.2.3
2.Go here
https://addons.mozilla.org/extensions/moreinfo.php?id=491&os=Windows&application=firefox&version=1.4&go=Go
and the version is 0.2.5
3.

Actual Results:  
It is possible to see that the same extensions has two diufferent number of
release in depending on wich page are you looking.

Expected Results:  
To see the latest version on both pages.

Comment 1

[Michael Morgan [:morgamic]]

Looks like the queries used to select the most recent version are not matchin up
on the two pages.  Will have to debug this.

Comment 2

[Giuliano Masseroni [:jooliaan]]

I forgot to say that even the extension details are not updated on this webpage:
https://addons.mozilla.org/extensions/showlist.php?application=firefox&category=Miscellaneous&numpg=10&pageid=2

Comment 3

[Michael Morgan [:morgamic]]

This problem happens for all extensions, so it is a larger issue.

Comment 4

[Dave Miller [:justdave]]

The database replication process stopped a week ago and the cron job we're
supposed to have monitoring replication and yelling at us when it quits never
got set up (this is a relatively new database server).  Replication has
restarted, but as mentioned, it's a week behind, it's probably going to be a few
hours to catch up.

I've spot-checked the other two database servers that do replication, and the
other two did have the cron job set up already.  This one does now, too.

Comment 5

[Dave Miller [:justdave]]

OK, after watching it going for a while, at the rate it's going, it's probably
going to take about 2 days to get caught up.  There are currently 7 days 19
hours 30 minutes worth of replication logs waiting to be processed.  It was at
about 8 days even when I posted my last comment.

Comment 6

[Dave Miller [:justdave]]

We started doing some query analysis to find out why the hell there were enough
queries in the replication log for it to take 2 days to catch up...  and
discovered there's two multiple-row deletes based on a date range and two
inserts and/or updates happening every time someone downloads an extension,
(which on average happens 4.5 times per second).  The deletes were cleaning out
old entries more than 8 days old.  Every time someone downloaded an extension. 
Most of it was just maintenance type stuff that could easily be done from an
hourly cron job (thus running said delete 53 times in 53 hours instead of 1.8
million times in that 53 hours).

I just killed the replication process and did a full "load data from master"
database reload on the slave, and it took all of three seconds.  So replication
is now caught up.  Mike Morgan and I are working on getting those maintenance
queries moved into a cron job.

Comment 7

[Jens Bannmann]

*** Bug 308003 has been marked as a duplicate of this bug. ***

Comment 8

[Michael Morgan [:morgamic]]

Attachment: Patch to simplify install.php

Attached is a patch to simplify install.php by taking redundant tasks and
putting those into maintenance.php, which would be called by cron.  This means
that garbage collection and counter updates would not have to happen _every
time_ someone downloaded something.  This eliminates the excessive DELETE
statements and frees up the logs a bit, as Dave stated above.

Comment 9

[Dave Miller [:justdave]]

Is there any way to rig the maintenance script so it does nothing (or redirects
to the home page or something) if you hit it from the web, but it works properly
from the command line when I run it from cron?

Comment 10

[Jens Bannmann]

(In reply to comment #9)
> Is there any way to rig the maintenance script so it does nothing (or redirects
> to the home page or something) if you hit it from the web, but it works properly
> from the command line when I run it from cron?

An easy way might be to check for the presence of the usual server environment
variables that are unlikely to be set from cron, for example
_SERVER["HTTP_ACCEPT"] or _SERVER["HTTP_USER_AGENT"].

Comment 11

[Michael Morgan [:morgamic]]

(In reply to comment #9)
> Is there any way to rig the maintenance script so it does nothing from the web?
Ah - I forgot about this part.  I will find the best way to do this and submit
an updated patch.

Comment 12

[Jens Bannmann]

Comment on attachment 196694 [details] [diff] [review]
Patch to simplify install.php

>Index: core/maintenance.php
>===================================================================
>+// The Initial Developer of the Original Code is
>+// Chris "Wolf" Crews.
>+// Portions created by the Initial Developer are Copyright (C) 2004
>+// the Initial Developer. All Rights Reserved.
>+//
>+// Contributor(s):
>+//   Chris "Wolf" Crews <[e-mail]>

I think you want to change this :-)

Comment 13

[Michael Morgan [:morgamic]]

Attachment: Updated patch to limit requests to command line and/or cron

Also made a couple of fixes and added notes / TODOs.

Comment 14

[Michael Morgan [:morgamic]]

Attachment: Updated and improved maintenance/gc scripts.

New patch separates GC into its own script so it can be run seperately from
maintenance.php.  maintenance.php updated to properly handle and preserve all
count information.  This has been tested by me and verified, but I would like
another set of eyes to look at this and make sure it behaves properly. Also
fixed a webpath issue with the install() functions that utilize
XMLHttpRequest().

Comment 15

[Michael Morgan [:morgamic]]

Comment on attachment 198071 [details] [diff] [review]
Updated and improved maintenance/gc scripts.

Moved to bug 310631.