310631 - Install script should not do maintenance tasks

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: Public Pages, defect)

Description

[Michael Morgan [:morgamic]]

install.php performs maintenance tasks for statistics that should be instead
pushed out to another separate script(s) called by cron instead.  By separating
these tasks, we could eliminate a lot of queries that don't need to happen
_everytime_ someone downloads an addon.  This would dramatically lighten the
load on osdb01 and osdb02 and increase the overall performance of the entire
addons site as a result.

Comment 1

[Michael Morgan [:morgamic]]

Attachment: Maintenance scripts and an updated install.php

See also the conversation in bug 308691.  This patch is supposed to be a better
way to maintain statistics when addons are downloaded.	Mike, could you please
do some sanity checking on this script?  Dave, might want to have a look as
well.  Thanks.

Comment 2

[Mike Connor [:mconnor]]

Comment on attachment 198075 [details] [diff] [review]
Maintenance scripts and an updated install.php

r=me, with the tweak from IRC

it'd be nice in theory to clean up the formatting, but not worth the time at
this juncture

Comment 3

[Michael Morgan [:morgamic]]

Attachment: Fixed a minor JS error w/ themes installation.

This should be ready for primetime.

Comment 4

[Michael Morgan [:morgamic]]

The count scripts should work, I have tested them and am about to check them in.
 Main concern becomes performance for these, as they are huge sets of queries. 
That said, indexing, timeouts and scheduling become interesting challenges. 
Below is a plan that might help us hash this out.

Upgrade plan for `downloads`:
1) ID to int(11) UNSIGNED
2) user_agent to a varchar(255) or less -- or ignore it?
3) add INDEX for ID, date, vID, user_ip, user_agent
4) add INDEX for `counted`
5) remove all local changes to UMO .php files (except for config.php)
6) update from -rMOZILLA_UPDATE_1_0_BRANCH
7) check timeouts for php and mysql so we don't have maintenance.php die halfway
8) set up cron for maintenance tasks
    a) php -q core/maintenance.php weekly -- daily?
    b) php -q core/maintenance.php total -- 20 min?
    c) php -q core/maintenance.php gc -- daily?
9) re-verify that it works in production

Comment 5

[Michael Morgan [:morgamic]]

Comment on attachment 198323 [details] [diff] [review]
Fixed a minor JS error w/ themes installation.

CVS contains the most recent copy.

Comment 6

[Dave Miller [:justdave]]

(In reply to comment #4)
> 8) set up cron for maintenance tasks
>     a) php -q core/maintenance.php weekly -- daily?
>     b) php -q core/maintenance.php total -- 20 min?
>     c) php -q core/maintenance.php gc -- daily?

b) is in root's crontab to run every 20 minutes.
a) and c) have been placed in /etc/cron.daily/addons_daily_maint.cron, which
contains the following:

#!/bin/sh
cd /opt/update/core
# update the last-7-days download counts
php -q maintenance.php weekly
# garbage collection
php -q maintenance.php gc

Everything else was deployed as above.

I removed the existing index on ID in the downloads table, as it's redundant
with the new multi-index, since ID is the first column of that index.

Comment 7

[Dave Miller [:justdave]]

for the record, weekly and gc combined take about 20 seconds to run, so I've switched them to hourly instead of daily.

The total one that we're running every 20 minutes takes 3 to 4 minutes to run on average, so 20 minutes is probably a good period for that (since it'll probably continue to take longer as our usage grows).