511321 - [AMO] Add new cron jobs to preview.amo

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Wil Clouser [:clouserw]]

There are three new cron jobs for AMO:

1) This is a maintenance.php job, "share_count_totals".  It takes around 56s to run and I think it can go every 15 minutes.

2) /bin/build-recommendations.py.  This takes no arguments and could take around a minute to run.  Running every 30 minutes would be fine.  This requires python 2.6 and mysql connectivity.

3) /bin/recommend.py.  This also takes no arguements and close to no time to run.    Every 30 minutes should be good here too and this also requires python 2.6 with mysql.

Bug 510051 is fixed which should put the python we need on preview and production, but it only mentions it being on khan.  If it's not on preview we should reopen that bug and make this dependent on it.

Comment 1

[Jeff Balogh (:jbalogh)]

(In reply to comment #0)
> There are three new cron jobs for AMO:

Just 2, actually: `php maintenance.php share_count_totals` and `python build-recommendations.py`

> 3) /bin/recommend.py.  This also takes no arguements and close to no time to
> run.    Every 30 minutes should be good here too and this also requires python
> 2.6 with mysql.

This is not a script, it's a supporting library.

Comment 2

[Shyam Mani [:fox2mike]]

(In reply to comment #0)

> Bug 510051 is fixed which should put the python we need on preview and
> production, but it only mentions it being on khan.  If it's not on preview we
> should reopen that bug and make this dependent on it.

Nah, leave that bug alone! ;) (unless you really want to track using that...)

I'm working on getting the python you need into puppet so when preview.amo is updated, it'll get pulled in. I'll comment here when done, I'm using this to track that.

Comment 3

[Wil Clouser [:clouserw]]

> 1) This is a maintenance.php job, "share_count_totals".  It takes around 56s to
> run and I think it can go every 15 minutes.

This is not dependent on python.  Can we get this added?

Also, please rename the current tag_totals maintenance job to category_totals.  Thanks.

Comment 4

[Shyam Mani [:fox2mike]]

As a side note, I'm no longer working on the python bits since Dave and Jeremy felt an RPM would be best.

CC'ing Dave, he was working on the RPM.

Comment 5

[Wil Clouser [:clouserw]]

(In reply to comment #3)
> > 1) This is a maintenance.php job, "share_count_totals".  It takes around 56s to
> > run and I think it can go every 15 minutes.
> 
> This is not dependent on python.  Can we get this added?
> 
> Also, please rename the current tag_totals maintenance job to category_totals. 
> Thanks.

Neither of these depend on python.  Can we get them added please?  I didn't know this bug was going to turn into a python tracking bug but if we get those two out of the way we can focus on the python part.

Regarding the python part, we are code freezing today and expecting to launch on production next Thursday.  What's the status of the python?

Comment 6

[Wil Clouser [:clouserw]]

blocking QA, ->critical

Comment 7

[Jeremy Orem [:oremj]]

So far I've added:
*/15 * * * * root cd /data/amo/www/addons.mozilla.org-preview/bin; php -f maintenance.php share_count_totals

Comment 8

[Justin Scott [:fligtar]]

We are planning to launch the features that require this in production this Thursday night and only have a few days left of QA time. Is there any status update?

Comment 9

[Wil Clouser [:clouserw]]

(In reply to comment #5)
> (In reply to comment #3)
> > Also, please rename the current tag_totals maintenance job to category_totals. 

This still hasn't been done.  

Also, can you please add another maintenance.php job that runs every 8 minutes (it takes less than a second right now, but it will grow rapidly):  collection_stats

Comment 10

[Justin Scott [:fligtar]]

(In reply to comment #8)
> Is there any status update?

Please?

Comment 11

[Wil Clouser [:clouserw]]

(In reply to comment #9)
> (In reply to comment #5)
> > (In reply to comment #3)
> > > Also, please rename the current tag_totals maintenance job to category_totals. 
> 
> This still hasn't been done.  
> 
> Also, can you please add another maintenance.php job that runs every 8 minutes
> (it takes less than a second right now, but it will grow rapidly): 
> collection_stats

So we don't clutter up this bug more, this is now bug 512494.  This bug is now only about the python script:

2) /bin/build-recommendations.py.  This takes no arguments and could take
around a minute to run.  Running every 30 minutes would be fine.  This requires
python 2.6 and mysql connectivity.

Comment 12

[Dave Miller [:justdave]]

chizu found and fixed the problem we were having with the RPM of Python 2.6.2 we were trying to use before, so I'm going to go ahead and deploy that.  Working on setup right now and should hopefully have this on the preview site within the next hour, and barring any issues showing up there, can probably push it to the rest of the production servers later tonight.

Comment 13

[Dave Miller [:justdave]]

oremj wasn't here to ask, and I had trouble getting into the load balancer to look it up myself, so I just now found the server this is on (pm-app-amo24).  Unfortunately I'm out of time because I have a meeting to attend.  I should be back in about 2.5 hours and will resume then.  If someone else wants to try before I get back, the RPMs are all staged in mrepo/5Server-i386/RPMS.mozilla, just need to add that repo on that machine, then

yum install python26 MySQL-python26

Then try out the jobs listed above and make sure they work.  If that all checks out then we'll configure puppet to push it to the entire cluster later tonight.

Comment 14

[Jeremy Orem [:oremj]]

After tonight preview will be deployed against all the app nodes again. If you need we can continue to run it off one machine.

Comment 15

[Wil Clouser [:clouserw]]

This is blocking QA and we're launching tomorrow.  How can we help get this done ASAP?

Comment 16

[Dave Miller [:justdave]]

-bash-3.2$ python26 bin/build-recommendations.py 
/usr/lib/python2.6/site-packages/MySQLdb/__init__.py:34: DeprecationWarning: the sets module is deprecated
  from sets import ImmutableSet
Could not open input file: connection.php
Traceback (most recent call last):
  File "bin/build-recommendations.py", line 64, in <module>
    main()
  File "bin/build-recommendations.py", line 42, in main
    for s in ('read', 'write')]
  File "/usr/lib/python2.6/json/__init__.py", line 307, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.6/json/decoder.py", line 319, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.6/json/decoder.py", line 338, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

Comment 17

[Dave Miller [:justdave]]

aha, need to actually be in the bin directory.

ok, it's in cron every 30 minutes on pm-amo-app24.

The machine where the cron jobs usually run is x86_64, still working on building that version.

Comment 18

[matthew zeier [:mrz]]

This is spewing email to my inbox - anyone need that output?  Can it be dumped to /dev/null ?

Comment 19

[Wil Clouser [:clouserw]]

(In reply to comment #18)
> This is spewing email to my inbox - anyone need that output?  Can it be dumped
> to /dev/null ?

Is it errors?  Can you forward one of them to me?

Comment 20

[Wil Clouser [:clouserw]]

You can redirect the output for now.

Once this is in place on the x86_64 box the output should go to amo-developers with the rest of the cron jobs.

Thanks

Comment 21

[Wil Clouser [:clouserw]]

We're planning on doing the live push shortly - what is the status of this?

Comment 22

[Dave Miller [:justdave]]

got python deployed on mradm02 which is where the production cron jobs will run.