692115 - Setup cron and config for feed imports on mozilla.org/firefox pages

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Steven Garrity [:sgarrity]]

A few pages that import RSS feeds on mozilla.org/firefox pages have now been cleaned up and can now use a common cron job.

The cron script is: includes/feeds/feed-cache-cron.php

It can run at any frequency as it uses cache timeouts to know when to refresh data. We suggest every 2 minutes.

The cron script uses a config file. A config.ini.dist was committed with r95582. It should be renamed config.ini and modified as necessary.

Comment 1

[Jake Maul [:jakem]]

It looks like these changes have not been pushed to tags/production, so we can't do this just yet.

Also, there are 2 other feed-related cron's for mozilla.org:

*/10 * * * * root cd /data/static/www/www.mozilla.org-merged/org/includes/; ./feed-parser.php -c ./feed-parser-config.xml
25 * * * *   root cd /data/static/www/www.mozilla.org-merged/tools; php -f refresh-press-center-feed.php

Does this obsolete either of those?

Comment 2

[Shyam Mani [:fox2mike]]

Also, this should be on stage before it goes to prod.

Comment 3

[Jeremy Orem [:oremj]]

Jake, looks like you know what is going on here, so assigning to you.

Comment 4

[Michael Gauthier [:gauthierm]]

Jake, this new cron script does NOT obsolete either of those.

Comment 5

[Steven Garrity [:sgarrity]]

(In reply to Shyam Mani [:fox2mike] from comment #2)
> Also, this should be on stage before it goes to prod.

Yeah, can you set this up on stage first?

Comment 6

[Jake Maul [:jakem]]

I still can't do this. This hasn't been pushed to tags/production.

Furthermore, if you want to test in stage, it needs to be pushed to tags/stage as well.

Comment 7

[Steven Garrity [:sgarrity]]

(In reply to Jake Maul [:jakem] from comment #6)
> Furthermore, if you want to test in stage, it needs to be pushed to
> tags/stage as well.

Merged to stage in r96360.

Comment 8

[Jake Maul [:jakem]]

This has been completed on dev and stage. Please let me know how it looks and when this is pushed to tags/production, and I can get it done there too. Thanks!

Comment 9

[Steven Garrity [:sgarrity]]

(In reply to Jake Maul [:jakem] from comment #8)
> This has been completed on dev and stage. Please let me know how it looks
> and when this is pushed to tags/production, and I can get it done there too.

Thanks. Looks good on stage, and the code is now pushed to production (see Bug 683934). Please implement this cron on production and we're all set.

Comment 10

[Jake Maul [:jakem]]

Hmm, there's a problem with this...

[root@mradm02 feeds]# php feed-cache-cron.php
Feed cache cron config '/data/www/www.mozilla.org-merged/includes/feeds/cron-config.ini' missing.

The problem is that feed-cache-cron.php uses the "file_root" variable from includes/config.inc.php. The path in that file works on the web nodes, but not the admin node where the cron job runs.

This works on www.allizom.org and www-dev.allizom.org because those are both one node only... they don't have a separate admin node. That will change in the near future with the move to the bedrock cluster, so those will break in the future as well.

I see file_root is also referenced once in FeedCache.php.


Could this be reworked to use a relative directory and not rely on file_root ?

Comment 11

[Michael Gauthier [:gauthierm]]

Reworked to use relative directories in r96837 in trunk.

Comment 12

[Jake Maul [:jakem]]

Excellent, this still seems to work properly on www-dev.allizom.org, and I think it'll be good on prod now, too. Can we get this pushed to tags/stage and tags/production?

Comment 13

[Steven Garrity [:sgarrity]]

James, can you help us get this pushed to production?

Comment 14

[James Long (:jlongster)]

pushed to production r97199

Comment 15

[Steven Garrity [:sgarrity]]

(In reply to Jake Maul [:jakem] from comment #12)
> Excellent, this still seems to work properly on www-dev.allizom.org, and I
> think it'll be good on prod now, too. Can we get this pushed to tags/stage
> and tags/production?

Jake, can you confirm that this cron is in place in production?

Comment 16

[Jake Maul [:jakem]]

No, it was not. It is now.

It took a very long time to run the first time, but seems quick now. Guessing it needed to populate some past data or something.