Closed Bug 759564 Opened 12 years ago Closed 12 years ago

Please use product-details to update bedrock download links regularly

Categories

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Product:
Infrastructure & Operations Graveyard
Component:
WebOps: Other
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: akeybl, Assigned: bburton)

References

Details

(Whiteboard: [waiting on code changes]) 

Please use product-details to update bedrock download links regularly. This will prevent us from needing to request that links are updated whenever we release a new version of Firefox or Fennec.

This cron job should happen very regularly - as regularly as production rollouts upon checkin (once every ~15min).

James should have more details if necessary.
Blocks: 759577
James, more detail here please?  This is greek to me.   :(

I don't see anything about product-details in our update/push procedures (unless it is just a part of the script).  Please help supplement our docs with this info.

https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=1802733
product-details is a set of JSON files that are tracked in git. These files specify all of the current version of firefox to serve. We usually update them with a django management command and then commit.

The problem is that now that the site isn't auto-updating, every time any version of Firefox is bumped (minor versions, beta, aurora, etc) we need to do coordinate a push.

This has the potential to be automated, but since we track the JSON in git, I'm not sure how that would look. I'm currently talking with solarce on IRC about it.
Alex, the key problem is that we are moving away from automatically updating the sites every 15 minutes. We are moving towards continuous deployment, but that takes work and Bedrock doesn't have that set up yet. So for now we just need IT to push a lot.

The product-details is so easy to push that maybe IT can do it:

./manage.py update_product_details
git add -A
git commit -m 'update product-details'
git push origin master
<push to webheads>

Alex, then all you have to do is file a bug and IT can jump on it quickly. This will be solved in the coming months when we have CI and can push ourselves, but for now that shouldn't be hard. Corey, any thoughts?
(In reply to James Long (:jlongster) from comment #3)
> Alex, then all you have to do is file a bug and IT can jump on it quickly.
> This will be solved in the coming months when we have CI and can push
> ourselves, but for now that shouldn't be hard. Corey, any thoughts?

To set expectations properly, this is something that we'll have to request 2-4 times a week.
(In reply to James Long (:jlongster) from comment #3)
> Alex, the key problem is that we are moving away from automatically updating
> the sites every 15 minutes. We are moving towards continuous deployment, but
> that takes work and Bedrock doesn't have that set up yet. So for now we just
> need IT to push a lot.
> 
> The product-details is so easy to push that maybe IT can do it:
> 
> ./manage.py update_product_details
> git add -A
> git commit -m 'update product-details'
> git push origin master
> <push to webheads>
> 
> Alex, then all you have to do is file a bug and IT can jump on it quickly.
> This will be solved in the coming months when we have CI and can push
> ourselves, but for now that shouldn't be hard. Corey, any thoughts?

I've cc'd Corey and Jake to get their input, as this sounds like it'd require us to have a user that could commit back to the github repo too
I'm confused. :)

We have lots of sites that run an update_product_details cron job... none of them commit back upstream. Why would this one be different?
(In reply to Jake Maul [:jakem] from comment #6)
> I'm confused. :)
> 
> We have lots of sites that run an update_product_details cron job... none of
> them commit back upstream. Why would this one be different?

Do those sites track the json changes in git? Probably not. Maybe we don't have to either, and a quick `./manage.py update_product_details` needs to be added to the installation docs.

That sounds fine with me (remove from git, add to .gitignore, make a cron job)
Blocks: 760737
James mentioned today that we should probably start pulling these files directly from SVN and not need them to be committed to Github as well

Given that possible path, how should we proceed?
Assignee: server-ops → nobody
Component: Server Operations: Web Operations → Operations
Product: mozilla.org → Mozilla Services
QA Contact: cshields → operations
Version: other → unspecified
Assignee: nobody → server-ops
Component: Operations → Server Operations: Web Operations
Product: Mozilla Services → mozilla.org
QA Contact: operations → cshields
Version: unspecified → other
Assignee: server-ops → bburton
It's not that we should pull them directly from SVN (because that version has a bunch of PHP stuff), but rather that we shouldn't track it in git or SVN at all and simply run the update_product_details management command periodically.

We'll make that command part of the installation of bedrock as well, so devs will get a copy when they check out the site.

I removed the json from git in this PR: https://github.com/mozilla/bedrock/pull/153

Once that's merged to master, you can push it out to the webheads, and run `./manage.py update_product_details` yourself. Then set up a cronjob to do it automatically.
Once that's done, we should be able to make a cron job, like we have for the svn locale stuff, that runs the command and just deploys the json files
It should be merged to master now, so place do a push and manual `./manage.py update_product_details`. We should do this soon before we forget to do that on the next push, which would completely break the site.

Also set up the cron job.
Working on this, to start, does this look correct?

[root@bedrockadm.private.phx1 bedrock]# git pull
remote: Counting objects: 125, done.
remote: Compressing objects: 100% (39/39), done.
remote: Total 84 (delta 56), reused 67 (delta 39)
Unpacking objects: 100% (84/84), done.
From git://github.com/mozilla/bedrock
   7e5b172..977d4cf  master     -> origin/master
   e45718e..e237115  dev        -> origin/dev
 * [new branch]      responsive -> origin/responsive
Updating 7e5b172..977d4cf
Fast-forward
 apps/firefox/templates/firefox/central.html                            |    1 +
 docs/install.rst                                                       |    6 ++++++
 lib/product_details_json/.gitignore                                    |    1 +
 lib/product_details_json/.last_update                                  |    1 -
 lib/product_details_json/firefox_beta_builds.json                      |    1 -
 lib/product_details_json/firefox_history_development_releases.json     |    1 -
 lib/product_details_json/firefox_history_major_releases.json           |    1 -
 lib/product_details_json/firefox_history_stability_releases.json       |    1 -
 lib/product_details_json/firefox_primary_builds.json                   |    1 -
 lib/product_details_json/firefox_versions.json                         |    1 -
 lib/product_details_json/languages.json                                |    1 -
 lib/product_details_json/mobile_details.json                           |    1 -
 lib/product_details_json/mobile_history_development_releases.json      |    1 -
 lib/product_details_json/mobile_history_major_releases.json            |    1 -
 lib/product_details_json/mobile_history_stability_releases.json        |    1 -
 lib/product_details_json/regions/.last_update                          |    1 -
 lib/product_details_json/regions/af.json                               |    1 -
 lib/product_details_json/regions/ak.json                               |    1 -
 lib/product_details_json/regions/ar.json                               |    1 -
 lib/product_details_json/regions/as.json                               |    1 -
 lib/product_details_json/regions/ast.json                              |    1 -
 lib/product_details_json/regions/be.json                               |    1 -
 lib/product_details_json/regions/bg.json                               |    1 -
 lib/product_details_json/regions/bn-BD.json                            |    1 -
 lib/product_details_json/regions/bn-IN.json                            |    1 -
 lib/product_details_json/regions/br.json                               |    1 -
 lib/product_details_json/regions/ca.json                               |    1 -
 lib/product_details_json/regions/cs.json                               |    1 -
 lib/product_details_json/regions/cy.json                               |    1 -
 lib/product_details_json/regions/da.json                               |    1 -
 lib/product_details_json/regions/de.json                               |    1 -
 lib/product_details_json/regions/el.json                               |    1 -
 lib/product_details_json/regions/en-GB.json                            |    1 -
 lib/product_details_json/regions/en-US.json                            |    1 -
 lib/product_details_json/regions/en-ZA.json                            |    1 -
 lib/product_details_json/regions/eo.json                               |    1 -
 lib/product_details_json/regions/es-AR.json                            |    1 -
 lib/product_details_json/regions/es-CL.json                            |    1 -
 lib/product_details_json/regions/es-ES.json                            |    1 -
 lib/product_details_json/regions/es-MX.json                            |    1 -
 lib/product_details_json/regions/et.json                               |    1 -
 lib/product_details_json/regions/eu.json                               |    1 -
 lib/product_details_json/regions/fa.json                               |    1 -
 lib/product_details_json/regions/fi.json                               |    1 -
 lib/product_details_json/regions/fr.json                               |    1 -
 lib/product_details_json/regions/fy-NL.json                            |    1 -
 lib/product_details_json/regions/ga-IE.json                            |    1 -
 lib/product_details_json/regions/gd.json                               |    1 -
 lib/product_details_json/regions/gl.json                               |    1 -
 lib/product_details_json/regions/gu-IN.json                            |    1 -
 lib/product_details_json/regions/he.json                               |    1 -
 lib/product_details_json/regions/hi-IN.json                            |    1 -
 lib/product_details_json/regions/hr.json                               |    1 -
 lib/product_details_json/regions/hu.json                               |    1 -
 lib/product_details_json/regions/hy-AM.json                            |    1 -
 lib/product_details_json/regions/id.json                               |    1 -
 lib/product_details_json/regions/is.json                               |    1 -
 lib/product_details_json/regions/it.json                               |    1 -
 lib/product_details_json/regions/ka.json                               |    1 -
 lib/product_details_json/regions/kk.json                               |    1 -
 lib/product_details_json/regions/kn.json                               |    1 -
 lib/product_details_json/regions/ko.json                               |    1 -
 lib/product_details_json/regions/ku.json                               |    1 -
 lib/product_details_json/regions/lg.json                               |    1 -
 lib/product_details_json/regions/lt.json                               |    1 -
 lib/product_details_json/regions/lv.json                               |    1 -
 lib/product_details_json/regions/mai.json                              |    1 -
 lib/product_details_json/regions/mk.json                               |    1 -
 lib/product_details_json/regions/ml.json                               |    1 -
 lib/product_details_json/regions/mn.json                               |    1 -
 lib/product_details_json/regions/mr.json                               |    1 -
 lib/product_details_json/regions/nb-NO.json                            |    1 -
 lib/product_details_json/regions/nl.json                               |    1 -
 lib/product_details_json/regions/nn-NO.json                            |    1 -
 lib/product_details_json/regions/nso.json                              |    1 -
 lib/product_details_json/regions/oc.json                               |    1 -
 lib/product_details_json/regions/or.json                               |    1 -
 lib/product_details_json/regions/pa-IN.json                            |    1 -
 lib/product_details_json/regions/pl.json                               |    1 -
 lib/product_details_json/regions/pt-BR.json                            |    1 -
 lib/product_details_json/regions/pt-PT.json                            |    1 -
 lib/product_details_json/regions/rm.json                               |    1 -
 lib/product_details_json/regions/ro.json                               |    1 -
 lib/product_details_json/regions/ru.json                               |    1 -
 lib/product_details_json/regions/si.json                               |    1 -
 lib/product_details_json/regions/sk.json                               |    1 -
 lib/product_details_json/regions/sl.json                               |    1 -
 lib/product_details_json/regions/son.json                              |    1 -
 lib/product_details_json/regions/sq.json                               |    1 -
 lib/product_details_json/regions/sr.json                               |    1 -
 lib/product_details_json/regions/sv-SE.json                            |    1 -
 lib/product_details_json/regions/ta-LK.json                            |    1 -
 lib/product_details_json/regions/ta.json                               |    1 -
 lib/product_details_json/regions/te.json                               |    1 -
 lib/product_details_json/regions/th.json                               |    1 -
 lib/product_details_json/regions/tr.json                               |    1 -
 lib/product_details_json/regions/uk.json                               |    1 -
 lib/product_details_json/regions/vi.json                               |    1 -
 lib/product_details_json/regions/zh-CN.json                            |    1 -
 lib/product_details_json/regions/zh-TW.json                            |    1 -
 lib/product_details_json/regions/zu.json                               |    1 -
 lib/product_details_json/thunderbird_beta_builds.json                  |    1 -
 lib/product_details_json/thunderbird_history_development_releases.json |    1 -
 lib/product_details_json/thunderbird_history_major_releases.json       |    1 -
 lib/product_details_json/thunderbird_history_stability_releases.json   |    1 -
 lib/product_details_json/thunderbird_primary_builds.json               |    1 -
 lib/product_details_json/thunderbird_versions.json                     |    1 -
 media/css/sandstone/buttons.less                                       |    9 ++++++---
 media/img/landing/devices/firefox-phone.png                            |  Bin 91054 -> 120119 bytes
 109 files changed, 14 insertions(+), 107 deletions(-)
 create mode 100644 lib/product_details_json/.gitignore
 delete mode 100644 lib/product_details_json/.last_update
 delete mode 100644 lib/product_details_json/firefox_beta_builds.json
 delete mode 100644 lib/product_details_json/firefox_history_development_releases.json
 delete mode 100644 lib/product_details_json/firefox_history_major_releases.json
 delete mode 100644 lib/product_details_json/firefox_history_stability_releases.json
 delete mode 100644 lib/product_details_json/firefox_primary_builds.json
 delete mode 100644 lib/product_details_json/firefox_versions.json
 delete mode 100644 lib/product_details_json/languages.json
 delete mode 100644 lib/product_details_json/mobile_details.json
 delete mode 100644 lib/product_details_json/mobile_history_development_releases.json
 delete mode 100644 lib/product_details_json/mobile_history_major_releases.json
 delete mode 100644 lib/product_details_json/mobile_history_stability_releases.json
 delete mode 100644 lib/product_details_json/regions/.last_update
 delete mode 100644 lib/product_details_json/regions/af.json
 delete mode 100644 lib/product_details_json/regions/ak.json
 delete mode 100644 lib/product_details_json/regions/ar.json
 delete mode 100644 lib/product_details_json/regions/as.json
 delete mode 100644 lib/product_details_json/regions/ast.json
 delete mode 100644 lib/product_details_json/regions/be.json
 delete mode 100644 lib/product_details_json/regions/bg.json
 delete mode 100644 lib/product_details_json/regions/bn-BD.json
 delete mode 100644 lib/product_details_json/regions/bn-IN.json
 delete mode 100644 lib/product_details_json/regions/br.json
 delete mode 100644 lib/product_details_json/regions/ca.json
 delete mode 100644 lib/product_details_json/regions/cs.json
 delete mode 100644 lib/product_details_json/regions/cy.json
 delete mode 100644 lib/product_details_json/regions/da.json
 delete mode 100644 lib/product_details_json/regions/de.json
 delete mode 100644 lib/product_details_json/regions/el.json
 delete mode 100644 lib/product_details_json/regions/en-GB.json
 delete mode 100644 lib/product_details_json/regions/en-US.json
 delete mode 100644 lib/product_details_json/regions/en-ZA.json
 delete mode 100644 lib/product_details_json/regions/eo.json
 delete mode 100644 lib/product_details_json/regions/es-AR.json
 delete mode 100644 lib/product_details_json/regions/es-CL.json
 delete mode 100644 lib/product_details_json/regions/es-ES.json
 delete mode 100644 lib/product_details_json/regions/es-MX.json
 delete mode 100644 lib/product_details_json/regions/et.json
 delete mode 100644 lib/product_details_json/regions/eu.json
 delete mode 100644 lib/product_details_json/regions/fa.json
 delete mode 100644 lib/product_details_json/regions/fi.json
 delete mode 100644 lib/product_details_json/regions/fr.json
 delete mode 100644 lib/product_details_json/regions/fy-NL.json
 delete mode 100644 lib/product_details_json/regions/ga-IE.json
 delete mode 100644 lib/product_details_json/regions/gd.json
 delete mode 100644 lib/product_details_json/regions/gl.json
 delete mode 100644 lib/product_details_json/regions/gu-IN.json
 delete mode 100644 lib/product_details_json/regions/he.json
 delete mode 100644 lib/product_details_json/regions/hi-IN.json
 delete mode 100644 lib/product_details_json/regions/hr.json
 delete mode 100644 lib/product_details_json/regions/hu.json
 delete mode 100644 lib/product_details_json/regions/hy-AM.json
 delete mode 100644 lib/product_details_json/regions/id.json
 delete mode 100644 lib/product_details_json/regions/is.json
 delete mode 100644 lib/product_details_json/regions/it.json
 delete mode 100644 lib/product_details_json/regions/ka.json
 delete mode 100644 lib/product_details_json/regions/kk.json
 delete mode 100644 lib/product_details_json/regions/kn.json
 delete mode 100644 lib/product_details_json/regions/ko.json
 delete mode 100644 lib/product_details_json/regions/ku.json
 delete mode 100644 lib/product_details_json/regions/lg.json
 delete mode 100644 lib/product_details_json/regions/lt.json
 delete mode 100644 lib/product_details_json/regions/lv.json
 delete mode 100644 lib/product_details_json/regions/mai.json
 delete mode 100644 lib/product_details_json/regions/mk.json
 delete mode 100644 lib/product_details_json/regions/ml.json
 delete mode 100644 lib/product_details_json/regions/mn.json
 delete mode 100644 lib/product_details_json/regions/mr.json
 delete mode 100644 lib/product_details_json/regions/nb-NO.json
 delete mode 100644 lib/product_details_json/regions/nl.json
 delete mode 100644 lib/product_details_json/regions/nn-NO.json
 delete mode 100644 lib/product_details_json/regions/nso.json
 delete mode 100644 lib/product_details_json/regions/oc.json
 delete mode 100644 lib/product_details_json/regions/or.json
 delete mode 100644 lib/product_details_json/regions/pa-IN.json
 delete mode 100644 lib/product_details_json/regions/pl.json
 delete mode 100644 lib/product_details_json/regions/pt-BR.json
 delete mode 100644 lib/product_details_json/regions/pt-PT.json
 delete mode 100644 lib/product_details_json/regions/rm.json
 delete mode 100644 lib/product_details_json/regions/ro.json
 delete mode 100644 lib/product_details_json/regions/ru.json
 delete mode 100644 lib/product_details_json/regions/si.json
 delete mode 100644 lib/product_details_json/regions/sk.json
 delete mode 100644 lib/product_details_json/regions/sl.json
 delete mode 100644 lib/product_details_json/regions/son.json
 delete mode 100644 lib/product_details_json/regions/sq.json
 delete mode 100644 lib/product_details_json/regions/sr.json
 delete mode 100644 lib/product_details_json/regions/sv-SE.json
 delete mode 100644 lib/product_details_json/regions/ta-LK.json
 delete mode 100644 lib/product_details_json/regions/ta.json
 delete mode 100644 lib/product_details_json/regions/te.json
 delete mode 100644 lib/product_details_json/regions/th.json
 delete mode 100644 lib/product_details_json/regions/tr.json
 delete mode 100644 lib/product_details_json/regions/uk.json
 delete mode 100644 lib/product_details_json/regions/vi.json
 delete mode 100644 lib/product_details_json/regions/zh-CN.json
 delete mode 100644 lib/product_details_json/regions/zh-TW.json
 delete mode 100644 lib/product_details_json/regions/zu.json
 delete mode 100644 lib/product_details_json/thunderbird_beta_builds.json
 delete mode 100644 lib/product_details_json/thunderbird_history_development_releases.json
 delete mode 100644 lib/product_details_json/thunderbird_history_major_releases.json
 delete mode 100644 lib/product_details_json/thunderbird_history_stability_releases.json
 delete mode 100644 lib/product_details_json/thunderbird_primary_builds.json
 delete mode 100644 lib/product_details_json/thunderbird_versions.json
Status: NEW → ASSIGNED
ran ./manage.py update_product_details

Pushed to prod

[root@bedrockadm.private.phx1 bedrock]# /data/bedrock/src/update-www.mozilla.org-django.sh 
+ git pull -q
+ git submodule -q sync
+ git submodule -q update --init
+ pushd vendor
+ git submodule -q sync
+ git submodule -q update --init
+ popd
+ python manage.py compress_assets
/data/bedrock/src/www.mozilla.org-django/bedrock/vendor/src/funfactory/funfactory/manage.py:47: UserWarning: You're using an old-style Playdoh layout with a top level __init__.py and apps directories. This is error prone and fights the Zen of Python. See http://playdoh.readthedocs.org/en/latest/upgrading.html.
  warnings.warn("You're using an old-style Playdoh layout with a top "
To turn on cache busting, use settings.CACHEBUST_IMGS
Minifying /data/bedrock/src/www.mozilla.org-django/bedrock/media/css/persona-all.css (using YUI Compressor)
Minifying /data/bedrock/src/www.mozilla.org-django/bedrock/media/css/firefox-all.css (using YUI Compressor)
Minifying /data/bedrock/src/www.mozilla.org-django/bedrock/media/css/firefox_central-all.css (using YUI Compressor)
Minifying /data/bedrock/src/www.mozilla.org-django/bedrock/media/css/landing_devices-all.css (using YUI Compressor)
Minifying /data/bedrock/src/www.mozilla.org-django/bedrock/media/css/firefox_happy-all.css (using YUI Compressor)
Minifying /data/bedrock/src/www.mozilla.org-django/bedrock/media/css/firefox_security-all.css (using YUI Compressor)
Minifying /data/bedrock/src/www.mozilla.org-django/bedrock/media/css/b2g-all.css (using YUI Compressor)
Minifying /data/bedrock/src/www.mozilla.org-django/bedrock/media/css/firefox_customize-all.css (using YUI Compressor)
Minifying /data/bedrock/src/www.mozilla.org-django/bedrock/media/css/firefox_fx-all.css (using YUI Compressor)
Minifying /data/bedrock/src/www.mozilla.org-django/bedrock/media/css/firefox_new-all.css (using YUI Compressor)
Minifying /data/bedrock/src/www.mozilla.org-django/bedrock/media/css/privacy-all.css (using YUI Compressor)
Minifying /data/bedrock/src/www.mozilla.org-django/bedrock/media/css/firefox_features-all.css (using YUI Compressor)
Minifying /data/bedrock/src/www.mozilla.org-django/bedrock/media/css/sandstone_guide-all.css (using YUI Compressor)
Minifying /data/bedrock/src/www.mozilla.org-django/bedrock/media/css/firefox_speed-all.css (using YUI Compressor)
Minifying /data/bedrock/src/www.mozilla.org-django/bedrock/media/css/firefox_updates-all.css (using YUI Compressor)
Minifying /data/bedrock/src/www.mozilla.org-django/bedrock/media/css/firefox_geolocation-all.css (using YUI Compressor)
Minifying /data/bedrock/src/www.mozilla.org-django/bedrock/media/css/dnt-all.css (using YUI Compressor)
Minifying /data/bedrock/src/www.mozilla.org-django/bedrock/media/css/firefox_technology-all.css (using YUI Compressor)
Minifying /data/bedrock/src/www.mozilla.org-django/bedrock/media/css/firefox_organizations-all.css (using YUI Compressor)
Minifying /data/bedrock/src/www.mozilla.org-django/bedrock/media/css/common-all.css (using YUI Compressor)
Minifying /data/bedrock/src/www.mozilla.org-django/bedrock/media/css/firefox_channel-all.css (using YUI Compressor)
Unchanged files skipped for minification: 34
+ /data/bedrock/src/update-www.mozilla.org-svn-locale.sh
+ touch wsgi/playdoh.wsgi
+ /data/bedrock/deploy www.mozilla.org-django
+ issue-multi-command bedrock /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi
[2012-06-07 12:48:15] [bedrock1.webapp.scl3.mozilla.com] running: /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi
[2012-06-07 12:48:15] [bedrock2.webapp.scl3.mozilla.com] running: /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi
[2012-06-07 12:48:15] [bedrock3.webapp.scl3.mozilla.com] running: /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi
[2012-06-07 12:48:15] [bedrock4.webapp.scl3.mozilla.com] running: /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi
[2012-06-07 12:48:15] [bedrock5.webapp.scl3.mozilla.com] running: /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi
[2012-06-07 12:48:15] [bedrock1.webapp.phx1.mozilla.com] running: /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi
[2012-06-07 12:48:15] [bedrock2.webapp.phx1.mozilla.com] running: /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi
[2012-06-07 12:48:15] [bedrock3.webapp.phx1.mozilla.com] running: /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi
[2012-06-07 12:48:16] [bedrock2.webapp.phx1.mozilla.com] finished: /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi (1.165s)
[2012-06-07 12:48:17] [bedrock1.webapp.phx1.mozilla.com] finished: /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi (1.397s)
[2012-06-07 12:48:19] [bedrock3.webapp.phx1.mozilla.com] finished: /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi (3.908s)
[2012-06-07 12:48:19] [bedrock3.webapp.scl3.mozilla.com] finished: /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi (4.046s)
[2012-06-07 12:48:19] [bedrock1.webapp.scl3.mozilla.com] finished: /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi (4.056s)
[2012-06-07 12:48:19] [bedrock4.webapp.scl3.mozilla.com] finished: /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi (4.056s)
[2012-06-07 12:48:19] [bedrock2.webapp.scl3.mozilla.com] finished: /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi (4.067s)
[2012-06-07 12:48:19] [bedrock5.webapp.scl3.mozilla.com] finished: /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi (4.052s)
Cron deployed via puppet to run the following script

#!/bin/sh

python /data/bedrock/src/www.mozilla.org-django/bedrock/manage.py update_product_details &> /dev/null
echo -e "finished at $(date)" > /data/bedrock/src/www.mozilla.org-django/bedrock/media/product_details_finished.txt
/data/bedrock/deploy www.mozilla.org-django &> /dev/null


Will run at 15 after every hour

15 * * * * root /data/bedrock/src/update-www.mozilla.org-product-details.sh

You can check the last time it ran at http://www.mozilla.org/media/product_details_finished.txt (may take a few minutes to update after push, due to Zeus caching)
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Blocks: 763029
(In reply to Brandon Burton [:solarce] from comment #14)
> Cron deployed via puppet to run the following script
> 
> #!/bin/sh
> 
> python /data/bedrock/src/www.mozilla.org-django/bedrock/manage.py
> update_product_details &> /dev/null
> echo -e "finished at $(date)" >
> /data/bedrock/src/www.mozilla.org-django/bedrock/media/
> product_details_finished.txt
> /data/bedrock/deploy www.mozilla.org-django &> /dev/null
> 
> 
> Will run at 15 after every hour
> 
> 15 * * * * root /data/bedrock/src/update-www.mozilla.org-product-details.sh
> 
> You can check the last time it ran at
> http://www.mozilla.org/media/product_details_finished.txt (may take a few
> minutes to update after push, due to Zeus caching)

Can we do this more regularly? Perhaps every 15 minutes? Waiting up to an hour will cause us to have to continue to request manual pushes (like bug 763029).
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Runs every 15 minutes now.
Status: REOPENED → RESOLVED
Closed: 12 years ago → 12 years ago
Resolution: --- → FIXED
Okay, it seems like this maybe isn't really feasible.

As far as I can tell, the app has to be reloaded to notice changes to the product-details files. That takes a small amount of time (up to 15s or so), during which time the app is unresponsive. Hourly this might not be a major issue, whereas every 15 minutes it might be a concern.

Comments jlongster?
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(In reply to Jake Maul [:jakem] from comment #17)
> Okay, it seems like this maybe isn't really feasible.
> 
> As far as I can tell, the app has to be reloaded to notice changes to the
> product-details files. That takes a small amount of time (up to 15s or so),
> during which time the app is unresponsive. Hourly this might not be a major
> issue, whereas every 15 minutes it might be a concern.
> 
> Comments jlongster?

After reviewing things and discussing with :jakem, I have to lend a +1 to this, due to the way the code loads the .json files, restarting all wsgi daemons is required and that has a performance slowdown every time it is done.

If getting the updates within a 15-20 minute time frame of when they're committed to SVN is needed, then we'd recommend the code be reviewed to see if it can be modified to load the files on request
Thanks guys. Makes sense. I agree that we don't want to be restarting the processes every 15 minutes.

I should be able to modify the product-details library to support memcache, and we can just set a TTL on it for 15 minutes.
Waiting on memcache, bug 738381, and code changes outlined above, will review on 6/20
Status: REOPENED → ASSIGNED
Depends on: 738381
Whiteboard: [review 6/20]
Waiting on memcache, bug 738381, and code changes outlined above, will review on 6/20
Blocks: 765377
(In reply to Brandon Burton [:solarce] from comment #21)
> Waiting on memcache, bug 738381, and code changes outlined above, will
> review on 6/20

:jlongster, I just wanted to follow up on the code and when that's expected to be ready to test and deploy
Whiteboard: [review 6/20] → [review 6/25]
I'm uncomfortable with this pattern we're falling into of cron jobs that update memcache. I think we need to find a better approach.

MY THOUGHTS ON MEMCACHE USAGE:

Typically memcache is used as a cache for faster access, rather than the sole source of truth. If memcache goes down (or just doesn't have the relevant key), speed should be impacted but functionality should not be (at least, unless the alternative is so slow as to be nonfunctional).

If we have to restart the memcached daemon for some reason (server move, hardware failure, etc), the cache will be flushed. If the admin node is down and the cron doesn't run, the keys will expire. Anything that relies on memcache and doesn't have a "real" source to go to will fail... in this case, that seems very bad.

This all leads me to this: if memcache doesn't have the data, you should be able to go to the real source and then populate memcache with it. This is a much more resilient design.



POTENTIAL SOLUTIONS:

Off the top of my head, maybe the product details data could be periodically inserted into a database table, and bedrock could read from that (and optionally cache in memcache)? We do have plans for bedrock to eventually need a database anyway...

Perhaps we can inject some logic to the current (file-based, non-memcache-using) product-details cron job, and have it intelligently reload mod_wsgi only if needed. A simple "find . -type f -mnewer" command might suffice as a trigger, even. This might be the most direct solution and require the fewest changes.

Another potential fix to this whole scenario is to finally get around to setting up Chief (bug 730008). This would enable selected (non-webops) people to update the site as needed, without relying on a simple 15-minute timer. This is another thing that will eventually happen anyway (and soon, I hope).
Whiteboard: [review 6/25] → [waiting on code changes]
I think we can close this one given that Chief is now set up (bug 730008).

Alex: If anyone needs a push to refresh product-details data, you can ask me in #www. (and pmac or mkelly if I'm not around)
Status: ASSIGNED → RESOLVED
Closed: 12 years ago → 12 years ago
Resolution: --- → FIXED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.