773739 - (fx14push) [Bedrock] [Fx14] (mozilla.org) push on [7/17] [8:00 AM PST]

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Mike Alexis [:malexis]]

Web Ops Bedrock push for Tuesday July 17.

Fx14 targeting 8:00 AM PST launch so need coordinate. Details:

- Vidyo to war room in Release Coordination Vidyo room.
- For those supporting the release, please plan on arriving at your chosen location at 7:30 AM PST. Target launch time is 8AM PST.
- If you wish to call in, you can dial into : 1-800-707-2533 (pin 369) Conf# 99951 (US)
- You can also follow along in the following IRC channel:
• #planning (general)
• #Fx14launch (Web deployment)

Comment 1

[Anthony Ricaud (:rik)]

Assigning to Dan after a discussion with Brandon.

Comment 2

[Anthony Ricaud (:rik)]

Here are the redirects to put in place just after the push.

RewriteRule ^/en-US/projects/mozilla-based(/?)$ /b/en-US/projects/mozilla-based/ [PT]
RewriteRule ^/(\w{2,3}(?:-\w{2}(?:-mac)?)?/)?projects/mozilla-based(/?)$ /en-US/projects/mozilla-based/ [L,R=302]

RewriteRule ^/en-US/products(/?)$ /b/en-US/products/ [PT]
RewriteRule ^/(\w{2,3}(?:-\w{2}(?:-mac)?)?/)?products(/?)$ /en-US/products/ [L,R=302]

RewriteRule ^/(\w{2,3}(?:-\w{2}(?:-mac)?)?/)?firefox(/(?:\d+\.\d+\.?(?:\d+)?\.?(?:\d+)?(?:[a|b]?)(?:\d*)(?:pre)?(?:\d)?))?/whatsnew(/?)$ /b/$1firefox$2/whatsnew/ [PT]

Comment 3

[Anthony Ricaud (:rik)]

Also, I'll need to push r107518 from bug 763665.

Comment 4

[Daniel Maher [:phrawzty]]

The last time that we pushed this it took upwards of 15 minutes to compile and apply the Puppet manifests against all of the machines.  I *think* this was because I used commander to initialise a Puppet run on all of the machines at the same time - something that the Puppetmaster was unable to deal with.  In order to test this hypothesis, I'm going to run each Puppet update one at a time, which should also have the side-effect of a faster TTC for applying the modifications.

Comment 5

[Anthony Ricaud (:rik)]

Because of the perf issue in bug 760570, I'm returing 301s with Vary: User-Agent for the whatsnew page. This should reduce the number of requests we send to the origin servers.

But it could also fill the proxy cache because of the number of different user-agents that we see and not be helpful.

Comment 6

[Daniel Maher [:phrawzty]]

[[e-mail]1 ~]$ sudo /data/bedrock/src/update-www.mozilla.org-django.sh
+ git pull -q
+ git submodule -q sync
+ git submodule -q update --init
+ pushd vendor
+ git submodule -q sync
+ git submodule -q update --init
+ popd
+ python manage.py compress_assets
/data/bedrock/src/www.mozilla.org-django/bedrock/vendor/src/funfactory/funfactory/manage.py:47: UserWarning: You're using an old-style Playdoh layout with a top level __init__.py and apps directories. This is error prone and fights the Zen of Python. See http://playdoh.readthedocs.org/en/latest/upgrading.html.
  warnings.warn("You're using an old-style Playdoh layout with a top "
To turn on cache busting, use settings.CACHEBUST_IMGS
Minifying /data/bedrock/src/www.mozilla.org-django/bedrock/media/css/projects_mozilla_based-all.css (using YUI Compressor)
Minifying /data/bedrock/src/www.mozilla.org-django/bedrock/media/css/firefox_whatsnew-all.css (using YUI Compressor)
Minifying /data/bedrock/src/www.mozilla.org-django/bedrock/media/css/products-all.css (using YUI Compressor)
Unchanged files skipped for minification: 66
+ python manage.py update_product_details
/data/bedrock/src/www.mozilla.org-django/bedrock/vendor/src/funfactory/funfactory/manage.py:47: UserWarning: You're using an old-style Playdoh layout with a top level __init__.py and apps directories. This is error prone and fights the Zen of Python. See http://playdoh.readthedocs.org/en/latest/upgrading.html.
  warnings.warn("You're using an old-style Playdoh layout with a top "
Product Details were up to date.
+ /data/bedrock/src/update-www.mozilla.org-svn-locale.sh
+ /data/bedrock/deploy www.mozilla.org-django
+ touch wsgi/playdoh.wsgi
+ issue-multi-command bedrock /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi
[2012-07-17 07:42:44] [bedrock1.webapp.scl3.mozilla.com] running: /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi
[2012-07-17 07:42:44] [bedrock2.webapp.scl3.mozilla.com] running: /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi
[2012-07-17 07:42:44] [bedrock3.webapp.scl3.mozilla.com] running: /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi
[2012-07-17 07:42:44] [bedrock4.webapp.scl3.mozilla.com] running: /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi
[2012-07-17 07:42:44] [bedrock5.webapp.scl3.mozilla.com] running: /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi
[2012-07-17 07:42:44] [bedrock1.webapp.phx1.mozilla.com] running: /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi
[2012-07-17 07:42:44] [bedrock2.webapp.phx1.mozilla.com] running: /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi
[2012-07-17 07:42:44] [bedrock3.webapp.phx1.mozilla.com] running: /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi
[2012-07-17 07:42:45] [bedrock3.webapp.phx1.mozilla.com] finished: /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi (0.900s)
[2012-07-17 07:42:46] [bedrock2.webapp.phx1.mozilla.com] finished: /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi (1.088s)
[2012-07-17 07:42:46] [bedrock1.webapp.scl3.mozilla.com] finished: /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi (1.160s)
[2012-07-17 07:42:46] [bedrock2.webapp.scl3.mozilla.com] finished: /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi (1.160s)
[2012-07-17 07:42:46] [bedrock3.webapp.scl3.mozilla.com] finished: /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi (1.204s)
[2012-07-17 07:42:46] [bedrock5.webapp.scl3.mozilla.com] finished: /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi (1.195s)
[2012-07-17 07:42:46] [bedrock4.webapp.scl3.mozilla.com] finished: /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi (1.203s)
[2012-07-17 07:42:47] [bedrock1.webapp.phx1.mozilla.com] finished: /bin/touch /data/www/www.mozilla.org-django/bedrock/wsgi/playdoh.wsgi (2.067s)

Comment 7

[Daniel Maher [:phrawzty]]

(In reply to Daniel Maher [:phrawzty] from comment #4)
> deal with.  In order to test this hypothesis, I'm going to run each Puppet
> update one at a time, which should also have the side-effect of a faster TTC
> for applying the modifications.

Indeed, executing the Puppet runs one at a time on each webhead was *much* faster than trying to compile them all at once.  Clearly the Puppetmasters can't deal with compiling more than one or two manifests at a time.  :/

In any case, Puppet has updated the Apache configs on all of the webheads.