Last Comment Bug 611180 - Push Firefox Input 2.0 to production on 11/18
: Push Firefox Input 2.0 to production on 11/18
Status: VERIFIED FIXED
11/18/2010 @ 4pm
:
Product: Infrastructure & Operations
Classification: Other
Component: WebOps: Other (show other bugs)
: other
: All All
: -- normal (vote)
: ---
Assigned To: Justin Lazaro [:jlaz]
: matthew zeier [:mrz]
Mentors:
Depends on:
Blocks: 582866
  Show dependency treegraph
 
Reported: 2010-11-10 17:02 PST by Fred Wenzel [:wenzel]
Modified: 2013-10-09 10:29 PDT (History)
10 users (show)
mzeier: needs‑downtime+
See Also:
Due Date:
QA Whiteboard:
Iteration: ---
Points: ---
Cab Review: ServiceNow Change Request (use flag)


Attachments

Description Fred Wenzel [:wenzel] 2010-11-10 17:02:46 PST
We need a push of input.stage.mozilla.com to production on Tuesday, 11/16.

The only step to perform now (thanks to the amazing shyam) is to run:

/root/bin/input_update.sh 


We're flexible on time, but suggest 4pm.
Comment 1 Fred Wenzel [:wenzel] 2010-11-12 11:31:52 PST
Oremj, will we be able to add long expire headers as well as gzipping during this downtime? (bug 582866 comment 7). W.r.t. something to go off of, we can look what AMO is doing and probably just reuse their settings.
Comment 2 matthew zeier [:mrz] 2010-11-12 11:32:44 PST
User impacting?
Comment 3 Fred Wenzel [:wenzel] 2010-11-12 12:03:07 PST
Sure, the entire push contains dozens of changes to the app that'll affect the user. Isn't that why you've set needs-downtime+ ?
Comment 4 Dave Dash [:davedash, :dd] (assign all bugs to mbrandt) 2010-11-12 12:08:41 PST
Actually mrz, this shouldn't need downtime.  THe gzipping, etc can be done at anytime, but I don't know if that'll require downtime.

The script won't produce downtime if all goes to plan.
Comment 5 Fred Wenzel [:wenzel] 2010-11-12 12:13:27 PST
(In reply to comment #0)
> The only step to perform now (thanks to the amazing shyam) is to run:
> 
> /root/bin/input_update.sh 

We need to make sure this script contains:
./manage.py compress_assets

I don't think we minified before, so that's new. It uses Java, but I had that checked in bug 607460.
Comment 6 Justin Dow [:jabba] 2010-11-15 11:23:39 PST
So, no user facing downtime? Who will be around when I do the push to test and verify? I presume we have a rollback plan if something goes south?
Comment 7 Stephen Donner [:stephend] 2010-11-15 11:37:19 PST
(In reply to comment #6)
> So, no user facing downtime? Who will be around when I do the push to test and
> verify? I presume we have a rollback plan if something goes south?

I'll be around; about to certify, and send an email to input@mozilla.com (per the release checklist [1]), then sign off in this bug.

[1] https://wiki.mozilla.org/Firefox/Input/Release_Checklist
Comment 8 Stephen Donner [:stephend] 2010-11-15 12:42:39 PST
r/a=stephend for 2.0.
Comment 9 Justin Dow [:jabba] 2010-11-15 15:50:50 PST
Something came up and I won't be able to do this push tomorrow. Punting back to server-ops, in hopes that someone else will be able to do it.
Comment 10 Aakash Desai [:aakashd] 2010-11-15 16:00:07 PST
I'll move this a higher priority if no one picks it up by tomorrow mid-morning.
Comment 11 Aakash Desai [:aakashd] 2010-11-16 12:23:49 PST
We're releasing at 4pm PST, need to have someone assigned here.
Comment 12 Dave Dash [:davedash, :dd] (assign all bugs to mbrandt) 2010-11-16 16:35:19 PST
So one step that I think is missing in the script is that sphinx needs to be udpated as well and the compress assets command needs to run.

Fred, can you verify that that's all that needs to be done.

We're babied by preview which auto-updates and runs other scripts automatically.

-d
Comment 13 Dave Dash [:davedash, :dd] (assign all bugs to mbrandt) 2010-11-16 16:38:11 PST
http://it.pastebin.mozilla.org/854706


Fred and I can work on updating this script, and storing it in our repo.
Comment 14 matthew zeier [:mrz] 2010-11-16 16:44:15 PST
(In reply to comment #4)
> Actually mrz, this shouldn't need downtime.  

That wasn't the case.  What was missing from this push?

Next push will require an announced downtime window.
Comment 15 Dave Dash [:davedash, :dd] (assign all bugs to mbrandt) 2010-11-16 16:49:25 PST
(In reply to comment #14)
> (In reply to comment #4)
> > Actually mrz, this shouldn't need downtime.  
> 
> That wasn't the case.  What was missing from this push?
> 
> Next push will require an announced downtime window.

tl;dr version:
That's fair.  Sphinx needed to be updated first.

the long version:
We relied on the update script which didn't update the sphinx nodes.

The script didn't update the sphinx nodes, because the sphinx config generally never changes.

This push relied on some big changes to our sphinx configuration.
Comment 16 Shyam Mani [:fox2mike] 2010-11-16 20:28:16 PST
If you're talking about the script in comment #0, that's never going to be modified to update sphinx configs, since sphinx configs are pushed out by puppet and will have to be done separately (by hand, for production).
Comment 17 Dave Dash [:davedash, :dd] (assign all bugs to mbrandt) 2010-11-16 20:30:42 PST
Shyam - good to know.  Then in our regular push instructions for Input we'll just need to make note of that.

-d
Comment 18 Aakash Desai [:aakashd] 2010-11-17 07:49:24 PST
Matt, is there going to be an announced downtime window for today? There are a number of fixes for the next version of Input, 2.1, that are waiting to merge onto our trunk once we go live and we'd like to ramp up development as those fixes are needed for Mobile Firefox Beta 3.
Comment 19 matthew zeier [:mrz] 2010-11-17 10:24:50 PST
I didn't catch the Whiteboard's date - default windows are Tuesday/Thursdays - was expecting this to be on tomorrow's schedule.
Comment 20 Fred Wenzel [:wenzel] 2010-11-17 13:20:01 PST
Okay changing the release date then. Please announce a downtime window for Input for tomorrow then.
Comment 21 Justin Lazaro [:jlaz] 2010-11-18 17:29:12 PST
Push is live!
Comment 22 Stephen Donner [:stephend] 2010-11-18 17:40:02 PST
Apparently, I didn't do enough regression-testing on 610576, so we're living with that while we get 2.1 ready, but, verified FIXED, as the rest of the changes are in and working.

Sorry :-(
Comment 23 Dave Dash [:davedash, :dd] (assign all bugs to mbrandt) 2010-11-18 20:34:35 PST
Phong wanted me to update this:

I made a number of changes to the push script.  I found it somewhat confusing that it was pulling into two different places (maybe for the settings_local.py) and then rsync (--delete)ing one version over the other - that was in effect removing our compressed assets.  We rewrote it to build the compressed assets in /python so they could rsync over to the /django directory.

In any case we (fwenzel and I) figured this out fairly quickly, and were able to jointly push with jlazaro and phong's help.

Good teamwork :)

Note You need to log in before you can comment on or make changes to this bug.