611180 - Push Firefox Input 2.0 to production on 11/18

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Fred Wenzel [:wenzel]]

We need a push of input.stage.mozilla.com to production on Tuesday, 11/16.

The only step to perform now (thanks to the amazing shyam) is to run:

/root/bin/input_update.sh 


We're flexible on time, but suggest 4pm.

Comment 1

[Fred Wenzel [:wenzel]]

Oremj, will we be able to add long expire headers as well as gzipping during this downtime? (bug 582866 comment 7). W.r.t. something to go off of, we can look what AMO is doing and probably just reuse their settings.

Comment 2

[matthew zeier [:mrz]]

User impacting?

Comment 3

[Fred Wenzel [:wenzel]]

Sure, the entire push contains dozens of changes to the app that'll affect the user. Isn't that why you've set needs-downtime+ ?

Comment 4

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

Actually mrz, this shouldn't need downtime.  THe gzipping, etc can be done at anytime, but I don't know if that'll require downtime.

The script won't produce downtime if all goes to plan.

Comment 5

[Fred Wenzel [:wenzel]]

(In reply to comment #0)
> The only step to perform now (thanks to the amazing shyam) is to run:
> 
> /root/bin/input_update.sh 

We need to make sure this script contains:
./manage.py compress_assets

I don't think we minified before, so that's new. It uses Java, but I had that checked in bug 607460.

Comment 6

[Justin Dow [:jabba]]

So, no user facing downtime? Who will be around when I do the push to test and verify? I presume we have a rollback plan if something goes south?

Comment 7

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to comment #6)
> So, no user facing downtime? Who will be around when I do the push to test and
> verify? I presume we have a rollback plan if something goes south?

I'll be around; about to certify, and send an email to [e-mail] (per the release checklist [1]), then sign off in this bug.

[1] https://wiki.mozilla.org/Firefox/Input/Release_Checklist

Comment 8

[Stephen Donner [:stephend] Not actively reading bugmail]

r/a=stephend for 2.0.

Comment 9

[Justin Dow [:jabba]]

Something came up and I won't be able to do this push tomorrow. Punting back to server-ops, in hopes that someone else will be able to do it.

Comment 10

[Aakash Desai [:aakashd]]

I'll move this a higher priority if no one picks it up by tomorrow mid-morning.

Comment 11

[Aakash Desai [:aakashd]]

We're releasing at 4pm PST, need to have someone assigned here.

Comment 12

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

So one step that I think is missing in the script is that sphinx needs to be udpated as well and the compress assets command needs to run.

Fred, can you verify that that's all that needs to be done.

We're babied by preview which auto-updates and runs other scripts automatically.

-d

Comment 13

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

http://it.pastebin.mozilla.org/854706


Fred and I can work on updating this script, and storing it in our repo.

Comment 14

[matthew zeier [:mrz]]

(In reply to comment #4)
> Actually mrz, this shouldn't need downtime.  

That wasn't the case.  What was missing from this push?

Next push will require an announced downtime window.

Comment 15

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

(In reply to comment #14)
> (In reply to comment #4)
> > Actually mrz, this shouldn't need downtime.  
> 
> That wasn't the case.  What was missing from this push?
> 
> Next push will require an announced downtime window.

tl;dr version:
That's fair.  Sphinx needed to be updated first.

the long version:
We relied on the update script which didn't update the sphinx nodes.

The script didn't update the sphinx nodes, because the sphinx config generally never changes.

This push relied on some big changes to our sphinx configuration.

Comment 16

[Shyam Mani [:fox2mike]]

If you're talking about the script in comment #0, that's never going to be modified to update sphinx configs, since sphinx configs are pushed out by puppet and will have to be done separately (by hand, for production).

Comment 17

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

Shyam - good to know.  Then in our regular push instructions for Input we'll just need to make note of that.

-d

Comment 18

[Aakash Desai [:aakashd]]

Matt, is there going to be an announced downtime window for today? There are a number of fixes for the next version of Input, 2.1, that are waiting to merge onto our trunk once we go live and we'd like to ramp up development as those fixes are needed for Mobile Firefox Beta 3.

Comment 19

[matthew zeier [:mrz]]

I didn't catch the Whiteboard's date - default windows are Tuesday/Thursdays - was expecting this to be on tomorrow's schedule.

Comment 20

[Fred Wenzel [:wenzel]]

Okay changing the release date then. Please announce a downtime window for Input for tomorrow then.

Comment 21

[Justin Lazaro [:jlaz] (use needinfo)]

Push is live!

Comment 22

[Stephen Donner [:stephend] Not actively reading bugmail]

Apparently, I didn't do enough regression-testing on 610576, so we're living with that while we get 2.1 ready, but, verified FIXED, as the rest of the changes are in and working.

Sorry :-(

Comment 23

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

Phong wanted me to update this:

I made a number of changes to the push script.  I found it somewhat confusing that it was pulling into two different places (maybe for the settings_local.py) and then rsync (--delete)ing one version over the other - that was in effect removing our compressed assets.  We rewrote it to build the compressed assets in /python so they could rsync over to the /django directory.

In any case we (fwenzel and I) figured this out fairly quickly, and were able to jointly push with jlazaro and phong's help.

Good teamwork :)