Push Firefox Input 2.0 to production on 11/18

VERIFIED FIXED

Status

Infrastructure & Operations
WebOps: Other
VERIFIED FIXED
7 years ago
4 years ago

People

(Reporter: wenzel, Assigned: jlaz)

Tracking

other
Bug Flags:
needs-downtime +

Details

(Whiteboard: 11/18/2010 @ 4pm)

(Reporter)

Description

7 years ago
We need a push of input.stage.mozilla.com to production on Tuesday, 11/16.

The only step to perform now (thanks to the amazing shyam) is to run:

/root/bin/input_update.sh 


We're flexible on time, but suggest 4pm.

Updated

7 years ago
Whiteboard: 11/16

Updated

7 years ago
Severity: major → normal

Updated

7 years ago
Flags: needs-downtime+
Whiteboard: 11/16 → 11/16/2010 @ 4pm
(Reporter)

Comment 1

7 years ago
Oremj, will we be able to add long expire headers as well as gzipping during this downtime? (bug 582866 comment 7). W.r.t. something to go off of, we can look what AMO is doing and probably just reuse their settings.

Comment 2

7 years ago
User impacting?
(Reporter)

Comment 3

7 years ago
Sure, the entire push contains dozens of changes to the app that'll affect the user. Isn't that why you've set needs-downtime+ ?
Actually mrz, this shouldn't need downtime.  THe gzipping, etc can be done at anytime, but I don't know if that'll require downtime.

The script won't produce downtime if all goes to plan.
(Reporter)

Comment 5

7 years ago
(In reply to comment #0)
> The only step to perform now (thanks to the amazing shyam) is to run:
> 
> /root/bin/input_update.sh 

We need to make sure this script contains:
./manage.py compress_assets

I don't think we minified before, so that's new. It uses Java, but I had that checked in bug 607460.
(Reporter)

Updated

7 years ago
Blocks: 582866

Comment 6

7 years ago
So, no user facing downtime? Who will be around when I do the push to test and verify? I presume we have a rollback plan if something goes south?
Assignee: server-ops → jdow
(In reply to comment #6)
> So, no user facing downtime? Who will be around when I do the push to test and
> verify? I presume we have a rollback plan if something goes south?

I'll be around; about to certify, and send an email to input@mozilla.com (per the release checklist [1]), then sign off in this bug.

[1] https://wiki.mozilla.org/Firefox/Input/Release_Checklist
r/a=stephend for 2.0.

Comment 9

7 years ago
Something came up and I won't be able to do this push tomorrow. Punting back to server-ops, in hopes that someone else will be able to do it.
Assignee: jdow → server-ops
I'll move this a higher priority if no one picks it up by tomorrow mid-morning.
We're releasing at 4pm PST, need to have someone assigned here.
Severity: normal → blocker
(Assignee)

Updated

7 years ago
Assignee: server-ops → jlazaro
So one step that I think is missing in the script is that sphinx needs to be udpated as well and the compress assets command needs to run.

Fred, can you verify that that's all that needs to be done.

We're babied by preview which auto-updates and runs other scripts automatically.

-d
http://it.pastebin.mozilla.org/854706


Fred and I can work on updating this script, and storing it in our repo.
(In reply to comment #4)
> Actually mrz, this shouldn't need downtime.  

That wasn't the case.  What was missing from this push?

Next push will require an announced downtime window.
Severity: blocker → normal
Summary: Push Firefox Input 2.0 to production on 11/16 → Push Firefox Input 2.0 to production on 11/17
Whiteboard: 11/16/2010 @ 4pm → 11/17/2010 @ 4pm

Updated

7 years ago
Assignee: jlazaro → server-ops
(In reply to comment #14)
> (In reply to comment #4)
> > Actually mrz, this shouldn't need downtime.  
> 
> That wasn't the case.  What was missing from this push?
> 
> Next push will require an announced downtime window.

tl;dr version:
That's fair.  Sphinx needed to be updated first.

the long version:
We relied on the update script which didn't update the sphinx nodes.

The script didn't update the sphinx nodes, because the sphinx config generally never changes.

This push relied on some big changes to our sphinx configuration.
If you're talking about the script in comment #0, that's never going to be modified to update sphinx configs, since sphinx configs are pushed out by puppet and will have to be done separately (by hand, for production).
Shyam - good to know.  Then in our regular push instructions for Input we'll just need to make note of that.

-d
Matt, is there going to be an announced downtime window for today? There are a number of fixes for the next version of Input, 2.1, that are waiting to merge onto our trunk once we go live and we'd like to ramp up development as those fixes are needed for Mobile Firefox Beta 3.
I didn't catch the Whiteboard's date - default windows are Tuesday/Thursdays - was expecting this to be on tomorrow's schedule.
(Reporter)

Comment 20

7 years ago
Okay changing the release date then. Please announce a downtime window for Input for tomorrow then.
Summary: Push Firefox Input 2.0 to production on 11/17 → Push Firefox Input 2.0 to production on 11/18
Whiteboard: 11/17/2010 @ 4pm → 11/18/2010 @ 4pm
(Assignee)

Updated

7 years ago
Assignee: server-ops → jlazaro
(Assignee)

Comment 21

7 years ago
Push is live!
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
Apparently, I didn't do enough regression-testing on 610576, so we're living with that while we get 2.1 ready, but, verified FIXED, as the rest of the changes are in and working.

Sorry :-(
Status: RESOLVED → VERIFIED
Phong wanted me to update this:

I made a number of changes to the push script.  I found it somewhat confusing that it was pulling into two different places (maybe for the settings_local.py) and then rsync (--delete)ing one version over the other - that was in effect removing our compressed assets.  We rewrote it to build the compressed assets in /python so they could rsync over to the /django directory.

In any case we (fwenzel and I) figured this out fairly quickly, and were able to jointly push with jlazaro and phong's help.

Good teamwork :)
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.