We need a push of input.stage.mozilla.com to production on Tuesday, 11/16.
The only step to perform now (thanks to the amazing shyam) is to run:
We're flexible on time, but suggest 4pm.
Oremj, will we be able to add long expire headers as well as gzipping during this downtime? (bug 582866 comment 7). W.r.t. something to go off of, we can look what AMO is doing and probably just reuse their settings.
Sure, the entire push contains dozens of changes to the app that'll affect the user. Isn't that why you've set needs-downtime+ ?
Actually mrz, this shouldn't need downtime. THe gzipping, etc can be done at anytime, but I don't know if that'll require downtime.
The script won't produce downtime if all goes to plan.
(In reply to comment #0)
> The only step to perform now (thanks to the amazing shyam) is to run:
We need to make sure this script contains:
I don't think we minified before, so that's new. It uses Java, but I had that checked in bug 607460.
So, no user facing downtime? Who will be around when I do the push to test and verify? I presume we have a rollback plan if something goes south?
(In reply to comment #6)
> So, no user facing downtime? Who will be around when I do the push to test and
> verify? I presume we have a rollback plan if something goes south?
I'll be around; about to certify, and send an email to email@example.com (per the release checklist ), then sign off in this bug.
r/a=stephend for 2.0.
Something came up and I won't be able to do this push tomorrow. Punting back to server-ops, in hopes that someone else will be able to do it.
I'll move this a higher priority if no one picks it up by tomorrow mid-morning.
We're releasing at 4pm PST, need to have someone assigned here.
So one step that I think is missing in the script is that sphinx needs to be udpated as well and the compress assets command needs to run.
Fred, can you verify that that's all that needs to be done.
We're babied by preview which auto-updates and runs other scripts automatically.
Fred and I can work on updating this script, and storing it in our repo.
(In reply to comment #4)
> Actually mrz, this shouldn't need downtime.
That wasn't the case. What was missing from this push?
Next push will require an announced downtime window.
(In reply to comment #14)
> (In reply to comment #4)
> > Actually mrz, this shouldn't need downtime.
> That wasn't the case. What was missing from this push?
> Next push will require an announced downtime window.
That's fair. Sphinx needed to be updated first.
the long version:
We relied on the update script which didn't update the sphinx nodes.
The script didn't update the sphinx nodes, because the sphinx config generally never changes.
This push relied on some big changes to our sphinx configuration.
If you're talking about the script in comment #0, that's never going to be modified to update sphinx configs, since sphinx configs are pushed out by puppet and will have to be done separately (by hand, for production).
Shyam - good to know. Then in our regular push instructions for Input we'll just need to make note of that.
Matt, is there going to be an announced downtime window for today? There are a number of fixes for the next version of Input, 2.1, that are waiting to merge onto our trunk once we go live and we'd like to ramp up development as those fixes are needed for Mobile Firefox Beta 3.
I didn't catch the Whiteboard's date - default windows are Tuesday/Thursdays - was expecting this to be on tomorrow's schedule.
Okay changing the release date then. Please announce a downtime window for Input for tomorrow then.
Push is live!
Apparently, I didn't do enough regression-testing on 610576, so we're living with that while we get 2.1 ready, but, verified FIXED, as the rest of the changes are in and working.
Phong wanted me to update this:
I made a number of changes to the push script. I found it somewhat confusing that it was pulling into two different places (maybe for the settings_local.py) and then rsync (--delete)ing one version over the other - that was in effect removing our compressed assets. We rewrote it to build the compressed assets in /python so they could rsync over to the /django directory.
In any case we (fwenzel and I) figured this out fairly quickly, and were able to jointly push with jlazaro and phong's help.
Good teamwork :)