Last Comment Bug 593280 - Push Firefox Input to production on 9/28
: Push Firefox Input to production on 9/28
09/28/2010 @ 7pm
Product: Infrastructure & Operations
Classification: Other
Component: WebOps: Other (show other bugs)
: other
: All All
: -- blocker (vote)
: ---
Assigned To: Shyam Mani [:fox2mike]
: matthew zeier [:mrz]
: 597001 (view as bug list)
Depends on: 600070
  Show dependency treegraph
Reported: 2010-09-02 23:01 PDT by Fred Wenzel [:wenzel]
Modified: 2013-10-09 10:29 PDT (History)
6 users (show)
mzeier: needs‑downtime+
See Also:
Due Date:
QA Whiteboard:
Iteration: ---
Points: ---
Cab Review: ServiceNow Change Request (use flag)


Description Fred Wenzel [:wenzel] 2010-09-02 23:01:40 PDT
We need a push of to production on Tuesday, 9/8.

The only step to perform now (thanks to the amazing shyam) is to run:


We're flexible on time, but I am traveling and will be at a conference in the Pacific time zone that day, so we'll need to find a time that plays well with that. Alternatively, Dave could take care of the push like he did the last time?
Comment 1 Shyam Mani [:fox2mike] 2010-09-03 00:00:07 PDT
Tuesday is the 7th.
Comment 2 Aakash Desai [:aakashd] 2010-09-07 10:23:05 PDT
Ok, we're going to push this back to Thursday, 9/16, per the usual time. I'm leaving the whiteboard entry as is until you're ok with it, Shyam.
Comment 3 Shyam Mani [:fox2mike] 2010-09-07 10:28:36 PDT
If you're okay with it, so am I :)
Comment 4 Shyam Mani [:fox2mike] 2010-09-16 08:47:03 PDT
Okay, so there were too many changes to paste here, looking at this :

Updating reporter...
HEAD is now at 7f57260... Needed to disable Frisian as Babel does not know fy_NL.
Running migration 5:
CREATE TABLE `celery_taskmeta` (
    `task_id` varchar(255) NOT NULL UNIQUE,
    `status` varchar(50) NOT NULL,
    `result` longtext,
    `date_done` datetime NOT NULL,
    `traceback` longtext
CREATE TABLE `celery_tasksetmeta` (
    `taskset_id` varchar(255) NOT NULL UNIQUE,
    `result` longtext NOT NULL,
    `date_done` datetime NOT NULL

That took 0.29 seconds

Running migration 2:
ALTER TABLE `website_issues_sitesummary`
    ADD COLUMN `os` varchar(30) DEFAULT NULL;
CREATE INDEX `sitesummary_eab31616`
    ON `website_issues_sitesummary` (`url`,`os`,`version`);
Error: Had trouble running this: BEGIN;
ALTER TABLE `website_issues_sitesummary`
    ADD COLUMN `os` varchar(30) DEFAULT NULL;
CREATE INDEX `sitesummary_eab31616`
    ON `website_issues_sitesummary` (`url`,`os`,`version`);
UPDATE schema_version SET version = 2;
stderr: ERROR 1290 (HY000) at line 2: The MySQL server is running with the --read-only option so it cannot execute this statement

returncode: 1
Comment 5 Michael Kurze [:michaelk] 2010-09-16 09:02:36 PDT
About the problem with the second migration: I can only guess that schematic tries to talk to a slave, because my script was able to push data to the master (no read-only there). Maybe the settings need to be adjusted temporarily not to use replication, but that’s only a guess. 
If I can do anything here, I’ll have time to look at this again later today.
Comment 6 Shyam Mani [:fox2mike] 2010-09-16 09:22:50 PDT
Reverted this push.
Comment 7 Aakash Desai [:aakashd] 2010-09-16 15:52:35 PDT
We're pushing tonight. Best time would be after 11pm.
Comment 8 matthew zeier [:mrz] 2010-09-16 16:33:14 PDT
This needs to hold until Tuesday - you missed the cutoff for the announcement.
Comment 9 Michael Kurze [:michaelk] 2010-09-16 16:34:35 PDT
aakash: I’ll make sure someone from metrics knows what to do.
Comment 10 Stephen Donner [:stephend] 2010-09-16 16:41:42 PDT
Can we please update both this bug and to reflect the date change?
Comment 11 Aakash Desai [:aakashd] 2010-09-16 16:44:55 PDT
Matt, what announcement?
Comment 12 matthew zeier [:mrz] 2010-09-16 17:00:31 PDT
I send out maintenance notes a day before the Tuesday/Thursday windows.  I sent notification out lastnight for Thursday and this wasn't included in that.

We're also in the midst of a Fx release and I generally don't like pushing changes during release times.
Comment 13 Aakash Desai [:aakashd] 2010-09-16 17:42:15 PDT
Matt, we've had this bug filed and planned for 9/2. Shyam knows we're going to be releasing tonight and has done this push in the past many times. We also have Fred, Michael and myself available tonight in order to do this push. I don't see where there's an issue. If there is no support, that'd be one thing, but that's not the case here.

As for being in the midst of a Fx release, that's not for Fx 4.0 Beta which is what Input is a dependent on, so that doesn't apply here.
Comment 14 matthew zeier [:mrz] 2010-09-16 18:56:10 PDT
> As for being in the midst of a Fx release, that's not for Fx 4.0 Beta which is
> what Input is a dependent on, so that doesn't apply here.

We do a change-control freeze during Firefox releases.  Your schedule got pushed because we had a chemspill release.
Comment 15 Aakash Desai [:aakashd] 2010-09-16 19:03:59 PDT
Ah, understood. Thanks for the info, Matt. What can we do to this bug to make sure we get pushed out on Tuesday?
Comment 16 matthew zeier [:mrz] 2010-09-16 19:26:25 PDT
Nothing, it's marked.  There is a webdev calendar that this should go on too.
Comment 17 Aakash Desai [:aakashd] 2010-09-20 13:56:53 PDT
What is the time period tomorrow for this release?
Comment 18 Aakash Desai [:aakashd] 2010-09-20 14:21:04 PDT
*** Bug 597001 has been marked as a duplicate of this bug. ***
Comment 19 Fred Wenzel [:wenzel] 2010-09-20 15:10:46 PDT
Shyam: I'll ping you first thing in my morning. Shouldn't take long.
Comment 20 Fred Wenzel [:wenzel] 2010-09-21 06:50:44 PDT
Shyam is busy today. I am up for rescheduling if there's someone else available. Let me know.
Comment 21 Aakash Desai [:aakashd] 2010-09-21 07:50:48 PDT
Upping the priority as we need this by today.
Comment 22 Dave Miller [:justdave] ( 2010-09-21 07:52:51 PDT
downing the priority as it's already on the downtime schedule and there's no need to page me at 7:50 in the morning for something that's not happening until 7pm.
Comment 23 Aakash Desai [:aakashd] 2010-09-21 08:13:50 PDT
I never got an answer to comment #17. We need the time to make sure we have a dev available for the push.
Comment 24 Dave Miller [:justdave] ( 2010-09-21 08:24:22 PDT
Morgamic's calendar says it's at 8pm.
Comment 25 Aakash Desai [:aakashd] 2010-09-21 10:17:15 PDT
So, I just talked to mrz and it should be ok to move this push to 7pm. Here's the plan:

Primary: Push at 7pm PST with the dave's (dash and miller).

Back-up: Push at 11pm PST with wenzel and someone from IT (requesting oremj).
Comment 26 Aakash Desai [:aakashd] 2010-09-21 10:42:02 PDT
Per Jeremy's approval, added time on whiteboard and assignment.
Comment 27 Aakash Desai [:aakashd] 2010-09-21 19:03:45 PDT
We're not safe to push to production today, so we'll need to delay this until QA has had enough time to test and get out for a quality release. 

I'll update this bug with a suggested date after some due diligence. Sorry for the extra leg work.
Comment 28 matthew zeier [:mrz] 2010-09-22 14:26:36 PDT
Re-open when we have a push date scheduled.
Comment 29 Aakash Desai [:aakashd] 2010-09-22 17:04:02 PDT
Tuesday, 9/28, will work. I'll suggest a time once we have a dev committed to being available for a certain time period on that day.
Comment 30 Aakash Desai [:aakashd] 2010-09-28 19:02:19 PDT
Who is picking this up?
Comment 31 Shyam Mani [:fox2mike] 2010-09-28 19:28:57 PDT
I am!

And it's done.
Comment 32 Stephen Donner [:stephend] 2010-09-28 20:09:50 PDT
We pushed this, and it stuck; aren't yet sure about bug 600448, since staging is currently down: bug 600444.  I might wait till that's figured out before verifying the push.

Note You need to log in before you can comment on or make changes to this bug.