593280 - Push Firefox Input to production on 9/28

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Fred Wenzel [:wenzel]]

We need a push of input.stage.mozilla.com to production on Tuesday, 9/8.

The only step to perform now (thanks to the amazing shyam) is to run:

/root/bin/input_update.sh 


We're flexible on time, but I am traveling and will be at a conference in the Pacific time zone that day, so we'll need to find a time that plays well with that. Alternatively, Dave could take care of the push like he did the last time?

Comment 1

[Shyam Mani [:fox2mike]]

Tuesday is the 7th.

Comment 2

[Aakash Desai [:aakashd]]

Ok, we're going to push this back to Thursday, 9/16, per the usual time. I'm leaving the whiteboard entry as is until you're ok with it, Shyam.

Comment 3

[Shyam Mani [:fox2mike]]

If you're okay with it, so am I :)

Comment 4

[Shyam Mani [:fox2mike]]

Okay, so there were too many changes to paste here, looking at this :

Updating reporter...
HEAD is now at 7f57260... Needed to disable Frisian as Babel does not know fy_NL.
Running migration 5:
BEGIN;
CREATE TABLE `celery_taskmeta` (
    `id` integer AUTO_INCREMENT NOT NULL PRIMARY KEY,
    `task_id` varchar(255) NOT NULL UNIQUE,
    `status` varchar(50) NOT NULL,
    `result` longtext,
    `date_done` datetime NOT NULL,
    `traceback` longtext
)
;
CREATE TABLE `celery_tasksetmeta` (
    `id` integer AUTO_INCREMENT NOT NULL PRIMARY KEY,
    `taskset_id` varchar(255) NOT NULL UNIQUE,
    `result` longtext NOT NULL,
    `date_done` datetime NOT NULL
)
;
COMMIT;

That took 0.29 seconds
################################################## 

Running migration 2:
ALTER TABLE `website_issues_sitesummary`
    ADD COLUMN `os` varchar(30) DEFAULT NULL;
CREATE INDEX `sitesummary_eab31616`
    ON `website_issues_sitesummary` (`url`,`os`,`version`);
Error: Had trouble running this: BEGIN;
ALTER TABLE `website_issues_sitesummary`
    ADD COLUMN `os` varchar(30) DEFAULT NULL;
CREATE INDEX `sitesummary_eab31616`
    ON `website_issues_sitesummary` (`url`,`os`,`version`);
UPDATE schema_version SET version = 2;
COMMIT;
stdout: 
stderr: ERROR 1290 (HY000) at line 2: The MySQL server is running with the --read-only option so it cannot execute this statement

returncode: 1

Comment 5

[Michael Kurze [:michaelk]]

About the problem with the second migration: I can only guess that schematic tries to talk to a slave, because my script was able to push data to the master (no read-only there). Maybe the settings need to be adjusted temporarily not to use replication, but that’s only a guess. 
If I can do anything here, I’ll have time to look at this again later today.

Comment 6

[Shyam Mani [:fox2mike]]

Reverted this push.

Comment 7

[Aakash Desai [:aakashd]]

We're pushing tonight. Best time would be after 11pm.

Comment 8

[matthew zeier [:mrz]]

This needs to hold until Tuesday - you missed the cutoff for the announcement.

Comment 9

[Michael Kurze [:michaelk]]

aakash: I’ll make sure someone from metrics knows what to do.

Comment 10

[Stephen Donner [:stephend] Not actively reading bugmail]

Can we please update both this bug and https://mail.mozilla.com/home/[e-mail]/Webdev%20Releases.html to reflect the date change?

Comment 11

[Aakash Desai [:aakashd]]

Matt, what announcement?

Comment 12

[matthew zeier [:mrz]]

I send out maintenance notes a day before the Tuesday/Thursday windows.  I sent notification out lastnight for Thursday and this wasn't included in that.

We're also in the midst of a Fx release and I generally don't like pushing changes during release times.

Comment 13

[Aakash Desai [:aakashd]]

Matt, we've had this bug filed and planned for 9/2. Shyam knows we're going to be releasing tonight and has done this push in the past many times. We also have Fred, Michael and myself available tonight in order to do this push. I don't see where there's an issue. If there is no support, that'd be one thing, but that's not the case here.

As for being in the midst of a Fx release, that's not for Fx 4.0 Beta which is what Input is a dependent on, so that doesn't apply here.

Comment 14

[matthew zeier [:mrz]]

> As for being in the midst of a Fx release, that's not for Fx 4.0 Beta which is
> what Input is a dependent on, so that doesn't apply here.

We do a change-control freeze during Firefox releases.  Your schedule got pushed because we had a chemspill release.

Comment 15

[Aakash Desai [:aakashd]]

Ah, understood. Thanks for the info, Matt. What can we do to this bug to make sure we get pushed out on Tuesday?

Comment 16

[matthew zeier [:mrz]]

Nothing, it's marked.  There is a webdev calendar that this should go on too.

Comment 17

[Aakash Desai [:aakashd]]

What is the time period tomorrow for this release?

Comment 19

[Fred Wenzel [:wenzel]]

Shyam: I'll ping you first thing in my morning. Shouldn't take long.

Comment 20

[Fred Wenzel [:wenzel]]

Shyam is busy today. I am up for rescheduling if there's someone else available. Let me know.

Comment 21

[Aakash Desai [:aakashd]]

Upping the priority as we need this by today.

Comment 22

[Dave Miller [:justdave]]

downing the priority as it's already on the downtime schedule and there's no need to page me at 7:50 in the morning for something that's not happening until 7pm.

Comment 23

[Aakash Desai [:aakashd]]

I never got an answer to comment #17. We need the time to make sure we have a dev available for the push.

Comment 24

[Dave Miller [:justdave]]

Morgamic's calendar says it's at 8pm.

Comment 25

[Aakash Desai [:aakashd]]

So, I just talked to mrz and it should be ok to move this push to 7pm. Here's the plan:

Primary: Push at 7pm PST with the dave's (dash and miller).

Back-up: Push at 11pm PST with wenzel and someone from IT (requesting oremj).

Comment 26

[Aakash Desai [:aakashd]]

Per Jeremy's approval, added time on whiteboard and assignment.

Comment 27

[Aakash Desai [:aakashd]]

We're not safe to push to production today, so we'll need to delay this until QA has had enough time to test and get out for a quality release. 

I'll update this bug with a suggested date after some due diligence. Sorry for the extra leg work.

Comment 28

[matthew zeier [:mrz]]

Re-open when we have a push date scheduled.

Comment 29

[Aakash Desai [:aakashd]]

Tuesday, 9/28, will work. I'll suggest a time once we have a dev committed to being available for a certain time period on that day.

Comment 30

[Aakash Desai [:aakashd]]

Who is picking this up?

Comment 31

[Shyam Mani [:fox2mike]]

I am!

And it's done.

Comment 32

[Stephen Donner [:stephend] Not actively reading bugmail]

We pushed this, and it stuck; aren't yet sure about bug 600448, since staging is currently down: bug 600444.  I might wait till that's figured out before verifying the push.