Push Firefox Input to production on 9/28

VERIFIED FIXED

Status

Infrastructure & Operations
WebOps: Other
--
blocker
VERIFIED FIXED
7 years ago
4 years ago

People

(Reporter: wenzel, Assigned: fox2mike)

Tracking

other
Bug Flags:
needs-downtime +

Details

(Whiteboard: 09/28/2010 @ 7pm)

(Reporter)

Description

7 years ago
We need a push of input.stage.mozilla.com to production on Tuesday, 9/8.

The only step to perform now (thanks to the amazing shyam) is to run:

/root/bin/input_update.sh 


We're flexible on time, but I am traveling and will be at a conference in the Pacific time zone that day, so we'll need to find a time that plays well with that. Alternatively, Dave could take care of the push like he did the last time?
(Assignee)

Comment 1

7 years ago
Tuesday is the 7th.
Assignee: server-ops → shyam
Summary: Push Firefox Input to production on 9/8 → Push Firefox Input to production on 9/7
Whiteboard: 09/07/2010
Ok, we're going to push this back to Thursday, 9/16, per the usual time. I'm leaving the whiteboard entry as is until you're ok with it, Shyam.
Summary: Push Firefox Input to production on 9/7 → Push Firefox Input to production on 9/16
(Assignee)

Comment 3

7 years ago
If you're okay with it, so am I :)
Whiteboard: 09/07/2010 → 09/16/2010
(Assignee)

Comment 4

7 years ago
Okay, so there were too many changes to paste here, looking at this :

Updating reporter...
HEAD is now at 7f57260... Needed to disable Frisian as Babel does not know fy_NL.
Running migration 5:
BEGIN;
CREATE TABLE `celery_taskmeta` (
    `id` integer AUTO_INCREMENT NOT NULL PRIMARY KEY,
    `task_id` varchar(255) NOT NULL UNIQUE,
    `status` varchar(50) NOT NULL,
    `result` longtext,
    `date_done` datetime NOT NULL,
    `traceback` longtext
)
;
CREATE TABLE `celery_tasksetmeta` (
    `id` integer AUTO_INCREMENT NOT NULL PRIMARY KEY,
    `taskset_id` varchar(255) NOT NULL UNIQUE,
    `result` longtext NOT NULL,
    `date_done` datetime NOT NULL
)
;
COMMIT;

That took 0.29 seconds
################################################## 

Running migration 2:
ALTER TABLE `website_issues_sitesummary`
    ADD COLUMN `os` varchar(30) DEFAULT NULL;
CREATE INDEX `sitesummary_eab31616`
    ON `website_issues_sitesummary` (`url`,`os`,`version`);
Error: Had trouble running this: BEGIN;
ALTER TABLE `website_issues_sitesummary`
    ADD COLUMN `os` varchar(30) DEFAULT NULL;
CREATE INDEX `sitesummary_eab31616`
    ON `website_issues_sitesummary` (`url`,`os`,`version`);
UPDATE schema_version SET version = 2;
COMMIT;
stdout: 
stderr: ERROR 1290 (HY000) at line 2: The MySQL server is running with the --read-only option so it cannot execute this statement

returncode: 1
About the problem with the second migration: I can only guess that schematic tries to talk to a slave, because my script was able to push data to the master (no read-only there). Maybe the settings need to be adjusted temporarily not to use replication, but that’s only a guess. 
If I can do anything here, I’ll have time to look at this again later today.
(Assignee)

Comment 6

7 years ago
Reverted this push.
We're pushing tonight. Best time would be after 11pm.

Comment 8

7 years ago
This needs to hold until Tuesday - you missed the cutoff for the announcement.
aakash: I’ll make sure someone from metrics knows what to do.
Can we please update both this bug and https://mail.mozilla.com/home/morgamic@mozilla.com/Webdev%20Releases.html to reflect the date change?
Matt, what announcement?
I send out maintenance notes a day before the Tuesday/Thursday windows.  I sent notification out lastnight for Thursday and this wasn't included in that.

We're also in the midst of a Fx release and I generally don't like pushing changes during release times.
Flags: needs-downtime+
Matt, we've had this bug filed and planned for 9/2. Shyam knows we're going to be releasing tonight and has done this push in the past many times. We also have Fred, Michael and myself available tonight in order to do this push. I don't see where there's an issue. If there is no support, that'd be one thing, but that's not the case here.

As for being in the midst of a Fx release, that's not for Fx 4.0 Beta which is what Input is a dependent on, so that doesn't apply here.
> As for being in the midst of a Fx release, that's not for Fx 4.0 Beta which is
> what Input is a dependent on, so that doesn't apply here.

We do a change-control freeze during Firefox releases.  Your schedule got pushed because we had a chemspill release.
Ah, understood. Thanks for the info, Matt. What can we do to this bug to make sure we get pushed out on Tuesday?
Nothing, it's marked.  There is a webdev calendar that this should go on too.
Whiteboard: 09/16/2010 → 09/21/2010
Summary: Push Firefox Input to production on 9/16 → Push Firefox Input to production on 9/21
What is the time period tomorrow for this release?
Duplicate of this bug: 597001
(Reporter)

Comment 19

7 years ago
Shyam: I'll ping you first thing in my morning. Shouldn't take long.
(Reporter)

Comment 20

7 years ago
Shyam is busy today. I am up for rescheduling if there's someone else available. Let me know.
Assignee: shyam → server-ops
Upping the priority as we need this by today.
Severity: normal → blocker
downing the priority as it's already on the downtime schedule and there's no need to page me at 7:50 in the morning for something that's not happening until 7pm.
Severity: blocker → normal
I never got an answer to comment #17. We need the time to make sure we have a dev available for the push.
Morgamic's calendar says it's at 8pm.
So, I just talked to mrz and it should be ok to move this push to 7pm. Here's the plan:

Primary: Push at 7pm PST with the dave's (dash and miller).

Back-up: Push at 11pm PST with wenzel and someone from IT (requesting oremj).
Per Jeremy's approval, added time on whiteboard and assignment.
Assignee: server-ops → jeremy.orem+bugs
Whiteboard: 09/21/2010 → 09/21/2010 7:00 PM PST
We're not safe to push to production today, so we'll need to delay this until QA has had enough time to test and get out for a quality release. 

I'll update this bug with a suggested date after some due diligence. Sorry for the extra leg work.

Updated

7 years ago
Summary: Push Firefox Input to production on 9/21 → Push Firefox Input to production
Whiteboard: 09/21/2010 7:00 PM PST
Re-open when we have a push date scheduled.
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → INCOMPLETE

Updated

7 years ago
Assignee: jeremy.orem+bugs → server-ops
Tuesday, 9/28, will work. I'll suggest a time once we have a dev committed to being available for a certain time period on that day.
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Summary: Push Firefox Input to production → Push Firefox Input to production on 9/28

Updated

7 years ago
Whiteboard: 09/28/2010 @ 7pm
Depends on: 600070
Who is picking this up?
Severity: normal → blocker
(Assignee)

Updated

7 years ago
Assignee: server-ops → shyam
(Assignee)

Comment 31

7 years ago
I am!

And it's done.
Status: REOPENED → RESOLVED
Last Resolved: 7 years ago7 years ago
Resolution: --- → FIXED
We pushed this, and it stuck; aren't yet sure about bug 600448, since staging is currently down: bug 600444.  I might wait till that's figured out before verifying the push.
Status: RESOLVED → VERIFIED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.