948693 - Downtime required to deploy new EditComments extension to production BMO servers

Status: RESOLVED FIXED

(Infrastructure & Operations :: Change Requests, task)

Description

[David Lawrence [:dkl]]

We'll need to schedule some downtime to deploy the code changes that will add the ability to privileged users to edit comments in BMO.

It will require taking the site down for 30 mins or less to perform the necessary schema changes and code push. We have discussed this with the sheriffs and they confirmed that a tree closure was not strictly necessary if it could be done in a small amount of time. As no time is really a great time as far as BMO usage, we propose to perform the update slightly after hours PST on any given day which works best for IT.

A cab review bug should be filed as well by the IT person who will be performing the migration.

Thanks
dkl

Comment 1

[David Lawrence [:dkl]]

- date, time, duration of maintenance

TBD due to need to know who will be doing the maintenance and when they are available. It has already been cleared with the tree sheriffs in that a 30 min or so downtime will not be a major impact on their work,

- system(s) affected

bugzilla.mozilla.org and third-party systems that access it such as BzAPI, TBPL, etc.

- end-user impact

Consumers of bugzilla.mozilla.org will need to wait til maintenance has completed. No bug access.

- maintenance plan and timeline (link to a wiki or etherpad is fine)

Basically what is in comment 0. I can create an etherpad to outline in more detail what the steps will be if necessary. It will basically resemble other similar weekly production code pushes except we can do this one in a single push since the site will be down temporarily. Normally we have to do two back to back pushes, one to do the schema update only, and the second for the actual code change. 

- rollback plan / rollback point (at which point will you determine to roll back)

For big migrations, normal practice is to to remove a slave from the replication pool, perform the update, and then if something goes wrong the slave is promoted to master to roll back to where we were. We will be able to do basic testing right after the update to make sure things are working properly.

- notification mechanisms

tools.bmo, dev.planning, announce header on BMO itself if needed.

- who will be point, who else will be involved 

WebOps TBD, BMO devel will be dkl

Comment 2

[David Lawrence [:dkl]]

We decided not a good idea to perform this update before the holiday break so we will push this off til after. Maybe we could do it during the Jan 11th, 2014 tree closure?

dkl

Comment 3

[Sheeri Cabral [:sheeri]]

:cyborgshadow will be the DBA oncall then.

Comment 4

[Sheeri Cabral [:sheeri]]

putting in a needinfo from Kendall - there should be a CAB bug made....

Comment 5

[Kendall Libby [:fubar] (he/him)]

This is the CAB bug

Comment 6

[Sheeri Cabral [:sheeri]]

Hrm, it doesn't seem it was reviewed in the CAB meeting this week :( Do we know why not?

Comment 7

[Kendall Libby [:fubar] (he/him)]

It was discussed; initially it was going to be done before change freeze, but all involved decided that wasn't the best idea. Afterwards, it was suggested to be added to the window on the 11th. 

It needn't be, since the sheriffs have already given their ok for a short BMO downtime outside of a tree closure window, but BMO already has to have a short outage because of the NFS work on it.

Comment 8

[Gervase Markham [:gerv]]

So are we doing this on Jan 11th, or was it decided not to?

Gerv

Comment 9

[Kendall Libby [:fubar] (he/him)]

:fox2mike hasn't updated the bug, but yes it's going this weekend.

Comment 10

[Shyam Mani [:fox2mike]]

(In reply to Kendall Libby [:fubar] from comment #9)
> :fox2mike hasn't updated the bug, but yes it's going this weekend.

Yes.

Approved by the CAB on 8 Jan to be carried out during the tree closing window of 11 Jan.

Comment 11

[Kendall Libby [:fubar] (he/him)]

Push completed.