842269 - Move bugzilla.mozilla.org to SCL3 and upgrade it to 4.2

Status: RESOLVED FIXED

(Infrastructure & Operations :: Change Requests, task)

Description

[Shyam Mani [:fox2mike]]

Should be discussed in the CAB meeting of 19 Feb 2013.

*date, time, duration of maintenance
 Friday, 8 March 2013, 17:30:00 to 23:30:00 PST

 Begin times in other Mozilla cities :

 San Francisco - Friday, 8 March 2013, 17:30:00 PST (UTC -8)
 Vancouver - Friday, 8 March 2013, 17:30:00 PST (UTC -8)
 Toronto - Friday, 8 March 2013, 20:30:00 EST (UTC -5)
 London - Saturday, 9 March 2013, 01:30:00 GMT UTC
 Beijing - Saturday, 9 March 2013, 09:30:00 CST (UTC +8)
 Tokyo - Saturday, 9 March 2013, 10:30:00 JST (UTC +9)
 Auckland - Saturday, 9 March 2013, 14:30:00 NZDT (UTC +13)

*system(s) affected 
 All of Bugzilla in phx1 and scl3, i.e.;
 PHX1
 ----
 pp-app-bugs[01-03]
 tp-bugs01-master01/slave01/slave02
 Zeus Load balancers in phx1 (front and back end, config changes only) 

 SCL3
 ----    
 web[1-5].bugs.scl3
 bugzilla[1-4].db.scl3
 bugzillaadm.private.scl3
 bugzillapush1.bugs.scl3
 jobqueue1.bugs.scl3
 Zeus Load balancers in scl3 (front and back end, config changes only) 

*end-user impact 
The site (bugzilla.mozilla.org) will be completely offline for the duration of the maintenance window. Users will NOT be able to view, edit or file new bugs.Any systems/dashboards which interface with BMO will also be impacted, including but not limited to BzAPI consumers.

*maintenance plan and timeline (link to a wiki or etherpad is fine)
 Please see https://etherpad.mozilla.org/bugzillamove2scl3

*rollback plan / rollback point (at which point will you determine to roll back) 

Rollback point to be checked @ 2100 (prior to making scl3 instance live)
 Flip DNS back to phx1
 Remove Zeus hardhat in phx1

"Rollback" 2 (absolute worst case scenario)
 Switch back to phx, lose N days of data
 Upgrading from Bugzilla 4.0 to 4.2 alters the schema and the data.
 Downgrading is not supported.

*notification mechanisms - Posts will be drafted and sent to : 
 Blog post on blog.mozilla.org/it
 Yammer referencing the blog post
 E-mail [e-mail]
 E-mail [e-mail]
 E-mail [e-mail]
 E-mail [e-mail]
 E-mail [e-mail]
 Short one-liner will be posted on bugzilla.mozilla.org via announcehtml

Comment 1

[u429623]

Sheriffs are requesting tree closure, as they can't update the bugs that caused a failure otherwise.

Comment 2

[Ed Morley [:emorley]]

(In reply to Hal Wine [:hwine] from comment #1)
> Sheriffs are requesting tree closure, as they can't update the bugs that
> caused a failure otherwise.

(This has been since discussed via IRC/emails, but adding here for posterity, for people CCing on this bug)

The primary reason for requiring closure of the main trees is due to the TBPL bug suggestions, without which, we'll be left in the dark as to whether a failed job is a known intermittent failure, or a new failure. With our current all-time-high intermittent-failure rate (aka "OrangeFactor") of 7 per push, this becomes even more problematic:
http://brasstacks.mozilla.com/orangefactor/

Comment 3

[Shyam Mani [:fox2mike]]

All revised notifications were sent out 27/28th of Feb, and this is now supposed to proceed on the 5th March 2013 b/w 1730 and 2330 PST.

Comment 4

[Shyam Mani [:fox2mike]]

This was completed. Plan etherpad is up to date, bugzilla.mozilla.org was down for 3 hours and 26 mins despite a network issue in SCL3 interrupting the process.

Comment 5

[Shyam Mani [:fox2mike]]

Any issues arising from this move should be filed as new bugs and block bug 848218