732142 - QA and deploy BrowserID train-2012.03.01 to production

Status: RESOLVED FIXED

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Description

[Lloyd Hilaiel [:lloyd]]

Version: 451e675 (0.2012.03.01.1)

Tests pass: http://travis-ci.org/#!/mozilla/browserid/builds/775301

ChangeLog including issues resolved: 
https://github.com/mozilla/browserid/blob/train-2012.03.01/ChangeLog#L1-24

[QA] Suggested areas of focus for QA:
  * normal stuff 
  * disable cookies, ensure good error messaging - all devices/browsers
  * abuse mysql (SIGSTOP, restart, shutdown, route TCP packets to the bitbucket) - ensure you see reasonable error messages but that the service stays up
  * mobile firefox nightlies
  * stop the keysigner, ensure that the service stays up and you see reasonable error messages (log out, try to log into a site in this state)

[ops] deployment issues:
  * compress.sh and compress-locales.sh now all implemented in compress - rpm updated - no action should be required.
  * see mysql changes mentioned above, you now have more options in how you manage database service/failover, but proper application failover to a new ip is still not possible without restart

Comment 1

[Lloyd Hilaiel [:lloyd]]

hotfix for rpm build added - 52a367c - 0.2012.03.01.2 - https://github.com/mozilla/browserid/issues/1235

Comment 2

[Pete Fritchman [:petef]]

deployed to stage.

Comment 3

[James Bonacci [:jbonacci]]

QA accepts this train for testing in Stage:
52a367c bump version number and document hotfix for issue #1235
locale svn r102416

Comment 4

[James Bonacci [:jbonacci]]

QA will work with petef next week on the following items:
* abuse mysql (SIGSTOP, restart, shutdown, route TCP packets to the bitbucket) - ensure you see reasonable error messages but that the service stays up

* stop the keysigner, ensure that the service stays up and you see reasonable error messages (log out, try to log into a site in this state)

Comment 5

[James Bonacci [:jbonacci]]

Processes, logs, and initial checks of Stage look good.
Proceeding with sanity tests...

Comment 6

[James Bonacci [:jbonacci]]

Sanity tests are good....moving on to bug regression.

Comment 7

[James Bonacci [:jbonacci]]

Hot Fix 2 (with rollback to 2012-02-16):
1258: method to probe dbwriter's database connection health
1259: Issue1258
version:
7558d69 document db conn monitoring hotfix and bump version
locale svn r102380

Tested and passed.

Comment 8

[James Bonacci [:jbonacci]]

Post-HotFix roll to 2012-03-01 release:
e327804 document db conn monitoring hotfix and bump version
locale svn r102598

Sanity testing is in progress...

Comment 9

[James Bonacci [:jbonacci]]

Sanity testing was completed yesterday. Bug verification continues...

Comment 10

[James Bonacci [:jbonacci]]

Bug verification and trolling for older closed bugs has completed.

Comment 11

[James Bonacci [:jbonacci]]

Covered the following in today's load testing + Stage breakage:
* abuse mysql (SIGSTOP, restart, shutdown, route TCP packets to the bitbucket) - ensure you see reasonable error messages but that the service stays up
* stop the keysigner, ensure that the service stays up and you see reasonable error messages (log out, try to log into a site in this state)
(from below)
* database (mysql) improvements, handle temporary outages without taking down the server
990: handle stalling in mysql driver
1011: bin/browserid on stage webhead dies on network issue
1211: update database code and mysql driver to handle stalled connections

Comment 12

[James Bonacci [:jbonacci]]

Testing is mostly finished. QA is blocked on the following:
1133: iOS - must click on radio button to select email address.
1286: Unable to select other accounts on Android 

Final bits of locale testing should happen today:
New locales:
lt: Lithuanian (100%)
pt_BR: Brazilian (Portuguese) (100%)
rm: Romansh (100%)
uk: Ukrainian (100%)

Locales removed:
fy: Frisian
af: Afrikaans

Comment 13

[James Bonacci [:jbonacci]]

QA signs off on the test of this train in Stage, noting the following email thread that details the derailing of this release: "BrowserID dev: derailing train-2012.03.01"