703596 - QA and deploy BrowserID train-2011.11.17 to production

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Labs, task)

Description

[Lloyd Hilaiel [:lloyd]]

ChangeLog including issues resolved: 
https://github.com/mozilla/browserid/blob/77c743/ChangeLog#L1-16

[QA] Suggested areas of focus for QA:
  * android mobile 2.2 and 2.3 (should work)
  * spawning dialog twice from an RP page without reloading the page (should work)
  * test new "required email" feature, visible through http://beta.myfavoriteshow.org
  * test new "keep me signed in" feature, visible through http://beta.myfavoritebooze.org
  * inspect logs and general functioning of dbwriter process.
  * compare current beta with new staging environment
  * normal regression of closed issues

[RELEASE ENGINEERING]:
  * no new steps, but dbwriter process is not yet in prod as train-2011.11.10 won't roll - follow deployment steps in bug # 701934

Comment 1

[James Bonacci [:jbonacci]]

QA accepts this train for testing in Beta.

Here is some new information for the current Beta environment:
Client: http://beta.myfavoritebeer.org/
New client: http://beta.myfavoritebooze.org/ (for keep me signed in)
New client: http://beta.myfavoriteshow.org/ (for required email)
Server: https://diresworb.org/

And here is the client and server for the new Stage environment:
Main server: https://stage-browserid.services.mozilla.com
Stage RP: http://carrera.databits.net:9999
(must be reached through Mozilla-MPT)

Comment 2

[James Bonacci [:jbonacci]]

Stage is offline at the moment, so only showing information for Beta:
Code and Deployment Verification: BETA:
/usr/local/bin/node /home/browserid/code/bin/verifier
/usr/local/bin/node /home/browserid/code/bin/keysigner
/usr/local/bin/node /home/browserid/code/bin/dbwriter
/usr/local/bin/node /home/browserid/code/bin/browserid

Beta:
https://diresworb.org/ver.txt
77c7435 revert accidental change introduced in commit d0af61dd - don't generate ephemeral DSA1024 domain keys - all ephemeral domain keys are RSA1024 till we decide to change this (based on thought, data, and stuff)

Old Dev:
https://dev.diresworb.org/ver.txt
35eae1d add 'domain_key_creation_date' to session_context wsapi - issue #599

Old Prod:
https://browserid.org/ver.txt
67098c0 merge verifier hotfix into production

Server heartbeats:
Beta: https://diresworb.org/__heartbeat__
ok

A quick check of the log files on Beta shows some evidence of the following errors (new to me anyway):
browserid.log
Saw one instance of this error (today): ECONNREFUSED
Fri, 18 Nov 2011 16:25:03 GMT
And one instance of this error (11/17): ETIMEDOUT

dbwriter.log
Saw one instance of this error (today): ECONNRESET

All of these were very near server restarts...

Comment 3

[James Bonacci [:jbonacci]]

Stage stats,processes, and logs look pretty close to Beta.
Noting extra errors in dbwriter.log due to backend issues yesterday/earlier today.

Also noting that we have no log rotation/archiving on Stage (or Dev for that matter).
Pete will be working on this soon...
REF:
Bug 695963 - browserid: coordinate log collection with metrics
Bug 695967 - browserid: setup monitoring

Comment 4

[James Bonacci [:jbonacci]]

All Closed/Fixed issues for this train were either Verified or reopened.
Testing now will focus on beta.myfavoritebooze.org and beta.myfavoriteshow.org.
After that, we move on to a bit more sanity testing of the new Dev environment.
When Stage is back up and running smoothly (minus all the dbwriter nastiness), deep dive testing will happen there...

Comment 5

[James Bonacci [:jbonacci]]

beta.myfavoriteshow.org has been thoroughly tested.
There was some initial confusion that prevented quicker testing and less issues.

This needs a lot of UI work.

REF:
626: Relay frame error on beta.myfavoriteshow.org on browser restart
631: Second attempt to log in to beta.myfavoriteshow.org causes apparent hang or no action on diresworb.org
632: beta.myfavoriteshow.org email field allows all matter of silliness...
633: No apparent way to log out of beta.myfavoriteshow.org
636: BrowserID stops working on beta.myfavoriteshow after cancelling a sign in
637: Inconsistent behavior when logging into beta.myfavoriteshow.org multiple times

Comment 6

[James Bonacci [:jbonacci]]

beta.myfavoritebooze.org
The "keep me signed in" feature seems to work pretty well.

Win7: IE9, FF, Chrome all work
IE9 fails on user logout/login
Did not try an OS restart for Win7

Linux (Ubuntu 11): FF, Opera, and Chrome all work
Did not try an OS restart

Mac: FF, Opera, Chrome, and Safari
All browsers work as expected except for Opera (not sure why)
Did not try an OS restart

General:
You must remember to not clear browser settings (history, data, cache, etc).
You do not need to remember or save the open tabs/windows.
You must stay logged in before closing the browser and/or restarting the OS (or logging out/in).
Interesting "side effect": each login is unique per beta.myfavoritebooze.org and diresweb.org.
I was logged in differently on IE9, FF, and Chrome, and the corresponding diresworb.org view on each was also unique.

w/o this setting, logging into beer or booze or diresworb.org shows the same account information across all sites on all browsers.

Tested new accounts and current accounts.
Tested accounts with single email and with multiple emails.

REF:
639: IE9: Beer and Booze names are not being stored/used.
640: IE9: beta.myfavoritebooze.org gets error after user logs out/in.
641: Mac Opera is not holding beta.myfavoritebooze.org information

Comment 7

[James Bonacci [:jbonacci]]

Completed:
Sanity testing of Firefox 8.0.1
New sites (booze, show) with iPhone, iPad, and tablet
Sanity testing on Fedora 16
beta.myfavoritebooze.org with shared accounts/emails
More sanity testing of Dev
Retesting of some issues

Still to do for this train:
Full sanity/stability testing of Stage
Load Test monitoring

Comment 8

[James Bonacci [:jbonacci]]

QA signs off on this (final) Beta train.
Many issues were closed.
Many issues were opened as we moved closer to full OPs environments.

Comment 9

[Lloyd Hilaiel [:lloyd]]

as agreed in triage, blocking IE8 for this release.

Comment 10

[Lloyd Hilaiel [:lloyd]]

deployment starts now...

Comment 11

[Lloyd Hilaiel [:lloyd]]

deployment complete.  smoke testing looks good.  QA, please confirm at your leisure.

Comment 12

[James Bonacci [:jbonacci]]

Prod looks good.
Verified browserid.org
myfavoritebeer.org
myfavoritebooze.org
myfavoriteshow.org