Closed Bug 719243 Opened 12 years ago Closed 12 years ago

QA and deploy BrowserID train-2012.01.18 to production

Categories

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Product:
Cloud Services
Component:
Operations: Deployment Requests - DEPRECATED
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: lloyd, Assigned: petef)

References

Details

(Whiteboard: [qa?][target 2/2]) 

ChangeLog including issues resolved: 
https://github.com/mozilla/browserid/blob/train-2012.01.18/ChangeLog#L1-12

[QA] Suggested areas of focus for QA:
  * Full and complete regression testing across all devices now that primary support is merged.
  * testing of required email features via myfavoriteshow.org
  * testing of persistent sign in features via myfavoritebooze.org
  * testing of primary support using an @eyedee.me address (to be completed and documented by EOD thurs).

[ops] Manual deployment steps
  * we need to be able to disable primary support rapidly if a critical issue is discovered - that means being able to rapidly deploy a change to .disable_primary_support: true
  * we need HTTP proxies in place that accept internal traffic over HTTP and will fetch https:// resources ( only https://<domain>/.well-known/browserid need be proxied ever )
  * must configure both dbwriter and webhead with http proxy servers: https://github.com/mozilla/browserid/blob/train-2012.01.18/lib/configuration.js#L85-90
Assignee: nobody → petef
No longer depends on: 713961
QA Contact: operations-deploy-requests → jbonacci
petef, we're ready for deploy on staging now, branch train-2012.01.18 - sha -  ab55a682 - version 0.2012.01.18-1
Changed title to match deployment.
Summary: QA and deploy BrowserID train-2012.01.05 to production → QA and deploy BrowserID train-2012.01.18 to production
We are currently held up in Stage by some issues with webheads, idle timeout, mysql bugs.
In addition, we are blocked by NetOPs in order to set up proxies.

QA will work with OPs today to get Stage back up and running on current deployment (primaries off) with load_gen changes so we can focus on those fixes.
deployed to stage with primary support off
QA excepts this train to Stage for testing.
Build: ab55a68 update changelog and browserid.spec for 2012.01.18

Primary support is still disabled, but other new fixes are in, based on ChangeLog#L1-12 (please add any further changes to this ticket).

First big test will be load to Stage to try to duplicate Issue #875.

Once the Dev branch has the latest load_gen fixes, we will pick that up and continue various load tests (QPS issues, max load/stress testing, etc).
Process and log checks are good on Stage.
Unit tests passed.
Sanity tests passed.
Local install, unit testing, and load testing passed.
Also tested the new example RP and Primary on local install.
Load Testing on Stage could again be considered a fail due to the following:
1. QPS again showed a consistent drop on webheads across 40+ hours (see #875)
2. load_gen exited on client4 with an apparent out of memory error

Next up:
1. Bug verification
2. Suggested areas of focus for QA

Also, we need some help checking the following issue: #927
Check the path where an account is created on one browser then used on another browser.
We need to look at this two ways:
a. All browsers are on the same platform
b. Browsers are on separate platforms
primary support enabled, proxies configured
Assignee: petef → jbonacci
Super quick verification of primary support and proxies by testing against eyedee.me and diresworb.org.
QA needs to dig deeper on this but the basic functionality seems to be there.
Status: NEW → ASSIGNED
petef, a hotfix is available.  0.2012.01.18.2 - https://github.com/mozilla/browserid/tree/1df6d4b877411077893b4ecdfb4901919f4318e7

please deploy when you get a chance.
(In reply to Lloyd Hilaiel from comment #9)
> petef, a hotfix is available.  0.2012.01.18.2 -
> https://github.com/mozilla/browserid/tree/
> 1df6d4b877411077893b4ecdfb4901919f4318e7
> 
> please deploy when you get a chance.

live on stage
petef thanks. QA will verify HotFixes shortly.
Issues list: 932, 933, 934, and 935

After that QA moves on to the following:
1. Sanity/Smoke Stage on desktop and mobile
2. Load Testing to check for upper limits
3. Initial verification of primary support using eyedee.me
4. All the other stage testing
petef, a hotfix is available.  0.2012.01.18.3 - https://github.com/mozilla/browserid/tree/ae5c7766eb12b39e2f9f869dd4d253de5ef56430

Full list of changes that have gone in in this batch:


  * (hotfix 2012.01.24) Fix "slow script" error on IE8 during keygen on behalf of primary: #956
  * (hotfix 2012.01.24) Publish javascript API to provide a native-support compatible for primaries' auth pages: #909
  * (hotfix 2012.01.24) Allow load testing hooks to be enabled with minified resources: #939
  * (hotfix 2012.01.24) IE8 fixes for primary flow: #962, #961, #958, #955
petef, note, when you deploy can you please update 'env' to 'production' so we have minified resources served?  notice the fix for 909 which will remove the mutual exclusivity of 'BROWSERID_FAKE_VERIFICATION' and production.

you may want to add a check that '/wsapi/fake_verification' always returns a 404 in production to ensure we never ship this feature enabled.
petef, a hotfix is available.  0.2012.01.18.4 - https://github.com/mozilla/browserid/tree/4acea5fe38a7dd0808601066b54792bcb5a2090a

Adds: 
  * (hotfix 2012.01.24) print correct url for where the user will be directed: #964
(In reply to Lloyd Hilaiel from comment #14)
> petef, a hotfix is available.  0.2012.01.18.4 -
> https://github.com/mozilla/browserid/tree/
> 4acea5fe38a7dd0808601066b54792bcb5a2090a
> 
> Adds: 
>   * (hotfix 2012.01.24) print correct url for where the user will be
> directed: #964

pushed, and NODE_ENV=production now on stage webheads (yay!).
QA picks up this deployment including HotFix Set 2 for verification.
QA has completed verification of issues from HotFix Set 1:
Issues list: 929, 932, 933, 934, and 935

QA has completed work on issues from HotFix Set 2
Details

Verified:
* Fix "slow script" error on IE8 during keygen on behalf of primary: #956
* Publish javascript API to provide a native-support compatible for primaries' auth pages: #909
* Allow load testing hooks to be enabled with minified resources: #939
* print correct url for where the user will be directed: #964

Outstanding items:
#955 needs feedback from stomlinson.
#961 (blocked by #955)
#958 needs feedback from stomlinson.
#962 (blocked by #958)
Need further feedback for
#955 needs feedback from stomlinson.
#961 (blocked by #955)

#958 and #962 have been verified.

Bug verification continues, along with sanity/smoke testing.
Tomorrow (Thu) we will run the big load test.
QA signs off on this train.
It's been a very full and focused 11 days of testing.
Assignee: jbonacci → petef
Whiteboard: [qa+]
Whiteboard: [qa+] → [qa?]
HotFix #1:
972: [regression] 'Always sign in with this email' doesn't persist login across browser restarts

Pushed to Stage, Verified.
HotFix #2 pushed to Stage for testing.

Details:
848: A dialog with unresponsive buttons is opened when confirming an email from a secondary browser
973: Mobile: Trouble verifying email using remote browser
1026: completing registration in a browser other than the initiating browser results in non-functional dialog

RPM: browserid-server-0.2012.01.18-6.el6.x86_64.rpm
Commit: https://github.com/mozilla/browserid/commit/a63e9c4194aba45710a2aa545072df68981cbb07
targeting a deploy tomorrow, 2/2 2-3pm PST window.
Whiteboard: [qa?] → [qa?][target 2/2]
HotFix #2 is done and passed.
Back over to petef for deployment to Prod.
This was just pushed to Prod by petef:
f55084d update changelog and version for hotfix for issue #1026 and issue #972

QA signs off on this push.
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Status: RESOLVED → VERIFIED
Blocks: 723755
Blocks: 727995
No longer blocks: 727995
You need to log in before you can comment on or make changes to this bug.