Closed
Bug 776304
Opened 13 years ago
Closed 13 years ago
browserid: QA and deploy BrowserID train-2012.07.20 to production
Categories
(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)
Cloud Services
Operations: Deployment Requests - DEPRECATED
x86_64
Linux
Tracking
(Not tracked)
RESOLVED
INCOMPLETE
People
(Reporter: lhilaiel, Assigned: Atoll)
References
Details
(Whiteboard: [qa+])
Attachments
(1 file)
520 bytes,
patch
|
Details | Diff | Splinter Review |
Version: 78b9df364ea (0.2012.07.20.1)
available here: https://github.com/mozilla/browserid/tree/train-2012.07.20
Tests pass: http://travis-ci.org/#!/mozilla/browserid/builds/1924330
ChangeLog including issues resolved:
https://github.com/mozilla/browserid/blob/train-2012.07.20/ChangeLog#L1-29
[QA] Suggested additional areas of focus for QA:
* English language review and link checking on main site
* Verification of i18n of main site (use debug language? it-CH)
* Verify all static resources are served from correct domain
* Explore the new forgot password flow!
* Test all permutations of email addition, user accnt creation, email re-confirmation, and password reset on a different browser than where the action is initiated
* double check metrics fixes. Are lines in router-metrics.log correct? Is there one every time the dialog pops up, even when your browser cache is full?
* how do fonts look on IE8? On Windows Firefox?
* verify that other locales look correct. Lots of strings have changed and will be untranslated, but let's make sure the i18n system is working properly in staging environment asap.
[ops] deployment issues:
* new static process introduced as proposed in bug 768169
* metrics file name has changed. no longer is it browserid-metrics.log but router-metrics.log that the metrics team should parse.
* 'locale_directory' no longer a neccesary nor allowed configuration parameter
* please confirm with QA that the "debug locale" is configured on in our staging environment
Comment 1•13 years ago
|
||
Reassigning to :atoll who is covering for :gene on this Monday deployment.
Also:
:atoll - we normally deply to the l10n environment and the clientX machines at the same time we deploy this to Stage.
Assignee: gene → rsoderberg
Comment 2•13 years ago
|
||
:lloyd and zack.carter,
Are there specific steps for addressing bug 768169 ?
It is listed as a dependency, so just want to get everything here that is needed...
Reporter | ||
Comment 3•13 years ago
|
||
:jbonacci,
From an ops perspective it's adding a new process, statsd reporting, rsbac rules etc.
From a QA perspective it should be zero visible change, we just should not have broken anything. W.R.T. logs, you should now see all requests for static resources in static.log rather than browserid.log.
Comment 4•13 years ago
|
||
Release Wiki for this train is here:
https://wiki.mozilla.org/QA/Browser_Technologies/Services/Releases/BrowserID_07202012
Test Plan Spreadsheet is here:
https://docs.google.com/spreadsheet/ccc?key=0As8GUPwNNWMFdDJZSjYzMUxDSnZVQW5vZy0wMjU0Mmc#gid=0
Comment 5•13 years ago
|
||
Is there an ETA for deploying this to stage environment? atoll?
Working on this today. Has a Metrics bug been filed regarding the new filename, or should I do so?
Status: NEW → ASSIGNED
Summary: QA and deploy BrowserID train-2012.07.20 to production → browserid: QA and deploy BrowserID train-2012.07.20 to production
(In reply to Lloyd Hilaiel [:lloyd] from comment #0)
> * please confirm with QA that the "debug locale" is configured on in our
> staging environment
For this particular parameter, "debug_lang" defaults to "it-CH", we do not override the default, and so this should work automatically in staging. It *probably* shouldn't be in production, and QA should test for its absence, but it also probably isn't a rollback-grade issue if it's there by accident.
Comment 9•13 years ago
|
||
Verified the correct version for this train:
78b9df3 changes in train-2012.07.20
locale svn r107851
QA accepts this train in Stage for testing...
Comment 10•13 years ago
|
||
On the webheads, verified the following processes:
node bin/static
node bin/verifier
node bin/router
node bin/browserid
And, the following services:
browserid-router -> /var/services/browserid-router
browserid-static -> /var/services/browserid-static
browserid-verifier -> /var/services/browserid-verifier
browserid-webhead -> /var/services/browserid-webhead
browserid-webhead-loadtest -> /var/services/browserid-webhead-loadtest
All other servers had the expected processes and services.
Comment 11•13 years ago
|
||
Logging check:
web*
/var/log/browserid/browserid.log
/var/log/browserid/browserid-metrics.json
/var/log/browserid/router.log
/var/log/browserid/static.log
/var/log/browserid/verifier-compute.log
/var/log/browserid/verifier.log
/var/log/browserid/verifier-metrics.json
Note: I am not seeing router-metrics.json
And, further, the browserid-metrics.json file looks new/updated.
Perhaps that file will show/be created once we start hitting Stage with traffic
sweb*
/var/log/browserid/dbwriter.log
sign*
/var/log/browserid/keysigner-compute.log
/var/log/browserid/keysigner.log
Attachment #645641 -
Attachment description: draft: remove locale_directory from stage config (v1) → puppet: remove locale_directory from stage config (v1)
![]() |
Assignee | |
Comment 12•13 years ago
|
||
Deployed train-2012.07.20 to stage with bug 768169's puppet (v7), sysadmins (v1) patches, and this bug's puppet (v1) patch.
Ops is not yet ready to signoff on this deployment due to unfinished work in bug 768169, however the missing components (metrics, monitoring) are not required for QA to begin testing.
Comment 13•13 years ago
|
||
And, it looks like we got all the services up and running about the same time:
web1.idweb.scl2.stage.svc.mozilla.com
{"level":"info","message":"browserid server starting up","timestamp":"2012-07-27T02:14:18.862Z"}
{"level":"info","message":"router server starting up","timestamp":"2012-07-27T02:14:18.791Z"}
{"level":"info","message":"static starting up","timestamp":"2012-07-27T02:14:18.836Z"}
{"level":"info","message":"verifier server starting up","timestamp":"2012-07-27T02:14:18.745Z"}
web2.idweb.scl2.stage.svc.mozilla.com
{"level":"info","message":"browserid server starting up","timestamp":"2012-07-27T02:15:27.158Z"}
{"level":"info","message":"router server starting up","timestamp":"2012-07-27T02:15:27.101Z"}
{"level":"info","message":"static starting up","timestamp":"2012-07-27T02:15:27.149Z"}
{"level":"info","message":"verifier server starting up","timestamp":"2012-07-27T02:15:27.057Z"}
web3.idweb.scl2.stage.svc.mozilla.com
{"level":"info","message":"browserid server starting up","timestamp":"2012-07-27T02:16:35.287Z"}
{"level":"info","message":"router server starting up","timestamp":"2012-07-27T02:16:35.217Z"}
{"level":"info","message":"static starting up","timestamp":"2012-07-27T02:16:35.277Z"}
{"level":"info","message":"verifier server starting up","timestamp":"2012-07-27T02:16:35.192Z"}
sweb1.idsecweb.scl2.stage.svc.mozilla.com
{"level":"info","message":"dbwriter starting up","timestamp":"2012-07-27T02:17:43.205Z"}
sweb2.idsecweb.scl2.stage.svc.mozilla.com
{"level":"info","message":"dbwriter starting up","timestamp":"2012-07-27T02:18:25.860Z"}
sweb3.idsecweb.scl2.stage.svc.mozilla.com
{"level":"info","message":"dbwriter starting up","timestamp":"2012-07-27T02:19:33.357Z"}
sign1.idkeysign.scl2.stage.svc.mozilla.com
{"level":"info","message":"Certs will be issued from: login.anosrep.org","timestamp":"2012-07-27T02:20:42.684Z"}
sign2.idkeysign.scl2.stage.svc.mozilla.com
{"level":"info","message":"Certs will be issued from: login.anosrep.org","timestamp":"2012-07-27T02:21:52.230Z"}
sign3.idkeysign.scl2.stage.svc.mozilla.com
{"level":"info","message":"Certs will be issued from: login.anosrep.org","timestamp":"2012-07-27T02:23:00.332Z"}
![]() |
Assignee | |
Comment 14•13 years ago
|
||
stage is currently down due to undocumented schema alteration fd46d9e964912122b6e3b150f20afd90d5f1afa1 "add .verified to list_emails response and database", will review for any other changes and then prepare an alter script
![]() |
Assignee | |
Comment 15•13 years ago
|
||
A bug was found in the RPM build process, that prevents a clean rpm build from a fresh checkout. The workaround and fix are documented in https://github.com/mozilla/browserid/issues/2165 and a new locales build r107854 or later will be pushed out tomorrow.
Deployed train-2012.07.20 to client3..9.
![]() |
Assignee | |
Comment 16•13 years ago
|
||
This bug (and its associated train, 2012.07.20) will be RESOLVED WONTFIX no later than Aug 03 unless it is possible to reschedule train-2012.08.03 to Aug 10 or later, with a production deploy date of Aug 22 or later.
train-2012.07.20 will not be shipped to production, as stage testing by Ops and QA will be derailed no more than 2 days after we successfully deploy it to staging, by train-2012.08.03 landing on Aug 03.
Ops and QA will continue to work on train-2012.07.20 until train-2012.08.03 lands on Aug 03. All ops engineering work for train-2012.07.20 will be carried forward to train-2012.08.03.
Work on metrics and cepmon for the new static process continues in bug 768169, and work on the schema change continues in this bug. The schema alteration adds a new testing requirement, "old code w/ new schema". Work on this schema change will be reused for train-2012.08.03 as it includes a schema change as well.
Comment 17•13 years ago
|
||
We got the go ahead to do some late testing on this derailed release.
I was able to sign in, sign up, and delete and account.
Looking at logs for 2012.07.29
A quick check of the static.log files on web* shows some 304s and 404s for today.
The 304s are a mix of
/favicon.ico
/sign_in
/ver.txt
The 404s are a mix of
/sdk
/robots.txt
/admin/
The rest of the logs look good.
Comment 18•13 years ago
|
||
Also verified that clientX boxes have been updated, so we can run load at any time...
Comment 19•13 years ago
|
||
I started a load_gen from client5 at 22:23 PDT
Comment 20•13 years ago
|
||
Closing this one down to not block bug 779264.
I am rolling some PR/issues from this release into the next for Verification:
* 1909: Static process
* 1757: Create new process for serving static or "cacheable" resources
* 2019: Minimalist implementation of support in core for Proxy IDP (a.k.a. "BigTent")
* 1862: Translate the main site
* 1824: mixed language in dialog
* 2093: Train-2012.07.20 l10n issues
* 1954: setting a timeout for provisioning of 20s
* 1570: Certificate provisioning doesn't time out
* 2109: returnTo, siteName, and siteLogo only should work with the observer API
* 2086: siteName and siteLogo not scrubbed from id.get()
Status: ASSIGNED → RESOLVED
Closed: 13 years ago
Resolution: --- → INCOMPLETE
You need to log in
before you can comment on or make changes to this bug.
Description
•