deploy train-2012.11.23 version 3, sha b74cf6ead to production. Note: I'm going to recheck some stuff tonight, and we have a meeting together in the morning. I'm thinking we're good to go at this point, but will give the final signoff tomorrow.
I didn't set gene as the assigned, so he didn't have this on his radar. We synced up at 3pm and I wasn't feeling to good (physically) to push this through. So we will get this out on Monday. My fault.
12:59pm dns drained traffic at SCL2
1:06pm began scl2 code deploy
1:10pm scl2 updated, please test and approve
This change is similarly required in production puppet for config/production.json https://bugzilla.mozilla.org/show_bug.cgi?id=816208#c2
Config change has been made (sysadmins r54178) and deployed at scl2. All processes have been restarted.
Okay, I have login/logout with bugz,mdn,affiliates,etherpad.mozilla.org,marketplace,mozillians,mfbeer,123done,voost,dolumar,times-xword,openphoto with scl2. I can do signup,forgetpass,changepass,addemail,cancelaccount. https://login.persona.org/.well-known/browserid returns a pubkey, and verifier /verifier returns a response. And I get l10n content when I choose a non-en language. Not seeing errors on pencil. Okay gene, we can switch traffic to phx1.
3:37 switched from scl2 to phx1
3:42 switched back to phx1 per jrgm
rolling back to browserid-server-0.2012.10.12-6.el6_110339.x86_64 at scl2 per jrgm
Yes, thanks gene. Also let's undo the puppet change in prod for adding proxy_idps https://bugzilla.mozilla.org/show_bug.cgi?id=816208#c2 We'll need to figure out why bugzilla was unhappy.
4:41 rollback at scl2 complete
Okay, starting validation. Will update soon.
Okay, repeated validations noted above. Gene you can add scl2 back into public operation.
5:44pm scl2 added back to dns load balance group.
Status: ASSIGNED → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → INCOMPLETE
The reason for the rollback was that we had a change in the verifier service that was not compatible with how we were being called by some RPs, including bugzilla. The fix for that is in bug 820446 and https://github.com/mozilla/browserid/pull/2855 and a new build of train-2012.11.23 is in stage for testing now.
You need to log in before you can comment on or make changes to this bug.