939961 - please deploy browserid train-2013.10.23 to production

Status: RESOLVED DUPLICATE of bug 958719

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Description

[Karl Thiessen [:kthiessen, he/him]]

from bug 930783:

sha: fdfdd553fa3c2a417c80d7457ced984a05fb9120

ChangeLog: https://github.com/mozilla/browserid/blob/train-2013.10.23/ChangeLog#L2-L11

Schedule: http://personatra.in/#2013.10.23

RPM now on r6:

/home/kthiessen/Library/github/mozilla/browserid_private/
rpmbuild/RPMS/x86_64/browserid-server-0.2013.10.23-1.el6_122101.x86_64.rpm

(with no line-break in the path)

Comment 1

[Gene Wood [:gene]]

Stacks 1119 built at both us-west-2 and us-east-1 in production. When would you like to plan to switch traffic to these?

Comment 2

[Karl Thiessen [:kthiessen, he/him]]

:jrgm is running load tests on staging today/tomorrow morning.  After we've had a chance to check those results and do a quick overall sanity check, I'd like to switch prod.

I'd say some time between 2 and 3 tomorrow (Tue) afternoon, if that works ok for you.

Comment 3

[Gene Wood [:gene]]

Sounds good

Comment 4

[Karl Thiessen [:kthiessen, he/him]]

This is going to be delayed due to problems found by :jrgm with the load tests.  Details to follow in email from :jrgm.

Comment 5

[John Morrison [:jrgm]]

There's not a great bit of detail to follow. Basically, the toobusy stuff, which never fires during a 250K ADU load test, instead was firing about 100 times per hour (i.e., something was stalling the event loop for >40 ms).

Comment 6

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

this bug hasn't moved in a couple of days. is anyone working on reproducing the load test errors or finding the offending changes via git bisect?

Comment 7

[Karl Thiessen [:kthiessen, he/him]]

This deployment is blocked by https://github.com/mozilla/persona/issues/4027 -- see that issue for details.

Comment 8

[John Morrison [:jrgm]]

We haven't found a definitive resolution to the 'too busy' problem, and this is increasingly looking like it is a problem with AWS. So I put in a pull request to increase maximum_event_loop_lag to 100ms.

Gene: could you build a new stage stack with this change https://github.com/mozilla/identity-ops/pull/138
(and keeping the same rpm for train-2013.10.23).

Comment 9

[John Morrison [:jrgm]]

Hmm, I now see this is the production deploy ticket. Nevermind.

Comment 10

[Gene Wood [:gene]]

With this train going live in stage last week, when should I plan to put this into production?

Comment 11

[John Morrison [:jrgm]]

Yeah Gene, if you could build production stacks for this train, that would be good, and if we can find a time that works we can ship this week.

Comment 12

[Karl Thiessen [:kthiessen, he/him]]

Seconding :jrgm's comments -- he's got as much context here as I do.

Comment 13

[Gene Wood [:gene]]

Stack 1217 is up in both datacenters

Comment 14

[John Morrison [:jrgm]]

Fuck it. This rpm does not have this commit - https://github.com/mozilla/persona/pull/4035.
Closing this bug.

Comment 15

[John Morrison [:jrgm]]

This train is derailed. Merry Christmas.

Comment 16

[Gene Wood [:gene]]

Ok, I'll tear down those prod stacks

Comment 17

[Edwin Wong [:edwong]]

Talked to jrgm and karl.  We're going to work on hot patching still but heavily considering waiting till 2014 for deploy to reduce any risk over the holiday break.  We'll revisit this tomorrow to see where we're at.

Comment 18

[Karl Thiessen [:kthiessen, he/him]]

Marking as dupe to forward people to the new production ticket.