1010453 - [tracking] Multi-home bouncer (download.mozilla.org)

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Product Delivery, task)

Comment 1

[Philippe M. Chiasson (:gozer)]

download2.dynect.mozilla.net is now up and running, as well as ready to handle download traffic.

Just need to coordinate with the right folks to flip this on.

Comment 2

[Philippe M. Chiasson (:gozer)]

Turned up for load testing at a distribution rate of 30/2, so 6.6% of traffic. Complete success.

Comment 3

[Philippe M. Chiasson (:gozer)]

Given the remaining blockers are just soft-blockers (not really blocking, just improvements), asking for CAB review to turn this on for good.

Comment 4

[Corey Shields [:cshields]]

The test in #c2 ran for about 30 mins to verify success from an infra point of view, and there was no negative impact that we could tell.

CAB proposal is to run that same 30/2 test for however long it takes to discover any impact through ADI metrics (1-2 days?), after which if there is no negative impact it can go to a full split.

Pinging Lawrence for sign-off on this plan.

Comment 5

[Lawrence Mandel [:lmandel] (use needinfo)]

I'm OK with this plan. I would prefer that IT be confident in this change more than a few days before Firefox 30 is released (June 10) to minimize the risk that something will break. If we need to wait for the release, let's talk again about an appropriate time to do a full split in production.

Comment 6

[Philippe M. Chiasson (:gozer)]

So, since the release is now in 8 days, can I get a nod of approval to turn up the 30/2 test for the next 48 hours?

Comment 7

[Lukas Blakk [:lsblakk] use ?needinfo]

(In reply to Philippe M. Chiasson (:gozer) from comment #6)
> So, since the release is now in 8 days, can I get a nod of approval to turn
> up the 30/2 test for the next 48 hours?

Nod on behalf of RelMan.

Comment 8

[Philippe M. Chiasson (:gozer)]

download.mozilla.org now split 30/2 PHX1/SCL3 as of now.

Comment 9

[Philippe M. Chiasson (:gozer)]

download.mozilla.org now back to PHX1 only.

Comment 10

[Philippe M. Chiasson (:gozer)]

Who should I be chasing down to get final approval to move forward with this?

Comment 11

[Philippe M. Chiasson (:gozer)]

We will be moving forward with this starting tonight, and increasing the ratio of traffic to SCL3 over the week-end until we hit a 50/50 distribution.

Comment 12

[Philippe M. Chiasson (:gozer)]

Right now, we are at:
 - SCL3: 25%
 - PHX1: 75%

Comment 13

[hwine]

adding :nthomas to cc

Comment 14

[Philippe M. Chiasson (:gozer)]

Right now, we are at:
 - SCL3: 30%
 - PHX1: 70%

Comment 15

[Philippe M. Chiasson (:gozer)]

Right now, we are at:
 - SCL3: 40%
 - PHX1: 60%

Comment 16

[Philippe M. Chiasson (:gozer)]

Right now, we are at:
 - SCL3: 45%
 - PHX1: 55%

Comment 17

[Philippe M. Chiasson (:gozer)]

Right now, we are at:
 - SCL3: 50%
 - PHX1: 50%