747374 - build bouncer prod env in scl3

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P4)

Description

[Corey Shields [:cshields]]

Need the AUS dist cluster dual-homed between PHX1 and SCL3  (whereas before it was between PHX1 and SJC1). Also when doing this let's bring dist up to current puppet standards.  The nodes for this in SCL3 were kicked in 744685

Will need an adm host for this, for the sake of simplicity in the new cluster we will do that in scl3. Filed 747373 for this.

So here's the thing:  currently we have an adm, dev, and stage environment for the new aus(4) in phx1.  adm will get replaced with this VM.  It can continue to manage dev and stage for the new environment in phx1, and we will also want to puppetize the new dist cluster for aus2/3 with this module and manage it appropriately.  I'll draw some "before" and "after" graphs to make sure we are all on the same page.

Assigning this to Dan since he will probably have better coverage with nthomas to test this as we go along.

Comment 2

[Jake Maul [:jakem]]

This is a WebOps Q3 goal, so P1.

Comment 3

[Corey Shields [:cshields]]

this is morphing a bit - we are going to spin bouncer off into its own cluster first (which is currently half of the aus/dist cluster).  See https://bugzilla.mozilla.org/show_bug.cgi?id=786820

Comment 4

[Jake Maul [:jakem]]

This is not the proper Q3 goal bug... this is about bringing dist up in SCL3. The goal is to get the aus/dist cluster up to current standards. Bug 786820 is the proper goal bug, and it's already tagged as such.

This might be a suitable 2012Q4 or 2013Q1 bug.

Comment 5

[Jake Maul [:jakem]]

Need some clarification on this... are we more interested in dual-homing bouncer or AUS? I feel like bouncer is the more important of the two apps, because while AUS affects updating, bouncer affects updating as well as new installs.

I'm renaming this for bouncer, so if we also want AUS to be multi-homed let's file another bug. Alternatively, now that the new bouncer cluster is up to current standards, it's feasible to run both AUS and bouncer on the same cluster as we always have, with the maintainability improvements inherent in the newer cluster design.

Wherever we end up putting AUS, it should get Chief so we can get away from the "blocker" AUS2 push bugs we get regularly. Bouncer should get this too, just on general principles. If we put them back on the same cluster, we only need one Chief instance for both apps. :)

Comment 6

[Corey Shields [:cshields]]

Ideally, both should be multi-homed, but I would start with bouncer given the same severity you pointed out.

Comment 7

[Chris Turra [:cturra]]

bouncer has been multi-homed for some time already. aus is currently only hosted out of phx1, but there is currently work going on in releng to review product delivery end-to-end. i am going to mark this bug as r/fixed.