1035951 - [tracking] Multi-home MozTrap (moztrap.mozilla.org)

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Philippe M. Chiasson (:gozer)]

This is to track the process of multi-homing MozTrap.

Comment 1

[Brandon Johnson [:cyborgshadow]]

I'm working on the databases this week and am looking to be done by next week.

After discussion with camd and peterbe, we'll need webheads in each datacenter (We're using generic phx1 webheads for now, but there's no generic scl3 webheads).

More info in the db multi-homing bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=998308

Comment 2

[Brandon Johnson [:cyborgshadow]]

Meeting scheduled for tomorrow to plan the rest of this deployment with gozer, camd, and I. The database setup work is done. Still to go is to setup front ends, load balancers, and make the move.

Initial communication of downtime for the move has begun in emails with Clint Talbert.

Comment 3

[Brandon Johnson [:cyborgshadow]]

Met with camd and gozer today. We discussed the timeline of events and schedule for the migration.


All parties (dev, QA, DBA, and webops) believe we can meet a deadline of making a production migration next Friday, August 22nd, at 6 PM PDT which will require a small downtime of ~5 to no more than 30 minutes.

Pre-requisites:
- Moztrap DB servers in each DC are created, monitored, and scheduled for daily backups.
- Moztrap Webheads in each DC are created, monitored, and allow netflows for Chef pushes.

Maintenance plan and Emergency contacts:

Contacts:
- cyborgshadow - DBA
- gozer - Webops
- Clint Talbert - QA

Maintenance Plan:
- Webops will point the DNS to the newly deployed MozTrap webheads, which will not be serving connections. (Begin downtime)
- DBA will confirm no new connections to the moztrap database are being issued (can't set read only as the cluster serves other databases)
- DBA will migrate the database from the generic cluster to the newly deployed MozTrap cluster.
- DBA will ensure that database deployment reaches all 4 nodes (2 in scl3, 2 in phx1)
- DBA will run checksums to ensure integrity of deployment.
- DBA will give all clear to Webops to begin serving connections.
- Webops will allow connections to the new environment.
- DBA and Webops will confirm traffic and data flow.
- QA will test and validate traffic.

Once all the above steps are completed, the maintenance will be marked as complete.

On the following Monday, Dev (camd) will push a dummy change via Chef to ensure automated pushes are working properly.

Comment 4

[Brandon Johnson [:cyborgshadow]]

Post maintenance work:

- DBA to Decomm generic MozTrap database.
- Webops to Decomm MozTrap deployment on generic webheads.

Comment 5

[Stephen Donner [:stephend] Not actively reading bugmail]

I'm a little late with this, but #moztrap is where the real-time collaboration/corroboration can happen.

Comment 6

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Stephen Donner [:stephend] from comment #5)
> I'm a little late with this, but #moztrap is where the real-time
> collaboration/corroboration can happen.

From Web QA's standpoint, this looks good; thanks!

Comment 7

[Philippe M. Chiasson (:gozer)]

Done, and moztrap.mozilla.org is now fronted by Dynect, so web failover to SCL3 should be automagic.