891996 - Launch refactored snippets service

Status: VERIFIED FIXED

(Snippets :: Service, defect)

Description

[Justin Crawford [:hoosteeno] [:jcrawford]]

Tentative launch window: 

14:00:00 Thursday July 18, 2013, 2 hrs

Rough launch plan follows:

Evening before...
* notify wbowden to hold all changes to snippets content 
* export data from prod and reload new snippets db (clone this: https://bugzilla.mozilla.org/show_bug.cgi?id=891682)
* run data verification automation

Day of...
* add new servers to the pool, allow them to share the load, observe their behavior
* verify load capabilities
* verify functionality (QA)
* remove old servers from the pool
* notify all on successful launch
* extra attention to monitoring tools for ~24 hrs

Comment 1

[Justin Crawford [:hoosteeno] [:jcrawford]]

[That 14:00 is UTC.]

Comment 2

[Justin Crawford [:hoosteeno] [:jcrawford]]

Update: Our VMs are sufficient to carry production load, but we need more for redundancy. We are holding the launch window open while more VMs are provisioned.

Comment 3

[Justin Crawford [:hoosteeno] [:jcrawford]]

We've delayed the launch to allow virtual machines to be built, connected to the network, set up with puppet & chief, and tested with live data. The entire sequence of events now hinges on the network flows (bug 895456). Once those are ready, we will reschedule launch based on the critical path outlined below. 

We need to:

1. Follow up on/verify https://bugzilla.mozilla.org/show_bug.cgi?id=895456 to make sure it is done by 4 days before launch.
2. Verify that the two new VMs referenced in https://bugzilla.mozilla.org/show_bug.cgi?id=895420 are fully puppetised and rebooted by 3 days before launch (bug 895551).
3. Verify that the two new VMs can take a chief deploy of the new snippets application by 3 days before launch (bug 895554).
4. Conduct a live test of the infrastructure 2 days before launch (along with the team in #snippets) and making any needed adjustments immediately (bug 895558)
5. Load data into the database one day before launch (bug 895559)
6. Launch the application (along with the team in #snippets) by moving VMs into the pool; observing their behavior, and if it is satisfactory to all parties, removing the old snippets servers from the pool (leaving them running) (this bug).

Comment 4

[Justin Crawford [:hoosteeno] [:jcrawford]]

The new snippets servers with the new service are live; the old snippets servers are no longer handling requests.

Comment 5

[raymond [:retornam] (inactive)]

verified fixed. Thanks everyone