921161 - Telemetry Server cutover checklist

Status: RESOLVED FIXED

(Webtools Graveyard :: Telemetry Server, defect)

Description

[Mark Reid [:mreid]]

Oct 1, 2013 - Telemetry Server cutover plan
===========================================

1.  Set up production EC2 telemetry-server instances and Load Balancer
2.  Run tests on servers
3.  Clean prod S3 buckets / SQS Queue
4.  Cut over DNS for incoming.telemetry.mozilla.org to point to production AWS LB
5.  Verify we're getting data
6.  Remove "cturra" header for the Zeus rule (and update target) to redirect from data.mozilla.com/submit/telemetry to incoming.telemetry.mozilla.org (see Bug 916243)
7.  Verify no more data is going to Bagheera / Kafka
8.  Shut down the relay infrastructure
9.  Verify everything is working as expected
10. Switch http://telemetry-dash.mozilla.org/ to point at S3
11. Clean up

Bugs with details on the above points to follow.

Comment 1

[Mark Reid [:mreid]]

(In reply to Mark Reid [:mreid] from comment #0)
> 1.  Set up production EC2 telemetry-server instances and Load Balancer
> 2.  Run tests on servers
> 3.  Clean prod S3 buckets / SQS Queue
These three points are covered by bug 921164

> 4.  Cut over DNS for incoming.telemetry.mozilla.org to point to production
> AWS LB
> 5.  Verify we're getting data
These two follow on work in bug 911309, the actual change to be done is bug 921167

> 6.  Remove "cturra" header for the Zeus rule (and update target) to redirect
> from data.mozilla.com/submit/telemetry to incoming.telemetry.mozilla.org
> (see Bug 916243)
As stated, this is bug 916243

> 7.  Verify no more data is going to Bagheera / Kafka
> 8.  Shut down the relay infrastructure
> 9.  Verify everything is working as expected
These three are bug 921170

> 10. Switch http://telemetry-dash.mozilla.org/ to point at S3
This is bug 921077

> 11. Clean up
I'll take care of cleanup as part of closing this bug, and it will include
things like:
- Shut down prototype "relay" server instance (sending to mreid-telemery-incoming)
- Migrate S3 data from mreid-telemetry-published to production telemetry bucket

Comment 2

[Mark Reid [:mreid]]

> 10. Switch http://telemetry-dash.mozilla.org/ to point at S3

This doesn't need to block the "cutover" state, so it can be handled separately.

> 11. Clean up
Relay instances have been shut down. Data doesn't need to be migrated from the pre-release bucket, since we will be exporting data from Mango (per Bug 922745) and don't want to end up with duplicates from when the systems were running in parallel.

I'm calling this bug "done". Thanks to everyone who helped!