861411 - Deploy BigTent to production with train-2013.01.17 Release 17

Status: RESOLVED FIXED

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Description

[Austin King [:ozten]]

SHA: fee6d772086f678062c459ca1418448dc767d40e
tree: https://github.com/mozilla/browserid-bigtent/tree/train-2012.01.17

Previously deployed to stage and tested: bug #861004

gene: Note that we want to collect all of our debugging logs during the production deployment. Maybe as the first step?

We should delete these debugging information once it's been collected and shared with lloyd.

Comment 1

[John Morrison [:jrgm]]

QA signs off on this going to production. Gene has said the existing prod stack will remain available so  we switch back to it if needed.

Comment 2

[Gene Wood [:gene]]

Ok spun these new stacks in us-east-1 and us-west-2

instances
{u'i-312d1959': {'Name': u'identity-prod-prod-admin-0412',
                 'private_ip_address': u'10.146.36.93'},
 u'i-352d195d': {'Name': u'identity-prod-prod-proxy-0412',
                 'private_ip_address': u'10.146.37.22'},
 u'i-3b2d1953': {'Name': u'identity-prod-prod-webhead-0412',
                 'private_ip_address': u'10.146.37.106'},
 u'i-3b7c4059': {'Name': u'identity-prod-prod-proxy-0412',
                 'private_ip_address': u'10.146.33.199'},
 u'i-4675fe23': {'Name': u'identity-prod-prod-bigtent-0412',
                 'private_ip_address': u'10.146.35.131'},
 u'i-54c68e3f': {'Name': u'identity-prod-prod-keysign-0412',
                 'private_ip_address': u'10.146.33.252'},
 u'i-68e94f03': {'Name': u'identity-prod-prod-bigtent-0412',
                 'private_ip_address': u'10.146.33.22'},
 u'i-76e94f1d': {'Name': u'identity-prod-prod-dbwrite-0412',
                 'private_ip_address': u'10.146.33.196'},
 u'i-d1baa3b9': {'Name': u'identity-prod-prod-dbread-univ',
                 'private_ip_address': u'10.146.33.213'},
 u'i-d9701db6': {'Name': u'identity-prod-prod-dbread-univ',
                 'private_ip_address': u'10.146.37.43'},
 u'i-f6f46e97': {'Name': u'identity-prod-prod-db-univ',
                 'private_ip_address': u'10.146.35.96'},
 u'i-f888ab99': {'Name': u'identity-prod-prod-bigtent-0412',
                 'private_ip_address': u'10.146.37.151'}}
load balancers
{u'browserid-org-0412': {'dns_name': u'browserid-org-0412-1804145172.us-east-1.elb.amazonaws.com',
                         'instances': []},
 u'bt-login-persona-org-0412': {'dns_name': u'bt-login-persona-org-0412-198838778.us-east-1.elb.amazonaws.com',
                                'instances': [{'Name': u'identity-prod-prod-bigtent-0412',
                                               'id': u'i-4675fe23',
                                               'private_ip_address': u'10.146.35.131'},
                                              {'Name': u'identity-prod-prod-bigtent-0412',
                                               'id': u'i-68e94f03',
                                               'private_ip_address': u'10.146.33.22'},
                                              {'Name': u'identity-prod-prod-bigtent-0412',
                                               'id': u'i-f888ab99',
                                               'private_ip_address': u'10.146.37.151'}]},
 u'dbwrite-0412': {'dns_name': u'internal-dbwrite-0412-1289992602.us-east-1.elb.amazonaws.com',
                   'instances': [{'Name': u'identity-prod-prod-dbwrite-0412',
                                  'id': u'i-76e94f1d',
                                  'private_ip_address': u'10.146.33.196'}]},
 u'keysign-0412': {'dns_name': u'internal-keysign-0412-2025986634.us-east-1.elb.amazonaws.com',
                   'instances': [{'Name': u'identity-prod-prod-keysign-0412',
                                  'id': u'i-54c68e3f',
                                  'private_ip_address': u'10.146.33.252'}]},
 u'persona-org-0412': {'dns_name': u'persona-org-0412-1922840836.us-east-1.elb.amazonaws.com',
                       'instances': [{'Name': u'identity-prod-prod-webhead-0412',
                                      'id': u'i-3b2d1953',
                                      'private_ip_address': u'10.146.37.106'}]},
 u'proxy-0412': {'dns_name': u'internal-proxy-0412-787986314.us-east-1.elb.amazonaws.com',
                 'instances': [{'Name': u'identity-prod-prod-proxy-0412',
                                'id': u'i-352d195d',
                                'private_ip_address': u'10.146.37.22'},
                               {'Name': u'identity-prod-prod-proxy-0412',
                                'id': u'i-3b7c4059',
                                'private_ip_address': u'10.146.33.199'}]}}
instance_ip_list
u'10.146.35.131 10.146.37.151 10.146.33.213 10.146.37.106 10.146.33.196 10.146.33.252 10.146.37.43 10.146.37.22 10.146.33.22 10.146.36.93 10.146.35.96 10.146.33.199'


instances
{u'i-04667b36': {'Name': u'identity-prod-prod-dbread-univ',
                 'private_ip_address': u'10.148.37.14'},
 u'i-78607d4a': {'Name': u'identity-prod-prod-dbread-univ',
                 'private_ip_address': u'10.148.35.29'},
 u'i-80d7ceb2': {'Name': u'identity-prod-prod-webhead-0412',
                 'private_ip_address': u'10.148.35.202'},
 u'i-86d7ceb4': {'Name': u'identity-prod-prod-keysign-0412',
                 'private_ip_address': u'10.148.37.175'},
 u'i-8ad7ceb8': {'Name': u'identity-prod-prod-dbwrite-0412',
                 'private_ip_address': u'10.148.35.207'},
 u'i-8ed7cebc': {'Name': u'identity-prod-prod-bigtent-0412',
                 'private_ip_address': u'10.148.37.232'},
 u'i-92d7cea0': {'Name': u'identity-prod-prod-proxy-0412',
                 'private_ip_address': u'10.148.35.151'},
 u'i-96d7cea4': {'Name': u'identity-prod-prod-proxy-0412',
                 'private_ip_address': u'10.148.33.73'},
 u'i-acd7ce9e': {'Name': u'identity-prod-prod-admin-0412',
                 'private_ip_address': u'10.148.32.88'},
 u'i-e24d50d0': {'Name': u'identity-prod-prod-db-univ',
                 'private_ip_address': u'10.148.33.176'},
 u'i-f0d7cec2': {'Name': u'identity-prod-prod-bigtent-0412',
                 'private_ip_address': u'10.148.33.168'},
 u'i-f2d7cec0': {'Name': u'identity-prod-prod-bigtent-0412',
                 'private_ip_address': u'10.148.35.220'}}
load balancers
{u'browserid-org-0412': {'dns_name': u'browserid-org-0412-1848803836.us-west-2.elb.amazonaws.com',
                         'instances': []},
 u'bt-login-persona-org-0412': {'dns_name': u'bt-login-persona-org-0412-1871203801.us-west-2.elb.amazonaws.com',
                                'instances': [{'Name': u'identity-prod-prod-bigtent-0412',
                                               'id': u'i-8ed7cebc',
                                               'private_ip_address': u'10.148.37.232'},
                                              {'Name': u'identity-prod-prod-bigtent-0412',
                                               'id': u'i-f0d7cec2',
                                               'private_ip_address': u'10.148.33.168'},
                                              {'Name': u'identity-prod-prod-bigtent-0412',
                                               'id': u'i-f2d7cec0',
                                               'private_ip_address': u'10.148.35.220'}]},
 u'dbwrite-0412': {'dns_name': u'internal-dbwrite-0412-1846556457.us-west-2.elb.amazonaws.com',
                   'instances': [{'Name': u'identity-prod-prod-dbwrite-0412',
                                  'id': u'i-8ad7ceb8',
                                  'private_ip_address': u'10.148.35.207'}]},
 u'keysign-0412': {'dns_name': u'internal-keysign-0412-81970408.us-west-2.elb.amazonaws.com',
                   'instances': [{'Name': u'identity-prod-prod-keysign-0412',
                                  'id': u'i-86d7ceb4',
                                  'private_ip_address': u'10.148.37.175'}]},
 u'persona-org-0412': {'dns_name': u'persona-org-0412-325274653.us-west-2.elb.amazonaws.com',
                       'instances': [{'Name': u'identity-prod-prod-webhead-0412',
                                      'id': u'i-80d7ceb2',
                                      'private_ip_address': u'10.148.35.202'}]},
 u'proxy-0412': {'dns_name': u'internal-proxy-0412-664847413.us-west-2.elb.amazonaws.com',
                 'instances': [{'Name': u'identity-prod-prod-proxy-0412',
                                'id': u'i-96d7cea4',
                                'private_ip_address': u'10.148.33.73'},
                               {'Name': u'identity-prod-prod-proxy-0412',
                                'id': u'i-92d7cea0',
                                'private_ip_address': u'10.148.35.151'}]}}
instance_ip_list
u'10.148.37.14 10.148.37.232 10.148.35.207 10.148.35.151 10.148.33.176 10.148.35.29 10.148.35.220 10.148.35.202 10.148.33.73 10.148.37.175 10.148.33.168 10.148.32.88'

Comment 3

[Gene Wood [:gene]]

Here are the two names you're looking for to test this
bt-login-persona-org-0412-198838778.us-east-1.elb.amazonaws.com
bt-login-persona-org-0412-1871203801.us-west-2.elb.amazonaws.com

Comment 4

[John Morrison [:jrgm]]

Okay, with my local dns mapping yahoo.login.persona.org to either of those two regions, they both checkout okay, including showing the improved text in the UI for a failure case.

This can be switched in production.

Comment 5

[Gene Wood [:gene]]

0412 stack switched live in DNS. I'll leave the old email logging stack up till Monday and then collect all logs and destroy it

Comment 6

[Gene Wood [:gene]]

I'm going to leave this open till monday

If we need to rollback we want to go to :

bt-login-persona-org-0410-2059723931.us-west-2.elb.amazonaws.com
bt-login-persona-org-0410-785858900.us-east-1.elb.amazonaws.com

Comment 7

[John Morrison [:jrgm]]

Both kthiessen and I checked for normal operation and it looked good. (I actually had a bit of weirdness where my FF would not re-resolve to the new IPs for many minutes, even though the mozilla resolvers knew about the change in address. A restart cleared it up).