695960 - browserid: deploy to production

Status: VERIFIED FIXED

(Cloud Services :: Operations: Miscellaneous, task)

Description

[Pete Fritchman [:petef]]

- deploy an infra-sec reviewed build to prod boxes
- run base-line loadtests
- final QA pass
- browserid.org dns re-point, data migration
- whiskey

Comment 1

[Pete Fritchman [:petef]]

* lower TTL on browserid.org records to 30 seconds (currently 300)
* wait 300 seconds after change pushes
* put existing browserid MySQL in read-only mode (set global read_only=true
** how does the UI handle this if we try to write?
** maybe we are better off putting up a "sorry, we're migrating" page
** /verify and js bundle serving should still work during this
* import database into new prod environment
* verify replication/data in new environment
* switch DNS to new environment, everything becomes read-write, people win
** people using lame TTL-disobeying dns servers will hit the old r/o env
* bump TTL back up to normal
* drink excellent scotch

of course before we actually do this, we should do a dry-run and import
current prod data, then have jbonacci QA with a hacked /etc/hosts. I think
we can do this early next week, after we loadtest the prod cluster.

Comment 2

[Pete Fritchman [:petef]]

deployed!  72 second outage.

Comment 3

[James Bonacci [:jbonacci]]

Done and running.
Verified by jrgm and myself.