469215 - setup postgres replication for breakpad in staging.

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Aravind Gottipati [:aravind]]

Please setup postgres replication for breakpad in the staging environment.  The staging db lives on tm-stage01-master01.  The purpose of this replicated environment is to be able to make backups of the db from the slave without impacting the production db performance.  Once the staging env is setup and work, we can then deploy this to production.

Comment 1

[chizu]

My plan for this is shipping completed write ahead logs to a slave. The slave will replay the logs regularly and we'll backup the write ahead logs and the slave state.

I'll be deploying and verifying this on stage in the next couple days.

Comment 2

[matthew zeier [:mrz]]

chizu, for scheduling/planning, do you have an ETA on this?

Comment 3

[chizu]

This just needs an ACL from tm-stage01-master01 to dracula, the rest is ready.

Comment 4

[chizu]

This is replicating and I've made and restored backups from it. It just needs some more polish and documentation before I can call this done.

Comment 5

[Michael Morgan [:morgamic]]

Great news, Trevor!  Thanks for picking this up. <3

Comment 6

[chizu]

I should note how this is a different than MySQL replication. The master keeps write ahead logs normally. To make a slave we checkpoint one, rsync the database directory over to the slave, and put the slave into recovery mode pulling in those logs over nfs or rsync. Because the slave is in recovery mode, it's not serving queries. This makes it great for backups and failover in case the master dies, but it can't take read-only queries while replicating. Recovery mode can be stopped to run queries in batches.

Comment 7

[chizu]

Documented and running.