754458 - Get replayDB working, copying to StageDB

Status: RESOLVED FIXED

(Socorro :: Database, task)

Description

[[:jberkus] Josh Berkus]

ReplayDB is currently not running.  This bug has two parts:

1) get ReplayDB running and doing file-based replication from master01.

2) set up the script to do binary snapshot cloning to StageDB.

Comment 1

[Matt Pressman [:mpressman]]

ReplayDB is currently running and has been since 5/4

Comment 2

[[:jberkus] Josh Berkus]

Is it on a separate machine, then?  Because I only see two postmasters on dev.db, one of them for the crash-stats-dev database, and one of the for the scratch DevDB.

Comment 3

[Matt Pressman [:mpressman]]

ahh, I see the confusion, it's on the stage host. ReplayDB is running on socorro1.stage.db.phx1:4321

The original plan was indeed for it to go on the DevDB host. That was changed to socorro1.stage.db.phx1 after I had to move the crash-stats-dev instance off of socorro1.dev.dmz.phx1 a couple weeks ago because of the lack of storage available.

Comment 4

[[:jberkus] Josh Berkus]

Putting ReplayDB on the same box as StageDB is pretty far from ideal.  One of the goals of StageDB is to have it as similar as possible to production.  Running to Postgres instances on the box kind of hampers that goal, and makes checking for things like performance regressions impossible.  Did you discuss this with Laura before you did it?

Comment 5

[Laura Thomson :laura]

So we need four different databases, outside of production:
- scratch devdb.  Purpose: developers able to play with stuff
- csd: Purpose: run master branch
- stage: Purpose: run release branch for deployment and QA testing.  This should be as close as possible to prod.  We have learned this the hard way.
- replaydb: Purpose: copy prod to other places (stage, dev, backups).

I'm now really unclear what's running where, as it seems to have changed multiple times since we first had the discussion, but I will say:
- stage shouldn't have a second instance of PG on there.
- replaydb needs to be taken seriously since it's a source of backups.

Comment 6

[[:jberkus] Josh Berkus]

Laura,

I don't really know what's going on.  When I met with mpressman about this a couple months ago, this was the plan: https://mana.mozilla.org/wiki/display/websites/PHX+Staging+And+Dev+DB+Config

Based on that plan, dev.db would be running 2 instances and 3 databases, and stageDB would be almost a copy of the production database servers.

I don't know when exactly the plan changed, or why.

Comment 7

[Matt Pressman [:mpressman]]

Laura,
I will remove the replaydb frorm socorro1.stage.db.phx1 and move it to socorro1.dev.db.phx1. This will be completed first thing tomorrow morning

Comment 8

[Laura Thomson :laura]

(In reply to Matt Pressman [:mpressman] from comment #7)
> Laura,
> I will remove the replaydb frorm socorro1.stage.db.phx1 and move it to
> socorro1.dev.db.phx1. This will be completed first thing tomorrow morning

Thanks Matt!

Comment 9

[Matt Pressman [:mpressman]]

The replaydb has been moved off of socorro1.stage.db.phx1 and has been transferred to socorro1.dev.db.phx1. The directory structure is the same as was on stage in that the PGDATA directory is at /pgdata/pgslave/9.0/data. The stage host still requires some minor cleanup so as to not cause confusion in the future that this host contains a replicating instance. This includes removing the data directory, the postgresql-pgslave init scripts and the corresponding sysconfig variables.

Additionally, the automation script will be ironed out tomorrow for copying over to stage, dev and backups. Since this is pretty much close to completion, I want to ensure that it runs cleanly on socorro1.dev.db as it was worked on from stage. This shouldn't be much different, but I want to be sure before signing off.

Comment 10

[Matt Pressman [:mpressman]]

replayDB is currently named pgslave on socorro1.dev.db.phx1 it's PGDATA directory is /pgdata//pgslave/9.0/data/ it's PGPORT port is 5499.

Comment 11

[Matt Pressman [:mpressman]]

The script to copy is part of a larger script, this script uses netcat as is much quicker to perform the data transfer than running pg_restore. However quicker, it is more complex as it requires a listener process on stage and a sender on devdb. This is now working and time from stopping the service to bringing it up with the new dataset on stage is under 30 minutes.