427857 - Set up more realistic stage database

Status: RESOLVED WORKSFORME

(Socorro :: General, task)

Description

[Michael Morgan [:morgamic]]

The current stage server works fine but doesn't have a large enough data set to test things like topcrasher graphs over long periods or time or topcrasher queries in general for arbitrarily large datasets.

Would be helpful to set up a stage instance that pulls from either the entire prod db or a subset of it (maybe a month's worth).

Aravind -- is this possible to do in the short term?  Next 2-3 days?

Comment 1

[Aravind Gottipati [:aravind]]

I think this defeats the purpose of stage.  what you are suggesting is to use stage as another dev env to try things on large datasets.  I think khan should be the place for this.  If you want, I can get you a daily database dump drop from the prod db (or do you guys already have it?).

Or is this not what you are talking about?

Comment 2

[Michael Morgan [:morgamic]]

Thought it was huge?  Does khan have space for it?

Comment 3

[Michael Morgan [:morgamic]]

Two things:
1. stage should probably have this dataset too, since that is a more accurate representation of prod and they are supposed to be as similar as possible
2. agree that dev should have it as well, but throwing around a dataset that large is not something we really want or have to do -- so if we could grab a subset for khan, that'd be sufficient

Comment 4

[Aravind Gottipati [:aravind]]

I could grab a subset, but I am not sure how I'd do that other than doing some massive select from the prod db.  I don't see any options in pg_dump for doing getting a subset of the data out.

Regd. space on khan, we can probably get some chunk of space off the new storage servers to do this.  But thats going to take some time (like a week or so).

Comment 5

[Michael Morgan [:morgamic]]

Hey aravind, could you provide an update for this?  We need this to test partitioning schemes and the topcrasher cron jobs.

Comment 6

[Michael Morgan [:morgamic]]

We have a stage db and Austin is working on the test db script for test install setup/teardown.