652248 - create extraction script for socorro database (devdb)

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Robert Helmer [:rhelmer]]

It would be very helpful for development purposes to be able to extract a subset of the production database that would fit in a local development environment, such as a VM. 

40GB is a reasonable target.

Comment 1

[[:jberkus] Josh Berkus]

The way to do this is to do the extraction script I talked about doing for stage and didn't do then (implementing truncate_56.py instead).

Actually, this would consist of two scripts: one would selectively dump from the staging database (or the dev database) to a pg_dump file.  The other would restore the pg_dump file, selectively restoring TCBS, TCBU and Raw_ADU.

I'd say 3-4 hours to write this.

Comment 2

[Stephen Donner [:stephend] Not actively reading bugmail]

Per https://bugzilla.mozilla.org/show_bug.cgi?id=653609#c4, can we get this into 1.7.8 (and have it block bug 653587, if so?)

Comment 3

[[:jberkus] Josh Berkus]

We'll get it this week.

The extraction script is not tied to any specific release of Socorro, because it doesn't get deployed on production servers.

Comment 4

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to comment #3)
> We'll get it this week.
> 
> The extraction script is not tied to any specific release of Socorro, because
> it doesn't get deployed on production servers.

If it's blocking data being populated on staging, and therefore QA testing, it should block 1.7.8, unless there are other methods of populating staging with said data :-)

Comment 5

[[:jberkus] Josh Berkus]

I don't understand.  How is this blocking staging? There's already a procedure for updating staging.

Comment 6

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to comment #5)
> I don't understand.  How is this blocking staging? There's already a procedure
> for updating staging.

Again, I point to https://bugzilla.mozilla.org/show_bug.cgi?id=653609#c4.

Comment 7

[Robert Helmer [:rhelmer]]

(In reply to comment #5)
> I don't understand.  How is this blocking staging? There's already a procedure
> for updating staging.

We have two environments for QA now:

1) crash-stats.allizom.org (formerly crash-stats.stage.mozilla.com)
   Updated on-demand to release branch only

2) crash-stats-dev.allizom.org 
   Updates automatically to trunk

I think this is more blocking #2 than #1. #1 is only updated after code freeze and when backporting fixes to existing prod, #2 needs something lighter-weight than importing the whole DB.

Longer-term I'd like something more like bug 653609 (synthesize crash data) for #2 but that'll take some time.

Comment 8

[[:jberkus] Josh Berkus]

Let me make sure that I'm solving the right problem then.

An extraction script needs to pull from a live database.  That means that we need to either pull from DevDB or from Master01.   So the workflow would be:

1) Snapshot DevDB
2) Extract mini-DB from DevDB
3) Copy mini-DB around

If that works for you, then I'll proceed with the code I was already working on.  If that doesn't solve the real problem, I should stop now.

Comment 9

[Robert Helmer [:rhelmer]]

(In reply to comment #8)
> Let me make sure that I'm solving the right problem then.
> 
> An extraction script needs to pull from a live database.  That means that we
> need to either pull from DevDB or from Master01.   So the workflow would be:
> 
> 1) Snapshot DevDB
> 2) Extract mini-DB from DevDB
> 3) Copy mini-DB around
> 
> If that works for you, then I'll proceed with the code I was already working
> on.  If that doesn't solve the real problem, I should stop now.

Yes I think it will solve the problem I have in mind. Let's discuss further outside of the bug just to make sure.

Comment 10

[[:jberkus] Josh Berkus]

This task has been completed.  See wiki for instructions on using the script.