427820 - Run a script to collect data for bug 423968

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Benjamin Smedberg]

I've written a little script that will format the data kkovash wants into CSV format for him.

I've tested it on khan against the socorro development DB, and I'd like to get it run now against production, and the data uploaded somewhere:

the script is khan:/home/bsmedberg/dump-modules.py
You'll need to edit the header which has username/password information.
It will need to run on a host which has psycopg2 installed, which should be any of the socorro-capable VMs.

Just run it with "python dump-modules.py > data.csv". It dumps the reports and modules from the last week, which shouldn't be a terrible DB load since it can use the index pretty effectively.

Comment 1

[(not currently active) Ted Mielczarek]

That discussion is misplaced, should be in bug 423968.

Comment 2

[Aravind Gottipati [:aravind]]

Done.  Its a 70MB output file.

http://people.mozilla.com/~aravind/modules.csv.gz

Comment 3

[ken kovash]

Benjamin: it looks like I was a bit too ambitious in the volume of data I was requesting (a laptop won't be able to play around with it).  can I ask for an adjustment to script?

can you pull a smaller date range (e.g., one day rather than one week)?  the file had something like 250K+ rows/crashes.  if we can cut that down to something less than 100K, that'd be great.

also, can you omit the filename versions (i.e., just include the filename fields).  it looked like there were something like 1,200 fields/variables for each crash... if we can cut this down as well, the data would be more workable.

Comment 4

[Benjamin Smedberg]

Sure. We don't need to run the script again, I'll just text-process it down using python.

Comment 5

[Benjamin Smedberg]

Ken, http://benjamin.smedbergs.us/for-kkovash/ has what you need