559137 - Socorro hang report analysis: crash reason and hangid added to .csv files

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Benjamin Smedberg]

I want to do a more detailed analysis of the crash reports we're receiving which come from the hang detector. For now, let's just stick with Windows reports, because it's easier to identify them.

Query version is 3.6.3plugin1 or 3.6.4pre or 3.6.4
Crash reason is EXCEPTION_BREAKPOINT

Past week.

Even if it's just a set of crash IDs, I can run correlations using the JSON and raw dump data to try and break things down better. I'd love a more detailed report, but in order for it to be truly useful I'll need the detailed crash stack so I can try and construct some better signature data.

Comment 1

[Benjamin Smedberg]

Oh, and if it can be automated to send me this report every few days for a while, this would be really useful.

Comment 2

[chris hofmann]

one thought would be to add crash_reason to the .csv files.  that would allow a fair bit of analysis on its own and the .csv files also have uuid_url for digging up the individual reports and JSON raw dumps.

current contents of the .csv files are and we could tack crash_reason on the end.

1 signature
2 url
3 uuid_url
4 client_crash_date
5 date_processed
6 last_crash
7 product
8 version
9 build
10 branch
11 os_name
12 os_version
13 cpu_name
14 address
15 bug_list
16 user_comments
17 uptime_seconds
18 email
19 adu_count
20 topmost_filenames
21 addons_checked
22 flash_version

sanitized (urls edited) reports get published every night under  http://people.mozilla.com/crash_analysis/  with dbaron's correlation reports.

a simple awk, grep, or python script could extract all the "EXCEPTION_BREAKPOINT" entries along with uuid_url or any other fields of interest.

Would this option work?

Comment 3

[Benjamin Smedberg]

Is there data on the CSV files? Are they a sample of the crashes, or the full set? I don't really know anything about them, but since I'm going to have to pull full stack data and do manual HangID correlation until socorro grows those features, I was looking for the simplest solution.

Comment 4

[chris hofmann]

(In reply to comment #3)
> Is there data on the CSV files?

Its just a summary of each crash report with the 22 items in comment 2.

> Are they a sample of the crashes, or the full
> set? 

full set of all processed crashes each day.  (midnight to midnight)

> I don't really know anything about them, but since I'm going to have to
> pull full stack data and do manual HangID correlation until socorro grows those
> features, I was looking for the simplest solution.

it would be a pretty simple 4 or 5 line script to grab just the data you want, when you want it, from this source.

  scp or wget the_file from people
  gunzip the_file | awk -F\t '$23 ~ /EXCEPTION_BREAKPOINT/ {printf "%s\t%s\n" $1,$3}'

would give you a tab separated list of all

signatures uuid_urls

for all the hang reports.   If you decide you want to add os version or product version to help in filtering out the reports that's under your control, and not another request to the socorro team.

We are using these .csv files for other report prototyping like jst's  new crashes stuff http://people.mozilla.com/~jst/new-crashes/Firefox/latest/ and a few other one-off and experimental analysis reports.

Comment 5

[Benjamin Smedberg]

Sure, but the "data" I really wanted was "how can I get them"?

Comment 6

[chris hofmann]

sounds like HangID might be another good addtion to the .csv files if we can get that.

Comment 7

[Benjamin Smedberg]

FWIW, I'm now slurping up the data I need from the CSV files combined with the .json .jsonz files and it's in a couch. I don't need anything else for this bug, although chofmann may still want the HangID/crash reason stuff in the CSV if that isn't too hard to do.

Comment 8

[chris hofmann]

yeah, I'm still interested in crash reason being added to the .csv

Comment 9

[K Lars Lohn [:lars] [:klohn]]

1.6.2 backend changes are getting pushed to staging.  The included changes are:

1 - processor stores HangID in the reports table - hangid is indexed
2 - dailyUrl report now includes 'HangID' and 'reason'


Daily URL format:

1 signature
2 url
3 uuid_url
4 client_crash_date
5 date_processed
6 last_crash
7 product
8 version
9 build
10 branch
11 os_name
12 os_version
13 cpu_name
14 address
15 bug_list
16 user_comments
17 uptime_seconds
18 email
19 adu_count
20 topmost_filenames
21 addons_checked
22 flash_version
23 hangid
23 reason

Comment 10

[chris hofmann]

cool!  thanks lars.

Comment 11

[K Lars Lohn [:lars] [:klohn]]

marking this as resolved so we know we are "go" for production push.

Comment 12

[chris hofmann]

did 1.6.2 get pushed to production last night?  I'm not seeing hangid and reason in the .csv's produced early this morning.

Comment 13

[K Lars Lohn [:lars] [:klohn]]

bah, this was completed in code, but inadvertently omitted from the explicit production push instructions.  I'm refiling a bug to get that done.  See Bug 562856