976077 - Build report/index page from processed JSON's json_dump instead of pipe dump

Status: RESOLVED FIXED

(Socorro :: Webapp, task)

Description

[(not currently active) Ted Mielczarek]

Currently the report/index page gets built from the pipe dump output. We should switch it to use the processed JSON instead, since that has more data nowadays and we can do more interesting things (bug 929098, for example).

Comment 1

[Peter Bengtsson [:peterbe]]

We already use Processed JSON for the report index page. 
https://github.com/mozilla/socorro/blob/master/webapp-django/crashstats/crashstats/views.py#L980

We use that, and Raw Crash JSON and Correlation Signatures. 

We have https://bugzilla.mozilla.org/show_bug.cgi?id=959295 filed which might be of some interest to you. Once the unredacted JSON is available as a middleware endpoint I'm going to replace Processed JSON with that on the report index page (primary goal is to get the exploitability scoring stuff back). 

Invalid?

Comment 2

[(not currently active) Ted Mielczarek]

I think this is a terminology gap. :) That page is still using the pipe dump output:
https://github.com/mozilla/socorro/blob/master/webapp-django/crashstats/crashstats/views.py#L1018

I'd like it to use the JSON output from the stackwalker instead. I don't have a way to get at the raw processed JSON, so I can't point you at exactly what I mean unfortunately.

Comment 3

[Peter Bengtsson [:peterbe]]

Lars, can you help us bridge the above mentioned terminology gap between me and Ted? 
I'd love to work on this but not having been part of the big JSON/pipe/redacted/unredactored refactoring work I don't see this stuff clearly at all.

Comment 4

[K Lars Lohn [:lars] [:klohn]]

the processed_crash includes a key value pair 'dump': '<pipe dump>'.   the value is of the old format and exists solely for the use of the UI.  The data in the 'dump' is redundant: it exists elsewhere in the processed_crash in a better key:value form.  

For example: the method ```utils.parse_dump``` has to extract the 'os_name' from the pipe dump by parsing the awkward pipe dump format.  A more desirable method of getting the os_name is by accessing it directly from the processed_crash:  ```processed_crash['json_dump']['system_info']['os']```   

we want to move the UI to using the 'json_dump' instead of the 'dump' to get the values that populate the crash details page.

For a complete look of the layout of the processed crash, look at the last half of my blog posting:  http://www.twobraids.com/2013/12/socorro-support-classifiers.html

Comment 5

[(not currently active) Ted Mielczarek]

Aha, "json_dump"!

Comment 6

[Selena Deckelmann :selenamarie :selena]

Would love to push this forward in priority to ease disk utilization on production databases. We are currently storing this data twice - once in pipe-dump format and again in a more verbose JSON format. Fixing this will save us time and space!

Comment 7

[(not currently active) Ted Mielczarek]

I decided to try this after fixing bug xxx. Part 1 is up:
https://github.com/luser/socorro/commit/b947127f546517155bf5c59ef1778f49e9f21353

Part 1 changes utils.parse_dump's return value to match the stackwalker's JSON output, and changes the report/index template to match. Part 2 will use json_dump instead, which should be a fairly small patch on top of this.

Comment 8

[(not currently active) Ted Mielczarek]

... and by bug xxx I meant bug 1030276.

Comment 9

[(not currently active) Ted Mielczarek]

PR is up:
https://github.com/mozilla/socorro/pull/2192

Comment 10

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/7c4d4be61e8b7fd6a1c1a8c3c34f28547671d647
bug 976077 - Part 1: Change utils.parse_dump to return the same format as the stackwalker JSON output, and change the report/index template to match

https://github.com/mozilla/socorro/commit/c85d8ddf74c1ed1e815983fe47b46700972a7340
bug 976077 - Part 2: refactor bits of parse_dump out to be usable for enhancing the json_dump

https://github.com/mozilla/socorro/commit/c054623980415222ff8cad2729a624f35b2fa1f1
bug 976077 - Part 3: switch to using json_dump to build report/index