1061371 - UI to download memory file from crash reports

Status: VERIFIED FIXED

(Socorro :: General, task)

Description

[(Away)]

OOM crashes are sending a <uuid>.memory.json.gz as of bug 1007534. Socorro should offer a way to download them. (Presumably a link right next to the .dmp link?)

Bug 1026059 and possibly bug 1037512 may have helpful information.

Comment 1

[Benjamin Smedberg]

Can you include a sample crash id? I think this is exposed as part of the processed crash in the api.

Comment 2

[Robert Kaiser]

According to bug 1026059 comment #12 this is a reports that should have one: bp-a3b212d7-505f-44c7-8f62-0d41b2140831

Comment 3

[Robert Helmer [:rhelmer]]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #1)
> Can you include a sample crash id? I think this is exposed as part of the
> processed crash in the api.

I pointed to this in 1026059 comment 14 (which was good enough for the moment) but dmajor was asking in that bug if there was a way to grab the raw report too. It looks like we meant to make that available too via the API, but I can't get it to work (which would be a prereq to fixing this bug anyway)

I don't see any additional minidumps available for this particular crash, yet it has a memory report so there must be one right? I noticed that "additional_minidumps" is [] in https://crash-stats.mozilla.com/api/ProcessedCrash/?crash_id=a3b212d7-505f-44c7-8f62-0d41b2140831&datatype=processed

The "UI" could be as simple as just providing more links next to the json/dmp links under "Raw Dump" tab.

Comment 4

[(Away)]

The memory report is a .json.gz file so it's probably not counted as an additional minidump (.dmp file).

Comment 5

[Benjamin Smedberg]

Yeah, if you log in you have access to the data via https://crash-stats.mozilla.com/api/ProcessedCrash/?crash_id=a3b212d7-505f-44c7-8f62-0d41b2140831&datatype=processed (or if you use an access token tied to your account). I think that's sufficient for now; dmajor do you agree?

Comment 6

[(Away)]

For the very short term yes. I don't want to do it forever though! This bug is about making it nicer in the long term.

Comment 7

[(Away)]

Btw - I don't mind copy-pasting a UUID into that URL, that's fine. What is a pain to me is that it includes all the rest of the processed output, which I have to snip out by hand or with a script, before I can load the thing into the about:memory viewer.

Comment 8

[Benjamin Smedberg]

Oh you specifically care about loading it into a viewer? Could we just add a feature to the viewer to load directly from a crash UUID?

Comment 9

[Benjamin Smedberg]

To be clear, I was envisioning the ProcessedCrash API as the long-term solution for this.

Comment 10

[(Away)]

If somebody other than me writes the viewer code, I would accept that :)

I had assumed there would be a third link right on the Raw Dump tab, next to the .dmp and .json links -- that's what makes the most sense to me from a workflow perspective. But I guess I'd be ok with anything that lets me load reports easily.

Comment 11

[Benjamin Smedberg]

I don't mind that, but it's probably not coming this quarter (the similar request to make plugin-hang dumps downloadable has been around for a while). I think we should try to add this to the viewer code, since that should be relatively simple. Filed bug 1061960 on that.

Comment 12

[Nicholas Nethercote [inactive]]

I think the needinfo request in bug 1061960 subsumes this one.

Comment 13

[Ting-Yu Chou [:ting] (away)]

Bug 1061960 makes the viewer able to load the memory report from the file, but it's still annoying to replace the url, download and load. I'd like to make the memory report a link like what dmajor said in comment 10. It'd be even nicer if click the link will simply open an about:memory tab with the file loaded.

Comment 14

[Ting-Yu Chou [:ting] (away)]

(In reply to dmajor (away) from comment #10)
> I had assumed there would be a third link right on the Raw Dump tab, next to
> the .dmp and .json links -- that's what makes the most sense to me from a
> workflow perspective. But I guess I'd be ok with anything that lets me load

Can we add this link to Socorro?

Comment 15

[Paul Yang [:pyang] (away)]

Hi Peter,
This bug is about adding link for memory report in crash analysis page; detail in comment 5 and comment 10
Do you think there's any chance to add it?

Comment 16

[(not currently active) Ted Mielczarek]

Peter and I walked through some of this just now.

Here's a current crash report that contains a memory report: https://crash-stats.mozilla.com/report/index/92ac5591-c7ed-42b1-b65d-4a48d2160908

There's a 'ContainsMemoryReport' annotation when one of these is present. The file is submitted in a 'memory_report' form field. The collector stores all uploaded files in the raw crash storage.

This code generates the list of raw dumps for logged-in users with permission:
https://github.com/mozilla/socorro/blob/177a4d58ee5b728feabc2ef54c4e9922970a2d72/webapp-django/crashstats/crashstats/views.py#L1436

We'll need to add memory.json.gz there.

Then we also need to tweak the URL patterns for /rawdumps/ to allow memory.json.gz as an extension:
https://github.com/mozilla/socorro/blob/177a4d58ee5b728feabc2ef54c4e9922970a2d72/webapp-django/crashstats/crashstats/urls.py#L115

Comment 17

[Peter Bengtsson [:peterbe]]

Note-to-self; with the help of Ted, I think I've understood what needs to be done here. 
On the "Raw Dump" tab (of report index), there are 2 links. One to the raw crash (JSON) and the raw dump (binary). 
If the raw crash contains ContainsMemoryReport=1, we can expect to find a memory_report file in S3 by the same crash ID. If possible, make that third link available. It should "proxy" the file in S3 and set the Content-Disposition filename to be ":uuid.json.gz" so that it's easy to load into about:memory.

Comment 18

[Peter Bengtsson [:peterbe]]

There IS a way to get to the memory report MANUALLY. What this bug is about is me making this link available automatically (if possible) so you don't have to do these manual steps:

1) Find a crash that has ContainsMemoryReport=1 (e.g. https://crash-stats.mozilla.com/search/?contains_memory_report=%21__null__&product=Firefox&_sort=-date&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports)
2) Make sure you're signed in and have the View Personal Identifyable Information permission. 
3) Click the Raw Dumps tab and at the bottom should be two links. 
4) Copy the .json link, e.g. https://crash-stats.mozilla.com/rawdumps/24e56ff6-116c-49ae-94e3-486ca2160912.json
5) Insert the string "-memory_report" before the ".json", e.g. https://crash-stats.mozilla.com/rawdumps/24e56ff6-116c-49ae-94e3-486ca2160912-memory_report.json
6) Download it and gzip so it becomes .gz, e.g. 24e56ff6-116c-49ae-94e3-486ca2160912-memory_report.json.gz)
7) about:memory and Load the file you just gzipped.

Comment 19

[Peter Bengtsson [:peterbe]]

Gah! I take it back! The instructions above DON'T work. It just returns the raw JSON data. 
I fooled myself because I was testing this on a branch I was working on where it does work.

Comment 20

[Peter Bengtsson [:peterbe]]

Attachment: Link to Github pull-request: https://github.com/mozilla/socorro/pull/3468

Comment 21

[[github robot]]

Commit pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/6d0a79959cea23bcb57b044537e57ba8d2e0a66d
fixes bug 1061371 - UI to download memory file from crash reports (#3468)

* fixes bug 1061371 - UI to download memory file from crash reports

* check if memory_report exists more carefully

Comment 22

[Ting-Yu Chou [:ting] (away)]

Thanks for making this.

Comment 23

[Peter Bengtsson [:peterbe]]

I've verified this on stage and I can download it and load it into `about:memory`
http://jmp.sh/GUoUtNP

I think this is going to go live on Wednesday this week.

Comment 24

[Nicholas Nethercote [inactive]]

I verified it too. Thank you!