1026059 - Collect and store about:memory data for OOM crashes

Status: RESOLVED FIXED

(Socorro :: Backend, task)

Description

[Benjamin Smedberg]

The Firefox client will soon start sending about:memory data with crashes that appear to be OOM-related. This will require some changes in the collector and processor.

We were considering sending the about:memory data in the minidump itself, but that is going to be difficult. So instead we are going to send it as a separate file upload field when submitting. The format will be gzipped .json.

For now, we should treat the memory dump information as sensitive: we're going to be anonymizing it but we haven't really tested the anonymization.

It is likely that in the future we will want to use the memory data in classifiers and signature-generation, but we don't know yet because we'll need to explore the data first.

I don't know about storage. It might be expedient to somehow include the OOM data as part of the jDump, but it could also be its own field in the processed JSON. I'm hoping we can get away with not having new database fields for this data yet.

I'll file a separate bug for some basic supersearch enhancements.

Comment 1

[Benjamin Smedberg]

Lars requested size estimats for the about:memory reports. The size varies dramatically by the addons and number of tabs loaded, but the .json.gz reports on my systems are currently 83k, 186k, and 75k.

Comment 2

[K Lars Lohn [:lars] [:klohn]]

support for this feature will require no changes to the collector.  It already has the ability to save an arbitrary named file upload.  As it is a binary file, it could be treated just like the binary breakpad dumps until it gets to the processor. 

The processor will have to have a minor change.  When it iterates through the dumps and encounters the gzipped json binary blob, the processor could unzip and expand the data attaching it somewhere to the processed crash.  Since it is sensitive, the field in the processed crash would have to be added to the Redactor list of sensitive fields. 

All said and done, I'd imagine that this change, at least for the backend code (collectors, processors, crash storage), will require no more than an a dozen lines of new code.

Comment 3

[Benjamin Smedberg]

Excellent, thanks for the pointers lars! https://github.com/bsmedberg/socorro/compare/memoryinfo?expand=1 is my COMPLETELY UNTESTED work to fix this.

https://crash-stats.mozilla.com/report/index/6779a0ef-fff2-4f90-9655-c68932140703 was a crash I submitted with the new data: I don't know whether it can be turned back into a submission for testing purposes or not.

Comment 4

[Robert Helmer [:rhelmer]]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #3)
> Excellent, thanks for the pointers lars!
> https://github.com/bsmedberg/socorro/compare/memoryinfo?expand=1 is my
> COMPLETELY UNTESTED work to fix this.
> 
> https://crash-stats.mozilla.com/report/index/6779a0ef-fff2-4f90-9655-
> c68932140703 was a crash I submitted with the new data: I don't know whether
> it can be turned back into a submission for testing purposes or not.

Anyone with raw dump access should be able to take the .dmp and .json files from:
https://crash-stats.mozilla.com/report/index/6779a0ef-fff2-4f90-9655-c68932140703#rawdump

submitter_app.py can take these and re-submit them (may need to rename .dmp to .dump):
http://socorro.readthedocs.org/en/latest/installation/systemtest.html

Comment 5

[Benjamin Smedberg]

rhelmer, the memory report file doesn't show up at the link you mentioned, only the minidump.

Perhaps to confuse the issue, but it's like https://crash-stats.mozilla.com/report/index/c0416855-49cc-4c23-ade6-a78182140630#rawdump only lets you download the main minidump, not the other three hang dumps.

Comment 6

[Robert Helmer [:rhelmer]]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #5)
> rhelmer, the memory report file doesn't show up at the link you mentioned,
> only the minidump.
> 
> Perhaps to confuse the issue, but it's like
> https://crash-stats.mozilla.com/report/index/c0416855-49cc-4c23-ade6-
> a78182140630#rawdump only lets you download the main minidump, not the other
> three hang dumps.

Ah now I understand, looking at your patch helps.

So it looks like the middleware only knows how to fetch a single "raw" (upload_file_minidump) and "meta" (rest of the POST fields as JSON).

In the meantime we could pull these straight out of hbase, I don't see any way to get this out of the UI at the moment.

Comment 7

[K Lars Lohn [:lars] [:klohn]]

yes, we can get the crashes directly from hbase using the hbase client app:

   $ python socorro/external/hb/hbase_client.py --command get_raw_dumps <crash_id>

it will write all the dumps to disk under the names <crash_id>.<dump_name>.dump

it won't reconstitute the original .tar.gz extension for the OOM dump, you'll have to manually rename it.

It looks like the middleware does not have the ability to fetch anything but the default dump.

Comment 8

[Benjamin Smedberg]

Filed bug 1037512 on the ability to get all raw dumps from the middleware/API.

Comment 9

[Benjamin Smedberg]

Lars, are you able to take my prototype branch and finish this?

Comment 10

[K Lars Lohn [:lars] [:klohn]]

yeah, :bsmedberg, I can take it over, however, I'm on PTO beginning Monday Aug 11 - Aug 19. I'll give it a good look today to see if I can make short work of it...

Comment 11

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/50fbfd4de7225ec765ceec61e65d3942098bd286
Bug 1026059 - differentiate breadpad minidumps from memory-info dumps

https://github.com/mozilla/socorro/commit/7f1545d2bee5a52d016bfb4a6b4631d58631ef74
Merge pull request #2277 from twobraids/thanks-for-the-memories

Fixes Bug 1026059 - differentiate breakpad minidumps from memory_info dumps

Comment 12

[(Away)]

We've got our first memory report! bp-a3b212d7-505f-44c7-8f62-0d41b2140831

How can I download the report? (Also, I could have sworn that bsmedberg's demo crash had "ContainsMemoryReport" as a field under the Details tab. I don't see it there anymore, only in the Metadata tab.)

Comment 13

[Robert Helmer [:rhelmer]]

(In reply to David Major [:dmajor] from comment #12)
> We've got our first memory report! bp-a3b212d7-505f-44c7-8f62-0d41b2140831
> 
> How can I download the report? (Also, I could have sworn that bsmedberg's
> demo crash had "ContainsMemoryReport" as a field under the Details tab. I
> don't see it there anymore, only in the Metadata tab.)

I don't think there's any UI yet for downloading these (the issues in comment 6 are still open), we can fetch them manually for you in the meantime if that's helpful (note to self/Socorro team - comment 7 has the info on this).

Lonnen, is this feature in the works already? I don't see a bug for it.

Comment 14

[Robert Helmer [:rhelmer]]

(In reply to David Major [:dmajor] from comment #12)
> We've got our first memory report! bp-a3b212d7-505f-44c7-8f62-0d41b2140831
> 
> How can I download the report? (Also, I could have sworn that bsmedberg's
> demo crash had "ContainsMemoryReport" as a field under the Details tab. I
> don't see it there anymore, only in the Metadata tab.)

Hmm so actually looking at the processed JSON via the API just now (need to be logged in for this):
https://crash-stats.mozilla.com/api/ProcessedCrash/?crash_id=a3b212d7-505f-44c7-8f62-0d41b2140831&datatype=processed

Search for "memory_report" - is that what you are looking for?

I was expecting this to show up as an additional dump file based on commits and comments in this bug, but I don't see any "additional_minidumps" in the processed JSON. So, either I'm not sure how it's getting in there or "additional minidumps" isn't listing it as it should.

There might be a way to get at the original raw report via the API too once we get the above figured out.

Comment 15

[(Away)]

> Search for "memory_report" - is that what you are looking for?
Yes, it is, thanks! I can use this as a stopgap for now.

Is there a way to supersearch on "ContainsMemoryReport=1"? I can open a bug if not.

Comment 16

[Robert Helmer [:rhelmer]]

(In reply to David Major [:dmajor] from comment #15)
> > Search for "memory_report" - is that what you are looking for?
> Yes, it is, thanks! I can use this as a stopgap for now.
> 
> Is there a way to supersearch on "ContainsMemoryReport=1"? I can open a bug
> if not.

I don't see "ContainsMemoryReport" in the code right now or any bugs about this, please do open a bug for this!

Comment 17

[Lonnen :lonnen]

I'm not aware of any UI bugs for this. If they are desired, we need to file some.