Closed Bug 1409427 Opened 8 years ago Closed 7 years ago

calculate sha256 hash of upload_file_minidump

Categories

(Socorro :: Antenna, task, P1)

Product:
Socorro
Component:
Antenna
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: willkg, Assigned: willkg)

References

Details

Attachments

(2 files)

pr 261: fix bug 1409427 - switch to sha256 and add MinidumpSha256 checksum
52 bytes, text/x-github-pull-request
Details | Review
pr 4416: fix bug 1275374, 1409427 - add minidump_sha256_hash to telemetry crash data
53 bytes, text/x-github-pull-request
Details | Review
Bug #1408041 covers adding another crash id to the crash that we need to compute in Socorro that we can use to correlate crashes to other systems. This bug covers implementing a processor rule to calculate the sha256 hash of the upload_file_minidump file and add that to the processed crash data.
We should ignore crashes that don't have an upload_file_minidump (I'm pretty sure they exist, but maybe they don't). We need to figure out what to do with crashes that have a 0-byte upload_file_minidump. Maybe ignore them? Should we also ignore dumps that can't be parsed by minidump stackwalker? We need to figure out what key to use in the processed crash for the sha256 hash. Making this a P1 because having this value available in Elasticsearch (and searchable) and Telemetry enables helpful things that could be useful soon.
Component: General → Processor
Priority: -- → P1
I talked to Lonnen about this. My current understanding is that this is nice, but other pieces need to be done before this will help. Given that, I'm pushing this off by demoting it to P2..
Priority: P1 → P2
We should work on this this quarter. Based on conversations, here's my new brain dump: 1. Crashes that don't have an upload_file_minidump or a zero-byte upload_file_minidump will not get a sha256 hash. We'll calculate the sha256 hash for the upload_file_minidump for everything else. 2. We'll put the hash in the processed crash using some variation of the name that Telemetry uses: minidumpSha256Hash 3. This data needs to get to ES (for searchability), S3 (as part of the processed crash), and S3 (as part of the processed crash we send to Telemetry). Thus, it needs to be set up in socorro/external/es/super_search_fields.py. 4. After we finish this bug, we should create a bug to figure out how to surface this id in such a way that we can use it as a variant of the crash id. For the name, we could go with minidumpSha256Hash which is what Telemetry uses or minidump_sha256_hash which conforms to the scheme we use in Socorro processed crash keys.
Assignee: nobody → willkg
Priority: P2 → P1
Unassigning myself from bugs I'm not immediately working on and/or have some meaningful progress on.
Assignee: willkg → nobody
Lonnen and I talked about this a bit. We're going to break this up into a couple of parts. First part: 1. Add some code to Antenna (Socorro collector) to calculate the sha256 hash of non-empty upload_file_minidump files to the raw crash just like it does with the md5 checksum. 2. Change the Telemetry crash storage code to save this field, too. Second part (optional and needs more thought and likely to get bumped to a new bug): 1. Make the sha256 hash searchable via Super Search. 2. Surface the sha256 hash as an alternate to crash id so /report/index/<sha256-hash> also works or we give it a new endpoint or something.
Component: Processor → Antenna
Oops--meant to take this before I did it.
Assignee: nobody → willkg
Status: NEW → ASSIGNED
The Antenna part landed in https://github.com/mozilla-services/antenna/commit/862194cf4c5cba2fe486a150f2672e3916dcf3eb I still need to fix the Telemetry crash storage and the schema in Socorro.
Antenna got deployed to stage, but the MinidumpSha256 fields are empty. I need to fix that. I'll do it on Monday.
I messed up--my faux stage submitter had a bug where it wasn't renaming dump to upload_file_minidump. I fixed my faux stage submitter and re-tested and Antenna is fine. Yay!
We pushed Antenna to prod and it's generating MinidumpSha256Hash data. Example crash: https://crash-stats.mozilla.com/report/index/437bbcbb-92a1-4cbb-b9a1-a15870180424
Commits pushed to master at https://github.com/mozilla-services/socorro https://github.com/mozilla-services/socorro/commit/80fbcf8cfa5941200bea58f1e21d9bfd0b3131eb fix bug 1409427 - add minidump_sha256_hash to telemetry crash data https://github.com/mozilla-services/socorro/commit/3be01ae594a64d0291cf893101316339f611427a Merge pull request #4416 from willkg/1409427-sha256-processor fix bug 1275374, 1409427 - add minidump_sha256_hash to telemetry crash data
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: