981008 - [tracker] Deprecate pDump

Status: RESOLVED FIXED

(Socorro :: Backend, task)

Description

[Selena Deckelmann :selenamarie :selena]

We are currently storing both the pipe-delimited pDump and the fancy, friendly jsonDump in HBase, resulting in a 2-3x increase in storage size for this data.

We'd like to eliminate the dependency on pDump throughout the code so that we can save a bunch of disk space.

Comment 1

[K Lars Lohn [:lars] [:klohn]]

bsmedberg: i'm in the process of "acceptance testing" the rule based Processor2015.  I'm noticing some differences in the processed crash output.  However, I thinking that it is the current processor that is out of spec, not the new one.

the current "HybridProcessor" is applying post processing (like signature generation) to all of the breakpad dumps, not just the primary one.  HybridProcessor is saving the extra data in the processed crash buried in the associated dump's section.

1) processed_crash.signature   # the main signature used everywhere

2) processed_crash.upload_file_minidump_browser.signature  # a signature generated from the "browser" breakpad dump
3) processed_crash.upload_file_minidump_flash1.signature  # a signature generated from the "flash1" breakpad dump
4) processed_crash.upload_file_minidump_flash2.signature  # a signature generated from the "flash2" breakpad dump

Am I correct is saying that 2-4 are a waste of time and the new processor is correct by not doing it?

Comment 2

[Benjamin Smedberg]

It is necessary to process all the incoming dumps to the point where we have the jdump data from it. So e.g. for crash https://crash-stats.mozilla.com/report/index/41c5a703-2757-4d19-8d71-71fd02141006

In the ProcessedCrash API, it's important to have data["upload_file_minidump_flash2"]["json_dump"]. Whether you calculate the *signature* for those dumps isn't particularly important, although if it's easy to do it might be helpful.

Comment 3

[Lonnen :lonnen]

done