Closed Bug 794540 Opened 12 years ago Closed 12 years ago

run exploitable tool during processing

Categories

(Socorro :: Backend, task)

Product:
Socorro
Component:
Backend
Platform:
x86_64
Linux
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: ted, Assigned: lars)

References

(
URL
)

Details

(Whiteboard: [qa+])

I have this exploitable tool (source in the URL) that reads a minidump file and spits out an exploitability analysis. At some point in the future (when we switch to JSON stackwalk) we can just get this data as part of normal processing. As a stopgap solution we can simply run this standalone tool on each dump after running minidump_stackwalk on it, which will produce a single piece of data: a rating of how exploitable this crash might be. The output looks like: exploitability: <value> or ERROR: <reason> Where <value> is one of: none, interesting, low, medium, high. This bug covers executing this binary as part of processing and storing the result in the processed json.
Depends on: 794544
Blocks: 794548
Blocks: 513427
the question is where to store this data. It can go in the database, but I'm not sure if it can go in the processed crash. We already bar sensitive info from the processed crash (url, email, ...). I suppose we'll have to add it to the raw crash json, though that seems wrong to put processed info into something called 'raw'.
Depends on: 807349
What do we do with the email, url, etc, at present?
the aforementioned columns are native to the raw crash json and remain there untouched. They are inserted into the 'reports' table in the database. Some values from the raw crash are copied into the processed crash json. Those columns are filtered out of the copy. A raw crash contains 'sensitive' information. A processed crash has been cleaned of 'sensitive' information.
Target Milestone: --- → 28
pending questions blocking implementation: 1) If the exploitablity tool returns the "ERROR: ..." message, what 'exploitablity' value should be associated with the crash 2) should the 'exploitabilty' value be saved in either the raw crash or processed crash? There are arguments against both places: * the processor has never written to the raw crash before. This will involve a new HBase interaction and the associated overhead (both in time an storage space). Other values that were changed (anything changed by the rewrite rule system, like 'version') will get committed to HBase, too. * 'sensitive' data is banned from the processed crash. 'exploitability' is sensitive, saving in the processed crash is likely out. In the future implementation of the json MDSW, where would this value be stored? possibilities: a) store in database only b) store in raw crash c) store in processed crash d) store in some new third location
(In reply to K Lars Lohn [:lars] [:klohn] from comment #4) > pending questions blocking implementation: > > 1) If the exploitablity tool returns the "ERROR: ..." message, what > 'exploitablity' value should be associated with the crash I think I mentioned this on IRC, but we should simply make the column NULL in this case and put the error reason into the processor_notes field.
as for my second question, perhaps we ought to make a 'sensitive' branch in the processed crash json. Rather than having a blanket ban on sensitive data in the processed crash, we segregate it so that it can be easily pruned. -- uuid: -- date_processed -- ... -- sensitive -- url -- email -- exploitablity -- signature -- ... when we serve the processed crash from the middleware, we prune the 'sensitive' branch unless the user is authenticated. If we want to go this direction, we ought to synchronize with the future jsonMDSW so we're not changing the format of the processed crash twice.
I would vote for a). You could also store it in HBase in a new column, if you wanted to.
We've decided to put it into the 'reports' table in the database. It cannot go in the processed crash due to it being "sensitive" data. Any other storage locations will depend on what the use of this data is to be. We need guidance as to any desired reports that may include this information.
Commits pushed to master at https://github.com/mozilla/socorro https://github.com/mozilla/socorro/commit/5ad3dd639a7eee34af05b8ce27020b9e3688f6bb Bug 794540 - added support for exploitability tool to original processor https://github.com/mozilla/socorro/commit/96bc91048b24ac4c3fa9dc11dab611dedf271771 Merge pull request #956 from twobraids/exploit2012 fixes Bug 794540 - added support for exploitability tool to original processor
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
attaching real crashes for QA testing - had to change bug to confidential...
Group: mozilla-corporation-confidential
Ugh, can we not use fake data?
theses six crashes are for QA testing of the exploitability feature. Each one will have a different outcome in postgres high.* -> exploitability: 'high'; processornotes: '' low.* -> exploitability: 'low'; processornotes: '' medium.* -> exploitability: 'medium': processornotes: '' none.* -> exploitability: 'none': processornotes: '' unable.* -> exploitability: NULL; processornotes: '/data/socorro/stackwalk/bin/exploitable: ERROR: unable to analyze dump' fail.* -> exploitablity: NULL; processornotes: '/data/socorro/stackwalk/bin/stackwalk.sh returned no header lines for reportid: 462141561; No thread was identified as the cause of the crash; CSignatureTool: No signature could be created because we do not know which thread crashed; No frame data available; /data/socorro/stackwalk/bin/stackwalk.sh failed with return code 1; /data/socorro/stackwalk/bin/exploitable: ERROR: could not process minidump; /data/socorro/stackwalk/bin/exploitable failed with return code 1'
Blocks: 815309
Whiteboard: [qa+]
Looks good! -[ RECORD 1 ]---+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- uuid | 130894ac-8f68-4967-a9d9-399022121128 exploitability | processor_notes | /data/socorro/stackwalk/bin/exploitable: ERROR: unable to analyze dump -[ RECORD 2 ]---+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- uuid | d56653f7-4b70-4c8a-b7e5-42ade2121128 exploitability | high processor_notes | -[ RECORD 3 ]---+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- uuid | 1d6795e1-0c54-4811-9024-7a43e2121128 exploitability | low processor_notes | -[ RECORD 4 ]---+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- uuid | 933398ab-78f0-4558-ac62-2aa712121128 exploitability | none processor_notes | -[ RECORD 5 ]---+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- uuid | 316ce374-e394-44df-aa90-2026c2121128 exploitability | medium processor_notes | -[ RECORD 6 ]---+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- uuid | 944c96b5-6c75-4a30-bb01-486a42121128 exploitability | processor_notes | /data/socorro/stackwalk/bin/stackwalk.sh returned no header lines for reportid: 465258553; No thread was identified as the cause of the crash; CSignatureTool: No signature could be created because we do not know which thread crashed; No frame data available; /data/socorro/stackwalk/bin/stackwalk.sh failed with return code 1; /data/socorro/stackwalk/bin/exploitable: ERROR: could not process minidump; /data/socorro/stackwalk/bin/exploitable failed with return code 1
Bumping to verified - thank you rhelmer and selena! QA verified per comment 15, comment 16, and comment 17
Status: RESOLVED → VERIFIED
Group: mozilla-corporation-confidential
Blocks: 497731
No longer blocks: 513427
Blocks: 762216
Ted I haven't been noticing exploitability ratings on crash dumps lately. I do see this weird thing in the processor section: "Processor Notes sp-processor05.phx1.mozilla.com_14525:2012; exploitability tool failed: 127" e.g. bp-86ebf382-e64a-4e82-a52a-ba7ce2130423
Flags: needinfo?(ted)
dbolter: I wonder if we regressed this with processor2012. Maybe Lars can take a look.
there is an error in the Processor2012 configuration. what is says: exploitability_tool_pathname='/data/socorro/stackwalk/bin/exploitability' what it should say: exploitability_tool_pathname='/data/socorro/stackwalk/bin/exploitable' filing an bug to get this corrected.
THANKS! You guys rock.
Flags: needinfo?(ted)
@davidb: do we need to reach back into the past and reprocess crashes? I understand that the tool only works on windows, but it would still be an unprecedentedly huge reprocessing job to go back that far. We can go back as far as you'd like, but the further back we go the more effort it takes.
Well we used to live okay without it so I'd think we can live without going back too far (or going backwards at all). Thanks for asking :)
the exploitable tool is now running again in production. example: https://crash-stats.mozilla.com/report/index/d68f80e4-e1fd-497a-8acb-44d182130424
You need to log in before you can comment on or make changes to this bug.