Closed Bug 497731 Opened 16 years ago Closed 11 years ago

Ability to query for exploitable crashes and display the rating in supersearch

Categories

(Socorro :: General, task)

Product:
Socorro
Component:
General
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jruderman, Assigned: adrian)

References

Details

(Keywords: sec-want, Whiteboard: [sg:want] admin-panel search )

I'd like a way to query for only exploitable crashes. This would let me explore the top few, most likely to be discovered by an adversary, or to go through many of them and see if I can reproduce any. This feature could be limited to people who are part of the core security group.
Whiteboard: [sg:want]
How do you define an exploitable crash?
Jesse has some combination of stack and memory information that can give a higher exploitablity likelihood to not ratio than random.
I don't think we have the ability currently to get any memory information from minidump_stackwalk and therefore socorro (it's certainly not in the DB currently).
http://www.squarefree.com/2006/11/02/determining-whether-a-crash-looks-exploitable/ If the top line of the stack is an unknown address, and we're not running JIT code (called by js_MonitorLoopEdge), the crash is very likely to be exploitable. Here's an example of such a crash: bp-1ece10e8-5c10-41c5-a1bb-596862090607 Most crash reporters also have an additional piece of information for segfaults, "address the code tried to access", which can identify other crashes as possibly exploitable. Are you saying Breakpad doesn't send that? :(
It does, and we appear to have it in the DB as "Crash Address". Do you want any crash report where that is non-zero?
I want all crash reports where: * The top of the stack (instruction pointer) is not a function, and js_MonitorLoopEdge is not the highest known function on the stack or * The crash address is not on page zero (indicating a null deref with an offset) or the guard page at the stack limit (indicating a too-much-recursion bug). If you don't know where the stack limit is, you can assume crash reports with more than ~500 frames are too-much-recursion crashes. Btw, I couldn't find any Mac crashes where the "crash address" was close to zero. What does that mean?
I think comment 6 is implementable, although since we removed frame info from the Socorro database (I think) it will require trolling through (and parsing) the raw dumps which is going to be time-consuming. I don't know about the mac thing: perhaps the mac exception handler doesn't have access to the crash address information in the same way.
The database contains the top ten frames of the crashing thread for each crash. After the implementation of the compound signature feature (Bug 411349), the UI for searching the frames was removed to reduce the load on the database. While Socorro is saving these stack frames, I don't believe anything is currently using them. This sounds to me like something that would best be implemented at Socorro crash processing time rather than on the fly. Reading over Comment #6, I think this would be relatively easy to implement at the same time that the compound signature is created. I'm imagining adding a boolean to the 'reports' table called something like 'potentiallyExploitable'. For back filling old data, if the 10 frames is insufficient, then the raw output from minidump_stackwalk can be retrieved and reanalyzed (with the caveat that they have already been edited/truncated with a 100 frame recursion depth limit). Of course, a 'potentiallyExploitable' flag in the database should not be shown to the general public. UI support for such a field may have to wait for our implementation of user authentication.
you don't want to show "potentially exploitable" just to avoid scaring people? The information isn't that hard to extract from the raw dump...
A bigger worry is a bad guy noticing. But the people looking at crash reports tend to be friendly users, and seeing "potentially exploitable" might be just what it takes to get the user to take the time to figure out steps to reproduce and report the bug.
On OS X and Linux, the "exception address" is always assigned the current PC: http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/google-breakpad/src/client/mac/handler/minidump_generator.cc#598 http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/google-breakpad/src/client/linux/handler/minidump_generator.cc#651 The MSDN docs for the minidump format define this field as "The address where the exception occurred", which is pretty vague: http://msdn.microsoft.com/en-us/library/ms680367%28VS.85%29.aspx It looks like OS X does make that data available for certain types of exceptions, but Breakpad isn't using it. I'll file a bug upstream on that: "EXC_BAD_ACCESS Could not access memory. subcode contains the bad memory address. " http://web.mit.edu/darwin/src/modules/xnu/osfmk/man/catch_exception_raise.html Also: (In reply to comment #10) > A bigger worry is a bad guy noticing. As bsmedberg said, they could already trawl Socorro and figure this out. Without the URLs, it's a lot harder to reproduce the crash (as we well know). If it's their own crash, well, they don't need Breakpad/Socorro to figure out if it's exploitable.
I've run my proof of concept script for the date range of '2009-06-01' to '2009-06-05'. total records: 724827 > The top of the stack (instruction pointer) is not a function, and > js_MonitorLoopEdge is not the highest known function on the stack 130575 records returned > The crash address is not on page zero (indicating a null deref with an > offset) or the guard page at the stack limit (indicating a too-much-recursion > bug). If you don't know where the stack limit is, you can assume crash > reports with more than ~500 frames are too-much-recursion crashes. 321182 records returned to see the actual returned results (150M file) see http://people.mozilla.com/~lars/exp/report.txt.gz
18% of crashes having bad-top-of-stack matches what I remember from querying Talkback years ago. But I had no idea 44% would have bad access addresses. Scary. Given these numbers, it might make sense to focus on the "bad top of stack" crashes, since those are the easiest to exploit. Can you make it possible to query for just those? 3087726f-4246-4893-bb63-679962090601 probably shouldn't show up (js_MonitorLoopEdge exclusion).
> Given these numbers, it might make sense to focus on the "bad top of stack" > crashes, since those are the easiest to exploit. Can you make it possible to > query for just those? yes, it would be possible to make that a query in the UI. It may need only an added index to the database and an appropriate UI. Now that we've worked through some feasibility, I will likely reformulate this bug as a UI enhancement request. However, until such time as that is implemented, you can periodically ask me to run my script and forward the results to you for the time periods that you need. > 3087726f-4246-4893-bb63-679962090601 probably shouldn't show up > (js_MonitorLoopEdge exclusion). yes, that should not have been part of the result set. The script didn't take into account the C++ parameters as part of the signature. I'll correct that.
So, I'd like to add a few more cases: --- _PR_MD_SEND - this is a class of issues relating to Rogue third party libraries (LSP) d5a8f9c3-6f9d-4277-97cc-5fd662090601 "SocketSend" cc3ec521-9fe2-42c4-9f00-0c7212090601 "_pr_init_ipv6" 6ba264bf-0405-4f15-bd0b-e268e2090601 "ws2_32.dll@0x38d9" anything with "lsp" in its call stack should go into that bucket, e.g.: 96140341-c2f4-4661-9af6-503952090601 "SGPrxy.dll@0x258e" NSPROVIDER/WSALookupServiceBeginW/WSALookupServiceBeginA also fall into this class, however I hadn't seen them before so it's good to know: ef674719-2829-4bdd-9fca-5d9d32090601 "NSPROVIDER::NSPLookupServiceBegin(_WSAQuerySetW*, _WSAServiceClassInfoW*, unsigned long, void**)" --- df0ff877-7b76-4c8f-af64-152482090601 "@0x0 | coFFPlgn.dll@0x2db6a" it doesn't look like your 0 page exclusion was applied --- @0x20202020 (and things near 0x2020yyyy) under JS are garbage collected pointers and should be flagged, they won't be exploitable (windows loads a non executable library near this address which is why it won't run code) but they're bad and should be highlighted. b39ef740-f789-4a6e-9668-dd7942090601 "@0x20202020 | regexp_exec_sub" --- Anything under a known plugin should go into its own bucket Flash Player, NPSWF32.dll, coFFPlgn.dll, NPMacX3dx, wmp.dll, npCortona, ... Flash_EnforceLocalSecurity (public symbol - not actually a library) --- UserCallWinProcCheckWow - things like this are generally unloaded third party modules (e.g. plugins) and deserve a bucket 4ec7c00b-4677-4878-b01b-f52a82090601 "UserCallWinProcCheckWow" --- user32.dll@* should trigger a get symbols cycle and a reprocess of the dump: 2efa7330-f02e-4949-9d67-c55cf2090601 "user32.dll@0x11911" --- Things with JNI or Java in them should go into a java bucket, in some ways it's like JavaScript's exclusion, you're running JIT'd code (Java instead of JavaScript) 22f40687-07f1-4791-a333-d8f512090601 "libclient.dylib@0x119289" e9116e5b-4b69-4b0d-b03f-026822090601 "jpinscp.dll@0x36a9" --- MOZ_gdk_display_close - this is some lame bug which should be identifiable by this function, and can get an exclusion for it 787b80c0-876e-40aa-8fb0-4ad662090601 "libgdk-x11-2.0.so.0.1000.13@0x35cbb" --- theme engines! 46248c34-d1d2-4ec9-8797-ed3dd2090601 "nsNativeThemeCocoa::DrawPushButton(CGContext*, CGRect const&, int, int, int)" 92f0fef9-69a1-4f69-9f8d-e17392090601 "uxtheme.dll@0x3547" special bucket --- printer bucket! 997742c4-4605-4e1c-89ee-a5d5c2090601 "CanonIJPDE@0x143f0" signature containing any printer vendor name :) --- btw, do we have bugs about WinampTBPlayer? :( --- shell extensions! 77d0989c-085e-45e2-873c-d26332090601 "CDefView::CreateViewWindow(...)" (yet another thing that should be run out of the gecko process so that when stupid things harm processes they don't harm us. CJones will save us!) --- non gecko threads: ca5513a2-05ed-4fb0-b5ac-9da562090601 "BaseThreadStart" ---
I don't understand most of what timeless is asking for, but I disagree with him about 0x20202020. GC hazards are exploitable, and it's only luck that sometimes you get the fill pattern instead of a newly allocated GC object.
Whiteboard: [sg:want] → [sg:want] admin-panel
Component: Socorro → General
Product: Webtools → Socorro
Jesse, if you still want this, it's now possible. Let me know.
Depends on: 794540
So yes, we're now running the "exploitable" tool during processing. This tool uses Breakpad's built-in exploitability analysis: http://code.google.com/p/google-breakpad/source/browse/trunk/src/processor/exploitability_win.cc#76 This only exists for Windows right now. If you're interested in tweaking the heuristics there or porting it to other platforms let me know, it should be pretty easy to upstream changes.
Note there is a (possibly dupe) shadow bug 819426 we can use for sensitive info.
(In reply to Laura Thomson :laura from comment #18) > Jesse, if you still want this, it's now possible. Let me know. I want this :) Jesse did you see the question in comment 18? (needinfoing you so it is on radar)
Flags: needinfo?(jruderman)
I want this, but mostly so you and the other critsmashers can use it :)
Flags: needinfo?(jruderman)
Assignee: nobody → adrian
Depends on: 678101, 901977
Whiteboard: [sg:want] admin-panel → [sg:want] admin-panel search
I'm going to morph this to what I think we want now: supersearch should be able to search based on the exploitability rating (for privileged users only) and display the rating in the search results.
Summary: Ability to query for only exploitable crashes → Ability to query for exploitable crashes and display the rating in supersearch
Commit pushed to master at https://github.com/mozilla/socorro https://github.com/mozilla/socorro/commit/03c3a1b8819cd7e5401819d177a252331366efc0 Fixes bug 497731 - Added exploitability to Super Search. r=peterbe
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Target Milestone: --- → 75
Commits pushed to master at https://github.com/mozilla/socorro https://github.com/mozilla/socorro/commit/1de9f5449c240cf68beb7a7f66e80b179ab08227 Fixes bug 497731 - Fixed list of possible values, removed all errors-related words. https://github.com/mozilla/socorro/commit/32c6f447b60ebce42029a480eb74959ae99847fb Merge pull request #1912 from AdrianGaudebert/497731-exploitability-super-search Fixes bug 497731 - Fixed list of possible values, removed all errors-related words.
Target Milestone: 75 → 76
You need to log in before you can comment on or make changes to this bug.