Crash in [@ google_breakpad::ExceptionHandler::ExceptionHandlerThreadMain]
Categories
(Toolkit :: Crash Reporting, defect)
Tracking
()
Tracking | Status | |
---|---|---|
firefox-esr91 | --- | unaffected |
firefox99 | --- | unaffected |
firefox100 | --- | fixed |
firefox101 | --- | fixed |
People
(Reporter: aryx, Assigned: gsvelto)
Details
(Keywords: crash)
Crash Data
Attachments
(1 file)
48 bytes,
text/x-phabricator-request
|
diannaS
:
approval-mozilla-beta+
|
Details | Review |
20-35 crashes per Beta version on Windows 10+11. Crash is new in Firefox 100 branch with isolated crashes before. First build ID which got a crash report is 100.0a1 20220320065911. 66% of crashes with 32-bit builds.
Crash report: https://crash-stats.mozilla.org/report/index/fa8cc985-0327-447d-9f06-5b8630220413
Reason: EXCEPTION_ACCESS_VIOLATION_READ
Top 8 frames of crashing thread:
0 None @0x70efaf3b
1 xul.dll google_breakpad::ExceptionHandler::WriteMinidumpWithExceptionForProcess toolkit/crashreporter/breakpad-client/windows/handler/exception_handler.cc:1033
2 xul.dll google_breakpad::ExceptionHandler::WriteMinidumpWithException toolkit/crashreporter/breakpad-client/windows/handler/exception_handler.cc:934
3 xul.dll static google_breakpad::ExceptionHandler::ExceptionHandlerThreadMain toolkit/crashreporter/breakpad-client/windows/handler/exception_handler.cc:428
4 kernel32.dll BaseThreadInitThunk
5 mozglue.dll patched_BaseThreadInitThunk toolkit/xre/dllservices/mozglue/WindowsDllBlocklist.cpp:572
6 ntdll.dll __RtlUserThreadStart
7 ntdll.dll _RtlUserThreadStart
Assignee | ||
Comment 1•2 years ago
|
||
A very quick survey of the crashes shows that we crashed within what would be a call to MiniDumpWriteDump()
, however the dbgcore.dll
library we read the function from appears to have been unloaded which is rather odd.
Assignee | ||
Comment 2•2 years ago
|
||
I got the previous comment slightly wrong, MiniDumpWriteDump()
is in dbghelp.dll
and that's not unloaded, however it's calling into dbgcore.dll
and here's the odd thing: it appears that the latter has been unloaded then re-loaded at a different address. dbghelp.dll
appears to be calling a function that's inside the old mapping. Here's an example from b9ac95c2-8195-4f07-9c0c-24cfd0220410:
dbgcore.dll
is mapped between 0x00007fffba080000 and 0x00007fffba0ac000- it also appear in the unloaded modules at offset 0x0000000000006a80
- the crash reason is
EXCEPTION_ACCESS_VIOLATION_EXEC
at address 0x00007fffba086a80, see how this aligns with the offset of the old mapping ofdbgcore.dll
Assignee | ||
Comment 3•2 years ago
|
||
Aha, this appears to be a shutdown race of some kind. See a137814b-e8bf-4e27-8122-021d10220408:
- The main thread is unsetting the exception handler here
- While the exception handler thread is trying to write a minidump here
- But the main thread has already called
FreeLibrary()
ondbghelp.dll
here
This crash is probably hiding a different crash that's happening during shutdown. Good thing we have the WER interceptor and it could catch this, otherwise it would have been a silent shutdown crash.
Assignee | ||
Updated•2 years ago
|
Assignee | ||
Comment 4•2 years ago
|
||
Assignee | ||
Comment 5•2 years ago
|
||
I just landed this on Phabricator. Looking at recent crashes in beta what's triggering this is not really a shutdown crash but rather a shutdown hang. The shutdown hang thread is trying to write a minidump before killing the main process. This happened after we've unloaded the libraries as I described in the previous comment and that's what was causing the race.
Pushed by gsvelto@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/12974735c4ac Unload dbghelp.dll only when we're sure we won't call into anymore r=rkraesig
Comment 7•2 years ago
|
||
bugherder |
Assignee | ||
Comment 8•2 years ago
|
||
Asking for uplift because this is happening a lot on beta and I don't want it to blow up on release
Assignee | ||
Comment 9•2 years ago
|
||
Comment on attachment 9272439 [details]
Bug 1764608 - Unload dbghelp.dll only when we're sure we won't call into anymore r=rkraesig
Beta/Release Uplift Approval Request
- User impact if declined: Firefox crashes on shutdown
- Is this code covered by automated tests?: No
- Has the fix been verified in Nightly?: Yes
- Needs manual test from QE?: No
- If yes, steps to reproduce:
- List of other uplifts needed: None
- Risk to taking this patch: Low
- Why is the change risky/not risky? (and alternatives if risky): The patch moves a few lines of code that are invoked only during shutdown to prevent a race
- String changes made/needed: none
Comment 10•2 years ago
|
||
Comment on attachment 9272439 [details]
Bug 1764608 - Unload dbghelp.dll only when we're sure we won't call into anymore r=rkraesig
Approved fro 100.0b8
Comment 11•2 years ago
|
||
bugherder uplift |
Updated•2 years ago
|
Description
•