Closed Bug 1764608 Opened 2 years ago Closed 2 years ago

Crash in [@ google_breakpad::ExceptionHandler::ExceptionHandlerThreadMain]

Categories

(Toolkit :: Crash Reporting, defect)

x86
Windows 11
defect

Tracking

()

RESOLVED FIXED
101 Branch
Tracking Status
firefox-esr91 --- unaffected
firefox99 --- unaffected
firefox100 --- fixed
firefox101 --- fixed

People

(Reporter: aryx, Assigned: gsvelto)

Details

(Keywords: crash)

Crash Data

Attachments

(1 file)

20-35 crashes per Beta version on Windows 10+11. Crash is new in Firefox 100 branch with isolated crashes before. First build ID which got a crash report is 100.0a1 20220320065911. 66% of crashes with 32-bit builds.

Crash report: https://crash-stats.mozilla.org/report/index/fa8cc985-0327-447d-9f06-5b8630220413

Reason: EXCEPTION_ACCESS_VIOLATION_READ

Top 8 frames of crashing thread:

0 None @0x70efaf3b 
1 xul.dll google_breakpad::ExceptionHandler::WriteMinidumpWithExceptionForProcess toolkit/crashreporter/breakpad-client/windows/handler/exception_handler.cc:1033
2 xul.dll google_breakpad::ExceptionHandler::WriteMinidumpWithException toolkit/crashreporter/breakpad-client/windows/handler/exception_handler.cc:934
3 xul.dll static google_breakpad::ExceptionHandler::ExceptionHandlerThreadMain toolkit/crashreporter/breakpad-client/windows/handler/exception_handler.cc:428
4 kernel32.dll BaseThreadInitThunk 
5 mozglue.dll patched_BaseThreadInitThunk toolkit/xre/dllservices/mozglue/WindowsDllBlocklist.cpp:572
6 ntdll.dll __RtlUserThreadStart 
7 ntdll.dll _RtlUserThreadStart 

A very quick survey of the crashes shows that we crashed within what would be a call to MiniDumpWriteDump(), however the dbgcore.dll library we read the function from appears to have been unloaded which is rather odd.

I got the previous comment slightly wrong, MiniDumpWriteDump() is in dbghelp.dll and that's not unloaded, however it's calling into dbgcore.dll and here's the odd thing: it appears that the latter has been unloaded then re-loaded at a different address. dbghelp.dll appears to be calling a function that's inside the old mapping. Here's an example from b9ac95c2-8195-4f07-9c0c-24cfd0220410:

  • dbgcore.dll is mapped between 0x00007fffba080000 and 0x00007fffba0ac000
  • it also appear in the unloaded modules at offset 0x0000000000006a80
  • the crash reason is EXCEPTION_ACCESS_VIOLATION_EXEC at address 0x00007fffba086a80, see how this aligns with the offset of the old mapping of dbgcore.dll

Aha, this appears to be a shutdown race of some kind. See a137814b-e8bf-4e27-8122-021d10220408:

  • The main thread is unsetting the exception handler here
  • While the exception handler thread is trying to write a minidump here
  • But the main thread has already called FreeLibrary() on dbghelp.dll here

This crash is probably hiding a different crash that's happening during shutdown. Good thing we have the WER interceptor and it could catch this, otherwise it would have been a silent shutdown crash.

Assignee: nobody → gsvelto
Status: NEW → ASSIGNED

I just landed this on Phabricator. Looking at recent crashes in beta what's triggering this is not really a shutdown crash but rather a shutdown hang. The shutdown hang thread is trying to write a minidump before killing the main process. This happened after we've unloaded the libraries as I described in the previous comment and that's what was causing the race.

Pushed by gsvelto@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/12974735c4ac
Unload dbghelp.dll only when we're sure we won't call into anymore r=rkraesig
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → 101 Branch

Asking for uplift because this is happening a lot on beta and I don't want it to blow up on release

Comment on attachment 9272439 [details]
Bug 1764608 - Unload dbghelp.dll only when we're sure we won't call into anymore r=rkraesig

Beta/Release Uplift Approval Request

  • User impact if declined: Firefox crashes on shutdown
  • Is this code covered by automated tests?: No
  • Has the fix been verified in Nightly?: Yes
  • Needs manual test from QE?: No
  • If yes, steps to reproduce:
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): The patch moves a few lines of code that are invoked only during shutdown to prevent a race
  • String changes made/needed: none
Attachment #9272439 - Flags: approval-mozilla-beta?

Comment on attachment 9272439 [details]
Bug 1764608 - Unload dbghelp.dll only when we're sure we won't call into anymore r=rkraesig

Approved fro 100.0b8

Attachment #9272439 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: