1420251 - Firefox crashes at startup with Lenovo OneKey Theater on 64-bit Windows 7

Status: NEW

(External Software Affecting Firefox :: Other, defect, P2)

Description

[Marco Castelluccio [:marco]]

Apparently we traded one crash for another, see bug 1369361 comment 62:

(In reply to Chris Peterson [:cpeterson] from bug 1369361 comment 62)
> Ted, do you know why Breakpad is not able to catch these "unknown software
> exception 0x80000003" crashes in firefox.exe? Exception code 0x80000003
> implies corrupted system files or buggy anti-virus hooks.
> 
> We used to catch these Lenovo users' crash reports when SQLite called
> Windows' rand_s() API. Removing the call to rand_s() turned these crashes
> into exception code 0x80000003 instead of just crashing in other Firefox
> code.
> 
> There is a screenshot of the exception dialog in attachment 8929992 [details]
> [details].

Comment 1

[Chris Peterson [:cpeterson]]

Ted or David, do you know why Breakpad is not able to catch these "unknown software exception 0x80000003" crashes in firefox.exe? IIUC, exception code 0x80000003 implies corrupted system files or buggy anti-virus hooks.

We used to catch these Lenovo users' crash reports when SQLite called Windows' rand_s() API and crashed in some bad software hooks for advapi32.dll. Removing the call to rand_s() turned these crashes into exception code 0x80000003 instead of crashing later in some other Firefox code.

There is a screenshot of the exception dialog in attachment 8929992 [details] (from bug 1369361).

Comment 2

[(not currently active) Ted Mielczarek]

That's EXCEPTION_BREAKPOINT, FYI:
https://chromium.googlesource.com/breakpad/breakpad/+/master/src/google_breakpad/common/minidump_exception_win32.h#57

Breakpad doesn't handle these by default, although apparently there's a way to enable it in the ExceptionHandler class:
https://dxr.mozilla.org/mozilla-central/rev/ba283baf4e98aa3a5f45a17981077b98697aa73a/toolkit/crashreporter/breakpad-client/windows/handler/exception_handler.cc#468

Comment 3

[(Away)]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #2)
> Breakpad doesn't handle these by default, although apparently there's a way
> to enable it in the ExceptionHandler class:

We definitely enable it; EXCEPTION_BREAKPOINT is used by MOZ_CRASH among other things:
https://dxr.mozilla.org/mozilla-central/search?q=set_handle_debug_exceptions
https://crash-stats.mozilla.com/search/?reason=%3DEXCEPTION_BREAKPOINT

EXCEPTION_BREAKPOINT is generic enough that I don't think we should make guesses about what's causing it. It would require a case-by-case investigation with a debugger.

Comment 4

[(not currently active) Ted Mielczarek]

Oh, you are totally correct. Sorry! It's totally possible that this is simply happening very early in startup, by some DLL that has injected itself into Firefox and then crashes this way before we actually get to run code.

It might be informative to look for these in Winqual (bug 1371286) to see if we can find any patterns.

Comment 5

[Gabriele Svelto [:gsvelto]]

If somebody can reproduce could we try and get a minidump of this using WER? So that we'd know what we're looking at.

Comment 6

[Chris Peterson [:cpeterson]]

Gabriele, how mpcan a non-technical user gather a Firefox minidump so we don't need to access WER? Yuehang is talking to some Lenovo laptop users that can reproduce this crash.

Comment 7

[Gabriele Svelto [:gsvelto]]

I've had other reporters send me minidumps by using WER locally, it requires editing the registry and then finding the file which shouldn't be too hard even for non-technical users. See bug 1420767 comment 1 for a description of how it's done.

Comment 8

[Chris Peterson [:cpeterson]]

(In reply to Gabriele Svelto [:gsvelto] from comment #7)
> I've had other reporters send me minidumps by using WER locally, it requires
> editing the registry and then finding the file which shouldn't be too hard
> even for non-technical users. See bug 1420767 comment 1 for a description of
> how it's done.

Yuehang, you've talked to some Lenovo users who are crashing. Are they technical enough to be able to follow Gabriele's instructions for capturing WER minidumps?

Comment 9

[yxu]

(In reply to Chris Peterson [:cpeterson] from comment #8)
> (In reply to Gabriele Svelto [:gsvelto] from comment #7)
> > I've had other reporters send me minidumps by using WER locally, it requires
> > editing the registry and then finding the file which shouldn't be too hard
> > even for non-technical users. See bug 1420767 comment 1 for a description of
> > how it's done.
> 
> Yuehang, you've talked to some Lenovo users who are crashing. Are they
> technical enough to be able to follow Gabriele's instructions for capturing
> WER minidumps?

It's very helpful. I will try to contact the user to get the minidumps. Thank you Gabriele.

Comment 10

[yxu]

(In reply to Gabriele Svelto [:gsvelto] from comment #7)
> I've had other reporters send me minidumps by using WER locally, it requires
> editing the registry and then finding the file which shouldn't be too hard
> even for non-technical users. See bug 1420767 comment 1 for a description of
> how it's done.

Hi Gabriele, I have contect a user try to get minidumps. But after done as you said in bug 1420767 comment 1, under %LOCALAPPDATA%\CrashDumps path didn't create any files. Do we need to create additional keys in regedit? Or this error occurred earlier lead to the report can not be generated.

Comment 11

[Gabriele Svelto [:gsvelto]]

The only key that is required is "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps", if that's present then WER should be working.

It is possible that we intercept the error but then our crash reporter fails in which case it needs to be disabled explicitly for WER to work. This can be done by launching firefox with the MOZ_CRASHREPORTER_DISABLE=1 environment variable set. If this also doesn't work then I don't know what we could do without being able to hook up a debugger directly.