1932088 - Crash in [@ mozilla::CollectSingleStepData<T>]

Status: RESOLVED FIXED

(Core :: Widget: Win32, defect, P3)

Description

[Gabriele Svelto [:gsvelto]]

Crash report: https://crash-stats.mozilla.org/report/index/926ee853-666b-40e8-b903-e2f8f0241113

Reason:

EXCEPTION_STACK_BUFFER_OVERRUN / FAST_FAIL_GUARD_ICALL_CHECK_FAILURE

Top 10 frames:

0  ntdll.dll  LdrpICallHandler
1  ntdll.dll  RtlpExecuteHandlerForException
2  ntdll.dll  RtlDispatchException
3  ntdll.dll  KiUserExceptionDispatch
4  ntdll.dll  LdrpDispatchUserCallTarget
5  mozglue.dll  mozilla::CollectSingleStepData<void (*)()>(void (*)(), std::function<bool (vo...  mozglue/misc/WindowsDiagnostics.h:131
6  mozglue.dll  mozilla::CollectStackWalkLocks(mozilla::Array<void*, 2>&)  toolkit/xre/dllservices/mozglue/WindowsStackWalkInitialization.cpp:102
6  mozglue.dll  mozilla::WindowsStackWalkInitialization()  toolkit/xre/dllservices/mozglue/WindowsStackWalkInitialization.cpp:59
7  xul.dll  mozilla::BackgroundHangMonitor::Startup()  toolkit/components/backgroundhangmonitor/BackgroundHangMonitor.cpp:606
8  xul.dll  NS_InitXPCOM(nsIServiceManager**, nsIFile*, nsIDirectoryServiceProvider*, bool)  xpcom/build/XPCOMInit.cpp:509

Looks like some kind of stack corruption while we collect single-step data. This was caught by WER but I noticed it by looking at crash ping data.

Comment 1

[Gabriele Svelto [:gsvelto]]

FYI in terms of pings this looks like 100's of crashes all coming from a single user, but the crashes we have on file on Socorro appear to be from two distinct ones.

Comment 2

[Ray Kraesig [:rkraesig]]

(In reply to Gabriele Svelto [:gsvelto] from comment #1)

FYI in terms of pings this looks like 100's of crashes all coming from a single user, but the crashes we have on file on Socorro appear to be from two distinct ones.

The two crashes in Socorro have different listed causes:

• The first is EXCEPTION_SINGLE_STEP. I assume this one came in through WER? If I had to guess, I'd guess that the process was being run under a debugger, and that our exception handler and theirs got crossed somehow. I don't actually see how that could happen, though.

• The second is EXCEPTION_STACK_BUFFER_OVERRUN / FAST_FAIL_GUARD_ICALL_CHECK_FAILURE. That one went off the rails when the indirection-check for the debugging function reported it to be a bad destination address. I have no hypothesis as to why, other than a single-event upset; I can't test that hypothesis, though, because it doesn't look like the address it was trying to jump to was stored in the minidump. (Or at least I couldn't find it.)

Comment 3

[Gabriele Svelto [:gsvelto]]

It's the EXCEPTION_STACK_BUFFER_OVERRUN / FAST_FAIL_GUARD_ICALL_CHECK_FAILURE that came via WER, because it's failing via __fastfail() so the regular exception handle can't catch it. The EXCEPTION_SINGLE_STEP one on the other hand is the one that has hundreds of crash pings corresponding to it.

Comment 4

[BugBot [:suhaib / :marco]]

Closing because no crashes reported for 12 weeks.

Comment 5

[Ryan VanderMeulen [:RyanVM]]

This seems to be spiking again on Nightly recently (currently the #1 overall topcrash by volume). But it looks like a huge number of reports coming from a single install, so I'm not entirely sure what to make of it.

Comment 6

[BugBot [:suhaib / :marco]]

The bug is linked to a topcrash signature, which matches the following criterion:

• Top 10 desktop browser crashes on nightly

:gstoll, could you consider increasing the severity of this top-crash bug?

For more information, please visit BugBot documentation.

Comment 7

[Greg Stoll :gstoll]

:yannis, any idea what might be going on here?

It's unfortunate that this is marked as a topcrash if only one person is hitting this, though...

Comment 8

[BugBot [:suhaib / :marco]]

The bug is linked to a topcrash signature, which matches the following criterion:

• Top 10 desktop browser crashes on nightly (startup)

For more information, please visit BugBot documentation.

Comment 9

[Yannis Juglaret [:yannis]]

These crashes all come from users of the Panda Security antivirus. We can see their DLLs (PSNInjHookMS64.dll, PSNInjTools64.dll, PSNInjComm64.dll, PSNInjHookPlg64.dll) being injected into Firefox processes, even in old crashes not from the recent single-user spike. They are likely interfering with vectored exception handlers in some way (in our case, we set up our SingleStepExceptionHandler and rely on it being called).

I'll see if I can reproduce locally by messing with vectored exception handlers myself, and write a patch to also catch the exception via SEH as a fallback.

This impacts Nightly more than other channels because this code path is used by the Background Hang Reporter. For Beta and Release, only using the Firefox Profiler can trigger this code path, so I won't be asking for uplifts.

Comment 10

[Yannis Juglaret [:yannis]]

Attachment: Bug 1932088 - Fail cleanly via SEH if a single-step exception escapes the VEH chain. r=#win-reviewers

On Windows, we use single-step execution to collect the addresses of
some ntdll-internal locks that are required for deadlock-free profiling.
We achieve this by setting up a vectored exception handler that handles
single-step exceptions.

Based on crash data, third-party antivirus software can get in the way
of our handler, preventing it from running and thus letting the
single-step exception escape the VEH chain. When that happens, the user
will crash.

This patch thus adds a SEH handler to catch the single-step exception
and fail cleanly in case the exception escapes the VEH chain.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/2b304ea0d98f
https://hg.mozilla.org/integration/autoland/rev/d9b42d8bb5f4
Fail cleanly via SEH if a single-step exception escapes the VEH chain. r=win-reviewers,gstoll

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/6cd7c2aacba6
https://hg.mozilla.org/integration/autoland/rev/b5786c639e2e
Revert "Bug 1932088 - Fail cleanly via SEH if a single-step exception escapes the VEH chain. r=win-reviewers,gstoll" for causing failures at WindowsDiagnostics.h.

Comment 13

[Atila Butkovits]

Backed out for causing failures at WindowsDiagnostics.h.

Backout link: https://hg.mozilla.org/integration/autoland/rev/b5786c639e2e

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=d9b42d8bb5f4114ccc971a1043c2c86b76f770eb&selectedTaskRun=JyjPzo-_RYqUI9RJHYlLaQ.0

Failure log: https://treeherder.mozilla.org/logviewer?job_id=566762394&repo=autoland&task=JyjPzo-_RYqUI9RJHYlLaQ.0&lineNumber=10142

Comment 14

[Yannis Juglaret [:yannis]]

Thanks for the backout. I'll fix the spidermonkey build issue.

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/f2ec13393e45
https://hg.mozilla.org/integration/autoland/rev/1a58a41533c6
Fail cleanly via SEH if a single-step exception escapes the VEH chain. r=win-reviewers,gstoll

Comment 16

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/1a58a41533c6