1469603 - Crash in shutdownhang | RtlAcquireSRWLockExclusive | mozilla::plugins::PluginModuleChromeParent::TakeFullMinidump

Status: VERIFIED FIXED

(Toolkit :: Crash Reporting, defect)

Description

[[:philipp]]

This bug was filed from the Socorro interface and is
report bp-ed9ae6eb-6327-49dd-97f6-18ec20180619.
=============================================================

Top 10 frames of crashing thread:

0 ntdll.dll NtWaitForAlertByThreadId 
1 ntdll.dll RtlAcquireSRWLockExclusive 
2 xul.dll mozilla::plugins::PluginModuleChromeParent::TakeFullMinidump dom/plugins/ipc/PluginModuleParent.cpp:1085
3 xul.dll mozilla::plugins::PluginModuleChromeParent::TerminateChildProcess dom/plugins/ipc/PluginModuleParent.cpp:1282
4 xul.dll mozilla::plugins::TerminatePlugin dom/plugins/ipc/PluginModuleParent.cpp:385
5 xul.dll static nsresult `anonymous namespace'::HangMonitoredProcess::TerminatePlugin dom/ipc/ProcessHangMonitor.cpp:1104
6 xul.dll XPTC__InvokebyIndex xpcom/reflect/xptcall/md/win32/xptcinvoke_asm_x86_64.asm:97
7  @0xfff9ffffffffffff 
8 xul.dll XPCWrappedNative::CallMethod js/xpconnect/src/XPCWrappedNative.cpp:1235
9 xul.dll XPC_WN_CallMethod js/xpconnect/src/XPCWrappedNativeJSOps.cpp:911

=============================================================

this shutdown hang signature is newly showing up from 64bit builds on windows. the user comment at bp-078776dd-4718-4e78-b922-210ad0180530 suggests that firefox is crashing repeatedly for affected users.

the signature first showed up starting with 62.0a1 build 20180511100427 and then 61.0b6. timing-wise perhaps this could indicate that the changes from bug 1364624 are playing a role?

Comment 1

[Eric Rahm [:erahm]]

Gabriel, it looks like there's a deadlock in the minidump code. We recently switched to SRWLock on Windows which doesn't support re-entrancy, so it's possible that's what's going on. We can see `TakeFullMinidump` blocking on the main thread (the stack is truncated unfortunately), but also multiple nested instances of `CrashReporter::CreatePairedChildMinidumpAsync`each waiting on a sync call to `Dispatch` [1] on the "Minidump Writer" thread.

I *think* the issue is that even if we're doing an async call we end up doing a sync dispatch so we still hold the `mCrashReporterMutex` lock when the callback code [2] is executed, but this code is a bit hard to follow so having someone with more familiarity take a look would be helpful.

[1] https://searchfox.org/mozilla-central/rev/f822a0b61631cbb38901569e69b4967176314aa8/toolkit/crashreporter/nsExceptionHandler.cpp#3805-3810
[2] https://searchfox.org/mozilla-central/rev/f822a0b61631cbb38901569e69b4967176314aa8/dom/plugins/ipc/PluginModuleParent.cpp#1108-1123

Comment 2

[Gabriele Svelto [:gsvelto]]

Yes, GenerateMinidumpAndPair() synchronously calls the callback, so if it's being scheduled synchronously we'd still be holding the lock. The assumption in the code was that the call would always be asynchronous and the lock wouldn't be held. Switching to a recursive mutex should solve the issue.

And yes, this code is really hard to read, just yesterday I was talking to Ted about rewriting the whole thing :)

Comment 3

[Phabricator Automation]

Attachment: Bug 1469603 - Use a recursive lock in crash reporter callbacks that might be called synchronously

Comment 4

[Phabricator Automation]

Comment on attachment 8986439 [details]
Bug 1469603 - Use a recursive lock in crash reporter callbacks that might be called synchronously

Eric Rahm [:erahm] (please no mozreview requests) has approved the revision.

https://phabricator.services.mozilla.com/D1724

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f966dedaa07c
Use a recursive lock in crash reporter callbacks that might be called synchronously r=erahm

Comment 6

[Stefan Hindli [:stefan_hindli]]

Backed out changeset f966dedaa07c (bug 1469603) for B bustages in /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/plugins/PluginModuleParent.h:324: on a CLOSED TREE

Problematic push: https://hg.mozilla.org/integration/autoland/rev/f966dedaa07cc97f0d2f28bd6414577e1767300d
Bustage: https://treeherder.mozilla.org/#/jobs?repo=autoland&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-classifiedState=unclassified&selectedJob=184106783
Log: https://treeherder.mozilla.org/logviewer.html#?job_id=184106783&repo=autoland&lineNumber=18983
Backout: https://hg.mozilla.org/integration/autoland/rev/5dbc311fb93c0f6d4c82eb14fa9062984251ccfe

[task 2018-06-21T07:38:05.164Z] 07:38:05     INFO -  In file included from /builds/worker/workspace/build/src/dom/plugins/base/nsNPAPIPlugin.cpp:78:
[task 2018-06-21T07:38:05.164Z] 07:38:05     INFO -  /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/plugins/PluginModuleParent.h:324:5: error: no type named 'RecursiveMutex' in namespace 'mozilla'; did you mean '__gnu_cxx::recursive_mutex'?
[task 2018-06-21T07:38:05.164Z] 07:38:05     INFO -      mozilla::RecursiveMutex mCrashReporterMutex;
[task 2018-06-21T07:38:05.164Z] 07:38:05     INFO -      ^~~~~~~~~~~~~~~~~~~~~~~
[task 2018-06-21T07:38:05.164Z] 07:38:05     INFO -      __gnu_cxx::recursive_mutex
[task 2018-06-21T07:38:05.164Z] 07:38:05     INFO -  /builds/worker/workspace/build/src/android-ndk/sources/cxx-stl/llvm-libc++/include/mutex:194:24: note: '__gnu_cxx::recursive_mutex' declared here
[task 2018-06-21T07:38:05.164Z] 07:38:05     INFO -  class _LIBCPP_TYPE_VIS recursive_mutex
[task 2018-06-21T07:38:05.164Z] 07:38:05     INFO -                         ^
[task 2018-06-21T07:38:05.164Z] 07:38:05     INFO -  1 error generated.
[task 2018-06-21T07:38:05.164Z] 07:38:05     INFO -  /builds/worker/workspace/build/src/config/rules.mk:1031: recipe for target 'nsNPAPIPlugin.o' failed
[task 2018-06-21T07:38:05.164Z] 07:38:05     INFO -  make[4]: *** [nsNPAPIPlugin.o] Error 1
[task 2018-06-21T07:38:05.164Z] 07:38:05     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/dom/plugins/base'
[task 2018-06-21T07:38:05.164Z] 07:38:05     INFO -  /builds/worker/workspace/build/src/config/recurse.mk:74: recipe for target 'dom/plugins/base/target' failed
[task 2018-06-21T07:38:05.164Z] 07:38:05     INFO -  make[3]: *** [dom/plugins/base/target] Error 2
[task 2018-06-21T07:38:05.164Z] 07:38:05     INFO -  make[3]: *** Waiting for unfinished jobs....
[task 2018-06-21T07:38:05.167Z] 07:38:05     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/dom/plugins/ipc'
[task 2018-06-21T07:38:05.168Z] 07:38:05     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/dom/plugins/ipc'
[task 2018-06-21T07:38:05.196Z] 07:38:05     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/dom/plugins/ipc'
[task 2018-06-21T07:38:05.196Z] 07:38:05     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/dom/plugins/ipc'
[task 2018-06-21T07:38:05.449Z] 07:38:05     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/dom/workers'
[task 2018-06-21T07:38:05.449Z] 07:38:05     INFO -  /builds/worker/workspace/build/src/sccache2/sccache /builds/worker/workspace/build/src/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ -std=gnu++14 --target=arm-linux-androideabi -o Unified_cpp_dom_workers1.o -c -I/builds/worker/workspace/build/src/obj-firefox/dist/stl_wrappers -I/builds/worker/workspace/build/src/obj-firefox/dist/system_wrappers -include /builds/worker/workspace/build/src/config/gcc_hidden.h -DNDEBUG=1 -DTRIMMED=1 -DOS_POSIX=1 -DOS_LINUX=1 -DENABLE_WASM_GC -DSTATIC_EXPORTABLE_JS_API -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -I/builds/worker/workspace/build/src/dom/workers -I/builds/worker/workspace/build/src/obj-firefox/dom/workers -I/builds/worker/workspace/build/src/caps -I/builds/worker/workspace/build/src/dom/base -I/builds/worker/workspace/build/src/dom/bindings -I/builds/worker/workspace/build/src/dom/system -I/builds/worker/workspace/build/src/js/xpconnect/loader -I/builds/worker/workspace/build/src/netwerk/base -I/builds/worker/workspace/build/src/xpcom/build -I/builds/worker/workspace/build/src/xpcom/threads -I/builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/_ipdlheaders -I/builds/worker/workspace/build/src/ipc/chromium/src -I/builds/worker/workspace/build/src/ipc/glue -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -Qunused-arguments -isystem /builds/worker/workspace/build/src/android-ndk/platforms/android-9/arch-arm/usr/include -isystem /builds/worker/workspace/build/src/android-ndk/platforms/android-9/arch-arm/usr/include -gcc-toolchain /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64 -D__ANDROID_API__=9 -Qunused-arguments -Wall -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wunreachable-code-return -Wwrite-strings -Wno-invalid-offsetof -Wclass-varargs -Wloop-analysis -Wc++1z-compat -Wcomma -Wimplicit-fallthrough -Werror=non-literal-null-conversion -Wstring-conversion -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wformat -Wformat-security -Wno-gnu-zero-variadic-macro-arguments -Wno-unknown-warning-option -Wno-return-type-c-linkage -fno-sized-deallocation -fno-short-enums -fno-exceptions -I/builds/worker/workspace/build/src/android-ndk/sources/cxx-stl/llvm-libc++/include -I/builds/worker/workspace/build/src/android-ndk/sources/android/support/include -I/builds/worker/workspace/build/src/android-ndk/sources/cxx-stl/llvm-libc++abi/include -march=armv7-a -mthumb -mfpu=vfpv3-d16 -mfloat-abi=softfp -mno-unaligned-access -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pipe -g -Oz -funwind-tables -Werror -Wno-error=shadow  -MD -MP -MF .deps/Unified_cpp_dom_workers1.o.pp   /builds/worker/workspace/build/src/obj-firefox/dom/workers/Unified_cpp_dom_workers1.cpp
[task 2018-06-21T07:38:05.449Z] 07:38:05     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/dom/workers'
[task 2018-06-21T07:38:05.687Z] 07:38:05     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/dom/messagechannel'
[task 2018-06-21T07:38:05.688Z] 07:38:05     INFO -  /builds/worker/workspace/build/src/sccache2/sccache /builds/worker/workspace/build/src/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ -std=gnu++14 --target=arm-linux-androideabi -o Unified_cpp_dom_messagechannel0.o -c -I/builds/worker/workspace/build/src/obj-firefox/dist/stl_wrappers -I/builds/worker/workspace/build/src/obj-firefox/dist/system_wrappers -include /builds/worker/workspace/build/src/config/gcc_hidden.h -DNDEBUG=1 -DTRIMMED=1 -DOS_POSIX=1 -DOS_LINUX=1 -DSTATIC_EXPORTABLE_JS_API -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -I/builds/worker/workspace/build/src/dom/messagechannel -I/builds/worker/workspace/build/src/obj-firefox/dom/messagechannel -I/builds/worker/workspace/build/src/dom/base -I/builds/worker/workspace/build/src/dom/events -I/builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/_ipdlheaders -I/builds/worker/workspace/build/src/ipc/chromium/src -I/builds/worker/workspace/build/src/ipc/glue -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -Qunused-arguments -isystem /builds/worker/workspace/build/src/android-ndk/platforms/android-9/arch-arm/usr/include -isystem /builds/worker/workspace/build/src/android-ndk/platforms/android-9/arch-arm/usr/include -gcc-toolchain /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64 -D__ANDROID_API__=9 -Qunused-arguments -Wall -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wunreachable-code-return -Wwrite-strings -Wno-invalid-offsetof -Wclass-varargs -Wloop-analysis -Wc++1z-compat -Wcomma -Wimplicit-fallthrough -Werror=non-literal-null-conversion -Wstring-conversion -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wformat -Wformat-security -Wno-gnu-zero-variadic-macro-arguments -Wno-unknown-warning-option -Wno-return-type-c-linkage -fno-sized-deallocation -fno-short-enums -fno-exceptions -I/builds/worker/workspace/build/src/android-ndk/sources/cxx-stl/llvm-libc++/include -I/builds/worker/workspace/build/src/android-ndk/sources/android/support/include -I/builds/worker/workspace/build/src/android-ndk/sources/cxx-stl/llvm-libc++abi/include -march=armv7-a -mthumb -mfpu=vfpv3-d16 -mfloat-abi=softfp -mno-unaligned-access -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pipe -g -Oz -funwind-tables -Werror  -MD -MP -MF .deps/Unified_cpp_dom_messagechannel0.o.pp   /builds/worker/workspace/build/src/obj-firefox/dom/messagechannel/Unified_cpp_dom_messagechannel0.cpp
[task 2018-06-21T07:38:05.688Z] 07:38:05     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/dom/messagechannel'
[task 2018-06-21T07:38:06.897Z] 07:38:06     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/dom/promise'
[task 2018-06-21T07:38:06.900Z] 07:38:06     INFO -  /builds/worker/workspace/build/src/sccache2/sccache /builds/worker/workspace/build/src/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ -std=gnu++14 --target=arm-linux-androideabi -o Unified_cpp_dom_promise0.o -c -I/builds/worker/workspace/build/src/obj-firefox/dist/stl_wrappers -I/builds/worker/workspace/build/src/obj-firefox/dist/system_wrappers -include /builds/worker/workspace/build/src/config/gcc_hidden.h -DNDEBUG=1 -DTRIMMED=1 -DOS_POSIX=1 -DOS_LINUX=1 -DSTATIC_EXPORTABLE_JS_API -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -I/builds/worker/workspace/build/src/dom/promise -I/builds/worker/workspace/build/src/obj-firefox/dom/promise -I/builds/worker/workspace/build/src/dom/base -I/builds/worker/workspace/build/src/dom/ipc -I/builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/_ipdlheaders -I/builds/worker/workspace/build/src/ipc/chromium/src -I/builds/worker/workspace/build/src/ipc/glue -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -Qunused-arguments -isystem /builds/worker/workspace/build/src/android-ndk/platforms/android-9/arch-arm/usr/include -isystem /builds/worker/workspace/build/src/android-ndk/platforms/android-9/arch-arm/usr/include -gcc-toolchain /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64 -D__ANDROID_API__=9 -Qunused-arguments -Wall -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wunreachable-code-return -Wwrite-strings -Wno-invalid-offsetof -Wclass-varargs -Wloop-analysis -Wc++1z-compat -Wcomma -Wimplicit-fallthrough -Werror=non-literal-null-conversion -Wstring-conversion -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wformat -Wformat-security -Wno-gnu-zero-variadic-macro-arguments -Wno-unknown-warning-option -Wno-return-type-c-linkage -fno-sized-deallocation -fno-short-enums -fno-exceptions -I/builds/worker/workspace/build/src/android-ndk/sources/cxx-stl/llvm-libc++/include -I/builds/worker/workspace/build/src/android-ndk/sources/android/support/include -I/builds/worker/workspace/build/src/android-ndk/sources/cxx-stl/llvm-libc++abi/include -march=armv7-a -mthumb -mfpu=vfpv3-d16 -mfloat-abi=softfp -mno-unaligned-access -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pipe -g -Oz -funwind-tables -Werror -Wno-error=shadow  -MD -MP -MF .deps/Unified_cpp_dom_promise0.o.pp   /builds/worker/workspace/build/src/obj-firefox/dom/promise/Unified_cpp_dom_promise0.cpp
[task 2018-06-21T07:38:06.900Z] 07:38:06     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/dom/promise'
[task 2018-06-21T07:38:11.327Z] 07:38:11     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/dom/plugins/ipc'
[task 2018-06-21T07:38:11.337Z] 07:38:11     INFO -  /builds/worker/workspace/build/src/sccache2/sccache /builds/worker/workspace/build/src/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ -std=gnu++14 --target=arm-linux-androideabi -o Unified_cpp_dom_plugins_ipc1.o -c -I/builds/worker/workspace/build/src/obj-firefox/dist/stl_wrappers -I/builds/worker/workspace/build/src/obj-firefox/dist/system_wrappers -include /builds/worker/workspace/build/src/config/gcc_hidden.h -DNDEBUG=1 -DTRIMMED=1 -DOS_POSIX=1 -DOS_LINUX=1 -DFORCE_PR_LOG -DSTATIC_EXPORTABLE_JS_API -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -I/builds/worker/workspace/build/src/dom/plugins/ipc -I/builds/worker/workspace/build/src/obj-firefox/dom/plugins/ipc -I/builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/_ipdlheaders -I/builds/worker/workspace/build/src/ipc/chromium/src -I/builds/worker/workspace/build/src/ipc/glue -I/builds/worker/workspace/build/src/dom/plugins/base -I/builds/worker/workspace/build/src/xpcom/base -I/builds/worker/workspace/build/src/xpcom/threads -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -Qunused-arguments -isystem /builds/worker/workspace/build/src/android-ndk/platforms/android-9/arch-arm/usr/include -isystem /builds/worker/workspace/build/src/android-ndk/platforms/android-9/arch-arm/usr/include -gcc-toolchain /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64 -D__ANDROID_API__=9 -Qunused-arguments -Wall -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wunreachable-code-return -Wwrite-strings -Wno-invalid-offsetof -Wclass-varargs -Wloop-analysis -Wc++1z-compat -Wcomma -Wimplicit-fallthrough -Werror=non-literal-null-conversion -Wstring-conversion -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wformat -Wformat-security -Wno-gnu-zero-variadic-macro-arguments -Wno-unknown-warning-option -Wno-return-type-c-linkage -fno-sized-deallocation -fno-short-enums -fno-exceptions -I/builds/worker/workspace/build/src/android-ndk/sources/cxx-stl/llvm-libc++/include -I/builds/worker/workspace/build/src/android-ndk/sources/android/support/include -I/builds/worker/workspace/build/src/android-ndk/sources/cxx-stl/llvm-libc++abi/include -march=armv7-a -mthumb -mfpu=vfpv3-d16 -mfloat-abi=softfp -mno-unaligned-access -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pipe -g -Oz -funwind-tables -Werror -I/builds/worker/workspace/build/src/obj-firefox/dist/include/cairo -I/builds/worker/workspace/build/src/obj-firefox/dist/include/cairo -Wno-error=shadow  -MD -MP -MF .deps/Unified_cpp_dom_plugins_ipc1.o.pp   /builds/worker/workspace/build/src/obj-firefox/dom/plugins/ipc/Unified_cpp_dom_plugins_ipc1.cpp
[task 2018-06-21T07:38:11.337Z] 07:38:11     INFO -  In file included from /builds/worker/workspace/build/src/obj-firefox/dom/plugins/ipc/Unified_cpp_dom_plugins_ipc1.cpp:2:
[task 2018-06-21T07:38:11.337Z] 07:38:11     INFO -  In file included from /builds/worker/workspace/build/src/dom/plugins/ipc/PluginScriptableObjectChild.cpp:8:
[task 2018-06-21T07:38:11.337Z] 07:38:11     INFO -  In file included from /builds/worker/workspace/build/src/dom/plugins/ipc/PluginScriptableObjectUtils.h:10:
[task 2018-06-21T07:38:11.338Z] 07:38:11     INFO -  /builds/worker/workspace/build/src/dom/plugins/ipc/PluginModuleParent.h:324:5: error: no type named 'RecursiveMutex' in namespace 'mozilla'; did you mean '__gnu_cxx::recursive_mutex'?

Comment 7

[Gabriele Svelto [:gsvelto]]

Oops, I forgot including mozilla/RecursiveMutex.h though I wonder why did it compile locally :|

Comment 8

[Gabriele Svelto [:gsvelto]]

I've updated the patch with the issue fixed, it seems to be building fine on both Android and Windows.

Comment 9

[Gabriele Svelto [:gsvelto]]

There doesn't seem to be a way to land the updated diff directly from Lando so I'll just push it manually.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/0ed1a6d400dc
Use a recursive lock in crash reporter callbacks that might be called synchronously; r=erahm

Comment 11

[Eric Rahm [:erahm]]

(In reply to Gabriele Svelto [:gsvelto] from comment #9)
> There doesn't seem to be a way to land the updated diff directly from Lando
> so I'll just push it manually.

Sadly I think you have to ask for another review, although I'm not sure if the UI lets you do that if you got an r+ and landed already. You might want file a bug if it won't at least let you ask for a new review.

Comment 12

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/0ed1a6d400dc

Comment 13

[Ryan VanderMeulen [:RyanVM]]

No crashes on Nightly since this landed. Please nominate this for Beta approval when you get a chance.

Comment 14

[Gabriele Svelto [:gsvelto]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #13)
> No crashes on Nightly since this landed. Please nominate this for Beta
> approval when you get a chance.

Let's do it right away!

Comment 15

[Gabriele Svelto [:gsvelto]]

Comment on attachment 8986439 [details]
Bug 1469603 - Use a recursive lock in crash reporter callbacks that might be called synchronously

Approval Request Comment
[Feature/Bug causing the regression]: bug 1364624
[User impact if declined]: Firefox crashes repeatedly during normal use
[Is this code covered by automated tests?]: No
[Has the fix been verified in Nightly?]: Yes
[Needs manual test from QE? If yes, steps to reproduce]: No, the crash happens during a specific race so it's probably impossible to reproduce it manually
[List of other uplifts needed for the feature/fix]: None
[Is the change risky?]: No
[Why is the change risky/not risky?]: No logic is affected, we only changed the type of a mutex so that it deals properly with recursive locking
[String changes made/needed]: None

Comment 16

[Liz Henry (:lizzard) (relman/hg->git project)]

Comment on attachment 8986439 [details]
Bug 1469603 - Use a recursive lock in crash reporter callbacks that might be called synchronously

Crash fix, let's take this for beta 5.

Comment 17

[Dorel Luca [:dluca]]

https://hg.mozilla.org/releases/mozilla-beta/rev/014b19aba4a5

Comment 18

[Gabriele Svelto [:gsvelto]]

Comment on attachment 8986439 [details]
Bug 1469603 - Use a recursive lock in crash reporter callbacks that might be called synchronously

See comment 15 for the approval request comment

Comment 19

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 8986439 [details]
Bug 1469603 - Use a recursive lock in crash reporter callbacks that might be called synchronously

Cleanup some new shutdown hangs introduced by bug 1364624. Approved for 61.0.1.

Comment 20

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-release/rev/2826a36d480d

Comment 21

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 8986439 [details]
Bug 1469603 - Use a recursive lock in crash reporter callbacks that might be called synchronously

Also approved for ESR 60.2.

Comment 22

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr60/rev/0d73faee431c