IPC marker deadlock in SendDestroyReleasedChunksAtOrBefore
Categories
(Core :: Gecko Profiler, defect, P3)
Tracking
()
Tracking | Status | |
---|---|---|
firefox-esr128 | --- | unaffected |
firefox135 | --- | wontfix |
firefox136 | --- | wontfix |
firefox137 | --- | fixed |
People
(Reporter: jrmuizel, Assigned: canova)
References
(Regression)
Details
(Keywords: regression, Whiteboard: [fxp])
Attachments
(1 file)
This is on a custom build with something like https://phabricator.services.mozilla.com/D231560 applied. See the following stack:
* thread #1, name = 'MainThread', queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
* frame #0: 0x000000019817ca9c libsystem_kernel.dylib`__psynch_mutexwait + 8
frame #1: 0x00000001981b83f8 libsystem_pthread.dylib`_pthread_mutex_firstfit_lock_wait + 84
frame #2: 0x00000001981b5dbc libsystem_pthread.dylib`_pthread_mutex_firstfit_lock_slow + 220
frame #3: 0x0000000100f69ba0 libmozglue.dylib`mozilla::detail::MutexImpl::lock() at Mutex_posix.cpp:94:3 [opt]
frame #4: 0x0000000100f69b9c libmozglue.dylib`mozilla::detail::MutexImpl::lock(this=<unavailable>) at Mutex_posix.cpp:116:43 [opt]
frame #5: 0x0000000114e9b440 XUL`mozilla::ProfileBufferBlockIndex mozilla::ProfileChunkedBuffer::PutObjects<mozilla::ProfileBufferEntryKind, mozilla::MarkerOptions, mozilla::ProfilerStringView<char>, mozilla::MarkerCategory, unsigned char, mozilla::MarkerPayloadType, unsigned int, Flow>(mozilla::ProfileBufferEntryKind const&, mozilla::MarkerOptions const&, mozilla::ProfilerStringView<char> const&, mozilla::MarkerCategory const&, unsigned char const&, mozilla::MarkerPayloadType const&, unsigned int const&, Flow const&) [inlined] mozilla::baseprofiler::detail::BaseProfilerMutex::Lock(this=<unavailable>) at BaseProfilerDetail.h:57:35 [opt]
frame #6: 0x0000000114e9b43c XUL`mozilla::ProfileBufferBlockIndex mozilla::ProfileChunkedBuffer::PutObjects<mozilla::ProfileBufferEntryKind, mozilla::MarkerOptions, mozilla::ProfilerStringView<char>, mozilla::MarkerCategory, unsigned char, mozilla::MarkerPayloadType, unsigned int, Flow>(mozilla::ProfileBufferEntryKind const&, mozilla::MarkerOptions const&, mozilla::ProfilerStringView<char> const&, mozilla::MarkerCategory const&, unsigned char const&, mozilla::MarkerPayloadType const&, unsigned int const&, Flow const&) [inlined] mozilla::baseprofiler::detail::BaseProfilerMaybeMutex::Lock(this=0x0000000100f9a340) at BaseProfilerDetail.h:156:20 [opt]
frame #7: 0x0000000114e9b428 XUL`mozilla::ProfileBufferBlockIndex mozilla::ProfileChunkedBuffer::PutObjects<mozilla::ProfileBufferEntryKind, mozilla::MarkerOptions, mozilla::ProfilerStringView<char>, mozilla::MarkerCategory, unsigned char, mozilla::MarkerPayloadType, unsigned int, Flow>(mozilla::ProfileBufferEntryKind const&, mozilla::MarkerOptions const&, mozilla::ProfilerStringView<char> const&, mozilla::MarkerCategory const&, unsigned char const&, mozilla::MarkerPayloadType const&, unsigned int const&, Flow const&) [inlined] mozilla::baseprofiler::detail::BaseProfilerMaybeAutoLock::BaseProfilerMaybeAutoLock(this=0x000000016efc0530, aMaybeMutex=0x0000000100f9a340) at BaseProfilerDetail.h:176:17 [opt]
frame #8: 0x0000000114e9b424 XUL`mozilla::ProfileBufferBlockIndex mozilla::ProfileChunkedBuffer::PutObjects<mozilla::ProfileBufferEntryKind, mozilla::MarkerOptions, mozilla::ProfilerStringView<char>, mozilla::MarkerCategory, unsigned char, mozilla::MarkerPayloadType, unsigned int, Flow>(mozilla::ProfileBufferEntryKind const&, mozilla::MarkerOptions const&, mozilla::ProfilerStringView<char> const&, mozilla::MarkerCategory const&, unsigned char const&, mozilla::MarkerPayloadType const&, unsigned int const&, Flow const&) [inlined] mozilla::baseprofiler::detail::BaseProfilerMaybeAutoLock::BaseProfilerMaybeAutoLock(this=0x000000016efc0530, aMaybeMutex=0x0000000100f9a340) at BaseProfilerDetail.h:175:34 [opt]
frame #9: 0x0000000114e9b424 XUL`mozilla::ProfileBufferBlockIndex mozilla::ProfileChunkedBuffer::PutObjects<mozilla::ProfileBufferEntryKind, mozilla::MarkerOptions, mozilla::ProfilerStringView<char>, mozilla::MarkerCategory, unsigned char, mozilla::MarkerPayloadType, unsigned int, Flow>(mozilla::ProfileBufferEntryKind const&, mozilla::MarkerOptions const&, mozilla::ProfilerStringView<char> const&, mozilla::MarkerCategory const&, unsigned char const&, mozilla::MarkerPayloadType const&, unsigned int const&, Flow const&) [inlined] decltype(std::forward<mozilla::ProfileBufferBlockIndex mozilla::ProfileChunkedBuffer::PutObjects<mozilla::ProfileBufferEntryKind, mozilla::MarkerOptions, mozilla::ProfilerStringView<char>, mozilla::MarkerCategory, unsigned char, mozilla::MarkerPayloadType, unsigned int, Flow>(mozilla::ProfileBufferEntryKind const&, mozilla::MarkerOptions const&, mozilla::ProfilerStringView<char> const&, mozilla::MarkerCategory const&, unsigned char const&, mozilla::MarkerPayloadType const&, unsigned int const&, Flow const&)::'lambda'(mozilla::Maybe<mozilla::ProfileBufferEntryWriter>&)>(fp0)(decltype(std::__declval<mozilla::Maybe<mozilla::ProfileBufferEntryWriter>&>(0)) std::__1::declval<mozilla::Maybe<mozilla::ProfileBufferEntryWriter>&>()())) mozilla::ProfileChunkedBuffer::ReserveAndPut<mozilla::ProfileBufferBlockIndex mozilla::ProfileChunkedBuffer::PutObjects<mozilla::ProfileBufferEntryKind, mozilla::MarkerOptions, mozilla::ProfilerStringView<char>, mozilla::MarkerCategory, unsigned char, mozilla::MarkerPayloadType, unsigned int, Flow>(mozilla::ProfileBufferEntryKind const&, mozilla::MarkerOptions const&, mozilla::ProfilerStringView<char> const&, mozilla::MarkerCategory const&, unsigned char const&, mozilla::MarkerPayloadType const&, unsigned int const&, Flow const&)::'lambda'(), mozilla::ProfileBufferBlockIndex mozilla::ProfileChunkedBuffer::PutObjects<mozilla::ProfileBufferEntryKind, mozilla::MarkerOptions, mozilla::ProfilerStringView<char>, mozilla::MarkerCategory, unsigned char, mozilla::MarkerPayloadType, unsigned int, Flow>(mozilla::ProfileBufferEntryKind const&, mozilla::MarkerOptions const&, mozilla::ProfilerStringView<char> const&, mozilla::MarkerCategory const&, unsigned char const&, mozilla::MarkerPayloadType const&, unsigned int const&, Flow const&)::'lambda'(mozilla::Maybe<mozilla::ProfileBufferEntryWriter>&)>(this=0x0000000100f9a340, aCallbackEntryBytes=0x000000016efc04c8, aCallback=0x000000016efc0488) at ProfileChunkedBuffer.h:310:53 [opt]
frame #10: 0x0000000114e9b424 XUL`mozilla::ProfileBufferBlockIndex mozilla::ProfileChunkedBuffer::PutObjects<mozilla::ProfileBufferEntryKind, mozilla::MarkerOptions, mozilla::ProfilerStringView<char>, mozilla::MarkerCategory, unsigned char, mozilla::MarkerPayloadType, unsigned int, Flow>(this=0x0000000100f9a340, aTs=0x000000016efc0568, aTs=0x000000016efc0860, aTs=0x000000016efc08a0, aTs=0x000000011bcfa760, aTs=0x000000016efc05b8, aTs=0x000000016efc06a8, aTs=0x000000016efc0840, aTs=0x000000016efc0850) at ProfileChunkedBuffer.h:357:12 [opt]
frame #11: 0x0000000114e9a7a0 XUL`mozilla::ProfileBufferBlockIndex mozilla::base_profiler_markers_detail::AddMarkerToBuffer<mozilla::ipc::IPCFlowMarker, unsigned int, Flow>(mozilla::ProfileChunkedBuffer&, mozilla::ProfilerStringView<char> const&, mozilla::MarkerCategory const&, mozilla::MarkerOptions&&, bool (*)(mozilla::ProfileChunkedBuffer&, mozilla::StackCaptureOptions), unsigned int const&, Flow const&) [inlined] mozilla::base_profiler_markers_detail::StreamFunctionTypeHelper<void (mozilla::baseprofiler::SpliceableJSONWriter&, unsigned int, Flow)>::Serialize(aBuffer=0x0000000100f9a340, aName=0x000000016efc08a0, aCategory=0x000000011bcfa760, aOptions=0x000000016efc0860, aDeserializerTag='\n', aAs=0x000000016efc0840, aAs=0x000000016efc0850) at BaseProfilerMarkersDetail.h:120:20 [opt]
frame #12: 0x0000000114e9a770 XUL`mozilla::ProfileBufferBlockIndex mozilla::base_profiler_markers_detail::AddMarkerToBuffer<mozilla::ipc::IPCFlowMarker, unsigned int, Flow>(mozilla::ProfileChunkedBuffer&, mozilla::ProfilerStringView<char> const&, mozilla::MarkerCategory const&, mozilla::MarkerOptions&&, bool (*)(mozilla::ProfileChunkedBuffer&, mozilla::StackCaptureOptions), unsigned int const&, Flow const&) [inlined] mozilla::ProfileBufferBlockIndex mozilla::base_profiler_markers_detail::MarkerTypeSerialization<mozilla::ipc::IPCFlowMarker>::Serialize<unsigned int, Flow>(aBuffer=0x0000000100f9a340, aName=0x000000016efc08a0, aCategory=0x000000011bcfa760, aOptions=0x000000016efc0860, aTs=0x000000016efc0840, aTs=0x000000016efc0850) at BaseProfilerMarkersDetail.h:166:12 [opt]
frame #13: 0x0000000114e9a750 XUL`mozilla::ProfileBufferBlockIndex mozilla::base_profiler_markers_detail::AddMarkerToBuffer<mozilla::ipc::IPCFlowMarker, unsigned int, Flow>(mozilla::ProfileChunkedBuffer&, mozilla::ProfilerStringView<char> const&, mozilla::MarkerCategory const&, mozilla::MarkerOptions&&, bool (*)(mozilla::ProfileChunkedBuffer&, mozilla::StackCaptureOptions), unsigned int const&, Flow const&) [inlined] mozilla::ProfileBufferBlockIndex mozilla::base_profiler_markers_detail::AddMarkerWithOptionalStackToBuffer<mozilla::ipc::IPCFlowMarker, unsigned int, Flow>(aBuffer=0x0000000100f9a340, aName=0x000000016efc08a0, aCategory=0x000000011bcfa760, aOptions=0x000000016efc0860, aTs=0x000000016efc0840, aTs=0x000000016efc0850) at BaseProfilerMarkersDetail.h:255:12 [opt]
frame #14: 0x0000000114e9a750 XUL`mozilla::ProfileBufferBlockIndex mozilla::base_profiler_markers_detail::AddMarkerToBuffer<mozilla::ipc::IPCFlowMarker, unsigned int, Flow>(aBuffer=0x0000000100f9a340, aName=0x000000016efc08a0, aCategory=0x000000011bcfa760, aOptions=0x000000016efc0860, aOptionalBacktraceCaptureFunction=<unavailable>, aTs=0x000000016efc0840, aTs=0x000000016efc0850) at BaseProfilerMarkersDetail.h:327:10 [opt]
frame #15: 0x0000000114e9a688 XUL`mozilla::ProfileBufferBlockIndex AddMarkerToBuffer<mozilla::ipc::IPCFlowMarker, unsigned int, Flow>(aBuffer=0x0000000100f9a340, aName=0x000000016efc08a0, aCategory=0x000000011bcfa760, aOptions=0x000000016efc0860, aMarkerType=<unavailable>, aPayloadArguments=0x000000016efc0840, aPayloadArguments=0x000000016efc0850) at ProfilerMarkers.h:109:10 [opt]
frame #16: 0x0000000114e85f3c XUL`mozilla::ProfileBufferBlockIndex profiler_add_marker_impl<mozilla::ipc::IPCFlowMarker, unsigned int, Flow>(aName=0x000000016efc08a0, aCategory=0x000000011bcfa760, aOptions=0x000000016efc0860, aMarkerType=<unavailable>, aPayloadArguments=0x000000016efc0840, aPayloadArguments=0x000000016efc0850) at ProfilerMarkers.h:184:10 [opt]
frame #17: 0x0000000114e84180 XUL`mozilla::ipc::MessageChannel::AddProfilerMarker(this=0x00000001605a42a0, aMessage=0x000000017c785300, aDirection=eSending) at MessageChannel.cpp:2204:5 [opt]
frame #18: 0x0000000114e83ecc XUL`mozilla::ipc::MessageChannel::Send(this=0x00000001605a42a0, aMsg=UniquePtr<IPC::Message, mozilla::DefaultDelete<IPC::Message> > @ 0x000000016efc0960, aSeqno=<unavailable>) at MessageChannel.cpp:739:3 [opt]
frame #19: 0x0000000114e94240 XUL`mozilla::ipc::IProtocol::ChannelSend(this=<unavailable>, aMsg=<unavailable>, aSeqno=<unavailable>) at ProtocolUtils.cpp:535:22 [opt]
frame #20: 0x0000000118749bdc XUL`mozilla::PProfilerParent::SendDestroyReleasedChunksAtOrBefore(this=0x00000001605a4220, timeStamp=<unavailable>) at PProfilerParent.cpp:684:21 [opt]
frame #21: 0x0000000118741cb8 XUL`mozilla::ProfileBufferGlobalController::HandleChunkManagerNonFinalUpdate(int, mozilla::ProfileBufferControlledChunkManager::Update&&, mozilla::ProfileBufferControlledChunkManager&) [inlined] mozilla::ProfileBufferGlobalController::HandleChunkManagerNonFinalUpdate(int, mozilla::ProfileBufferControlledChunkManager::Update&&, mozilla::ProfileBufferControlledChunkManager&)::$_0::operator()(this=0x000000016efc09f0, profilerParent=<unavailable>) const at ProfilerParent.cpp:495:39 [opt]
frame #22: 0x0000000118741cb0 XUL`mozilla::ProfileBufferGlobalController::HandleChunkManagerNonFinalUpdate(int, mozilla::ProfileBufferControlledChunkManager::Update&&, mozilla::ProfileBufferControlledChunkManager&) at ProfilerParent.cpp:627:9 [opt]
frame #23: 0x0000000118741c68 XUL`mozilla::ProfileBufferGlobalController::HandleChunkManagerNonFinalUpdate(this=0x000000012d2e9e50, aProcessId=<unavailable>, aUpdate=0x000000016efc0ab0, aParentChunkManager=0x0000000307719de0) at ProfilerParent.cpp:492:7 [opt]
frame #24: 0x0000000118741710 XUL`mozilla::ProfileBufferGlobalController::HandleChildChunkManagerUpdate(this=0x000000012d2e9e50, aProcessId=72520, aUpdate=0x000000016efc0ab0) at ProfilerParent.cpp:410:3 [opt]
frame #25: 0x000000011874257c XUL`mozilla::ProfilerParentTracker::ForwardChildChunkManagerUpdate(aProcessId=<unavailable>, aUpdate=<unavailable>) at ProfilerParent.cpp:645:30 [opt] [artificial]
frame #26: 0x0000000118758e4c XUL`mozilla::MozPromise<mozilla::ProfileBufferChunkManagerUpdate, mozilla::ipc::ResponseRejectReason, true>::ThenValue<mozilla::ProfilerParent::RequestChunkManagerUpdate()::$_0, mozilla::ProfilerParent::RequestChunkManagerUpdate()::$_1>::DoResolveOrRejectInternal(mozilla::MozPromise<mozilla::ProfileBufferChunkManagerUpdate, mozilla::ipc::ResponseRejectReason, true>::ResolveOrRejectValue&) at ProfilerParent.cpp:812:11 [opt]
frame #27: 0x0000000118758e20 XUL`mozilla::MozPromise<mozilla::ProfileBufferChunkManagerUpdate, mozilla::ipc::ResponseRejectReason, true>::ThenValue<mozilla::ProfilerParent::RequestChunkManagerUpdate()::$_0, mozilla::ProfilerParent::RequestChunkManagerUpdate()::$_1>::DoResolveOrRejectInternal(mozilla::MozPromise<mozilla::ProfileBufferChunkManagerUpdate, mozilla::ipc::ResponseRejectReason, true>::ResolveOrRejectValue&) [inlined] mozilla::detail::MethodTrait<void (mozilla::ProfilerParent::RequestChunkManagerUpdate()::$_0::*)(mozilla::ProfileBufferChunkManagerUpdate const&) const>::ReturnType mozilla::MozPromise<mozilla::ProfileBufferChunkManagerUpdate, mozilla::ipc::ResponseRejectReason, true>::InvokeMethod<mozilla::ProfilerParent::RequestChunkManagerUpdate()::$_0, void (mozilla::ProfilerParent::RequestChunkManagerUpdate()::$_0::*)(mozilla::ProfileBufferChunkManagerUpdate const&) const, mozilla::ProfileBufferChunkManagerUpdate>(aThisVal=0x000000012e7869f0, aMethod=0x00000000000000000000000000000000, aValue=0x000000040b4f0b88) at MozPromise.h:652:14 [opt]
frame #28: 0x0000000118758e20 XUL`mozilla::MozPromise<mozilla::ProfileBufferChunkManagerUpdate, mozilla::ipc::ResponseRejectReason, true>::ThenValue<mozilla::ProfilerParent::RequestChunkManagerUpdate()::$_0, mozilla::ProfilerParent::RequestChunkManagerUpdate()::$_1>::DoResolveOrRejectInternal(mozilla::MozPromise<mozilla::ProfileBufferChunkManagerUpdate, mozilla::ipc::ResponseRejectReason, true>::ResolveOrRejectValue&) [inlined] RefPtr<mozilla::MozPromise<mozilla::ProfileBufferChunkManagerUpdate, mozilla::ipc::ResponseRejectReason, true>> mozilla::MozPromise<mozilla::ProfileBufferChunkManagerUpdate, mozilla::ipc::ResponseRejectReason, true>::InvokeCallbackMethod<false, mozilla::MozPromise<mozilla::ProfileBufferChunkManagerUpdate, mozilla::ipc::ResponseRejectReason, true>, mozilla::ProfilerParent::RequestChunkManagerUpdate()::$_0, void (mozilla::ProfilerParent::RequestChunkManagerUpdate()::$_0::*)(mozilla::ProfileBufferChunkManagerUpdate const&) const, mozilla::ProfileBufferChunkManagerUpdate>(aThisVal=0x000000012e7869f0, aMethod=0x00000000000000000000000000000000, aValue=0x000000040b4f0b88) at MozPromise.h:666:7 [opt]
frame #29: 0x0000000118758e20 XUL`mozilla::MozPromise<mozilla::ProfileBufferChunkManagerUpdate, mozilla::ipc::ResponseRejectReason, true>::ThenValue<mozilla::ProfilerParent::RequestChunkManagerUpdate()::$_0, mozilla::ProfilerParent::RequestChunkManagerUpdate()::$_1>::DoResolveOrRejectInternal(this=0x000000012e7869b0, aValue=0x000000040b4f0b88) at MozPromise.h:857:17 [opt]
frame #30: 0x00000001187529cc XUL`mozilla::MozPromise<mozilla::ProfileBufferChunkManagerUpdate, mozilla::ipc::ResponseRejectReason, true>::ThenValueBase::ResolveOrRejectRunnable::Run(this=0x00000003f7b40090) at MozPromise.h:488:21 [opt]
frame #31: 0x0000000114740384 XUL`mozilla::SimpleTaskQueue::DrainTasks(this=0x0000000103bd18f0) at TaskDispatcher.h:44:10 [opt]
frame #32: 0x0000000114756ab8 XUL`nsThread::DrainDirectTasks(this=0x0000000103bd17c0) at nsThread.cpp:1357:16 [opt]
frame #33: 0x0000000114756088 XUL`nsThread::ProcessNextEvent(this=0x0000000103bd17c0, aMayWait=<unavailable>, aResult=0x000000016efc0da7) at nsThread.cpp:1182:3 [opt]
frame #34: 0x0000000114754138 XUL`NS_ProcessPendingEvents(aThread=0x0000000103bd17c0, aTimeout=10) at nsThreadUtils.cpp:445:19 [opt]
frame #35: 0x0000000117fb3a78 XUL`nsBaseAppShell::NativeEventCallback(this=0x000000010aa8a700) at nsBaseAppShell.cpp:87:3 [opt]
frame #36: 0x000000011802545c XUL`nsAppShell::ProcessGeckoEvents(aInfo=0x000000010aa8a700) at nsAppShell.mm:534:11 [opt]
Comment 1•1 month ago
|
||
Nazim will have a look, he had the issue in the past already.
Updated•1 month ago
|
Assignee | ||
Comment 2•12 days ago
|
||
Sorry for not looking at this earlier. I could just get around to it.
It looks like the new markers are not checking for !profiler_is_locked_on_current_thread()
. We have a similar check in the main IPC markers: https://searchfox.org/mozilla-central/rev/d1fbe983fb7720f0a4aca0e748817af11c1a374e/ipc/chromium/src/chrome/common/ipc_channel_utils.cc#25-30 and https://searchfox.org/mozilla-central/rev/d1fbe983fb7720f0a4aca0e748817af11c1a374e/ipc/glue/MessageChannel.cpp#2232 .
It's because while we are adding a marker, we might trigger a new IPC. We acquire the lock while adding a marker, so we are probably trying to add this flow marker for that profiler IPC and that's causing a deadlock because we already have the lock acquired in the same thread. So we should have the same check and do not add the marker if it's the case.
This will cause us to not capture the flow markers for profiler IPCs. But that's okay considering that the alternative is a deadlock.
Assignee | ||
Comment 3•12 days ago
|
||
This is similar to what we have for the real IPC markers:
https://searchfox.org/mozilla-central/rev/d1fbe983fb7720f0a4aca0e748817af11c1a374e/ipc/chromium/src/chrome/common/ipc_channel_utils.cc#25-30
and
https://searchfox.org/mozilla-central/rev/d1fbe983fb7720f0a4aca0e748817af11c1a374e/ipc/glue/MessageChannel.cpp#2232
We need this because sometimes we might send an IPC while we are adding a marker.
This causes the profiler to try to acquire the profiler lock while it's already
acquired. So it creates a deadlock in case a marker triggers an IPC.
After this change, we will lose some of the IPC flow markers for the profiler
itself, but this is an okay compromise considering that the alternative is a
deadlock. Also we these IPC messages are not important for the program execution.
Updated•12 days ago
|
Comment 4•12 days ago
|
||
(In reply to Nazım Can Altınova [:canova][:canaltinova on phabricator] from comment #3)
We need this because sometimes we might send an IPC while we are adding a marker.
!!! :(
Updated•12 days ago
|
Comment 5•12 days ago
|
||
Set release status flags based on info from the regressing bug 1934288
Updated•8 days ago
|
Comment 8•7 days ago
|
||
The patch landed in nightly and beta is affected.
:canova, is this bug important enough to require an uplift?
- If yes, please nominate the patch for beta approval.
- If no, please set
status-firefox136
towontfix
.
For more information, please visit BugBot documentation.
Assignee | ||
Comment 9•7 days ago
|
||
This is behind a disabled feature in about:profiling
currently. So I'm not sure if it needs an uplift, but if it's important for flow profiling we can do it.
:jrmuizel, do you think it's necessary?
Assignee | ||
Updated•7 days ago
|
Description
•