1610585 - Assertion failure: aIncrease || mBusyCount (Mismatched busy count mods!), at dom/workers/WorkerPrivate.cpp:1760

Status: RESOLVED FIXED

(Core :: DOM: Workers, defect, P2)

Description

[Tyson Smith [:tsmith]]

Attachment: testcase.html

Reproduced with m-c 20200121-875ae8e3ce6f

Assertion failure: aIncrease || mBusyCount (Mismatched busy count mods!), at dom/workers/WorkerPrivate.cpp:1760

0|0|libxul.so|mozilla::dom::WorkerPrivate::ModifyBusyCount(bool)|hg:hg.mozilla.org/mozilla
-central:dom/workers/WorkerPrivate.cpp:875ae8e3ce6f706c2e48d04375145d08b94e05bf|1760|0x29
0|1|libxul.so|mozilla::dom::WorkerRunnable::Run()|hg:hg.mozilla.org/mozilla-central:dom/wo
rkers/WorkerRunnable.cpp:875ae8e3ce6f706c2e48d04375145d08b94e05bf|369|0x14
0|2|libxul.so|mozilla::ThrottledEventQueue::Inner::ExecuteRunnable()|hg:hg.mozilla.org/moz
illa-central:xpcom/threads/ThrottledEventQueue.cpp:875ae8e3ce6f706c2e48d04375145d08b94e05b
f|252|0xe
0|3|libxul.so|mozilla::ThrottledEventQueue::Inner::Executor::Run()|hg:hg.mozilla.org/mozilla-central:xpcom/threads/ThrottledEventQueue.cpp:875ae8e3ce6f706c2e48d04375145d08b94e05bf|
80|0x11
0|4|libxul.so|nsThread::ProcessNextEvent(bool, bool*)|hg:hg.mozilla.org/mozilla-central:xpcom/threads/nsThread.cpp:875ae8e3ce6f706c2e48d04375145d08b94e05bf|1220|0x11
0|5|libxul.so|NS_ProcessNextEvent(nsIThread*, bool)|hg:hg.mozilla.org/mozilla-central:xpcom/threads/nsThreadUtils.cpp:875ae8e3ce6f706c2e48d04375145d08b94e05bf|486|0xc
0|6|libxul.so|nsThread::Shutdown()|hg:hg.mozilla.org/mozilla-central:xpcom/threads/nsThread.cpp:875ae8e3ce6f706c2e48d04375145d08b94e05bf|909|0x5c
0|7|libxul.so|mozilla::ChildProfilerController::ShutdownAndMaybeGrabShutdownProfileFirst(nsTString<char>*)|hg:hg.mozilla.org/mozilla-central:tools/profiler/gecko/ChildProfilerController.cpp:875ae8e3ce6f706c2e48d04375145d08b94e05bf|62|0x11
0|8|libxul.so|mozilla::ChildProfilerController::GrabShutdownProfileAndShutdown()|hg:hg.mozilla.org/mozilla-central:tools/profiler/gecko/ChildProfilerController.cpp:875ae8e3ce6f706c2e48d04375145d08b94e05bf|44|0xb
0|9|libxul.so|mozilla::dom::ContentChild::ShutdownInternal()|hg:hg.mozilla.org/mozilla-central:dom/ipc/ContentChild.cpp:875ae8e3ce6f706c2e48d04375145d08b94e05bf|3125|0x13
0|10|libxul.so|mozilla::dom::ContentChild::RecvShutdown()|hg:hg.mozilla.org/mozilla-central:dom/ipc/ContentChild.cpp:875ae8e3ce6f706c2e48d04375145d08b94e05bf|3069|0x8
0|11|libxul.so|mozilla::dom::PContentChild::OnMessageReceived(IPC::Message const&)|s3:gecko-generated-sources:9d707622d978faf4be5548f73a72e51c1038c18a025ff17995f17f48af16ef7f5d9529beb48f5e1e4bdef55edd1890ed556e2b8f4dbd2db654cee7ea764b91cb/ipc/ipdl/PContentChild.cpp:|11193|0x8
0|12|libxul.so|mozilla::ipc::MessageChannel::DispatchAsyncMessage(mozilla::ipc::ActorLifecycleProxy*, IPC::Message const&)|hg:hg.mozilla.org/mozilla-central:ipc/glue/MessageChannel.cpp:875ae8e3ce6f706c2e48d04375145d08b94e05bf|2214|0xd
0|13|libxul.so|mozilla::ipc::MessageChannel::DispatchMessage(IPC::Message&&)|hg:hg.mozilla.org/mozilla-central:ipc/glue/MessageChannel.cpp:875ae8e3ce6f706c2e48d04375145d08b94e05bf|2136|0x5
0|14|libxul.so|mozilla::ipc::MessageChannel::RunMessage(mozilla::ipc::MessageChannel::MessageTask&)|hg:hg.mozilla.org/mozilla-central:ipc/glue/MessageChannel.cpp:875ae8e3ce6f706c2e48d04375145d08b94e05bf|1975|0xb
0|15|libxul.so|mozilla::ipc::MessageChannel::MessageTask::Run()|hg:hg.mozilla.org/mozilla-central:ipc/glue/MessageChannel.cpp:875ae8e3ce6f706c2e48d04375145d08b94e05bf|2006|0xc
0|16|libxul.so|nsThread::ProcessNextEvent(bool, bool*)|hg:hg.mozilla.org/mozilla-central:xpcom/threads/nsThread.cpp:875ae8e3ce6f706c2e48d04375145d08b94e05bf|1220|0x11
0|17|libxul.so|NS_ProcessNextEvent(nsIThread*, bool)|hg:hg.mozilla.org/mozilla-central:xpcom/threads/nsThreadUtils.cpp:875ae8e3ce6f706c2e48d04375145d08b94e05bf|486|0xc
0|18|libxul.so|mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*)|hg:hg.mozilla.org/mozilla-central:ipc/glue/MessagePump.cpp:875ae8e3ce6f706c2e48d04375145d08b94e05bf|87|0x7
0|19|libxul.so|MessageLoop::RunInternal()|hg:hg.mozilla.org/mozilla-central:ipc/chromium/src/base/message_loop.cc:875ae8e3ce6f706c2e48d04375145d08b94e05bf|315|0x17
0|20|libxul.so|MessageLoop::Run()|hg:hg.mozilla.org/mozilla-central:ipc/chromium/src/base/message_loop.cc:875ae8e3ce6f706c2e48d04375145d08b94e05bf|290|0x8
0|21|libxul.so|nsBaseAppShell::Run()|hg:hg.mozilla.org/mozilla-central:widget/nsBaseAppShell.cpp:875ae8e3ce6f706c2e48d04375145d08b94e05bf|137|0xd
0|22|libxul.so|XRE_RunAppShell()|hg:hg.mozilla.org/mozilla-central:toolkit/xre/nsEmbedFunctions.cpp:875ae8e3ce6f706c2e48d04375145d08b94e05bf|943|0xe
0|23|libxul.so|mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate*)|hg:hg.mozilla.org/mozilla-central:ipc/glue/MessagePump.cpp:875ae8e3ce6f706c2e48d04375145d08b94e05bf|237|0x5
0|24|libxul.so|MessageLoop::RunInternal()|hg:hg.mozilla.org/mozilla-central:ipc/chromium/src/base/message_loop.cc:875ae8e3ce6f706c2e48d04375145d08b94e05bf|315|0x17
0|25|libxul.so|MessageLoop::Run()|hg:hg.mozilla.org/mozilla-central:ipc/chromium/src/base/message_loop.cc:875ae8e3ce6f706c2e48d04375145d08b94e05bf|290|0x8
0|27|firefox-bin|content_process_main(mozilla::Bootstrap*, int, char**)|hg:hg.mozilla.org/
mozilla-central:ipc/contentproc/plugin-container.cpp:875ae8e3ce6f706c2e48d04375145d08b94e05bf|56|0x11
0|28|firefox-bin|main|hg:hg.mozilla.org/mozilla-central:browser/app/nsBrowserApp.cpp:875ae8e3ce6f706c2e48d04375145d08b94e05bf|303|0x19
0|29|libc-2.27.so||||0x21b97
0|30|firefox-bin|<name omitted>|hg:hg.mozilla.org/mozilla-central:mfbt/UniquePtr.h:875ae8e3ce6f706c2e48d04375145d08b94e05bf|275|0x17

Comment 1

[Tyson Smith [:tsmith]]

Attachment: a.js

Comment 2

[Tyson Smith [:tsmith]]

The test case also requires dom.allow_scripts_to_close_windows=true and dom.disable_open_during_load=false to trigger the issue.

Comment 3

[Tyson Smith [:tsmith]]

A Pernosco session is available here: https://pernos.co/debug/x1H3yd2qNkBUcLAIep3L7g/index.html

Comment 5

[Perry Jiang [:perry] [no longer employee, use ni?]]

Amazing, looks like a dupe of 1564700 (or rather 1564700 is a dupe of this) which I couldn't ever get to reproduce.

Note to self to re-enable test_bug949946.html.

Comment 6

[Intermittent Failures Robot]

1 failures in 4004 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1610585&startday=2020-01-20&endday=2020-01-26&tree=all

Comment 7

[Perry Jiang [:perry] [no longer employee, use ni?]]

ni? myself try to reproduce, seems like there's no volume right now.

Comment 8

[Tyson Smith [:tsmith]]

FWIW the fuzzers are hitting this multiple times a day. If you need a new test case please let me know.

Comment 9

[Perry Jiang [:perry] [no longer employee, use ni?]]

Timeline of WorkerPrivate::mBusyCount increment/decrements:

• Increment (CompileScriptRunnable PreDispatch)

• Increment (InitializeWorkerRunnable PreDispatch)

• Increment (LoadAllScript's StrongWorkerRef ctor)

• Decrement (LoadAllScript's StrongWorkerRef dtor)

• Decrement (CompileScriptRunnable PostRun)

• Decrement (InitializeWorkerRunnable PostRun)

• Increment (MessagePortIdentifierRunnable creates a MessagePort, which creates a StrongWorkerRef)
  (importScript's LoadAllScript's StrongWorkerRef ctor) (no increment busy count b\c MessagePort already has StrongWorkerRef)

• Increment (NotifyRunnable PreDispatch)

• Decrement (NotifyRunnable PostRun) (note: on nested event loop, shouldn't make a difference)

• Decrement (MessagePortIdentifierRunnable's WorkerRunnable::Run called when WorkerPrivate::ClearMainEventQueue is on the stack) *** (NOT BALANCED!)

(MessagePort's StrongWorkerRef is destroyed) (no decrement busy count b\c importScript's LoadAllScripts already has StrongWorkerRef)

• Decrement (importScript's LoadAllScript's StrongWorkerRef dtor) mBusyCount == 0 -> assertion failure

Comment 10

[Perry Jiang [:perry] [no longer employee, use ni?]]

Attachment: Bug 1610585 - MessagePortIdentifierRunnable ensure balanced busy count modifications r=#dom-workers-and-storage-reviewers

If the MessagePortIdentifierRunnable's WorkerRunnable::Run is called when the
WorkerPrivate is doing ClearMainEventQueue, WorkerRunnable::Run will decrement
the busy count. It should have already been incremented by a (successful)
WorkerRunnable::PreDispatch though, but MessagePortIdentifierRunnable's
PreDispatch doesn't do anything, so in this case the busy count modifications
are not balanced.

It seemed like MessagePortIdentifierRunnable tried to avoid modifying the
busy count at all, but it doesn't work in the case described above. So, the
patch just makes MessagePortIdentifierRunnable do the normal busy count
modifications. (I also don't see any "bad assertions" referenced by the empty
{Pre,Post}{Dispatch,Run} methods.)

Comment 11

[Perry Jiang [:perry] [no longer employee, use ni?]]

With the patch applied I can't reproduce the crash on changeset 875ae8e3ce6f (without the patch I can reproduce).

Comment 12

[Intermittent Failures Robot]

1 failures in 4690 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• linux1804-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1610585&startday=2020-02-10&endday=2020-02-16&tree=all

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/eb1765cff1be
MessagePortIdentifierRunnable ensure balanced busy count modifications r=dom-workers-and-storage-reviewers,asuth

Comment 14

[Oana Pop-Rus]

https://hg.mozilla.org/mozilla-central/rev/eb1765cff1be

Comment 15

[Ryan VanderMeulen [:RyanVM]]

Is there a real-world impact from this bug which would make us want to consider Beta uplift or can this fix ride 75 to release?

Comment 16

[Perry Jiang [:perry] [no longer employee, use ni?]]

I don't think there will be a real world impact. The bug occurs when a shared worker terminates, but it should still terminate completely even with the bug.

Comment 17

[Intermittent Failures Robot]

1 failures in 4045 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 1

Platform breakdown:

• linux64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1610585&startday=2020-02-17&endday=2020-02-23&tree=all