1628842 - Crash in [@ <name omitted> | mozilla::dom::AutoJSAPI::~AutoJSAPI] while using audio worklet on Zoom

Status: RESOLVED FIXED

(Core :: Web Audio, defect, P1)

Description

[Rob Wu [:robwu]]

This bug is for crash report bp-27edd074-a3ba-46ae-ace5-575f40200409.

I am in contact with the user who experienced the crash; they claimed that they were having a Zoom call:
"About 20 minutes in on a listen-only video call, audio playback suddenly stopped, so I tried to restart it by stopping the audio first, and then joining it again. Once I clicked "Join audio from computer", the tab immediately crashed."

Top 10 frames of crashing thread:

0 XUL <name omitted> js/src/jsapi.cpp:547
1 XUL mozilla::dom::AutoJSAPI::~AutoJSAPI dom/script/ScriptSettings.cpp:260
2 XUL mozilla::WorkletImpl::GetGlobalScope dom/worklet/WorkletImpl.cpp:90
3 XUL mozilla::dom::ExecutionRunnable::RunOnWorkletThread dom/worklet/Worklet.cpp:387
4 XUL mozilla::dom::ExecutionRunnable::Run dom/worklet/Worklet.cpp:347
5 XUL mozilla::MediaTrackGraphImpl::OneIterationImpl dom/media/MediaTrackGraph.cpp:1392
6 XUL mozilla::GraphRunner::Run dom/media/GraphRunner.cpp:114
7 XUL nsThread::ProcessNextEvent xpcom/threads/nsThread.cpp:1200
8 XUL mozilla::ipc::MessagePumpForNonMainThreads::Run ipc/glue/MessagePump.cpp:332
9 XUL MessageLoop::Run ipc/chromium/src/base/message_loop.cc:290

Comment 1

[Rob Wu [:robwu]]

Karl, could you take a look?

Comment 2

[Karl Tomlinson (:karlt)]

Thank you for filing. The stack is not making much sense to me because I haven't found anywhere that the JSAutoRealm constructor might be called while in the AutoJSAPI destructor. Perhaps "name omitted" for the JSAutoRealm constructor code might indicate that this is not really in the JSAutoRealm constructor. The other stack frames are derived from "frame_pointer".

There is a possibility of crashes near here if the EnsureCycleCollectedJSContext() call from RunOnWorkletThread() failed to create a JSContext. That's not expected to fail except on OOM and there is 1.7 GB of available physical memory in the crash report. Such crashes would also be expected to show up in JS_NewGlobalObject() before reaching the AutoJSAPI destructor.

Comment 3

[Karl Tomlinson (:karlt)]

Only one crash report so far, but I'll keep a close eye on this.
I tried leaving and joining computer audio many times without succeeding in reproducing.

Comment 4

[Matthew Gregan [:kinetik]]

Using the steps described here, dwarfdump suggests the crash is in an inlined call to GetObjectClass:

0x31bd06c0: DW_TAG_compile_unit
              DW_AT_producer    ("clang version 9.0.1 ")
              DW_AT_language    (DW_LANG_C_plus_plus)
              DW_AT_name        ("/builds/worker/workspace/obj-build/js/src/Unified_cpp_js_src11.cpp")
              DW_AT_stmt_list   (0x04892e57)
              DW_AT_comp_dir    ("/builds/worker/workspace/obj-build/js/src")
              DW_AT_GNU_pubnames        (0x01)
              DW_AT_APPLE_optimized     (0x01)
              DW_AT_low_pc      (0x0000000004226950)
              DW_AT_high_pc     (0x0000000004264550)

0x31d1f0df:   DW_TAG_inlined_subroutine
                DW_AT_abstract_origin   (0x0000000031cd8a53 "_ZN2js14GetObjectClassEPK8JSObject")
                DW_AT_low_pc    (0x0000000004227742)
                DW_AT_high_pc   (0x0000000004227748)
                DW_AT_call_file ("/builds/worker/workspace/obj-build/dist/include/js/Proxy.h")
                DW_AT_call_line (382)
                DW_AT_call_column       (0x0a)
Line info: file 'checkouts/gecko/js/src/jsfriendapi.h', line 599, column 56, start line 598

...where the previous frame is in an inlined call to AutoJSAPI::ReportException where a JSAutoRealm is constructed:

0x24bc38d2: DW_TAG_compile_unit
              DW_AT_producer    ("clang version 9.0.1 ")
              DW_AT_language    (DW_LANG_C_plus_plus)
              DW_AT_name        ("/builds/worker/workspace/obj-build/dom/script/Unified_cpp_dom_script0.cpp")
              DW_AT_stmt_list   (0x03249383)
              DW_AT_comp_dir    ("/builds/worker/workspace/obj-build/dom/script")
              DW_AT_GNU_pubnames        (0x01)
              DW_AT_APPLE_optimized     (0x01)
              DW_AT_low_pc      (0x0000000002ced790)
              DW_AT_high_pc     (0x0000000002d07048)

0x24cb5b3f:   DW_TAG_inlined_subroutine
                DW_AT_abstract_origin   (0x0000000024ca42d1 "_ZN7mozilla3dom9AutoJSAPI15ReportExceptionEv")
                DW_AT_ranges    (0x02baf580
                   [0x0000000002cf969e, 0x0000000002cf96ab)
                   [0x0000000002cf98ad, 0x0000000002cf9a6a)
                   [0x0000000002cf9a7b, 0x0000000002cf9c94))
                DW_AT_call_file ("/builds/worker/checkouts/gecko/dom/script/ScriptSettings.cpp")
                DW_AT_call_line (270)
                DW_AT_call_column       (0x03)
Line info: file 'checkouts/gecko/dom/script/ScriptSettings.cpp', line 494, column 15, start line 471

GetObjectClass is reachable via the IsCrossCompartmentWrapper assert in the JSAutoRealm constructor, so this stack decoding seems plausible.

Comment 5

[Karl Tomlinson (:karlt)]

Thank you very much, Matthew. That is much clearer.

I don't know whether there are any reasons why creation of the global might fail other than OOM, but there is a path where the AutoJSAPI may exit with an exception but no global.

AutoJSAPI::ReportException() has measures to deal with that on the main thread, but not on a worklet thread.

Comment 6

[Karl Tomlinson (:karlt)]

Attachment: Bug 1628842 give up on reporting off-main-thread exceptions when global creation fails r?baku

and there is no fallback global available.

I assume there is little value in attempting to create another global if a
previous attempt failed.

Comment 7

[Karl Tomlinson (:karlt)]

Attachment: Bug 1628842 reject Worklet#addModule() promise with DOM UnknownError on failure to create global r?baku

The UnknownError is chosen based on
https://heycam.github.io/webidl/#unknownerror and suspected OOM.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/af0a1889007f
reject Worklet#addModule() promise with DOM UnknownError on failure to create global r=baku
https://hg.mozilla.org/integration/autoland/rev/df0094f810e1
give up on reporting off-main-thread exceptions when global creation fails r=baku

Comment 9

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/af0a1889007f
https://hg.mozilla.org/mozilla-central/rev/df0094f810e1

Comment 10

[Ryan VanderMeulen [:RyanVM]]

Is this something which affects 76 also?

Comment 11

[Karl Tomlinson (:karlt)]

Comment on attachment 9141148 [details]
Bug 1628842 reject Worklet#addModule() promise with DOM UnknownError on failure to create global r?baku

Beta/Release Uplift Approval Request

• User impact if declined: null-deref crash. The reason for the crash is not certain, but could occur in OOM situations.
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): The risk is low because the code path is not usually reached.

The feature is covered by automated tests, but this particular code path is
not covered by tests because we don't know a way to trigger this path.

Risk would involve recovery from error on this path. That should also be low because the error path soon merges with other paths, which do have testing.

• String changes made/needed: None.

Comment 12

[Karl Tomlinson (:karlt)]

Three of four 76.0b5 [@ JSAutoRealm::JSAutoRealm ] reports sampled are this crash.
e.g. https://crash-stats.mozilla.org/report/index/1f78da41-9649-4139-8e6a-a79350200420

Memory usage in the reports seems to indicate this is not OOM.

Comment 13

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9140959 [details]
Bug 1628842 give up on reporting off-main-thread exceptions when global creation fails r?baku

Fixes a crash with AudioWorklets, which are riding the 76 train to release. Approved for 76.0b7.

Comment 14

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/72c2deca675c
https://hg.mozilla.org/releases/mozilla-beta/rev/151ce7a1eabf