1753316 - Intermittent Assertion failure: !sHasShutdown, at /builds/worker/checkouts/gecko/dom/media/MediaManager.cpp:3106

Status: RESOLVED FIXED

(Core :: WebRTC: Audio/Video, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: ncsoregi [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=366509352&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/GinySsMhRj-p3rbdWrr3kw/runs/0/artifacts/public/logs/live_backing.log

---

[task 2022-02-02T21:58:14.998Z] 21:58:14     INFO - TEST-START | /xhr/access-control-basic-post-success-no-content-type.htm
[task 2022-02-02T21:58:15.005Z] 21:58:15     INFO - Closing window 21b0783e-fea4-4084-a5e8-946da120ff19
[task 2022-02-02T21:58:15.036Z] 21:58:15     INFO - Setting pref privacy.resistFingerprinting.reduceTimerPrecision.jitter (false)
[task 2022-02-02T21:58:15.081Z] 21:58:15     INFO - Setting pref privacy.reduceTimerPrecision (false)
[task 2022-02-02T21:58:15.100Z] 21:58:15     INFO - Setting pref media.peerconnection.ice.trickle_grace_period (10000)
[task 2022-02-02T21:58:15.124Z] 21:58:15     INFO - Setting pref media.peerconnection.ice.obfuscate_host_addresses (false)
[task 2022-02-02T21:58:15.148Z] 21:58:15     INFO - Closing window 9b27f447-87a7-43ac-8b1e-4d0ad3f4b9ed
[task 2022-02-02T21:58:15.206Z] 21:58:15     INFO - PID 18437 | Assertion failure: !sHasShutdown, at /builds/worker/checkouts/gecko/dom/media/MediaManager.cpp:3106
[task 2022-02-02T21:58:15.206Z] 21:58:15     INFO - PID 18437 | #01: ???[/builds/worker/workspace/build/application/firefox/libxul.so +0x985fd58]
[task 2022-02-02T21:58:15.215Z] 21:58:15     INFO - PID 18437 | #02: ???[/builds/worker/workspace/build/application/firefox/libxul.so +0x99f67a5]
[task 2022-02-02T21:58:15.215Z] 21:58:15     INFO - PID 18437 | #03: ???[/builds/worker/workspace/build/application/firefox/libxul.so +0x40f2367]
[task 2022-02-02T21:58:15.215Z] 21:58:15     INFO - PID 18437 | #04: ???[/builds/worker/workspace/build/application/firefox/libxul.so +0x411bfbc]
[task 2022-02-02T21:58:15.215Z] 21:58:15     INFO - PID 18437 | #05: ???[/builds/worker/workspace/build/application/firefox/libxul.so +0x410e01c]
[task 2022-02-02T21:58:15.215Z] 21:58:15     INFO - PID 18437 | #06: ???[/builds/worker/workspace/build/application/firefox/libxul.so +0x411890d]
[task 2022-02-02T21:58:15.215Z] 21:58:15     INFO - PID 18437 | #07: ???[/builds/worker/workspace/build/application/firefox/libxul.so +0x52bb1f9]
[task 2022-02-02T21:58:15.215Z] 21:58:15     INFO - PID 18437 | #08: ???[/builds/worker/workspace/build/application/firefox/libxul.so +0x51ca7e2]
[task 2022-02-02T21:58:15.216Z] 21:58:15     INFO - PID 18437 | #09: ???[/builds/worker/workspace/build/application/firefox/libxul.so +0x410674a]
[task 2022-02-02T21:58:15.216Z] 21:58:15     INFO - PID 18437 | #10: ???[/builds/worker/workspace/build/application/firefox/libnspr4.so +0x7eadf]
[task 2022-02-02T21:58:15.216Z] 21:58:15     INFO - PID 18437 | #11: ???[/lib/x86_64-linux-gnu/libpthread.so.0 +0x76db]
[task 2022-02-02T21:58:15.217Z] 21:58:15     INFO - PID 18437 | #12: clone[/lib/x86_64-linux-gnu/libc.so.6 +0x121a3f]
[task 2022-02-02T21:58:15.217Z] 21:58:15     INFO - PID 18437 | #13: ??? (???:???)
[task 2022-02-02T21:58:15.217Z] 21:58:15     INFO - PID 18437 | AddressSanitizer:DEADLYSIGNAL
[task 2022-02-02T21:58:15.218Z] 21:58:15     INFO - PID 18437 | =================================================================
[task 2022-02-02T21:58:15.223Z] 21:58:15    ERROR - PID 18437 | ==22170==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000001 (pc 0x7f42c8d23d88 bp 0x7f42ae825bf0 sp 0x7f42ae825ae0 T66)
[task 2022-02-02T21:58:15.248Z] 21:58:15     INFO - PID 18437 | ==22170==The signal is caused by a WRITE memory access.
[task 2022-02-02T21:58:15.249Z] 21:58:15     INFO - PID 18437 | ==22170==Hint: address points to the zero page.
[task 2022-02-02T21:58:15.583Z] 21:58:15     INFO - .
[task 2022-02-02T21:58:15.583Z] 21:58:15     INFO - TEST-OK | /xhr/access-control-basic-post-success-no-content-type.htm | took 585ms

Comment 1

[Karl Tomlinson (:karlt)]

The assertion tests sHasShutdown in GetBackend() on the media thread, even though the variable is set on the main thread.
In this report, GetBackend() is called from the lambda task for EnumerateRawDevices().

Assertion failure: !sHasShutdown, at /builds/worker/checkouts/gecko/dom/media/MediaManager.cpp:3106

==22170==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000001 (pc 0x7f42c8d23d88 bp 0x7f42ae825bf0 sp 0x7f42ae825ae0 T66)

#0 0x7f42c8d23d88 in mozilla::MediaManager::GetBackend() /builds/worker/checkouts/gecko/dom/media/MediaManager.cpp:3105:5
#1 0x7f42c8eba7a4 in operator() /builds/worker/checkouts/gecko/dom/media/MediaManager.cpp:1835:34
#2 0x7f42c8eba7a4 in mozilla::media::LambdaTask<mozilla::MediaManager::EnumerateRawDevices(mozilla::dom::MediaSourceEnum, mozilla::dom::MediaSourceEnum, mozilla::EnumSet<mozilla::MediaManager::EnumerationFlag, unsigned int>)::$_89>::Run() /builds/worker/workspace/obj-build/dist/include/mozilla/media/MediaTaskUtils.h:36:5
#3 0x7f42c35b6366 in mozilla::TaskQueue::Runner::Run() /builds/worker/checkouts/gecko/xpcom/threads/TaskQueue.cpp:206:20
#4 0x7f42c35dffbb in nsThreadPool::Run() /builds/worker/checkouts/gecko/xpcom/threads/nsThreadPool.cpp:305:14
#5 0x7f42c35d201b in nsThread::ProcessNextEvent(bool, bool*) /builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp:1189:16
#6 0x7f42c35dc90c in NS_ProcessNextEvent(nsIThread*, bool) /builds/worker/checkouts/gecko/xpcom/threads/nsThreadUtils.cpp:467:10
#7 0x7f42c477f1f8 in mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) /builds/worker/checkouts/gecko/ipc/glue/MessagePump.cpp:300:20
#8 0x7f42c468e7e1 in RunInternal /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:331:10
#9 0x7f42c468e7e1 in RunHandler /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:324:3
#10 0x7f42c468e7e1 in MessageLoop::Run() /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:306:3
#11 0x7f42c35ca749 in nsThread::ThreadFunc(void*) /builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp:391:10
#12 0x7f42e53d8ade in _pt_root /builds/worker/checkouts/gecko/nsprpub/pr/src/pthreads/ptthread.c:201:5

Comment 2

[Karl Tomlinson (:karlt)]

https://hg.mozilla.org/mozilla-central/rev/7aec151a5d61ac96d517adc73f723db0a73ae0b6#l1.205

Comment 3

[Intermittent Failures Robot]

10 failures in 3165 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 8
• mozilla-central: 2

Platform and build breakdown:

• linux1804-64-asan-qr: 10
  • opt: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1753316&startday=2022-01-31&endday=2022-02-06&tree=all

Comment 4

[Intermittent Failures Robot]

11 failures in 3711 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1
• autoland: 6
• mozilla-central: 3
• try: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 11
  • opt: 11

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1753316&startday=2022-02-07&endday=2022-02-13&tree=all

Comment 5

[Intermittent Failures Robot]

9 failures in 3281 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4
• mozilla-beta: 2
• try: 1
• mozilla-central: 2

Platform and build breakdown:

• linux1804-64-asan-qr: 9
  • opt: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1753316&startday=2022-02-14&endday=2022-02-20&tree=all

Comment 6

[Intermittent Failures Robot]

11 failures in 2992 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 7
• mozilla-central: 2
• mozilla-beta: 1
• try: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 11
  • opt: 11

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1753316&startday=2022-02-21&endday=2022-02-27&tree=all

Comment 7

[Karl Tomlinson (:karlt)]

https://treeherder.mozilla.org/jobs?repo=try&revision=cbe960dfb4e0bad9b5e1c564168b9ae78a9560cd

Comment 8

[Karl Tomlinson (:karlt)]

Attachment: Bug 1753316 remove pre-TaskQueue comments relating to Chromium Thread shutdown r?jib

Obsolete since
https://hg.mozilla.org/mozilla-central/rev/4af433832f071364df98d95bbef0a107d4dbd336#l1.299

Depends on D140030

Comment 9

[Karl Tomlinson (:karlt)]

Attachment: Bug 1753316 allow GetBackend() to continue as normal after main thread shutdown r?jib

GetBackend() is used only on the media thread, on which mBackend is shut down
after main thread MediaManager shutdown.

Creating a backend on the media thread shortly after sHasMainThreadShutdown is
set on the main thread is not necessarily any different from creating shortly
before sHasMainThreadShutdown is set. The simplest way to handle these is to
take the same code path in each case.

MediaEventListener::operator=() will assert if another backend is created
after a previous backend is shutdown.
https://searchfox.org/mozilla-central/rev/8e1eb5cbd89a2455c34032d921f8de28261d016b/dom/media/MediaEventSource.h#336
https://searchfox.org/mozilla-central/rev/2a0b0ababd4541ecffb74cbe0820a9d0a25da636/dom/media/MediaManager.cpp#3112

Depends on D140031

Comment 10

[Karl Tomlinson (:karlt)]

Attachment: Bug 1753316 Remove main thread block on MediaManager TaskQueue shutdown r?jib

The main thread should not block on the media task queue because media tasks
block on the main thread. https://searchfox.org/mozilla-central/rev/8e1eb5cbd89a2455c34032d921f8de28261d016b/dom/media/MediaManager.h#188

This synchronous TaskQueue shutdown was unreviewed.
https://phabricator.services.mozilla.com/D82159?vs=308672&id=309405#2539546

It was added apparently on a concern of a race of completion of MediaManager
shutdown with CubebDeviceEnumerator and/or persistent MTA thread singleton
shutdown. However both CubebDeviceEnumerator and MTA thread use
ShutdownPhase::XPCOMShutdownThreads, which will not race with MediaManager's
earlier xpcom-will-shutdown barrier.
https://searchfox.org/mozilla-central/rev/8e1eb5cbd89a2455c34032d921f8de28261d016b/dom/media/webrtc/CubebDeviceEnumerator.cpp#35
https://searchfox.org/mozilla-central/rev/b697834e78a3ef7613e2fa57c07624b1d9d1c909/ipc/mscom/EnsureMTA.cpp#177
https://searchfox.org/mozilla-central/rev/b697834e78a3ef7613e2fa57c07624b1d9d1c909/dom/media/systemservices/MediaUtils.cpp#24

Depends on D140032

Comment 11

[Karl Tomlinson (:karlt)]

Attachment: Bug 1753316 Use NS_NewRunnableFunction to simplify MediaEngine shutdown task r?jib

Depends on D140033

Comment 12

[Karl Tomlinson (:karlt)]

Attachment: Bug 1753316 remove unnecessary sSingletonMutex r?jib

This was added due to a mistaken reading of the thread on which sSingleton was
cleared.
https://phabricator.services.mozilla.com/D48512#1545100

Depends on D140034

Comment 13

[Intermittent Failures Robot]

1 failures in 3063 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1753316&startday=2022-02-28&endday=2022-03-06&tree=all

Comment 14

[Intermittent Failures Robot]

1 failures in 2837 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1753316&startday=2022-03-07&endday=2022-03-13&tree=all

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/52915824185a
remove pre-TaskQueue comments relating to Chromium Thread shutdown r=jib
https://hg.mozilla.org/integration/autoland/rev/86a0271dcefe
allow GetBackend() to continue as normal after main thread shutdown r=jib

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ea0660396ced
Remove main thread block on MediaManager TaskQueue shutdown r=jib
https://hg.mozilla.org/integration/autoland/rev/e66787ae5c56
Use NS_NewRunnableFunction to simplify MediaEngine shutdown task r=jib

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/eae985ca32d8
remove unnecessary sSingletonMutex r=jib

Comment 18

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/52915824185a
https://hg.mozilla.org/mozilla-central/rev/86a0271dcefe
https://hg.mozilla.org/mozilla-central/rev/ea0660396ced
https://hg.mozilla.org/mozilla-central/rev/e66787ae5c56
https://hg.mozilla.org/mozilla-central/rev/eae985ca32d8

Comment 19

[BugBot [:suhaib / :marco/ :calixte]]

Since the status are different for nightly and release, what's the status for beta?
For more information, please visit auto_nag documentation.

Comment 20

[Karl Tomlinson (:karlt)]

Doesn't affect release builds, where the content process exits in ContentChild::ActorDestroy().