Closed Bug 1551615 Opened 5 years ago Closed 4 years ago

Intermittent SUMMARY: AddressSanitizer: SEGV /builds/worker/workspace/build/src/ipc/glue/MessageChannel.cpp:764:9 in mozilla::ipc::MessageChannel::Clear()

Categories

(Core :: Audio/Video: Playback, defect, P2)

Product:
Core
Component:
Audio/Video: Playback
Type:
defect

Tracking

()

Status:
VERIFIED FIXED
Milestone:
83 Branch
Tracking Flags:
Tracking Status
firefox-esr68 - wontfix
firefox-esr78 83+ fixed
firefox74 --- wontfix
firefox75 --- wontfix
firefox76 + wontfix
firefox77 + wontfix
firefox78 + wontfix
firefox79 + wontfix
firefox80 + wontfix
firefox81 - wontfix
firefox82 - wontfix
firefox83 + verified

People

(Reporter: intermittent-bug-filer, Assigned: bryce)

References

Details

(Keywords: intermittent-failure, regression, sec-moderate, Whiteboard: [post-critsmash-triage][adv-main83+r][adv-esr78.5+r])

Attachments

(6 files, 1 obsolete file)

Bug 1551615 - Make PGMPContent protocol refcounted. r?nika
47 bytes, text/x-phabricator-request
Details | Review
Bug 1551615 - Disabled clearkey-mp4-playback-temporary-multikey-sequential-readyState.https.html and clearkey-mp4-update-disallowed-input.https.html on Linux. r?#intermittent-reviewers
47 bytes, text/x-phabricator-request
Details | Review
Bug 1551615 - Make PGMPService protocol refcounted. r?nika
47 bytes, text/x-phabricator-request
Details | Review
Bug 1551615 - Have GeckoMediaPluginServiceChild use async shutdown blocker.
47 bytes, text/x-phabricator-request
Details | Review
testcase.html
493 bytes, text/html
Details
Bug1551615-esr78rebase.patch
50.43 KB, patch
RyanVM
: approval-mozilla-esr78+
Details | Diff | Splinter Review

Filed by: csabou [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=246387590&repo=autoland
Full log: https://queue.taskcluster.net/v1/task/F7-g4zURSsKOemLz6iIu3g/runs/0/artifacts/public/logs/live_backing.log

[task 2019-05-14T14:09:46.214Z] 14:09:46 INFO - TEST-OK | /encrypted-media/clearkey-mp4-unique-origin.https.html | took 1551ms
[task 2019-05-14T14:09:46.216Z] 14:09:46 INFO - TEST-START | /encrypted-media/clearkey-mp4-update-disallowed-input.https.html
[task 2019-05-14T14:09:46.220Z] 14:09:46 INFO - Closing window 81604378625
[task 2019-05-14T14:09:47.298Z] 14:09:47 INFO - PID 27123 | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2019-05-14T14:09:47.298Z] 14:09:47 INFO - PID 27123 | AddressSanitizer:DEADLYSIGNAL
[task 2019-05-14T14:09:47.299Z] 14:09:47 INFO - PID 27123 | =================================================================
[task 2019-05-14T14:09:47.300Z] 14:09:47 ERROR - PID 27123 | ==29121==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000001 (pc 0x7f89334149a7 bp 0x7ffde5ddfce0 sp 0x7ffde5ddfba0 T0)
[task 2019-05-14T14:09:47.301Z] 14:09:47 INFO - PID 27123 | ==29121==The signal is caused by a WRITE memory access.
[task 2019-05-14T14:09:47.303Z] 14:09:47 INFO - PID 27123 | ==29121==Hint: address points to the zero page.
[task 2019-05-14T14:09:48.069Z] 14:09:48 INFO - PID 27123 | #0 0x7f89334149a6 in mozilla::ipc::MessageChannel::Clear() /builds/worker/workspace/build/src/ipc/glue/MessageChannel.cpp:764:9
[task 2019-05-14T14:09:48.069Z] 14:09:48 INFO - PID 27123 | #1 0x7f89334125d8 in mozilla::ipc::MessageChannel::~MessageChannel() /builds/worker/workspace/build/src/ipc/glue/MessageChannel.cpp:628:3
[task 2019-05-14T14:09:48.069Z] 14:09:48 INFO - PID 27123 | #2 0x7f893343eecd in mozilla::ipc::IToplevelProtocol::ToplevelState::~ToplevelState() /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/ipc/ProtocolUtils.h:406:9
[task 2019-05-14T14:09:48.070Z] 14:09:48 INFO - PID 27123 | #3 0x7f893343f52d in mozilla::ipc::IToplevelProtocol::ToplevelState::~ToplevelState() /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/ipc/ProtocolUtils.h:406:9
[task 2019-05-14T14:09:48.073Z] 14:09:48 INFO - PID 27123 | #4 0x7f8933437a3c in operator() /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/UniquePtr.h:486:5
[task 2019-05-14T14:09:48.073Z] 14:09:48 INFO - PID 27123 | #5 0x7f8933437a3c in reset /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/UniquePtr.h:323
[task 2019-05-14T14:09:48.074Z] 14:09:48 INFO - PID 27123 | #6 0x7f8933437a3c in operator= /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/UniquePtr.h:296
[task 2019-05-14T14:09:48.074Z] 14:09:48 INFO - PID 27123 | #7 0x7f8933437a3c in mozilla::ipc::IToplevelProtocol::~IToplevelProtocol() /builds/worker/workspace/build/src/ipc/glue/ProtocolUtils.cpp:577
[task 2019-05-14T14:09:48.116Z] 14:09:48 INFO - PID 27123 | #8 0x7f893929a93d in mozilla::gmp::GMPContentParent::~GMPContentParent() /builds/worker/workspace/build/src/dom/media/gmp/GMPContentParent.cpp:35:39
[task 2019-05-14T14:09:48.132Z] 14:09:48 INFO - PID 27123 | #9 0x7f893225a17c in operator() /builds/worker/workspace/build/src/xpcom/ds/PLDHashTable.cpp:304:7
[task 2019-05-14T14:09:48.132Z] 14:09:48 INFO - PID 27123 | #10 0x7f893225a17c in ForEachSlot<(lambda at /builds/worker/workspace/build/src/xpcom/ds/PLDHashTable.cpp:302:51)> /builds/worker/workspace/build/src/obj-firefox/dist/include/PLDHashTable.h:359
[task 2019-05-14T14:09:48.132Z] 14:09:48 INFO - PID 27123 | #11 0x7f893225a17c in ForEachSlot<(lambda at /builds/worker/workspace/build/src/xpcom/ds/PLDHashTable.cpp:302:51)> /builds/worker/workspace/build/src/obj-firefox/dist/include/PLDHashTable.h:349
[task 2019-05-14T14:09:48.135Z] 14:09:48 INFO - PID 27123 | #12 0x7f893225a17c in PLDHashTable::~PLDHashTable() /builds/worker/workspace/build/src/xpcom/ds/PLDHashTable.cpp:302
[task 2019-05-14T14:09:48.170Z] 14:09:48 INFO - PID 27123 | #13 0x7f89392ebf3b in ~nsTHashtable /builds/worker/workspace/build/src/obj-firefox/dist/include/nsTHashtable.h:384:43
[task 2019-05-14T14:09:48.171Z] 14:09:48 INFO - PID 27123 | #14 0x7f89392ebf3b in ~GMPServiceChild /builds/worker/workspace/build/src/dom/media/gmp/GMPServiceChild.cpp:430
[task 2019-05-14T14:09:48.171Z] 14:09:48 INFO - PID 27123 | #15 0x7f89392ebf3b in mozilla::gmp::GMPServiceChild::~GMPServiceChild() /builds/worker/workspace/build/src/dom/media/gmp/GMPServiceChild.cpp:430
[task 2019-05-14T14:09:48.171Z] 14:09:48 INFO - PID 27123 | #16 0x7f89392ea605 in operator() /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/UniquePtr.h:486:5
[task 2019-05-14T14:09:48.172Z] 14:09:48 INFO - PID 27123 | #17 0x7f89392ea605 in reset /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/UniquePtr.h:323
[task 2019-05-14T14:09:48.174Z] 14:09:48 INFO - PID 27123 | #18 0x7f89392ea605 in operator= /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/UniquePtr.h:296
[task 2019-05-14T14:09:48.174Z] 14:09:48 INFO - PID 27123 | #19 0x7f89392ea605 in mozilla::gmp::GeckoMediaPluginServiceChild::Observe(nsISupports*, char const*, char16_t const*) /builds/worker/workspace/build/src/dom/media/gmp/GMPServiceChild.cpp:361
[task 2019-05-14T14:09:48.174Z] 14:09:48 INFO - PID 27123 | #20 0x7f89392ea6ec in non-virtual thunk to mozilla::gmp::GeckoMediaPluginServiceChild::Observe(nsISupports*, char const*, char16_t const*) /builds/worker/workspace/build/src/dom/media/gmp/GMPServiceChild.cpp
[task 2019-05-14T14:09:48.176Z] 14:09:48 INFO - PID 27123 | #21 0x7f893227ea41 in nsObserverList::NotifyObservers(nsISupports*, char const*, char16_t const*) /builds/worker/workspace/build/src/xpcom/ds/nsObserverList.cpp:66:19
[task 2019-05-14T14:09:48.177Z] 14:09:48 INFO - PID 27123 | #22 0x7f89322846d5 in nsObserverService::NotifyObservers(nsISupports*, char const*, char16_t const*) /builds/worker/workspace/build/src/xpcom/ds/nsObserverService.cpp:295:19
[task 2019-05-14T14:09:48.177Z] 14:09:48 INFO - PID 27123 | #23 0x7f893241e8bf in mozilla::ShutdownXPCOM(nsIServiceManager*) /builds/worker/workspace/build/src/xpcom/build/XPCOMInit.cpp:633:24
[task 2019-05-14T14:09:48.177Z] 14:09:48 INFO - PID 27123 | #24 0x7f893e57625c in XRE_TermEmbedding() /builds/worker/workspace/build/src/toolkit/xre/nsEmbedFunctions.cpp:222:3
[task 2019-05-14T14:09:48.178Z] 14:09:48 INFO - PID 27123 | #25 0x7f893343cce2 in mozilla::ipc::ScopedXREEmbed::Stop() /builds/worker/workspace/build/src/ipc/glue/ScopedXREEmbed.cpp:90:5
[task 2019-05-14T14:09:48.179Z] 14:09:48 INFO - PID 27123 | #26 0x7f893e576cf5 in XRE_InitChildProcess(int, char**, XREChildData const*) /builds/worker/workspace/build/src/toolkit/xre/nsEmbedFunctions.cpp:761:16
[task 2019-05-14T14:09:48.182Z] 14:09:48 INFO - PID 27123 | #27 0x5582daea43a7 in content_process_main /builds/worker/workspace/build/src/browser/app/../../ipc/contentproc/plugin-container.cpp:56:28
[task 2019-05-14T14:09:48.183Z] 14:09:48 INFO - PID 27123 | #28 0x5582daea43a7 in main /builds/worker/workspace/build/src/browser/app/nsBrowserApp.cpp:263
[task 2019-05-14T14:09:48.240Z] 14:09:48 INFO - PID 27123 | #29 0x7f8952abc82f in __libc_start_main /build/glibc-LK5gWL/glibc-2.23/csu/../csu/libc-start.c:291
[task 2019-05-14T14:09:48.240Z] 14:09:48 INFO - PID 27123 | #30 0x5582dadc5af8 in _start (/builds/worker/workspace/build/application/firefox/firefox+0x2aaf8)
[task 2019-05-14T14:09:48.242Z] 14:09:48 INFO - PID 27123 | AddressSanitizer can not provide additional info.
[task 2019-05-14T14:09:48.243Z] 14:09:48 INFO - PID 27123 | SUMMARY: AddressSanitizer: SEGV /builds/worker/workspace/build/src/ipc/glue/MessageChannel.cpp:764:9 in mozilla::ipc::MessageChannel::Clear()

There are 23 total failures in the last 7 days.

asan: linux64

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=281811584&repo=autoland&lineNumber=7271

Flags: needinfo?(drno)

There have been 25 failures within the last 7 days:

  • 11 failures on Linux x64 asan
  • 14 failures on Linux 18.04 x64 asan opt

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=285431136&repo=mozilla-central&lineNumber=5748

Whiteboard: [stockwell needswork]

There are 22 total failures in the last 7 days on linux1804-64-asan opt

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=286247516&repo=autoland&lineNumber=5713

[task 2020-01-24T00:00:54.729Z] 00:00:54 INFO - TEST-START | /encrypted-media/clearkey-mp4-update-disallowed-input.https.html
[task 2020-01-24T00:00:54.736Z] 00:00:54 INFO - Closing window 95
[task 2020-01-24T00:00:55.842Z] 00:00:55 INFO - PID 12817 | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2020-01-24T00:00:55.842Z] 00:00:55 INFO - PID 12817 | AddressSanitizer:DEADLYSIGNAL
[task 2020-01-24T00:00:55.843Z] 00:00:55 INFO - PID 12817 | =================================================================
[task 2020-01-24T00:00:55.843Z] 00:00:55 ERROR - PID 12817 | ==14758==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000001 (pc 0x7ff1b3a043bf bp 0x7ffe295a9b30 sp 0x7ffe295a9960 T0)
[task 2020-01-24T00:00:55.843Z] 00:00:55 INFO - PID 12817 | ==14758==The signal is caused by a WRITE memory access.
[task 2020-01-24T00:00:55.843Z] 00:00:55 INFO - PID 12817 | ==14758==Hint: address points to the zero page.
[task 2020-01-24T00:00:56.517Z] 00:00:56 INFO - PID 12817 | #0 0x7ff1b3a043be in mozilla::ipc::MessageChannel::Clear() /builds/worker/workspace/build/src/ipc/glue/MessageChannel.cpp:786:9
[task 2020-01-24T00:00:56.518Z] 00:00:56 INFO - PID 12817 | #1 0x7ff1b3a02ed6 in mozilla::ipc::MessageChannel::~MessageChannel() /builds/worker/workspace/build/src/ipc/glue/MessageChannel.cpp:650:3
[task 2020-01-24T00:00:56.518Z] 00:00:56 INFO - PID 12817 | #2 0x7ff1b3a2658e in mozilla::ipc::IToplevelProtocol::~IToplevelProtocol() /builds/worker/workspace/build/src/ipc/glue/ProtocolUtils.cpp:593:1
[task 2020-01-24T00:00:56.534Z] 00:00:56 INFO - PID 12817 | #3 0x7ff1b84dd76d in mozilla::gmp::GMPContentParent::~GMPContentParent() /builds/worker/workspace/build/src/dom/media/gmp/GMPContentParent.cpp:35:39
[task 2020-01-24T00:00:56.535Z] 00:00:56 INFO - PID 12817 | #4 0x7ff1b28a1c23 in operator() /builds/worker/workspace/build/src/xpcom/ds/PLDHashTable.cpp:299:7
[task 2020-01-24T00:00:56.536Z] 00:00:56 INFO - PID 12817 | #5 0x7ff1b28a1c23 in ForEachSlot<(lambda at /builds/worker/workspace/build/src/xpcom/ds/PLDHashTable.cpp:297:51)> /builds/worker/workspace/build/src/obj-firefox/dist/include/PLDHashTable.h:365:9
[task 2020-01-24T00:00:56.536Z] 00:00:56 INFO - PID 12817 | #6 0x7ff1b28a1c23 in ForEachSlot<(lambda at /builds/worker/workspace/build/src/xpcom/ds/PLDHashTable.cpp:297:51)> /builds/worker/workspace/build/src/obj-firefox/dist/include/PLDHashTable.h:355:7
[task 2020-01-24T00:00:56.536Z] 00:00:56 INFO - PID 12817 | #7 0x7ff1b28a1c23 in PLDHashTable::~PLDHashTable() /builds/worker/workspace/build/src/xpcom/ds/PLDHashTable.cpp:297:15
[task 2020-01-24T00:00:56.557Z] 00:00:56 INFO - PID 12817 | #8 0x7ff1b852fb2e in ~nsTHashtable /builds/worker/workspace/build/src/obj-firefox/dist/include/nsTHashtable.h:382:43
[task 2020-01-24T00:00:56.558Z] 00:00:56 INFO - PID 12817 | #9 0x7ff1b852fb2e in ~GMPServiceChild /builds/worker/workspace/build/src/dom/media/gmp/GMPServiceChild.cpp:427:38
[task 2020-01-24T00:00:56.558Z] 00:00:56 INFO - PID 12817 | #10 0x7ff1b852fb2e in mozilla::gmp::GMPServiceChild::~GMPServiceChild() /builds/worker/workspace/build/src/dom/media/gmp/GMPServiceChild.cpp:427:37
[task 2020-01-24T00:00:56.559Z] 00:00:56 INFO - PID 12817 | #11 0x7ff1b852e8ce in operator() /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/UniquePtr.h:487:5
[task 2020-01-24T00:00:56.559Z] 00:00:56 INFO - PID 12817 | #12 0x7ff1b852e8ce in reset /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/UniquePtr.h:324:7
[task 2020-01-24T00:00:56.560Z] 00:00:56 INFO - PID 12817 | #13 0x7ff1b852e8ce in operator= /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/UniquePtr.h:297:5
[task 2020-01-24T00:00:56.562Z] 00:00:56 INFO - PID 12817 | #14 0x7ff1b852e8ce in mozilla::gmp::GeckoMediaPluginServiceChild::Observe(nsISupports*, char const*, char16_t const*) /builds/worker/workspace/build/src/dom/media/gmp/GMPServiceChild.cpp:358:21
[task 2020-01-24T00:00:56.562Z] 00:00:56 INFO - PID 12817 | #15 0x7ff1b852e9cc in non-virtual thunk to mozilla::gmp::GeckoMediaPluginServiceChild::Observe(nsISupports*, char const*, char16_t const*) /builds/worker/workspace/build/src/dom/media/gmp/GMPServiceChild.cpp
[task 2020-01-24T00:00:56.564Z] 00:00:56 INFO - PID 12817 | #16 0x7ff1b28c0673 in nsObserverList::NotifyObservers(nsISupports*, char const*, char16_t const*) /builds/worker/workspace/build/src/xpcom/ds/nsObserverList.cpp:65:19
[task 2020-01-24T00:00:56.564Z] 00:00:56 INFO - PID 12817 | #17 0x7ff1b28c752f in nsObserverService::NotifyObservers(nsISupports*, char const*, char16_t const*) /builds/worker/workspace/build/src/xpcom/ds/nsObserverService.cpp:291:19
[task 2020-01-24T00:00:56.565Z] 00:00:56 INFO - PID 12817 | #18 0x7ff1b2a62d68 in mozilla::ShutdownXPCOM(nsIServiceManager*) /builds/worker/workspace/build/src/xpcom/build/XPCOMInit.cpp:635:24
[task 2020-01-24T00:00:56.566Z] 00:00:56 INFO - PID 12817 | #19 0x7ff1bd06d61c in XRE_TermEmbedding() /builds/worker/workspace/build/src/toolkit/xre/nsEmbedFunctions.cpp:226:3
[task 2020-01-24T00:00:56.567Z] 00:00:56 INFO - PID 12817 | #20 0x7ff1b3a282cf in mozilla::ipc::ScopedXREEmbed::Stop() /builds/worker/workspace/build/src/ipc/glue/ScopedXREEmbed.cpp:90:5
[task 2020-01-24T00:00:56.567Z] 00:00:56 INFO - PID 12817 | #21 0x7ff1bd06e3cb in XRE_InitChildProcess(int, char**, XREChildData const*) /builds/worker/workspace/build/src/toolkit/xre/nsEmbedFunctions.cpp:782:16
[task 2020-01-24T00:00:56.569Z] 00:00:56 INFO - PID 12817 | #22 0x55af5372c176 in content_process_main /builds/worker/workspace/build/src/browser/app/../../ipc/contentproc/plugin-container.cpp:56:28
[task 2020-01-24T00:00:56.569Z] 00:00:56 INFO - PID 12817 | #23 0x55af5372c176 in main /builds/worker/workspace/build/src/browser/app/nsBrowserApp.cpp:303:18
[task 2020-01-24T00:00:56.698Z] 00:00:56 INFO - PID 12817 | #24 0x7ff1d1b14b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
[task 2020-01-24T00:00:56.698Z] 00:00:56 INFO - PID 12817 | #25 0x55af53681c58 in _start (/builds/worker/workspace/build/application/firefox/firefox+0x99c58)
[task 2020-01-24T00:00:56.698Z] 00:00:56 INFO - PID 12817 | AddressSanitizer can not provide additional info.
[task 2020-01-24T00:00:56.698Z] 00:00:56 INFO - PID 12817 | SUMMARY: AddressSanitizer: SEGV /builds/worker/workspace/build/src/ipc/glue/MessageChannel.cpp:786:9 in mozilla::ipc::MessageChannel::Clear()
[task 2020-01-24T00:00:56.698Z] 00:00:56 INFO - PID 12817 | ==14758==ABORTING
[task 2020-01-24T00:00:57.905Z] 00:00:57 INFO - .
[task 2020-01-24T00:00:57.905Z] 00:00:57 INFO - TEST-OK | /encrypted-media/clearkey-mp4-update-disallowed-input.https.html | took 3178ms

Bryce can you please assign someone to take a look?

Flags: needinfo?(drno) → needinfo?(bvandyk)
Whiteboard: [stockwell needswork] → [stockwell needswork:owner]

I believe this is an IPC + GMP bug rather than an issue with the test. There is some ongoing work that I hope will help with this: Bug 1493656 + Bug 1557739. Those bugs appear to have gained some traction recently, so I'd like to wait until we land something there and see if it changes our failure rates here. Marking this sec, as the related bugs are.

Group: core-security
Flags: needinfo?(bvandyk)

There are 28 total failures in the last 7 days on

  • linux1804-64-asan opt,
  • linux64 asan

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=287240692&repo=autoland&lineNumber=5760

[task 2020-02-02T17:45:56.460Z] 17:45:56 INFO - TEST-START | /encrypted-media/clearkey-mp4-update-disallowed-input.https.html
[task 2020-02-02T17:45:56.461Z] 17:45:56 INFO - Closing window 95
[task 2020-02-02T17:45:57.561Z] 17:45:57 INFO - PID 12876 | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2020-02-02T17:45:57.570Z] 17:45:57 INFO - PID 12876 | AddressSanitizer:DEADLYSIGNAL
[task 2020-02-02T17:45:57.571Z] 17:45:57 INFO - PID 12876 | =================================================================
[task 2020-02-02T17:45:57.572Z] 17:45:57 ERROR - PID 12876 | ==14791==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000001 (pc 0x7efe41dbe5bf bp 0x7ffce080a530 sp 0x7ffce080a360 T0)
[task 2020-02-02T17:45:57.573Z] 17:45:57 INFO - PID 12876 | ==14791==The signal is caused by a WRITE memory access.
[task 2020-02-02T17:45:57.574Z] 17:45:57 INFO - PID 12876 | ==14791==Hint: address points to the zero page.
[task 2020-02-02T17:45:58.118Z] 17:45:58 INFO - PID 12876 | #0 0x7efe41dbe5be in mozilla::ipc::MessageChannel::Clear() /builds/worker/workspace/build/src/ipc/glue/MessageChannel.cpp:786:9
[task 2020-02-02T17:45:58.118Z] 17:45:58 INFO - PID 12876 | #1 0x7efe41dbd0d6 in mozilla::ipc::MessageChannel::~MessageChannel() /builds/worker/workspace/build/src/ipc/glue/MessageChannel.cpp:650:3
[task 2020-02-02T17:45:58.119Z] 17:45:58 INFO - PID 12876 | #2 0x7efe41de078e in mozilla::ipc::IToplevelProtocol::~IToplevelProtocol() /builds/worker/workspace/build/src/ipc/glue/ProtocolUtils.cpp:593:1
[task 2020-02-02T17:45:58.140Z] 17:45:58 INFO - PID 12876 | #3 0x7efe468baccd in mozilla::gmp::GMPContentParent::~GMPContentParent() /builds/worker/workspace/build/src/dom/media/gmp/GMPContentParent.cpp:35:39
[task 2020-02-02T17:45:58.140Z] 17:45:58 INFO - PID 12876 | #4 0x7efe40c520f3 in operator() /builds/worker/workspace/build/src/xpcom/ds/PLDHashTable.cpp:299:7
[task 2020-02-02T17:45:58.140Z] 17:45:58 INFO - PID 12876 | #5 0x7efe40c520f3 in ForEachSlot<(lambda at /builds/worker/workspace/build/src/xpcom/ds/PLDHashTable.cpp:297:51)> /builds/worker/workspace/build/src/obj-firefox/dist/include/PLDHashTable.h:365:9
[task 2020-02-02T17:45:58.141Z] 17:45:58 INFO - PID 12876 | #6 0x7efe40c520f3 in ForEachSlot<(lambda at /builds/worker/workspace/build/src/xpcom/ds/PLDHashTable.cpp:297:51)> /builds/worker/workspace/build/src/obj-firefox/dist/include/PLDHashTable.h:355:7
[task 2020-02-02T17:45:58.141Z] 17:45:58 INFO - PID 12876 | #7 0x7efe40c520f3 in PLDHashTable::~PLDHashTable() /builds/worker/workspace/build/src/xpcom/ds/PLDHashTable.cpp:297:15
[task 2020-02-02T17:45:58.158Z] 17:45:58 INFO - PID 12876 | #8 0x7efe4690d09e in ~nsTHashtable /builds/worker/workspace/build/src/obj-firefox/dist/include/nsTHashtable.h:382:43
[task 2020-02-02T17:45:58.158Z] 17:45:58 INFO - PID 12876 | #9 0x7efe4690d09e in ~GMPServiceChild /builds/worker/workspace/build/src/dom/media/gmp/GMPServiceChild.cpp:427:38
[task 2020-02-02T17:45:58.159Z] 17:45:58 INFO - PID 12876 | #10 0x7efe4690d09e in mozilla::gmp::GMPServiceChild::~GMPServiceChild() /builds/worker/workspace/build/src/dom/media/gmp/GMPServiceChild.cpp:427:37
[task 2020-02-02T17:45:58.160Z] 17:45:58 INFO - PID 12876 | #11 0x7efe4690be3e in operator() /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/UniquePtr.h:487:5
[task 2020-02-02T17:45:58.160Z] 17:45:58 INFO - PID 12876 | #12 0x7efe4690be3e in reset /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/UniquePtr.h:324:7
[task 2020-02-02T17:45:58.162Z] 17:45:58 INFO - PID 12876 | #13 0x7efe4690be3e in operator= /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/UniquePtr.h:297:5
[task 2020-02-02T17:45:58.162Z] 17:45:58 INFO - PID 12876 | #14 0x7efe4690be3e in mozilla::gmp::GeckoMediaPluginServiceChild::Observe(nsISupports*, char const*, char16_t const*) /builds/worker/workspace/build/src/dom/media/gmp/GMPServiceChild.cpp:358:21
[task 2020-02-02T17:45:58.163Z] 17:45:58 INFO - PID 12876 | #15 0x7efe4690bf3c in non-virtual thunk to mozilla::gmp::GeckoMediaPluginServiceChild::Observe(nsISupports*, char const*, char16_t const*) /builds/worker/workspace/build/src/dom/media/gmp/GMPServiceChild.cpp
[task 2020-02-02T17:45:58.164Z] 17:45:58 INFO - PID 12876 | #16 0x7efe40c70b43 in nsObserverList::NotifyObservers(nsISupports*, char const*, char16_t const*) /builds/worker/workspace/build/src/xpcom/ds/nsObserverList.cpp:65:19
[task 2020-02-02T17:45:58.165Z] 17:45:58 INFO - PID 12876 | #17 0x7efe40c779ff in nsObserverService::NotifyObservers(nsISupports*, char const*, char16_t const*) /builds/worker/workspace/build/src/xpcom/ds/nsObserverService.cpp:291:19
[task 2020-02-02T17:45:58.166Z] 17:45:58 INFO - PID 12876 | #18 0x7efe40e13490 in mozilla::ShutdownXPCOM(nsIServiceManager*) /builds/worker/workspace/build/src/xpcom/build/XPCOMInit.cpp:640:24
[task 2020-02-02T17:45:58.166Z] 17:45:58 INFO - PID 12876 | #19 0x7efe4b449f4c in XRE_TermEmbedding() /builds/worker/workspace/build/src/toolkit/xre/nsEmbedFunctions.cpp:226:3
[task 2020-02-02T17:45:58.166Z] 17:45:58 INFO - PID 12876 | #20 0x7efe41de24cf in mozilla::ipc::ScopedXREEmbed::Stop() /builds/worker/workspace/build/src/ipc/glue/ScopedXREEmbed.cpp:90:5
[task 2020-02-02T17:45:58.168Z] 17:45:58 INFO - PID 12876 | #21 0x7efe4b44acfb in XRE_InitChildProcess(int, char**, XREChildData const*) /builds/worker/workspace/build/src/toolkit/xre/nsEmbedFunctions.cpp:782:16
[task 2020-02-02T17:45:58.168Z] 17:45:58 INFO - PID 12876 | #22 0x5569f6d36176 in content_process_main /builds/worker/workspace/build/src/browser/app/../../ipc/contentproc/plugin-container.cpp:56:28
[task 2020-02-02T17:45:58.169Z] 17:45:58 INFO - PID 12876 | #23 0x5569f6d36176 in main /builds/worker/workspace/build/src/browser/app/nsBrowserApp.cpp:303:18
[task 2020-02-02T17:45:58.270Z] 17:45:58 INFO - PID 12876 | #24 0x7efe5ff21b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
[task 2020-02-02T17:45:58.270Z] 17:45:58 INFO - PID 12876 | #25 0x5569f6c8bc58 in _start (/builds/worker/workspace/build/application/firefox/firefox+0x99c58)
[task 2020-02-02T17:45:58.271Z] 17:45:58 INFO - PID 12876 | AddressSanitizer can not provide additional info.
[task 2020-02-02T17:45:58.271Z] 17:45:58 INFO - PID 12876 | SUMMARY: AddressSanitizer: SEGV /builds/worker/workspace/build/src/ipc/glue/MessageChannel.cpp:786:9 in mozilla::ipc::MessageChannel::Clear()
[task 2020-02-02T17:45:58.271Z] 17:45:58 INFO - PID 12876 | ==14791==ABORTING
[task 2020-02-02T17:45:59.585Z] 17:45:59 INFO - .
[task 2020-02-02T17:45:59.585Z] 17:45:59 INFO - TEST-OK | /encrypted-media/clearkey-mp4-update-disallowed-input.https.html | took 3128ms

There are 30 total failures in the last 7 days on linux1804-64-asan

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=289273192&repo=autoland&lineNumber=5789

[task 2020-02-17T21:33:54.980Z] 21:33:54 INFO - TEST-START | /encrypted-media/clearkey-mp4-update-disallowed-input.https.html
[task 2020-02-17T21:33:54.982Z] 21:33:54 INFO - Closing window 94
[task 2020-02-17T21:33:56.067Z] 21:33:56 INFO - PID 14061 | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2020-02-17T21:33:56.067Z] 21:33:56 INFO - PID 14061 | AddressSanitizer:DEADLYSIGNAL
[task 2020-02-17T21:33:56.067Z] 21:33:56 INFO - PID 14061 | =================================================================
[task 2020-02-17T21:33:56.067Z] 21:33:56 ERROR - PID 14061 | ==16073==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000001 (pc 0x7f02f93c534f bp 0x7ffcce55fd50 sp 0x7ffcce55fb80 T0)
[task 2020-02-17T21:33:56.067Z] 21:33:56 INFO - PID 14061 | ==16073==The signal is caused by a WRITE memory access.
[task 2020-02-17T21:33:56.068Z] 21:33:56 INFO - PID 14061 | ==16073==Hint: address points to the zero page.
[task 2020-02-17T21:33:56.697Z] 21:33:56 INFO - PID 14061 | #0 0x7f02f93c534e in mozilla::ipc::MessageChannel::Clear() /builds/worker/workspace/build/src/ipc/glue/MessageChannel.cpp:786:9
[task 2020-02-17T21:33:56.697Z] 21:33:56 INFO - PID 14061 | #1 0x7f02f93c3e66 in mozilla::ipc::MessageChannel::~MessageChannel() /builds/worker/workspace/build/src/ipc/glue/MessageChannel.cpp:650:3
[task 2020-02-17T21:33:56.698Z] 21:33:56 INFO - PID 14061 | #2 0x7f02f93fd932 in mozilla::ipc::IToplevelProtocol::~IToplevelProtocol() /builds/worker/workspace/build/src/ipc/glue/ProtocolUtils.h:392:32
[task 2020-02-17T21:33:56.719Z] 21:33:56 INFO - PID 14061 | #3 0x7f02fdefbeed in mozilla::gmp::GMPContentParent::~GMPContentParent() /builds/worker/workspace/build/src/dom/media/gmp/GMPContentParent.cpp:35:39
[task 2020-02-17T21:33:56.719Z] 21:33:56 INFO - PID 14061 | #4 0x7f02f824a903 in operator() /builds/worker/workspace/build/src/xpcom/ds/PLDHashTable.cpp:299:7
[task 2020-02-17T21:33:56.719Z] 21:33:56 INFO - PID 14061 | #5 0x7f02f824a903 in ForEachSlot<(lambda at /builds/worker/workspace/build/src/xpcom/ds/PLDHashTable.cpp:297:51)> /builds/worker/workspace/build/src/obj-firefox/dist/include/PLDHashTable.h:361:9
[task 2020-02-17T21:33:56.719Z] 21:33:56 INFO - PID 14061 | #6 0x7f02f824a903 in ForEachSlot<(lambda at /builds/worker/workspace/build/src/xpcom/ds/PLDHashTable.cpp:297:51)> /builds/worker/workspace/build/src/obj-firefox/dist/include/PLDHashTable.h:351:7
[task 2020-02-17T21:33:56.719Z] 21:33:56 INFO - PID 14061 | #7 0x7f02f824a903 in PLDHashTable::~PLDHashTable() /builds/worker/workspace/build/src/xpcom/ds/PLDHashTable.cpp:297:15
[task 2020-02-17T21:33:56.735Z] 21:33:56 INFO - PID 14061 | #8 0x7f02fdf4e47e in ~nsTHashtable /builds/worker/workspace/build/src/obj-firefox/dist/include/nsTHashtable.h:95:27
[task 2020-02-17T21:33:56.736Z] 21:33:56 INFO - PID 14061 | #9 0x7f02fdf4e47e in ~GMPServiceChild /builds/worker/workspace/build/src/dom/media/gmp/GMPServiceChild.cpp:427:38
[task 2020-02-17T21:33:56.736Z] 21:33:56 INFO - PID 14061 | #10 0x7f02fdf4e47e in mozilla::gmp::GMPServiceChild::~GMPServiceChild() /builds/worker/workspace/build/src/dom/media/gmp/GMPServiceChild.cpp:427:37
[task 2020-02-17T21:33:56.737Z] 21:33:56 INFO - PID 14061 | #11 0x7f02fdf4d1be in operator() /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/UniquePtr.h:487:5
[task 2020-02-17T21:33:56.737Z] 21:33:56 INFO - PID 14061 | #12 0x7f02fdf4d1be in reset /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/UniquePtr.h:324:7
[task 2020-02-17T21:33:56.737Z] 21:33:56 INFO - PID 14061 | #13 0x7f02fdf4d1be in operator= /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/UniquePtr.h:297:5
[task 2020-02-17T21:33:56.738Z] 21:33:56 INFO - PID 14061 | #14 0x7f02fdf4d1be in mozilla::gmp::GeckoMediaPluginServiceChild::Observe(nsISupports*, char const*, char16_t const*) /builds/worker/workspace/build/src/dom/media/gmp/GMPServiceChild.cpp:358:21
[task 2020-02-17T21:33:56.738Z] 21:33:56 INFO - PID 14061 | #15 0x7f02fdf4d2bc in non-virtual thunk to mozilla::gmp::GeckoMediaPluginServiceChild::Observe(nsISupports*, char const*, char16_t const*) /builds/worker/workspace/build/src/dom/media/gmp/GMPServiceChild.cpp
[task 2020-02-17T21:33:56.739Z] 21:33:56 INFO - PID 14061 | #16 0x7f02f8269363 in nsObserverList::NotifyObservers(nsISupports*, char const*, char16_t const*) /builds/worker/workspace/build/src/xpcom/ds/nsObserverList.cpp:65:19
[task 2020-02-17T21:33:56.739Z] 21:33:56 INFO - PID 14061 | #17 0x7f02f827025f in nsObserverService::NotifyObservers(nsISupports*, char const*, char16_t const*) /builds/worker/workspace/build/src/xpcom/ds/nsObserverService.cpp:292:19
[task 2020-02-17T21:33:56.741Z] 21:33:56 INFO - PID 14061 | #18 0x7f02f840c3fb in mozilla::ShutdownXPCOM(nsIServiceManager*) /builds/worker/workspace/build/src/xpcom/build/XPCOMInit.cpp:639:24
[task 2020-02-17T21:33:56.741Z] 21:33:56 INFO - PID 14061 | #19 0x7f0302ac9f7c in XRE_TermEmbedding() /builds/worker/workspace/build/src/toolkit/xre/nsEmbedFunctions.cpp:226:3
[task 2020-02-17T21:33:56.742Z] 21:33:56 INFO - PID 14061 | #20 0x7f02f93f9e2f in mozilla::ipc::ScopedXREEmbed::Stop() /builds/worker/workspace/build/src/ipc/glue/ScopedXREEmbed.cpp:90:5
[task 2020-02-17T21:33:56.743Z] 21:33:56 INFO - PID 14061 | #21 0x7f0302acad3c in XRE_InitChildProcess(int, char**, XREChildData const*) /builds/worker/workspace/build/src/toolkit/xre/nsEmbedFunctions.cpp:782:16
[task 2020-02-17T21:33:56.745Z] 21:33:56 INFO - PID 14061 | #22 0x55a20b04c290 in content_process_main /builds/worker/workspace/build/src/browser/app/../../ipc/contentproc/plugin-container.cpp:56:28
[task 2020-02-17T21:33:56.745Z] 21:33:56 INFO - PID 14061 | #23 0x55a20b04c290 in main /builds/worker/workspace/build/src/browser/app/nsBrowserApp.cpp:303:18
[task 2020-02-17T21:33:56.853Z] 21:33:56 INFO - .
[task 2020-02-17T21:33:56.853Z] 21:33:56 INFO - TEST-OK | /encrypted-media/clearkey-mp4-update-disallowed-input.https.html | took 1872ms
[task 2020-02-17T21:33:56.854Z] 21:33:56 INFO - TEST-START | /encrypted-media/clearkey-mp4-waiting-for-a-key.https.html
[task 2020-02-17T21:33:56.854Z] 21:33:56 INFO - Closing window 96
[task 2020-02-17T21:33:56.870Z] 21:33:56 INFO - PID 14061 | #24 0x7f031764db96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
[task 2020-02-17T21:33:56.870Z] 21:33:56 INFO - PID 14061 | #25 0x55a20afa1c58 in _start (/builds/worker/workspace/build/application/firefox/firefox+0x99c58)
[task 2020-02-17T21:33:56.870Z] 21:33:56 INFO - PID 14061 | AddressSanitizer can not provide additional info.
[task 2020-02-17T21:33:56.871Z] 21:33:56 INFO - PID 14061 | SUMMARY: AddressSanitizer: SEGV /builds/worker/workspace/build/src/ipc/glue/MessageChannel.cpp:786:9 in mozilla::ipc::MessageChannel::Clear()
[task 2020-02-17T21:33:56.871Z] 21:33:56 INFO - PID 14061 | ==16073==ABORTING
[task 2020-02-17T21:33:58.650Z] 21:33:58 INFO - PID 14061 | -----------------------------------------------------
[task 2020-02-17T21:33:58.651Z] 21:33:58 INFO - PID 14061 | Suppressions used:
[task 2020-02-17T21:33:58.653Z] 21:33:58 INFO - PID 14061 | count bytes template
[task 2020-02-17T21:33:58.654Z] 21:33:58 INFO - PID 14061 | 27 832 nsComponentManagerImpl
[task 2020-02-17T21:33:58.654Z] 21:33:58 INFO - PID 14061 | 633 18287 libfontconfig.so
[task 2020-02-17T21:33:58.654Z] 21:33:58 INFO - PID 14061 | 1 11 libglib-2.0.so
[task 2020-02-17T21:33:58.655Z] 21:33:58 INFO - PID 14061 | -----------------------------------------------------
[task 2020-02-17T21:34:01.679Z] 21:34:01 INFO - .
[task 2020-02-17T21:34:01.679Z] 21:34:01 INFO - TEST-OK | /encrypted-media/clearkey-mp4-waiting-for-a-key.https.html | took 4829ms

Bryce can you assign someone to take a look?

Flags: needinfo?(bvandyk)

:jld, do you think this looks related to bug 1493656? I'm wondering if this is something we can fix in the media codebase, or if it's to do with the greater IPC machinery.

Flags: needinfo?(bvandyk) → needinfo?(jld)

Summarizing the discussion about this from the 2020-02-26 IPC meeting: Nika noticed that GMPContentParent::ActorDestroy is subtly wrong (it needs to wait for teardown to finish on the IPC thread, not spin the event loop on the actor thread, if I understand correctly), and making the actor refcounted in IPDL instead should fix this.

Flags: needinfo?(jld)
Group: core-security → media-core-security
Assignee: nobody → bvandyk

https://hg.mozilla.org/integration/autoland/rev/13ed73c1c33519108eb50ce92f331af7dd5af7fa
https://hg.mozilla.org/mozilla-central/rev/13ed73c1c335

Group: media-core-security → core-security-release
Status: NEW → RESOLVED
Closed: 4 years ago
status-firefox76: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla76

Changing the priority to p2 as the bug is tracked by a release manager for the current nightly.
See What Do You Triage for more information

Priority: P5 → P2

There are 150 total failures in the last 7 days on linux1804-64-asan opt with a spike starting with the 13th of April.

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=297705268&repo=mozilla-central&lineNumber=4955

[task 2020-04-15T09:56:11.484Z] 09:56:11 INFO - TEST-START | /encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential-readyState.https.html
[task 2020-04-15T09:56:11.500Z] 09:56:11 INFO - Closing window 46
[task 2020-04-15T09:56:11.762Z] 09:56:11 INFO - PID 10699 | AddressSanitizer:DEADLYSIGNAL
[task 2020-04-15T09:56:11.762Z] 09:56:11 INFO - PID 10699 | =================================================================
[task 2020-04-15T09:56:11.763Z] 09:56:11 ERROR - PID 10699 | ==12050==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000001 (pc 0x7f0ee3daa001 bp 0x7fffed5472f0 sp 0x7fffed547120 T0)
[task 2020-04-15T09:56:11.766Z] 09:56:11 INFO - PID 10699 | ==12050==The signal is caused by a WRITE memory access.
[task 2020-04-15T09:56:11.766Z] 09:56:11 INFO - PID 10699 | ==12050==Hint: address points to the zero page.
[task 2020-04-15T09:56:12.658Z] 09:56:12 INFO - PID 10699 | #0 0x7f0ee3daa000 in mozilla::ipc::MessageChannel::Clear() /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:769:9
[task 2020-04-15T09:56:12.658Z] 09:56:12 INFO - PID 10699 | #1 0x7f0ee3da8c55 in mozilla::ipc::MessageChannel::~MessageChannel() /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:633:3
[task 2020-04-15T09:56:12.658Z] 09:56:12 INFO - PID 10699 | #2 0x7f0ee3dcf832 in mozilla::ipc::IToplevelProtocol::~IToplevelProtocol() /builds/worker/workspace/obj-build/dist/include/mozilla/ipc/ProtocolUtils.h:391:32
[task 2020-04-15T09:56:12.695Z] 09:56:12 INFO - PID 10699 | #3 0x7f0ee8930d9d in mozilla::gmp::GMPContentParent::~GMPContentParent() /builds/worker/checkouts/gecko/dom/media/gmp/GMPContentParent.cpp:35:39
[task 2020-04-15T09:56:12.696Z] 09:56:12 INFO - PID 10699 | #4 0x7f0ee2bdb623 in operator() /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:292:7
[task 2020-04-15T09:56:12.696Z] 09:56:12 INFO - PID 10699 | #5 0x7f0ee2bdb623 in ForEachSlot<(lambda at /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:290:51)> /builds/worker/workspace/obj-build/dist/include/PLDHashTable.h:357:9
[task 2020-04-15T09:56:12.697Z] 09:56:12 INFO - PID 10699 | #6 0x7f0ee2bdb623 in ForEachSlot<(lambda at /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:290:51)> /builds/worker/workspace/obj-build/dist/include/PLDHashTable.h:347:7
[task 2020-04-15T09:56:12.699Z] 09:56:12 INFO - PID 10699 | #7 0x7f0ee2bdb623 in PLDHashTable::~PLDHashTable() /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:290:15
[task 2020-04-15T09:56:12.736Z] 09:56:12 INFO - PID 10699 | #8 0x7f0ee8982aee in ~nsTHashtable /builds/worker/workspace/obj-build/dist/include/nsTHashtable.h:170:27
[task 2020-04-15T09:56:12.736Z] 09:56:12 INFO - PID 10699 | #9 0x7f0ee8982aee in ~GMPServiceChild /builds/worker/checkouts/gecko/dom/media/gmp/GMPServiceChild.cpp:427:35
[task 2020-04-15T09:56:12.736Z] 09:56:12 INFO - PID 10699 | #10 0x7f0ee8982aee in mozilla::gmp::GMPServiceChild::~GMPServiceChild() /builds/worker/checkouts/gecko/dom/media/gmp/GMPServiceChild.cpp:427:35
[task 2020-04-15T09:56:12.736Z] 09:56:12 INFO - PID 10699 | #11 0x7f0ee8981871 in operator() /builds/worker/workspace/obj-build/dist/include/mozilla/UniquePtr.h:460:5
[task 2020-04-15T09:56:12.736Z] 09:56:12 INFO - PID 10699 | #12 0x7f0ee8981871 in reset /builds/worker/workspace/obj-build/dist/include/mozilla/UniquePtr.h:302:7
[task 2020-04-15T09:56:12.736Z] 09:56:12 INFO - PID 10699 | #13 0x7f0ee8981871 in operator= /builds/worker/workspace/obj-build/dist/include/mozilla/UniquePtr.h:275:5
[task 2020-04-15T09:56:12.737Z] 09:56:12 INFO - PID 10699 | #14 0x7f0ee8981871 in mozilla::gmp::GeckoMediaPluginServiceChild::Observe(nsISupports*, char const*, char16_t const*) /builds/worker/checkouts/gecko/dom/media/gmp/GMPServiceChild.cpp:358:21
[task 2020-04-15T09:56:12.737Z] 09:56:12 INFO - PID 10699 | #15 0x7f0ee898197c in non-virtual thunk to mozilla::gmp::GeckoMediaPluginServiceChild::Observe(nsISupports*, char const*, char16_t const*) /builds/worker/checkouts/gecko/dom/media/gmp/GMPServiceChild.cpp
[task 2020-04-15T09:56:12.737Z] 09:56:12 INFO - PID 10699 | #16 0x7f0ee2bf9da3 in nsObserverList::NotifyObservers(nsISupports*, char const*, char16_t const*) /builds/worker/checkouts/gecko/xpcom/ds/nsObserverList.cpp:65:19
[task 2020-04-15T09:56:12.739Z] 09:56:12 INFO - PID 10699 | #17 0x7f0ee2c00e12 in nsObserverService::NotifyObservers(nsISupports*, char const*, char16_t const*) /builds/worker/checkouts/gecko/xpcom/ds/nsObserverService.cpp:292:19
[task 2020-04-15T09:56:12.740Z] 09:56:12 INFO - PID 10699 | #18 0x7f0ee2da35e1 in mozilla::ShutdownXPCOM(nsIServiceManager*) /builds/worker/checkouts/gecko/xpcom/build/XPCOMInit.cpp:637:24
[task 2020-04-15T09:56:12.756Z] 09:56:12 INFO - PID 10699 | #19 0x7f0eed562d7c in XRE_TermEmbedding() /builds/worker/checkouts/gecko/toolkit/xre/nsEmbedFunctions.cpp:222:3
[task 2020-04-15T09:56:12.757Z] 09:56:12 INFO - PID 10699 | #20 0x7f0ee3de188f in mozilla::ipc::ScopedXREEmbed::Stop() /builds/worker/checkouts/gecko/ipc/glue/ScopedXREEmbed.cpp:90:5
[task 2020-04-15T09:56:12.757Z] 09:56:12 INFO - PID 10699 | #21 0x7f0eed563c67 in XRE_InitChildProcess(int, char**, XREChildData const*) /builds/worker/checkouts/gecko/toolkit/xre/nsEmbedFunctions.cpp:744:16
[task 2020-04-15T09:56:12.758Z] 09:56:12 INFO - PID 10699 | #22 0x55844375d00b in content_process_main /builds/worker/checkouts/gecko/browser/app/../../ipc/contentproc/plugin-container.cpp:56:28
[task 2020-04-15T09:56:12.758Z] 09:56:12 INFO - PID 10699 | #23 0x55844375d00b in main /builds/worker/checkouts/gecko/browser/app/nsBrowserApp.cpp:303:18
[task 2020-04-15T09:56:12.928Z] 09:56:12 INFO - PID 10699 | #24 0x7f0f0260bb96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
[task 2020-04-15T09:56:12.931Z] 09:56:12 INFO - PID 10699 | #25 0x5584436b1998 in _start (/builds/worker/workspace/build/application/firefox/firefox+0x9c998)
[task 2020-04-15T09:56:12.932Z] 09:56:12 INFO - PID 10699 | AddressSanitizer can not provide additional info.
[task 2020-04-15T09:56:12.932Z] 09:56:12 INFO - PID 10699 | SUMMARY: AddressSanitizer: SEGV /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:769:9 in mozilla::ipc::MessageChannel::Clear()
[task 2020-04-15T09:56:12.932Z] 09:56:12 INFO - PID 10699 | ==12050==ABORTING
[task 2020-04-15T09:56:18.142Z] 09:56:18 INFO - PID 12307 | 1586944578127 Marionette INFO Listening on port 45926

Bryce can you take a look or assign someone?

Flags: needinfo?(bvandyk)
Flags: needinfo?(bvandyk)

Acking that I'm still working on this.

Flags: needinfo?(bvandyk)

Thank you.

There are 200 total failures in the last 7 days on linux1804-64-asan opt

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=298934911&repo=autoland&lineNumber=5717

[task 2020-04-22T23:21:16.772Z] 23:21:16 INFO - TEST-START | /encrypted-media/clearkey-mp4-update-disallowed-input.https.html
[task 2020-04-22T23:21:16.775Z] 23:21:16 INFO - Closing window 94
[task 2020-04-22T23:21:17.863Z] 23:21:17 INFO - PID 11263 | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2020-04-22T23:21:17.863Z] 23:21:17 INFO - PID 11263 | AddressSanitizer:DEADLYSIGNAL
[task 2020-04-22T23:21:17.863Z] 23:21:17 INFO - PID 11263 | =================================================================
[task 2020-04-22T23:21:17.864Z] 23:21:17 ERROR - PID 11263 | ==17112==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000001 (pc 0x7f100a56a921 bp 0x7ffe4fbd7a70 sp 0x7ffe4fbd78a0 T0)
[task 2020-04-22T23:21:17.864Z] 23:21:17 INFO - PID 11263 | ==17112==The signal is caused by a WRITE memory access.
[task 2020-04-22T23:21:17.865Z] 23:21:17 INFO - PID 11263 | ==17112==Hint: address points to the zero page.
[task 2020-04-22T23:21:18.355Z] 23:21:18 INFO - PID 11263 | #0 0x7f100a56a920 in mozilla::ipc::MessageChannel::Clear() /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:769:9
[task 2020-04-22T23:21:18.355Z] 23:21:18 INFO - PID 11263 | #1 0x7f100a569575 in mozilla::ipc::MessageChannel::~MessageChannel() /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:633:3
[task 2020-04-22T23:21:18.355Z] 23:21:18 INFO - PID 11263 | #2 0x7f100a590152 in mozilla::ipc::IToplevelProtocol::~IToplevelProtocol() /builds/worker/workspace/obj-build/dist/include/mozilla/ipc/ProtocolUtils.h:391:32
[task 2020-04-22T23:21:18.371Z] 23:21:18 INFO - PID 11263 | #3 0x7f100f105a3d in mozilla::gmp::GMPContentParent::~GMPContentParent() /builds/worker/checkouts/gecko/dom/media/gmp/GMPContentParent.cpp:35:39
[task 2020-04-22T23:21:18.372Z] 23:21:18 INFO - PID 11263 | #4 0x7f1009396023 in operator() /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:292:7
[task 2020-04-22T23:21:18.373Z] 23:21:18 INFO - PID 11263 | #5 0x7f1009396023 in ForEachSlot<(lambda at /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:290:51)> /builds/worker/workspace/obj-build/dist/include/PLDHashTable.h:357:9
[task 2020-04-22T23:21:18.373Z] 23:21:18 INFO - PID 11263 | #6 0x7f1009396023 in ForEachSlot<(lambda at /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:290:51)> /builds/worker/workspace/obj-build/dist/include/PLDHashTable.h:347:7
[task 2020-04-22T23:21:18.374Z] 23:21:18 INFO - PID 11263 | #7 0x7f1009396023 in PLDHashTable::~PLDHashTable() /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:290:15
[task 2020-04-22T23:21:18.396Z] 23:21:18 INFO - PID 11263 | #8 0x7f100f15778e in ~nsTHashtable /builds/worker/workspace/obj-build/dist/include/nsTHashtable.h:170:27
[task 2020-04-22T23:21:18.396Z] 23:21:18 INFO - PID 11263 | #9 0x7f100f15778e in ~GMPServiceChild /builds/worker/checkouts/gecko/dom/media/gmp/GMPServiceChild.cpp:427:35
[task 2020-04-22T23:21:18.396Z] 23:21:18 INFO - PID 11263 | #10 0x7f100f15778e in mozilla::gmp::GMPServiceChild::~GMPServiceChild() /builds/worker/checkouts/gecko/dom/media/gmp/GMPServiceChild.cpp:427:35
[task 2020-04-22T23:21:18.396Z] 23:21:18 INFO - PID 11263 | #11 0x7f100f156511 in operator() /builds/worker/workspace/obj-build/dist/include/mozilla/UniquePtr.h:460:5
[task 2020-04-22T23:21:18.396Z] 23:21:18 INFO - PID 11263 | #12 0x7f100f156511 in reset /builds/worker/workspace/obj-build/dist/include/mozilla/UniquePtr.h:302:7
[task 2020-04-22T23:21:18.396Z] 23:21:18 INFO - PID 11263 | #13 0x7f100f156511 in operator= /builds/worker/workspace/obj-build/dist/include/mozilla/UniquePtr.h:275:5
[task 2020-04-22T23:21:18.396Z] 23:21:18 INFO - PID 11263 | #14 0x7f100f156511 in mozilla::gmp::GeckoMediaPluginServiceChild::Observe(nsISupports*, char const*, char16_t const*) /builds/worker/checkouts/gecko/dom/media/gmp/GMPServiceChild.cpp:358:21
[task 2020-04-22T23:21:18.397Z] 23:21:18 INFO - PID 11263 | #15 0x7f100f15661c in non-virtual thunk to mozilla::gmp::GeckoMediaPluginServiceChild::Observe(nsISupports*, char const*, char16_t const*) /builds/worker/checkouts/gecko/dom/media/gmp/GMPServiceChild.cpp
[task 2020-04-22T23:21:18.397Z] 23:21:18 INFO - PID 11263 | #16 0x7f10093b47a3 in nsObserverList::NotifyObservers(nsISupports*, char const*, char16_t const*) /builds/worker/checkouts/gecko/xpcom/ds/nsObserverList.cpp:65:19
[task 2020-04-22T23:21:18.398Z] 23:21:18 INFO - PID 11263 | #17 0x7f10093bb812 in nsObserverService::NotifyObservers(nsISupports*, char const*, char16_t const*) /builds/worker/checkouts/gecko/xpcom/ds/nsObserverService.cpp:292:19
[task 2020-04-22T23:21:18.399Z] 23:21:18 INFO - PID 11263 | #18 0x7f100955d8a1 in mozilla::ShutdownXPCOM(nsIServiceManager*) /builds/worker/checkouts/gecko/xpcom/build/XPCOMInit.cpp:637:24
[task 2020-04-22T23:21:18.408Z] 23:21:18 INFO - PID 11263 | #19 0x7f1013d4ba9c in XRE_TermEmbedding() /builds/worker/checkouts/gecko/toolkit/xre/nsEmbedFunctions.cpp:222:3
[task 2020-04-22T23:21:18.408Z] 23:21:18 INFO - PID 11263 | #20 0x7f100a5a21af in mozilla::ipc::ScopedXREEmbed::Stop() /builds/worker/checkouts/gecko/ipc/glue/ScopedXREEmbed.cpp:90:5
[task 2020-04-22T23:21:18.409Z] 23:21:18 INFO - PID 11263 | #21 0x7f1013d4c987 in XRE_InitChildProcess(int, char**, XREChildData const*) /builds/worker/checkouts/gecko/toolkit/xre/nsEmbedFunctions.cpp:744:16
[task 2020-04-22T23:21:18.410Z] 23:21:18 INFO - PID 11263 | #22 0x5607ed7b200b in content_process_main /builds/worker/checkouts/gecko/browser/app/../../ipc/contentproc/plugin-container.cpp:56:28
[task 2020-04-22T23:21:18.410Z] 23:21:18 INFO - PID 11263 | #23 0x5607ed7b200b in main /builds/worker/checkouts/gecko/browser/app/nsBrowserApp.cpp:303:18
[task 2020-04-22T23:21:18.466Z] 23:21:18 INFO - PID 11263 | #24 0x7f1028e94b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
[task 2020-04-22T23:21:18.467Z] 23:21:18 INFO - PID 11263 | #25 0x5607ed706998 in _start (/builds/worker/workspace/build/application/firefox/firefox+0x9c998)
[task 2020-04-22T23:21:18.467Z] 23:21:18 INFO - PID 11263 | AddressSanitizer can not provide additional info.
[task 2020-04-22T23:21:18.467Z] 23:21:18 INFO - PID 11263 | SUMMARY: AddressSanitizer: SEGV /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:769:9 in mozilla::ipc::MessageChannel::Clear()
[task 2020-04-22T23:21:18.468Z] 23:21:18 INFO - PID 11263 | ==17112==ABORTING
[task 2020-04-22T23:21:19.899Z] 23:21:19 INFO - .
[task 2020-04-22T23:21:19.899Z] 23:21:19 INFO - TEST-OK | /encrypted-media/clearkey-mp4-update-disallowed-input.https.html | took 3130ms

Looked in the last 20 logs and saw that this is mostly failing in

  • clearkey-mp4-playback-temporary-multikey-sequential-readyState.https.html
    and
  • clearkey-mp4-update-disallowed-input.https.html
    Should we try to disable these tests until there is a fix?

We can disable tests until I'm able to get a fix.

Keywords: leave-open
Whiteboard: [stockwell needswork:owner] → [stockwell needswork:owner][stockwell disabled]

Merged https://hg.mozilla.org/mozilla-central/rev/681820075af1
Disabled clearkey-mp4-playback-temporary-multikey-sequential-readyState.https.html and clearkey-mp4-update-disallowed-input.https.html on Linux. r=intermittent-reviewers,jmaher

Summary: Intermittent PID 27123 | SUMMARY: AddressSanitizer: SEGV /builds/worker/workspace/build/src/ipc/glue/MessageChannel.cpp:764:9 in mozilla::ipc::MessageChannel::Clear() → Intermittent SUMMARY: AddressSanitizer: SEGV /builds/worker/workspace/build/src/ipc/glue/MessageChannel.cpp:764:9 in mozilla::ipc::MessageChannel::Clear()

I've been continuing to work on this and it's proving thorny. I've noticed at least one more race against the IPC machinery, so have expanded refcounting of protocols in my WIP.

I believe the leak that resulted from the initial ref counting patch here is due to shutdown racing GMP creation. MediaKeys::Init() starts a process where the GMP machinery will stand up a new process. Doing this is complicated and involves IPC between content, parent, and the GMP process. Within each process there are also multiple threads to consider. Races typically involve crashes or worse, as we see in this bug, so turning a crash into a leak is progress in a sense.

XPCOM shutdown will result in us shutting down threads used by the GMP machinery and this can kill threads while an initialization is still in flight. There's a lot of handling to try and deal with this nicely, but it appears to be coming up short.

I'd previously tried to use shutdown blockers to fix similar issues and didn't have much success. However, it's gotten to the point where I believe I should have another go.

My current view of the problem is that we have the following states:

  1. Firefox is running and GMPs can be created.
  2. Firefox shutdown has begun and no new GMPs can be created, existing ones should move to shutdown.
  3. Firefox shutdown is progressing and GMP machinery should be torn down.

The GMP machinery will observe something like these states, but doesn't block progress through the states. This means you can rapidly move from state 1 to 3. This is particularly bad for GMPs that began initialization just before the transition started.

I'm going to see if blocking transition from 2 to 3 until everything is in a consistent state can help mitigate the issue.

In case it helps as an example, kmag recently fixed an issue with the hang monitor being in the middle of setting up when xpcom-shutdown happens in bug 1617709, by making the main thread wait for the initialization to finish.

There are 75 total failures in the last 7 days on linux1804-64-asan opt.

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=300491483&repo=autoland&lineNumber=4480

[task 2020-05-02T12:55:17.078Z] 12:55:17 INFO - TEST-START | /encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential.https.html
[task 2020-05-02T12:55:17.101Z] 12:55:17 INFO - Closing window 46
[task 2020-05-02T12:55:17.345Z] 12:55:17 INFO - PID 9275 | AddressSanitizer:DEADLYSIGNAL
[task 2020-05-02T12:55:17.347Z] 12:55:17 INFO - PID 9275 | =================================================================
[task 2020-05-02T12:55:17.348Z] 12:55:17 ERROR - PID 9275 | ==10589==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000001 (pc 0x7f053003f971 bp 0x7fff24ea4c50 sp 0x7fff24ea4a80 T0)
[task 2020-05-02T12:55:17.349Z] 12:55:17 INFO - PID 9275 | ==10589==The signal is caused by a WRITE memory access.
[task 2020-05-02T12:55:17.351Z] 12:55:17 INFO - PID 9275 | ==10589==Hint: address points to the zero page.
[task 2020-05-02T12:55:18.630Z] 12:55:18 INFO - PID 9275 | #0 0x7f053003f970 in mozilla::ipc::MessageChannel::Clear() /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:769:9
[task 2020-05-02T12:55:18.632Z] 12:55:18 INFO - PID 9275 | #1 0x7f053003e5c5 in mozilla::ipc::MessageChannel::~MessageChannel() /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:633:3
[task 2020-05-02T12:55:18.632Z] 12:55:18 INFO - PID 9275 | #2 0x7f0530064fb2 in mozilla::ipc::IToplevelProtocol::~IToplevelProtocol() /builds/worker/workspace/obj-build/dist/include/mozilla/ipc/ProtocolUtils.h:391:32
[task 2020-05-02T12:55:18.680Z] 12:55:18 INFO - PID 9275 | #3 0x7f0534c07a4d in mozilla::gmp::GMPContentParent::~GMPContentParent() /builds/worker/checkouts/gecko/dom/media/gmp/GMPContentParent.cpp:35:39
[task 2020-05-02T12:55:18.702Z] 12:55:18 INFO - PID 9275 | #4 0x7f052ee54a53 in operator() /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:292:7
[task 2020-05-02T12:55:18.702Z] 12:55:18 INFO - PID 9275 | #5 0x7f052ee54a53 in ForEachSlot<(lambda at /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:290:51)> /builds/worker/workspace/obj-build/dist/include/PLDHashTable.h:357:9
[task 2020-05-02T12:55:18.702Z] 12:55:18 INFO - PID 9275 | #6 0x7f052ee54a53 in ForEachSlot<(lambda at /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:290:51)> /builds/worker/workspace/obj-build/dist/include/PLDHashTable.h:347:7
[task 2020-05-02T12:55:18.702Z] 12:55:18 INFO - PID 9275 | #7 0x7f052ee54a53 in PLDHashTable::~PLDHashTable() /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:290:15
[task 2020-05-02T12:55:18.763Z] 12:55:18 INFO - PID 9275 | #8 0x7f0534c5a22e in ~nsTHashtable /builds/worker/workspace/obj-build/dist/include/nsTHashtable.h:170:27
[task 2020-05-02T12:55:18.763Z] 12:55:18 INFO - PID 9275 | #9 0x7f0534c5a22e in ~GMPServiceChild /builds/worker/checkouts/gecko/dom/media/gmp/GMPServiceChild.cpp:427:35
[task 2020-05-02T12:55:18.767Z] 12:55:18 INFO - PID 9275 | #10 0x7f0534c5a22e in mozilla::gmp::GMPServiceChild::~GMPServiceChild() /builds/worker/checkouts/gecko/dom/media/gmp/GMPServiceChild.cpp:427:35
[task 2020-05-02T12:55:18.768Z] 12:55:18 INFO - PID 9275 | #11 0x7f0534c58fb1 in operator() /builds/worker/workspace/obj-build/dist/include/mozilla/UniquePtr.h:460:5
[task 2020-05-02T12:55:18.768Z] 12:55:18 INFO - PID 9275 | #12 0x7f0534c58fb1 in reset /builds/worker/workspace/obj-build/dist/include/mozilla/UniquePtr.h:302:7
[task 2020-05-02T12:55:18.768Z] 12:55:18 INFO - PID 9275 | #13 0x7f0534c58fb1 in operator= /builds/worker/workspace/obj-build/dist/include/mozilla/UniquePtr.h:275:5
[task 2020-05-02T12:55:18.768Z] 12:55:18 INFO - PID 9275 | #14 0x7f0534c58fb1 in mozilla::gmp::GeckoMediaPluginServiceChild::Observe(nsISupports*, char const*, char16_t const*) /builds/worker/checkouts/gecko/dom/media/gmp/GMPServiceChild.cpp:358:21
[task 2020-05-02T12:55:18.768Z] 12:55:18 INFO - PID 9275 | #15 0x7f0534c590bc in non-virtual thunk to mozilla::gmp::GeckoMediaPluginServiceChild::Observe(nsISupports*, char const*, char16_t const*) /builds/worker/checkouts/gecko/dom/media/gmp/GMPServiceChild.cpp
[task 2020-05-02T12:55:18.771Z] 12:55:18 INFO - PID 9275 | #16 0x7f052ee72f73 in nsObserverList::NotifyObservers(nsISupports*, char const*, char16_t const*) /builds/worker/checkouts/gecko/xpcom/ds/nsObserverList.cpp:65:19
[task 2020-05-02T12:55:18.787Z] 12:55:18 INFO - PID 9275 | #17 0x7f052ee7a302 in nsObserverService::NotifyObservers(nsISupports*, char const*, char16_t const*) /builds/worker/checkouts/gecko/xpcom/ds/nsObserverService.cpp:288:19
[task 2020-05-02T12:55:18.788Z] 12:55:18 INFO - PID 9275 | #18 0x7f052f01d3b1 in mozilla::ShutdownXPCOM(nsIServiceManager*) /builds/worker/checkouts/gecko/xpcom/build/XPCOMInit.cpp:638:24
[task 2020-05-02T12:55:18.789Z] 12:55:18 INFO - PID 9275 | #19 0x7f05398b1adc in XRE_TermEmbedding() /builds/worker/checkouts/gecko/toolkit/xre/nsEmbedFunctions.cpp:222:3
[task 2020-05-02T12:55:18.791Z] 12:55:18 INFO - PID 9275 | #20 0x7f05300771ff in mozilla::ipc::ScopedXREEmbed::Stop() /builds/worker/checkouts/gecko/ipc/glue/ScopedXREEmbed.cpp:90:5
[task 2020-05-02T12:55:18.794Z] 12:55:18 INFO - PID 9275 | #21 0x7f05398b29c7 in XRE_InitChildProcess(int, char**, XREChildData const*) /builds/worker/checkouts/gecko/toolkit/xre/nsEmbedFunctions.cpp:744:16
[task 2020-05-02T12:55:18.811Z] 12:55:18 INFO - PID 9275 | #22 0x5595ea21800b in content_process_main /builds/worker/checkouts/gecko/browser/app/../../ipc/contentproc/plugin-container.cpp:56:28
[task 2020-05-02T12:55:18.812Z] 12:55:18 INFO - PID 9275 | #23 0x5595ea21800b in main /builds/worker/checkouts/gecko/browser/app/nsBrowserApp.cpp:303:18
[task 2020-05-02T12:55:18.933Z] 12:55:18 INFO - PID 10518 | -----------------------------------------------------
[task 2020-05-02T12:55:18.933Z] 12:55:18 INFO - PID 10518 | Suppressions used:
[task 2020-05-02T12:55:18.934Z] 12:55:18 INFO - PID 10518 | count bytes template
[task 2020-05-02T12:55:18.936Z] 12:55:18 INFO - PID 10518 | 24 744 nsComponentManagerImpl
[task 2020-05-02T12:55:18.936Z] 12:55:18 INFO - PID 10518 | 633 18287 libfontconfig.so
[task 2020-05-02T12:55:18.937Z] 12:55:18 INFO - PID 10518 | 1 11 libglib-2.0.so
[task 2020-05-02T12:55:18.937Z] 12:55:18 INFO - PID 10518 | -----------------------------------------------------
[task 2020-05-02T12:55:18.958Z] 12:55:18 INFO - PID 9275 | #24 0x7f054eb01b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
[task 2020-05-02T12:55:18.958Z] 12:55:18 INFO - PID 9275 | #25 0x5595ea16c998 in _start (/builds/worker/workspace/build/application/firefox/firefox+0xa2998)
[task 2020-05-02T12:55:18.958Z] 12:55:18 INFO - PID 9275 | AddressSanitizer can not provide additional info.
[task 2020-05-02T12:55:18.958Z] 12:55:18 INFO - PID 9275 | SUMMARY: AddressSanitizer: SEGV /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:769:9 in mozilla::ipc::MessageChannel::Clear()
[task 2020-05-02T12:55:18.958Z] 12:55:18 INFO - PID 9275 | ==10589==ABORTING
[task 2020-05-02T12:55:19.533Z] 12:55:19 INFO - .
[task 2020-05-02T12:55:19.533Z] 12:55:19 INFO - TEST-OK | /eventsource/eventsource-constructor-url-bogus.any.worker.html | took 2581ms

New patches up. I need to find some reviewers for GMP changes, but wanted them up to discuss them. I'm going to get a larger write up on this bug to better contextualize the GMP changes + for posterity.

Attached file testcase.html —

Attaching a test case I've been using from another bug. I've found this test case to trigger the bug here. This testcase will also trigger other issues, which has resulted in fixing the issue here being a bit of a yak shave.

To have this testcase work correctly you'll need to set dom.allow_scripts_to_close_windows before opening it. This seems to work best if the tab you open the test in is the only one in the browser, as you get a racy shutdown.

Attachment #9148134 - Attachment is obsolete: true

There are 74 total failures in the last 7 days all on linux1804-64-asan opt, with failure count going up on the 15th:

https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2020-05-10&endday=2020-05-17&tree=trunk&bug=1551615

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=302624975&repo=autoland&lineNumber=3508

[task 2020-05-17T07:50:55.470Z] 07:50:55 INFO - TEST-START | /encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential.https.html
[task 2020-05-17T07:50:55.499Z] 07:50:55 INFO - Closing window 47
[task 2020-05-17T07:50:55.804Z] 07:50:55 INFO - PID 5116 | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2020-05-17T07:50:55.825Z] 07:50:55 INFO - PID 5116 | AddressSanitizer:DEADLYSIGNAL
[task 2020-05-17T07:50:55.828Z] 07:50:55 INFO - PID 5116 | =================================================================
[task 2020-05-17T07:50:55.830Z] 07:50:55 ERROR - PID 5116 | ==6550==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000001 (pc 0x7fbeb89ee901 bp 0x7ffc9652fb50 sp 0x7ffc9652f980 T0)
[task 2020-05-17T07:50:55.830Z] 07:50:55 INFO - PID 5116 | ==6550==The signal is caused by a WRITE memory access.
[task 2020-05-17T07:50:55.830Z] 07:50:55 INFO - PID 5116 | ==6550==Hint: address points to the zero page.
[task 2020-05-17T07:50:56.796Z] 07:50:56 INFO - PID 5116 | #0 0x7fbeb89ee901 in mozilla::ipc::MessageChannel::Clear() /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:749:9
[task 2020-05-17T07:50:56.797Z] 07:50:56 INFO - PID 5116 | #1 0x7fbeb89ed3f6 in mozilla::ipc::MessageChannel::~MessageChannel() /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:633:3
[task 2020-05-17T07:50:56.801Z] 07:50:56 INFO - PID 5116 | #2 0x7fbeb8a133b2 in mozilla::ipc::IToplevelProtocol::~IToplevelProtocol() /builds/worker/workspace/obj-build/dist/include/mozilla/ipc/ProtocolUtils.h:391:32
[task 2020-05-17T07:50:56.821Z] 07:50:56 INFO - PID 5116 | #3 0x7fbebd77b7b7 in Release /builds/worker/checkouts/gecko/dom/media/gmp/GMPContentParent.h:25:3
[task 2020-05-17T07:50:56.822Z] 07:50:56 INFO - PID 5116 | #4 0x7fbebd77b7b7 in Release /builds/worker/workspace/obj-build/dist/include/mozilla/RefPtr.h:50:40
[task 2020-05-17T07:50:56.822Z] 07:50:56 INFO - PID 5116 | #5 0x7fbebd77b7b7 in Release /builds/worker/workspace/obj-build/dist/include/mozilla/RefPtr.h:381:36
[task 2020-05-17T07:50:56.822Z] 07:50:56 INFO - PID 5116 | #6 0x7fbebd77b7b7 in ~RefPtr /builds/worker/workspace/obj-build/dist/include/mozilla/RefPtr.h:81:7
[task 2020-05-17T07:50:56.823Z] 07:50:56 INFO - PID 5116 | #7 0x7fbebd77b7b7 in ~nsBaseHashtableET /builds/worker/workspace/obj-build/dist/include/nsBaseHashtable.h:72:32
[task 2020-05-17T07:50:56.825Z] 07:50:56 INFO - PID 5116 | #8 0x7fbebd77b7b7 in nsTHashtable<nsBaseHashtableET<nsUint64HashKey, RefPtr<mozilla::gmp::GMPContentParent> > >::s_ClearEntry(PLDHashTable*, PLDHashEntryHdr*) /builds/worker/workspace/obj-build/dist/include/nsTHashtable.h:512:37
[task 2020-05-17T07:50:56.842Z] 07:50:56 INFO - PID 5116 | #9 0x7fbeb77a9866 in operator() /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:292:7
[task 2020-05-17T07:50:56.843Z] 07:50:56 INFO - PID 5116 | #10 0x7fbeb77a9866 in ForEachSlot<(lambda at /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:290:51)> /builds/worker/workspace/obj-build/dist/include/PLDHashTable.h:357:9
[task 2020-05-17T07:50:56.845Z] 07:50:56 INFO - PID 5116 | #11 0x7fbeb77a9866 in ForEachSlot<(lambda at /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:290:51)> /builds/worker/workspace/obj-build/dist/include/PLDHashTable.h:347:7
[task 2020-05-17T07:50:56.847Z] 07:50:56 INFO - PID 5116 | #12 0x7fbeb77a9866 in PLDHashTable::~PLDHashTable() /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:290:15
[task 2020-05-17T07:50:56.848Z] 07:50:56 INFO - PID 5116 | #13 0x7fbebd74e64e in ~nsTHashtable /builds/worker/workspace/obj-build/dist/include/nsTHashtable.h:170:27
[task 2020-05-17T07:50:56.849Z] 07:50:56 INFO - PID 5116 | #14 0x7fbebd74e64e in ~GMPServiceChild /builds/worker/checkouts/gecko/dom/media/gmp/GMPServiceChild.cpp:426:35
[task 2020-05-17T07:50:56.850Z] 07:50:56 INFO - PID 5116 | #15 0x7fbebd74e64e in mozilla::gmp::GMPServiceChild::~GMPServiceChild() /builds/worker/checkouts/gecko/dom/media/gmp/GMPServiceChild.cpp:426:35
[task 2020-05-17T07:50:56.854Z] 07:50:56 INFO - PID 5116 | #16 0x7fbebd74d3bd in operator() /builds/worker/workspace/obj-build/dist/include/mozilla/UniquePtr.h:460:5
[task 2020-05-17T07:50:56.855Z] 07:50:56 INFO - PID 5116 | #17 0x7fbebd74d3bd in reset /builds/worker/workspace/obj-build/dist/include/mozilla/UniquePtr.h:302:7
[task 2020-05-17T07:50:56.856Z] 07:50:56 INFO - PID 5116 | #18 0x7fbebd74d3bd in operator= /builds/worker/workspace/obj-build/dist/include/mozilla/UniquePtr.h:275:5
[task 2020-05-17T07:50:56.857Z] 07:50:56 INFO - PID 5116 | #19 0x7fbebd74d3bd in mozilla::gmp::GeckoMediaPluginServiceChild::Observe(nsISupports*, char const*, char16_t const*) /builds/worker/checkouts/gecko/dom/media/gmp/GMPServiceChild.cpp:357:21
[task 2020-05-17T07:50:56.871Z] 07:50:56 INFO - PID 5116 | #20 0x7fbebd74d4cc in non-virtual thunk to mozilla::gmp::GeckoMediaPluginServiceChild::Observe(nsISupports*, char const*, char16_t const*) /builds/worker/checkouts/gecko/dom/media/gmp/GMPServiceChild.cpp
[task 2020-05-17T07:50:56.873Z] 07:50:56 INFO - PID 5116 | #21 0x7fbeb77c7bbe in nsObserverList::NotifyObservers(nsISupports*, char const*, char16_t const*) /builds/worker/checkouts/gecko/xpcom/ds/nsObserverList.cpp:65:19
[task 2020-05-17T07:50:56.874Z] 07:50:56 INFO - PID 5116 | #22 0x7fbeb77ceb91 in nsObserverService::NotifyObservers(nsISupports*, char const*, char16_t const*) /builds/worker/checkouts/gecko/xpcom/ds/nsObserverService.cpp:288:19
[task 2020-05-17T07:50:56.874Z] 07:50:56 INFO - PID 5116 | #23 0x7fbeb796a1e3 in mozilla::ShutdownXPCOM(nsIServiceManager*) /builds/worker/checkouts/gecko/xpcom/build/XPCOMInit.cpp:638:24
[task 2020-05-17T07:50:56.875Z] 07:50:56 INFO - PID 5116 | #24 0x7fbec254a96c in XRE_TermEmbedding() /builds/worker/checkouts/gecko/toolkit/xre/nsEmbedFunctions.cpp:222:3
[task 2020-05-17T07:50:56.876Z] 07:50:56 INFO - PID 5116 | #25 0x7fbeb8a25b9e in mozilla::ipc::ScopedXREEmbed::Stop() /builds/worker/checkouts/gecko/ipc/glue/ScopedXREEmbed.cpp:90:5
[task 2020-05-17T07:50:56.876Z] 07:50:56 INFO - PID 5116 | #26 0x7fbec254ba25 in XRE_InitChildProcess(int, char**, XREChildData const*) /builds/worker/checkouts/gecko/toolkit/xre/nsEmbedFunctions.cpp:744:16
[task 2020-05-17T07:50:56.890Z] 07:50:56 INFO - PID 5116 | #27 0x55daf4fcc16d in content_process_main(mozilla::Bootstrap*, int, char**) /builds/worker/checkouts/gecko/browser/app/../../ipc/contentproc/plugin-container.cpp:56:28
[task 2020-05-17T07:50:56.890Z] 07:50:56 INFO - PID 5116 | #28 0x55daf4fcc5aa in main /builds/worker/checkouts/gecko/browser/app/nsBrowserApp.cpp:303:18
[task 2020-05-17T07:50:57.003Z] 07:50:57 INFO - PID 5116 | #29 0x7fbed78c4b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
[task 2020-05-17T07:50:57.003Z] 07:50:57 INFO - PID 5116 | #30 0x55daf4f20cb5 in _start (/builds/worker/workspace/build/application/firefox/firefox+0x55cb5)
[task 2020-05-17T07:50:57.003Z] 07:50:57 INFO - PID 5116 | AddressSanitizer can not provide additional info.
[task 2020-05-17T07:50:57.008Z] 07:50:57 INFO - PID 5116 | SUMMARY: AddressSanitizer: SEGV /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:749:9 in mozilla::ipc::MessageChannel::Clear()
[task 2020-05-17T07:50:57.009Z] 07:50:57 INFO - PID 5116 | ==6550==ABORTING
[task 2020-05-17T07:50:57.087Z] 07:50:57 INFO - .
[task 2020-05-17T07:50:57.087Z] 07:50:57 INFO - TEST-OK | /encoding-detection/zh-Big5-late.tentative.html | took 3082ms

Looked in last 15 failure logs and this fails in:

  • html/cross-origin-embedder-policy/blob.https.html
  • encrypted-media/clearkey-mp4-waiting-for-a-key.https.html
  • encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential.https.html
  • eventsource/format-field-unknown.htm

Discussion of current patches:

Making PGMPContent + PGMPService refcounted

The GMP machinery has had reoccurring issues with racy shutdowns and IPC (like the one in this bug). It makes sense to me to refcount more and more actors involved to avoid these issues. I'm refcounting PGMPContent + PGMPService because I have triggered asserts and crashes related to these specifically while debugging using the testcase in comment 66.

I plan to revise https://phabricator.services.mozilla.com/D75178 once bug 1638124 lands as that patch can then be simplified.

Having GeckoMediaPluginServiceChild block shutdown

This has been touched on in comment 59, but I want to make sure my working is sane so I'm going to go into a lot more detail below. Chris Pearce's blog on EME/GMP is a useful reference here, and the diagram of the GMP architecture is something I often gaze at.

The GMP architecture is complicated, I personally find it quite difficult to reason about how it functions in a non-ideal-happy path shutdown. By blocking shutdown we can require that the GMP components are in a certain state before progressing. I.e. blocking at xpcom-will-shutdown until we're in a certain happy state makes it much easier to reason about what we need to deal with when we observe xpcom-shutdown-threads.

The specific failure case that this bug stems from is if we shutdown while the content process is requesting a GMP process be started to host the CDM. A quick version of what goes wrong:

  • We call GetCDM in the content process because a site is using the MediaKeys interface.
  • This will call into GetContentParent in the GMPServiceChild.
  • This begins the process of requesting a GMP process be started by the parent process.
    • Ideally the parent then passes back an endpoint to content, which will then bind that endpoint and allow communication directly from content to the GMP process.
  • We shutdown while the above is happening, in the content process we move through shutdown steps and observe xpcom-shutdown-threads so we shutdown/kill the GMP thread.
  • We run whatever remaining tasks we have queued on the GMP thread, and end up in a state where we'd need to run yet more tasks on the GMP thread to not leak some of our objects. However, we're out of luck, because our thread is dead and we're shutting down, so we trigger LSAN (seen prior to the backout of my first patch).

The parent process actor for the GMPService has code to block shutdown and prevent itself shutting down too early, but the content process actor does not. My patch tries to change that so content will block shutdown until the GMP process associated with it is no longer in use. I'm hoping this results in the following shutdown dependencies:

  • GMPServiceChild will not shutdown until its GMPContentParents are ready for shutdown. I.e. GMP machinery will block shutdown in the content process until GMP processes are happy.
  • GMPServiceParent will not shutdown its GMPServiceChildren are ready for shutdown. I.e. GMP machinery will block shutdown in the parent process until the content processes using GMP are happy. This should already be true prior to my patch.

This ideally results in a more deterministic shutdown of the machinery involved that avoids bugs like this one.

The current patch has comments + logs + asserts I'd like to land in an ideal world, but can yank them for sec and landing and return them later.

I'm happy to provide further details to the above and/or on how I debugged this to arrive at my conclusions. The testcase above + MOZ_LOG=EME:5,GMP:5 + an LSAN build and connecting different leaks and crashes with logging output were my primary tools.

I'm looking for a reviewer for the last patch. I'm not certain we have any suitable media folks ATM. Nika, would you feel comfortable reviewing https://phabricator.services.mozilla.com/D75180 or have any suggestions as to a suitable reviewer?

Flags: needinfo?(nika)

I would generally prefer if someone more familiar with GMP reviewed that patch, due to the number of GMP-lifecycle-specific changes which it makes. If there's nobody else who'd be up to review it, I suppose I could dig in & do it, but it might take a while.

Flags: needinfo?(nika)

Andrew, do you have any thoughts on who would be a suitable reviewer for the last change?

Flags: needinfo?(continuation)

I don't know. What would you say are the difficult parts of the patch? Is anybody besides you familiar with the GMP lifecycle?

Flags: needinfo?(continuation)

Understanding all the different systems involved and making sure they're being used sanely during shutdown has been difficult for me. We have IPC + xpcom shutdown logic to handle, then the GMP building its own logic on top of them. Making sure I'm not doing anything unwise re IPC + xpcom would be good.

:gerald is the only person I'm aware of who is familiar with the GMP, however he is no longer with the media team. I believe :peterv has done some work on it, but I think it was before my time.

I'll do some more follow up outside this bug to see what they think. Any other suggestions would also be appreciated.

Marking as wontfix for 77 as we build RC tomorrow and this is a sec-moderate, feel free to reach out to me if you think the upcoming patch should be in the 77.0 release.

status-firefox77: affected → wontfix

Jon Bauman is helping me look at the GMP and the last patch in the series. Marking 78 as won't fix as I think the knowledge sharing involved will not be done in that time frame.

status-firefox78: affected → wontfix
status-firefox79: --- → affected

I'm going to optimistically assume we'll get these patches into the Fx79/78.1esr releases shipping in a bit over a month, but I'm not convinced this is going to be worth the effort of backporting to ESR68 given the sec rating and how close to EOL it is.

Status: REOPENED → ASSIGNED
status-firefox-esr68: affected → wontfix
tracking-firefox-esr68: ? → -
tracking-firefox-esr78: ? → 79+

Hi, where do we stand in getting the necessary reviews on these patches?

Flags: needinfo?(bvandyk)

I would estimate it will take weeks, with the possibility of 4+ weeks being within expected parameters.

Flags: needinfo?(bvandyk)

Jon reports running into Bug 1642735 while testing this. It sounds like we should clobber that to make sure testing of this is reliable.

I'm struggling to see much point in continually tracking this bug release after release when it doesn't seem to be moving at all.

status-firefox81: affected → fix-optional
status-firefox82: --- → fix-optional
status-firefox-esr78: affected → fix-optional
tracking-firefox81: + → -
tracking-firefox82: --- → -
tracking-firefox-esr78: 81+ → ?

There is work being done here, but it's largely about GMP learning so isn't very "visible". And unfortunately it's my #2 priority right now and it keeps getting bumped as security issues are discovered related to my #1 priority.

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #82)

Make PGMPContent protocol refcounted. r=nika
https://hg.mozilla.org/integration/autoland/rev/34e6bd2c67db5c52c671fa84414ad37b238c2d94
Make PGMPService protocol refcounted. r=nika
https://hg.mozilla.org/integration/autoland/rev/151d4618b8ea31a972713e51817d48e75645e8c7
Have GeckoMediaPluginServiceChild use async shutdown blocker. r=jbauman
https://hg.mozilla.org/integration/autoland/rev/4c92b9d3c7d8d58ef42148232edde5c30149f61e
https://hg.mozilla.org/mozilla-central/rev/34e6bd2c67db
https://hg.mozilla.org/mozilla-central/rev/151d4618b8ea
https://hg.mozilla.org/mozilla-central/rev/4c92b9d3c7d8

Is there any work left to be done in this bug?

No, thanks for the heads up. I forgot this had a leave open on it.

Status: ASSIGNED → RESOLVED
Closed: 4 years ago → 4 years ago
status-firefox83: --- → fixed
Flags: needinfo?(bvandyk)
Keywords: leave-open
Resolution: --- → FIXED

NI me to re-enable tests once this has baked for a bit. https://phabricator.services.mozilla.com/D72515 disabled some of the tests here and it makes sense to re-enable them, but I'd like this bake for a week or so to make sure it looks good.

Flags: needinfo?(bvandyk)

Bug 1670280 tracks re-enabling the tests mentioned in comment 85.

Flags: needinfo?(bvandyk)
Flags: qe-verify+
Whiteboard: [stockwell needswork:owner][stockwell disabled] → [stockwell needswork:owner][stockwell disabled][post-critsmash-triage]

Removing stockwell tags as we've patched this and re-enabled tests (bug 1670280).

Whiteboard: [stockwell needswork:owner][stockwell disabled][post-critsmash-triage] → [post-critsmash-triage]

Hello Bryce!
Can you please help me with some steps in order to verify this manually? I don't know for sure that doing the steps from comment 66 and using the test case on an asan build is all I have to do in order to verify/reproduce this.
Thank you!

Flags: needinfo?(bvandyk)

(In reply to Alexandru Trif, QA [:atrif] from comment #88)

Hello Bryce!
Can you please help me with some steps in order to verify this manually? I don't know for sure that doing the steps from comment 66 and using the test case on an asan build is all I have to do in order to verify/reproduce this.
Thank you!

Those are the best repro steps I have -- it's intermittent due to the bug involving races. When testing with a versions that would fault I could normally catch the bug within 5 or so runs. I'd set the needed pref and start Fx from the command line with the test file as an argument so it would load and immediately close. I'd then watch the asan output to determine if the issue occurred.

After the fix is applied I tried running 10+ times without issue. It's not a deterministic proof of this being fixed, but it's better than nothing.

If you want to be sure the steps can cause the fault you could try with an asan build from before the fix was applied to and get that faulting, then use the same steps with a later build (I did the same thing while working on this).

Flags: needinfo?(bvandyk)

(In reply to Bryce Seager van Dyk (:bryce) from comment #89)

(In reply to Alexandru Trif, QA [:atrif] from comment #88)

Hello Bryce!
Can you please help me with some steps in order to verify this manually? I don't know for sure that doing the steps from comment 66 and using the test case on an asan build is all I have to do in order to verify/reproduce this.
Thank you!

Those are the best repro steps I have -- it's intermittent due to the bug involving races. When testing with a versions that would fault I could normally catch the bug within 5 or so runs. I'd set the needed pref and start Fx from the command line with the test file as an argument so it would load and immediately close. I'd then watch the asan output to determine if the issue occurred.

After the fix is applied I tried running 10+ times without issue. It's not a deterministic proof of this being fixed, but it's better than nothing.

If you want to be sure the steps can cause the fault you could try with an asan build from before the fix was applied to and get that faulting, then use the same steps with a later build (I did the same thing while working on this).

You can also use grizzly for reproducing issues like these:

  1. pip install grizzly-framework
  2. python3 -m grizzly.replay --xvfb ~/builds/mc-asan/firefox --repeat 20 ./testcase.html --no-harness

The prefs :bryce mentioned should be enabled by default in grizzly replay.

Thank you a lot for the responses. They were a great help and sorry for the late reply.

I have managed to reproduce the issue with Firefox 79.0a1 (20200620013707) ASAN build on Ubuntu 18.04. Using the steps provided in comment 89 I reproduced the issue after a few tries. Also with python3 -m grizzly.replay --xvfb '/home/alexandru.trif/Desktop/tarasan/firefox/firefox' --repeat 20 '/home/alexandru.trif/Desktop/tarasan/firefox/testcase.html' --no-harness command it seems that I get the Result: [@ mozilla::ipc::MessageChannel::Clear] (1330a6f0:b2a9b53e), Result successfully reproduced message on first try.

I cannot reproduce the issue anymore with Firefox 83.0b2 (20201020173725) and Firefox 84.0a1 (20201022093646) ASAN builds by loading Firefox with the test case opened in terminal several times or by using the grizzly replay command. With the grizzly replay command, it seems that the 20 replays are satisfied:
Performing replay (20/20)... [2020-10-22 16:20:24] Target closed itself [2020-10-22 16:20:24] Failed to reproduce results [2020-10-22 16:20:24] Shutting down... [2020-10-22 16:20:24] Done.

Marking the issue as verified based on the above. Thank you!

Status: RESOLVED → VERIFIED
status-firefox83: fixed → verified
Flags: qe-verify+

Huzzah! Nice work, Bryce!

Is this something we should consider uplifting to ESR78? Looks like it's going to need some rebasing if so (mostly the last patch).

Flags: needinfo?(bvandyk)

(In reply to Ryan VanderMeulen [:RyanVM] from comment #93)

Is this something we should consider uplifting to ESR78? Looks like it's going to need some rebasing if so (mostly the last patch).

It's probably a good idea -- most of this bug was asserts and upsetting sanitizers, but there's at least one scary manifestation (bug 1558288) which it would be better to be safe than sorry around. Should I look at getting a happy rebased version ready?

Please do!

Rebasing now. Having to work around bug 1650696, bug 1654111, bug 1645982 and some others having landed before this. Think I'm close.

Rebased patches. The final patch required a bit of work, but hopefully is all sorted.

Comment on attachment 9185788 [details] [diff] [review]
Bug1551615-esr78rebase.patch

ESR Uplift Approval Request

  • If this is not a sec:{high,crit} bug, please state case for ESR consideration: Fixes a racy shutdown case in the GMP -- implicated in various sec bugs including possible sec-high UAF.
  • User impact if declined: Possible sec issues due to GMP shutdown and xpcom shutdown racing, this includes a UAF.
  • Fix Landed on Version: 83
  • Risk to taking this patch: Medium
  • Why is the change risky/not risky? (and alternatives if risky): The patch set is non-trivial and has had to be manually rebased. It also deals with complex code. The original code has undergone a lot of manual testing and we don't see any issues, however it's possible I've missed something in the rebase. I think the most likely result of me making a mistake would be shutdown hangs -- but wouldn't think that overly likely and I've done some manual testing and not seen any issues.
  • String or UUID changes made by this patch: None.
Flags: needinfo?(bvandyk)
Attachment #9185788 - Flags: approval-mozilla-esr78?

Just noticed the original landing here reintroduced some AbstractThread usage that we don't want. Will fix in a follow up.

Regressions: 1675365

Comment on attachment 9185788 [details] [diff] [review]
Bug1551615-esr78rebase.patch

Approved for 78.5esr, thanks.

Attachment #9185788 - Flags: approval-mozilla-esr78? → approval-mozilla-esr78+
Whiteboard: [post-critsmash-triage] → [post-critsmash-triage][adv-main83+r]
Whiteboard: [post-critsmash-triage][adv-main83+r] → [post-critsmash-triage][adv-main83+r][adv-esr78.5+r]
Group: core-security-release
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: