1661364 - Intermittent runner.py | application crashed [@ mozilla::dom::ContentParent::AssertNotInPool()]

Status: RESOLVED FIXED

(Core :: DOM: Content Processes, defect, P2)

Description

[Treeherder Bug Filer]

Filed by: nbeleuzu [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=314128672&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/EV6CHZBHRuW5tbgvInMkaA/runs/0/artifacts/public/logs/live_backing.log

---

[task 2020-08-26T22:34:53.862Z] 22:34:53 ERROR - PROCESS-CRASH | runner.py | application crashed [@ mozilla::dom::ContentParent::AssertNotInPool()]
[task 2020-08-26T22:34:53.862Z] 22:34:53 INFO - Mozilla crash reason: MOZ_RELEASE_ASSERT(group.GetKey()->GetHostProcess(mRemoteType) != this) (still a host process for one of our groups?)
[task 2020-08-26T22:34:53.862Z] 22:34:53 INFO - Crash dump filename: /var/folders/nk/c_lc4b9n5jg1t38z5vrwwnkm000017/T/tmpzcwSiu/profile/minidumps/58FDAF29-12AD-4B45-80C6-86010CB4665A.dmp
[task 2020-08-26T22:34:53.862Z] 22:34:53 INFO - Operating system: Mac OS X
[task 2020-08-26T22:34:53.863Z] 22:34:53 INFO - 10.14.5 18F132
[task 2020-08-26T22:34:53.863Z] 22:34:53 INFO - CPU: amd64
[task 2020-08-26T22:34:53.863Z] 22:34:53 INFO - family 6 model 69 stepping 1
[task 2020-08-26T22:34:53.863Z] 22:34:53 INFO - 4 CPUs
[task 2020-08-26T22:34:53.863Z] 22:34:53 INFO - GPU: UNKNOWN
[task 2020-08-26T22:34:53.863Z] 22:34:53 INFO - Crash reason: EXC_BAD_ACCESS / KERN_INVALID_ADDRESS
[task 2020-08-26T22:34:53.863Z] 22:34:53 INFO - Crash address: 0x0
[task 2020-08-26T22:34:53.863Z] 22:34:53 INFO - Process uptime: 6 seconds
[task 2020-08-26T22:34:53.864Z] 22:34:53 INFO - Thread 0 (crashed)
[task 2020-08-26T22:34:53.864Z] 22:34:53 INFO - 0 XUL!mozilla::dom::ContentParent::AssertNotInPool() [ContentParent.cpp:827f8ba12fdbabb7b19b36368a33dd27cd4954af : 1676 + 0x11]
[task 2020-08-26T22:34:53.864Z] 22:34:53 INFO - rax = 0x00000001196b1895 rdx = 0x0000000000000003
[task 2020-08-26T22:34:53.864Z] 22:34:53 INFO - rcx = 0x000000010fcf03a8 rbx = 0x0000000111c317e8
[task 2020-08-26T22:34:53.864Z] 22:34:53 INFO - rsi = 0x000000012130a1a8 rdi = 0x000000012130a1a8
[task 2020-08-26T22:34:53.864Z] 22:34:53 INFO - rbp = 0x00007ffedff91270 rsp = 0x00007ffedff911f0
[task 2020-08-26T22:34:53.864Z] 22:34:53 INFO - r8 = 0x0000000000000062 r9 = 0x0000000111c316c0
[task 2020-08-26T22:34:53.864Z] 22:34:53 INFO - r10 = 0x00000000030e8706 r11 = 0x000000011661f4c0
[task 2020-08-26T22:34:53.864Z] 22:34:53 INFO - r12 = 0x0000000111c31000 r13 = 0x0000000000000000
[task 2020-08-26T22:34:53.865Z] 22:34:53 INFO - r14 = 0x0000000111c316c0 r15 = 0x00007ffedff91220
[task 2020-08-26T22:34:53.865Z] 22:34:53 INFO - rip = 0x0000000115520f3f
[task 2020-08-26T22:34:53.865Z] 22:34:53 INFO - Found by: given as instruction pointer in context
[task 2020-08-26T22:34:53.865Z] 22:34:53 INFO - 1 XUL!mozilla::dom::ContentParent::TryToRecycle() [ContentParent.cpp:827f8ba12fdbabb7b19b36368a33dd27cd4954af : 1995 + 0x8]
[task 2020-08-26T22:34:53.865Z] 22:34:53 INFO - rbp = 0x00007ffedff912a0 rsp = 0x00007ffedff91280
[task 2020-08-26T22:34:53.865Z] 22:34:53 INFO - rip = 0x00000001155223c8
[task 2020-08-26T22:34:53.865Z] 22:34:53 INFO - Found by: previous frame's frame pointer
[task 2020-08-26T22:34:53.865Z] 22:34:53 INFO - 2 XUL!mozilla::dom::ContentParent::NotifyTabDestroyed(mozilla::dom::IdType<mozilla::dom::BrowserParent> const&, bool) [ContentParent.cpp:827f8ba12fdbabb7b19b36368a33dd27cd4954af : 2146 + 0x8]
[task 2020-08-26T22:34:53.865Z] 22:34:53 INFO - rbp = 0x00007ffedff912e0 rsp = 0x00007ffedff912b0
[task 2020-08-26T22:34:53.866Z] 22:34:53 INFO - rip = 0x000000011550db03
[task 2020-08-26T22:34:53.866Z] 22:34:53 INFO - Found by: previous frame's frame pointer
[task 2020-08-26T22:34:53.866Z] 22:34:53 INFO - 3 XUL!mozilla::dom::PBrowserParent::OnMessageReceived(IPC::Message const&) [PBrowserParent.cpp: : 4245 + 0x1f]
[task 2020-08-26T22:34:53.866Z] 22:34:53 INFO - rbp = 0x00007ffedff91550 rsp = 0x00007ffedff912f0
[task 2020-08-26T22:34:53.866Z] 22:34:53 INFO - rip = 0x00000001137afa55
[task 2020-08-26T22:34:53.866Z] 22:34:53 INFO - Found by: previous frame's frame pointer
[task 2020-08-26T22:34:53.866Z] 22:34:53 INFO - 4 XUL!mozilla::dom::PContentParent::OnMessageReceived(IPC::Message const&) [PContentParent.cpp: : 6674 + 0xd]
[task 2020-08-26T22:34:53.866Z] 22:34:53 INFO - rbp = 0x00007ffedff925c0 rsp = 0x00007ffedff91560
[task 2020-08-26T22:34:53.867Z] 22:34:53 INFO - rip = 0x00000001134e6c80
[task 2020-08-26T22:34:53.867Z] 22:34:53 INFO - Found by: previous frame's frame pointer
[task 2020-08-26T22:34:53.867Z] 22:34:53 INFO - 5 XUL!mozilla::ipc::MessageChannel::DispatchMessage(IPC::Message&&) [MessageChannel.cpp:827f8ba12fdbabb7b19b36368a33dd27cd4954af : 2074 + 0x7f]
[task 2020-08-26T22:34:53.867Z] 22:34:53 INFO - rbp = 0x00007ffedff928a0 rsp = 0x00007ffedff925d0
[task 2020-08-26T22:34:53.867Z] 22:34:53 INFO - rip = 0x0000000112676c7f
[task 2020-08-26T22:34:53.867Z] 22:34:53 INFO - Found by: previous frame's frame pointer
[task 2020-08-26T22:34:53.867Z] 22:34:53 INFO - 6 XUL!mozilla::ipc::MessageChannel::MessageTask::Run() [MessageChannel.cpp:827f8ba12fdbabb7b19b36368a33dd27cd4954af : 1953 + 0xf6]
[task 2020-08-26T22:34:53.867Z] 22:34:53 INFO - rbp = 0x00007ffedff928f0 rsp = 0x00007ffedff928b0
[task 2020-08-26T22:34:53.867Z] 22:34:53 INFO - rip = 0x0000000112677f8e
[task 2020-08-26T22:34:53.867Z] 22:34:53 INFO - Found by: previous frame's frame pointer
[task 2020-08-26T22:34:53.868Z] 22:34:53 INFO - 7 XUL!mozilla::TaskController::DoExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) [TaskController.cpp:827f8ba12fdbabb7b19b36368a33dd27cd4954af : 512 + 0xb9]
[task 2020-08-26T22:34:53.868Z] 22:34:53 INFO - rbp = 0x00007ffedff92dc0 rsp = 0x00007ffedff92900
[task 2020-08-26T22:34:53.868Z] 22:34:53 INFO - rip = 0x0000000112639d7b
[task 2020-08-26T22:34:53.868Z] 22:34:53 INFO - Found by: previous frame's frame pointer
[task 2020-08-26T22:34:53.868Z] 22:34:53 INFO - 8 XUL!mozilla::detail::RunnableFunction<mozilla::TaskController::InitializeInternal()::$_5>::Run() [nsThreadUtils.h:827f8ba12fdbabb7b19b36368a33dd27cd4954af : 577 + 0x1f]
[task 2020-08-26T22:34:53.868Z] 22:34:53 INFO - rbp = 0x00007ffedff92e00 rsp = 0x00007ffedff92dd0
[task 2020-08-26T22:34:53.868Z] 22:34:53 INFO - rip = 0x000000011263baaa
[task 2020-08-26T22:34:53.868Z] 22:34:53 INFO - Found by: previous frame's frame pointer
[task 2020-08-26T22:34:53.868Z] 22:34:53 INFO - 9 XUL!nsThread::ProcessNextEvent(bool, bool*) [nsThread.cpp:827f8ba12fdbabb7b19b36368a33dd27cd4954af : 1234 + 0x10]
[task 2020-08-26T22:34:53.869Z] 22:34:53 INFO - rbp = 0x00007ffedff93320 rsp = 0x00007ffedff92e10
[task 2020-08-26T22:34:53.869Z] 22:34:53 INFO - rip = 0x000000011263fc99
[task 2020-08-26T22:34:53.869Z] 22:34:53 INFO - Found by: previous frame's frame pointer
[task 2020-08-26T22:34:53.869Z] 22:34:53 INFO - 10 XUL!NS_ProcessPendingEvents(nsIThread*, unsigned int) [nsThreadUtils.cpp:827f8ba12fdbabb7b19b36368a33dd27cd4954af : 461 + 0x1e]
[task 2020-08-26T22:34:53.878Z] 22:34:53 INFO - rbp = 0x00007ffedff93370 rsp = 0x00007ffedff93330
[task 2020-08-26T22:34:53.878Z] 22:34:53 INFO - rip = 0x000000011263d910
[task 2020-08-26T22:34:53.878Z] 22:34:53 INFO - Found by: previous frame's frame pointer
[task 2020-08-26T22:34:53.878Z] 22:34:53 INFO - 11 XUL!nsAppShell::ProcessGeckoEvents(void*) [nsAppShell.mm:827f8ba12fdbabb7b19b36368a33dd27cd4954af : 429 + 0x50]
[task 2020-08-26T22:34:53.878Z] 22:34:53 INFO - rbp = 0x00007ffedff933d0 rsp = 0x00007ffedff93380
[task 2020-08-26T22:34:53.879Z] 22:34:53 INFO - rip = 0x000000011585e120
[task 2020-08-26T22:34:53.879Z] 22:34:53 INFO - Found by: previous frame's frame pointer
[task 2020-08-26T22:34:53.879Z] 22:34:53 INFO - 12 CoreFoundation + 0x58083
[task 2020-08-26T22:34:53.879Z] 22:34:53 INFO - rbp = 0x00007ffedff933e0 rsp = 0x00007ffedff933e0
[task 2020-08-26T22:34:53.879Z] 22:34:53 INFO - rip = 0x00007fff2ff33083
[task 2020-08-26T22:34:53.879Z] 22:34:53 INFO - Found by: previous frame's frame pointer
[task 2020-08-26T22:34:53.879Z] 22:34:53 INFO - 13 CoreFoundation + 0x58029
[task 2020-08-26T22:34:53.879Z] 22:34:53 INFO - rbp = 0x00007ffedff93410 rsp = 0x00007ffedff933f0
[task 2020-08-26T22:34:53.880Z] 22:34:53 INFO - rip = 0x00007fff2ff33029
[task 2020-08-26T22:34:53.880Z] 22:34:53 INFO - Found by: previous frame's frame pointer
[task 2020-08-26T22:34:53.880Z] 22:34:53 INFO - 14 CoreFoundation + 0x3b9eb
[task 2020-08-26T22:34:53.880Z] 22:34:53 INFO - rbp = 0x00007ffedff93480 rsp = 0x00007ffedff93420
[task 2020-08-26T22:34:53.880Z] 22:34:53 INFO - rip = 0x00007fff2ff169eb
[task 2020-08-26T22:34:53.880Z] 22:34:53 INFO - Found by: previous frame's frame pointer
[task 2020-08-26T22:34:53.880Z] 22:34:53 INFO - 15 CoreFoundation + 0x3afb5
[task 2020-08-26T22:34:53.880Z] 22:34:53 INFO - rbp = 0x00007ffedff94190 rsp = 0x00007ffedff93490
[task 2020-08-26T22:34:53.880Z] 22:34:53 INFO - rip = 0x00007fff2ff15fb5
[task 2020-08-26T22:34:53.881Z] 22:34:53 INFO - Found by: previous frame's frame pointer
[task 2020-08-26T22:34:53.881Z] 22:34:53 INFO - 16 CoreFoundation + 0x3a8be
[task 2020-08-26T22:34:53.881Z] 22:34:53 INFO - rbp = 0x00007ffedff94220 rsp = 0x00007ffedff941a0
[task 2020-08-26T22:34:53.881Z] 22:34:53 INFO - rip = 0x00007fff2ff158be
[task 2020-08-26T22:34:53.881Z] 22:34:53 INFO - Found by: previous frame's frame pointer
[task 2020-08-26T22:34:53.881Z] 22:34:53 INFO - 17 HIToolbox!RunCurrentEventLoopInMode + 0x124
[task 2020-08-26T22:34:53.881Z] 22:34:53 INFO - rbp = 0x00007ffedff94270 rsp = 0x00007ffedff94230
[task 2020-08-26T22:34:53.882Z] 22:34:53 INFO - rip = 0x00007fff2f20196b
[task 2020-08-26T22:34:53.882Z] 22:34:53 INFO - Found by: previous frame's frame pointer
[task 2020-08-26T22:34:53.882Z] 22:34:53 INFO - 18 HIToolbox!ReceiveNextEventCommon + 0x25b
[task 2020-08-26T22:34:53.882Z] 22:34:53 INFO - rbp = 0x00007ffedff942f0 rsp = 0x00007ffedff94280
[task 2020-08-26T22:34:53.882Z] 22:34:53 INFO - rip = 0x00007fff2f2016a5
[task 2020-08-26T22:34:53.882Z] 22:34:53 INFO - Found by: previous frame's frame pointer

Comment 1

[Nika Layzell [:nika] (ni? for response)]

This assertion failure was one added in bug 1660342 to figure out what the potential cause of that issue was. I think with this extra information I have a slightly better idea of what is going wrong, and may be able to fix it.

Comment 2

[Intermittent Failures Robot]

40 failures in 755 pushes (0.053 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 23
• mozilla-central: 17

Platform breakdown:

• macosx1014-64-shippable: 18
• linux64-shippable-qr: 6
• linux1804-64-qr: 1
• windows10-64-shippable-qr: 4
• linux64-shippable: 6
• windows7-32-shippable: 2
• windows10-64-shippable: 2
• linux1804-64-shippable-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1661364&startday=2020-08-27&endday=2020-08-27&tree=all

Comment 3

[Nika Layzell [:nika] (ni? for response)]

Attachment: Bug 1661364 - Part 1: Don't remove recycled content processes from the pool,

Previously, we would use the PreallocatedProcessManager to handle recycled
"web" content processes, which would remove them from the pool and try to make
them act like "prealloc" content processes.

Unlike true preallocated content processes, we can end up having content loaded
in recycled "web" content processes, such as due to a new pop-up being created
at an inconvenient time during process teardown. The complexity around this
semi-prealloc state caused some assertion failures in other related code.

This new approach doesn't remove the recycled process from the process selection
pool. Instead, after process selection decides to spawn a new "web" process, a
pointer to the specific "recycled" content process is fetched and returned by
the process selection algorithm before asking the preallocator.

This new approach should have similar behaviour, but avoids some of the pitfalls
caused by removing the entry from the process selection pool after it has begun
hosting content.

Comment 4

[Nika Layzell [:nika] (ni? for response)]

Attachment: Bug 1661364 - Part 2: Don't recycle web processes with Fission enabled,

It is uncommon to have long-lived "web" processes when Fission is enabled, so in
general we probably don't want to be performing content process recycling.

We also skip recycling web processes when E10s is disabled, as the process
preallocator would be disabled in that case, so recycling processes can lead to
mochitest-chrome test failures.

Comment 5

[Nika Layzell [:nika] (ni? for response)]

Attachment: Bug 1661364 - Part 3: Add a MaybeBeginShutDown method,

This helper method abstracts over the common tasks performed in every normal
shutdown codepath. This will be useful for making process recycling respect
preallocation being disabled.

Comment 6

[Nika Layzell [:nika] (ni? for response)]

Attachment: Bug 1661364 - Part 4: Disable process recycling when prealloc is disabled,

Various tests disable process preallocation in order to get "stable" process
counts for testing purposes. In order to keep this behaviour, we need to disable
and shut down the recycled E10S process whenever the preallocator is disabled.

This is accomplished by having the preallocator expose it's enabled status to
ContentParent, and by having the preallocator trigger ContentParent shutdown
when it is disabled.

This also has the benefit of hooking the recycler back into memory pressure
notifications, meaning it will be shut down if not in use when a memory pressure
event occurs.

Comment 7

[Nika Layzell [:nika] (ni? for response)]

Attachment: Bug 1661364 - Part 5: Remove MarkAsTroubled,

This method has no implementation, and is unused.

Comment 9

[Intermittent Failures Robot]

41 failures in 787 pushes (0.052 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 36
• mozilla-central: 5

Platform breakdown:

• linux64-shippable-qr: 6
• windows7-32-shippable: 11
• macosx1014-64-shippable: 13
• windows10-64-shippable-qr: 3
• linux64-shippable: 5
• windows10-64-shippable: 2
• linux1804-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1661364&startday=2020-08-28&endday=2020-08-28&tree=all

Comment 10

[Intermittent Failures Robot]

107 failures in 4046 pushes (0.026 failures/push) were associated with this bug in the last 7 days.

This is the #2 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 83
• mozilla-central: 24

Platform breakdown:

• macosx1014-64-shippable: 51
• linux64-shippable-qr: 14
• linux1804-64-qr: 1
• windows10-64-shippable-qr: 8
• linux64-shippable: 12
• windows7-32-shippable: 14
• windows10-64-shippable: 5
• linux1804-64-shippable-qr: 1
• linux1804-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1661364&startday=2020-08-24&endday=2020-08-30&tree=all

Comment 11

[Julien Cristau [:jcristau] (back April 22)]

This is by far the #1 top crash on nightly right now, we need the fix or a backout today.

Comment 12

[BugBot [:suhaib / :marco/ :calixte]]

This bug is tracked by a release manager but with a small severity so change it to major.
For more information, please visit auto_nag documentation.

Comment 13

[Julien Cristau [:jcristau] (back April 22)]

The assertions have been backed out, removing blocking flag.

Comment 14

[Intermittent Failures Robot]

2 failures in 2984 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform breakdown:

• linux64-shippable-qr: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1661364&startday=2020-08-31&endday=2020-09-06&tree=all

Comment 15

[Nika Layzell [:nika] (ni? for response)]

Attachment: Bug 1661364 - Part 6: Give the recycled process background process priority,

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/37b78e402c23
Part 1: Don't remove recycled content processes from the pool, r=kmag,jesup
https://hg.mozilla.org/integration/autoland/rev/9cb0e2dc8d8a
Part 2: Don't recycle web processes with Fission enabled, r=kmag,jesup
https://hg.mozilla.org/integration/autoland/rev/eac6bfc4a98d
Part 3: Add a MaybeBeginShutDown method, r=kmag,jesup
https://hg.mozilla.org/integration/autoland/rev/99234e013005
Part 4: Disable process recycling when prealloc is disabled, r=kmag,jesup
https://hg.mozilla.org/integration/autoland/rev/fc185ddca53b
Part 5: Remove MarkAsTroubled, r=kmag
https://hg.mozilla.org/integration/autoland/rev/a56449ddacc9
Part 6: Give the recycled process background process priority, r=kmag,jesup

Comment 17

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/37b78e402c23
https://hg.mozilla.org/mozilla-central/rev/9cb0e2dc8d8a
https://hg.mozilla.org/mozilla-central/rev/eac6bfc4a98d
https://hg.mozilla.org/mozilla-central/rev/99234e013005
https://hg.mozilla.org/mozilla-central/rev/fc185ddca53b
https://hg.mozilla.org/mozilla-central/rev/a56449ddacc9

Comment 18

[Bogdan Tara[:bogdan_tara | bogdant]]

Backed out 7 changesets (bug 1661364, bug 1660342) for making bug 1551615 perma fail.

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&selectedTaskRun=JOtk5CD-SiqCHnKUTcQJTQ.0&searchStr=linux%2C18.04%2Cx64%2Casan%2Copt%2Cweb%2Cplatform%2Ctests%2Ctest-linux1804-64-asan%2Fopt-web-platform-tests-e10s%2Cwpt18&fromchange=97bb7fd483e0e1802da6562ed37831198e036896&tochange=a56449ddacc90ad5136af43a455586df77988c1f

Backout link: https://hg.mozilla.org/integration/autoland/rev/433c4ac7d65a7ee98ae406cbc0421185a0c23021

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=315433318&repo=autoland&lineNumber=3109

[task 2020-09-11T10:43:39.147Z] 10:43:39     INFO - TEST-START | /encrypted-media/clearkey-mp4-setmediakeys-at-same-time.https.html
[task 2020-09-11T10:43:39.170Z] 10:43:39     INFO - Closing window 86
[task 2020-09-11T10:43:39.331Z] 10:43:39     INFO - PID 5745 | AddressSanitizer:DEADLYSIGNAL
[task 2020-09-11T10:43:39.331Z] 10:43:39     INFO - PID 5745 | =================================================================
[task 2020-09-11T10:43:39.331Z] 10:43:39    ERROR - PID 5745 | ==6830==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000001 (pc 0x7f315172c53b bp 0x7fff5f859190 sp 0x7fff5f858fe0 T0)
[task 2020-09-11T10:43:39.331Z] 10:43:39     INFO - PID 5745 | ==6830==The signal is caused by a WRITE memory access.
[task 2020-09-11T10:43:39.331Z] 10:43:39     INFO - PID 5745 | ==6830==Hint: address points to the zero page.
[task 2020-09-11T10:43:39.865Z] 10:43:39     INFO - PID 5745 | error: address range table at offset 0x5ac0 has an invalid tuple (length = 0) at offset 0x5ae0
[task 2020-09-11T10:43:40.042Z] 10:43:40     INFO - PID 5745 |     #0 0x7f315172c53b in mozilla::ipc::MessageChannel::Clear() /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:733:9
[task 2020-09-11T10:43:40.043Z] 10:43:40     INFO - PID 5745 |     #1 0x7f315172af3a in mozilla::ipc::MessageChannel::~MessageChannel() /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:630:3
[task 2020-09-11T10:43:40.051Z] 10:43:40     INFO - PID 5745 |     #2 0x7f3151a4076c in ~IToplevelProtocol /builds/worker/workspace/obj-build/dist/include/mozilla/ipc/ProtocolUtils.h:392:32
[task 2020-09-11T10:43:40.051Z] 10:43:40     INFO - PID 5745 |     #3 0x7f3151a4076c in mozilla::gmp::PGMPContentParent::~PGMPContentParent() /builds/worker/workspace/obj-build/ipc/ipdl/PGMPContentParent.cpp:56:1
[task 2020-09-11T10:43:40.072Z] 10:43:40     INFO - PID 5745 |     #4 0x7f31565ae0f7 in Release /builds/worker/checkouts/gecko/dom/media/gmp/GMPContentParent.h:25:3
[task 2020-09-11T10:43:40.073Z] 10:43:40     INFO - PID 5745 |     #5 0x7f31565ae0f7 in Release /builds/worker/workspace/obj-build/dist/include/mozilla/RefPtr.h:50:40
[task 2020-09-11T10:43:40.073Z] 10:43:40     INFO - PID 5745 |     #6 0x7f31565ae0f7 in Release /builds/worker/workspace/obj-build/dist/include/mozilla/RefPtr.h:381:36
[task 2020-09-11T10:43:40.074Z] 10:43:40     INFO - PID 5745 |     #7 0x7f31565ae0f7 in ~RefPtr /builds/worker/workspace/obj-build/dist/include/mozilla/RefPtr.h:81:7
[task 2020-09-11T10:43:40.074Z] 10:43:40     INFO - PID 5745 |     #8 0x7f31565ae0f7 in ~nsBaseHashtableET /builds/worker/workspace/obj-build/dist/include/nsBaseHashtable.h:72:32
[task 2020-09-11T10:43:40.074Z] 10:43:40     INFO - PID 5745 |     #9 0x7f31565ae0f7 in nsTHashtable<nsBaseHashtableET<nsUint64HashKey, RefPtr<mozilla::gmp::GMPContentParent> > >::s_ClearEntry(PLDHashTable*, PLDHashEntryHdr*) /builds/worker/workspace/obj-build/dist/include/nsTHashtable.h:512:37
[task 2020-09-11T10:43:40.081Z] 10:43:40     INFO - PID 5745 |     #10 0x7f315043e6b6 in operator() /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:292:7
[task 2020-09-11T10:43:40.082Z] 10:43:40     INFO - PID 5745 |     #11 0x7f315043e6b6 in ForEachSlot<(lambda at /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:290:51)> /builds/worker/workspace/obj-build/dist/include/PLDHashTable.h:357:9
[task 2020-09-11T10:43:40.083Z] 10:43:40     INFO - PID 5745 |     #12 0x7f315043e6b6 in ForEachSlot<(lambda at /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:290:51)> /builds/worker/workspace/obj-build/dist/include/PLDHashTable.h:347:7
[task 2020-09-11T10:43:40.083Z] 10:43:40     INFO - PID 5745 |     #13 0x7f315043e6b6 in PLDHashTable::~PLDHashTable() /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:290:15
[task 2020-09-11T10:43:40.083Z] 10:43:40     INFO - PID 5745 |     #14 0x7f315658234e in ~nsTHashtable /builds/worker/workspace/obj-build/dist/include/nsTHashtable.h:170:27
[task 2020-09-11T10:43:40.083Z] 10:43:40     INFO - PID 5745 |     #15 0x7f315658234e in ~GMPServiceChild /builds/worker/checkouts/gecko/dom/media/gmp/GMPServiceChild.cpp:426:35
[task 2020-09-11T10:43:40.083Z] 10:43:40     INFO - PID 5745 |     #16 0x7f315658234e in mozilla::gmp::GMPServiceChild::~GMPServiceChild() /builds/worker/checkouts/gecko/dom/media/gmp/GMPServiceChild.cpp:426:35
[task 2020-09-11T10:43:40.084Z] 10:43:40     INFO - PID 5745 |     #17 0x7f315658113d in operator() /builds/worker/workspace/obj-build/dist/include/mozilla/UniquePtr.h:460:5
[task 2020-09-11T10:43:40.084Z] 10:43:40     INFO - PID 5745 |     #18 0x7f315658113d in reset /builds/worker/workspace/obj-build/dist/include/mozilla/UniquePtr.h:302:7
[task 2020-09-11T10:43:40.085Z] 10:43:40     INFO - PID 5745 |     #19 0x7f315658113d in operator= /builds/worker/workspace/obj-build/dist/include/mozilla/UniquePtr.h:275:5
[task 2020-09-11T10:43:40.086Z] 10:43:40     INFO - PID 5745 |     #20 0x7f315658113d in mozilla::gmp::GeckoMediaPluginServiceChild::Observe(nsISupports*, char const*, char16_t const*) /builds/worker/checkouts/gecko/dom/media/gmp/GMPServiceChild.cpp:357:21
[task 2020-09-11T10:43:40.086Z] 10:43:40     INFO - PID 5745 |     #21 0x7f315658124c in non-virtual thunk to mozilla::gmp::GeckoMediaPluginServiceChild::Observe(nsISupports*, char const*, char16_t const*) /builds/worker/checkouts/gecko/dom/media/gmp/GMPServiceChild.cpp
[task 2020-09-11T10:43:40.087Z] 10:43:40     INFO - PID 5745 |     #22 0x7f315045c9de in nsObserverList::NotifyObservers(nsISupports*, char const*, char16_t const*) /builds/worker/checkouts/gecko/xpcom/ds/nsObserverList.cpp:70:19
[task 2020-09-11T10:43:40.088Z] 10:43:40     INFO - PID 5745 |     #23 0x7f3150463151 in nsObserverService::NotifyObservers(nsISupports*, char const*, char16_t const*) /builds/worker/checkouts/gecko/xpcom/ds/nsObserverService.cpp:287:19
[task 2020-09-11T10:43:40.089Z] 10:43:40     INFO - PID 5745 |     #24 0x7f3150611fb0 in mozilla::ShutdownXPCOM(nsIServiceManager*) /builds/worker/checkouts/gecko/xpcom/build/XPCOMInit.cpp:640:24
[task 2020-09-11T10:43:40.091Z] 10:43:40     INFO - PID 5745 |     #25 0x7f315b499dfc in XRE_TermEmbedding() /builds/worker/checkouts/gecko/toolkit/xre/nsEmbedFunctions.cpp:223:3
[task 2020-09-11T10:43:40.092Z] 10:43:40     INFO - PID 5745 |     #26 0x7f315174f72e in mozilla::ipc::ScopedXREEmbed::Stop() /builds/worker/checkouts/gecko/ipc/glue/ScopedXREEmbed.cpp:90:5
[task 2020-09-11T10:43:40.092Z] 10:43:40     INFO - PID 5745 |     #27 0x7f315b49aefd in XRE_InitChildProcess(int, char**, XREChildData const*) /builds/worker/checkouts/gecko/toolkit/xre/nsEmbedFunctions.cpp:748:16
[task 2020-09-11T10:43:40.099Z] 10:43:40     INFO - PID 5745 |     #28 0x5639631532cd in content_process_main(mozilla::Bootstrap*, int, char**) /builds/worker/checkouts/gecko/browser/app/../../ipc/contentproc/plugin-container.cpp:56:28
[task 2020-09-11T10:43:40.100Z] 10:43:40     INFO - PID 5745 |     #29 0x56396315370a in main /builds/worker/checkouts/gecko/browser/app/nsBrowserApp.cpp:303:18
[task 2020-09-11T10:43:40.120Z] 10:43:40     INFO - PID 5745 |     #30 0x7f3170e4cb96 in __libc_start_main /build/glibc-2ORdQG/glibc-2.27/csu/../csu/libc-start.c:310
[task 2020-09-11T10:43:40.120Z] 10:43:40     INFO - PID 5745 |     #31 0x5639630a6c85 in _start (/builds/worker/workspace/build/application/firefox/firefox+0x57c85)
[task 2020-09-11T10:43:40.121Z] 10:43:40     INFO - PID 5745 | AddressSanitizer can not provide additional info.
[task 2020-09-11T10:43:40.121Z] 10:43:40     INFO - PID 5745 | SUMMARY: AddressSanitizer: SEGV /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:733:9 in mozilla::ipc::MessageChannel::Clear()
[task 2020-09-11T10:43:40.121Z] 10:43:40     INFO - PID 5745 | ==6830==ABORTING
[task 2020-09-11T10:43:40.661Z] 10:43:40     INFO - PID 7344 | console.error: SearchCache: "_readCacheFile: Error reading cache file:" (new Error("", "(unknown module)"))
[task 2020-09-11T10:43:41.331Z] 10:43:41     INFO - .
[task 2020-09-11T10:43:41.331Z] 10:43:41     INFO - TEST-OK | /encrypted-media/clearkey-mp4-setmediakeys-at-same-time.https.html | took 2189ms

Comment 19

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

This triggered crashes with pthread_mutex_trylock | mozilla::detail::MutexImpl::lock | mozilla::ipc::MessageChannel::NotifyImpendingShutdown.

Comment 20

[BugBot [:suhaib / :marco/ :calixte]]

There are some r+ patches which didn't land and no activity in this bug for 2 weeks.
:nika, could you have a look please?
For more information, please visit auto_nag documentation.

Comment 21

[Neha Kochar [:neha]]

These crashes are not Fission-only, but we do want them fixed asap. Moving to M6c for tracking.

Comment 22

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

We're hoping to have the media work to prevent the test issues above landed shortly.

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b2cfa4e8c0ad
Part 1: Don't remove recycled content processes from the pool, r=kmag,jesup
https://hg.mozilla.org/integration/autoland/rev/9beb584e6b53
Part 2: Don't recycle web processes with Fission enabled, r=kmag,jesup
https://hg.mozilla.org/integration/autoland/rev/9ec6aba9aebc
Part 3: Add a MaybeBeginShutDown method, r=kmag,jesup
https://hg.mozilla.org/integration/autoland/rev/cba5f9325701
Part 4: Disable process recycling when prealloc is disabled, r=kmag,jesup
https://hg.mozilla.org/integration/autoland/rev/83189e460f5d
Part 5: Remove MarkAsTroubled, r=kmag
https://hg.mozilla.org/integration/autoland/rev/804277e7a302
Part 6: Give the recycled process background process priority, r=kmag,jesup

Comment 24

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/b2cfa4e8c0ad
https://hg.mozilla.org/mozilla-central/rev/9beb584e6b53
https://hg.mozilla.org/mozilla-central/rev/9ec6aba9aebc
https://hg.mozilla.org/mozilla-central/rev/cba5f9325701
https://hg.mozilla.org/mozilla-central/rev/83189e460f5d
https://hg.mozilla.org/mozilla-central/rev/804277e7a302

Comment 25

[BugBot [:suhaib / :marco/ :calixte]]

Since the status are different for nightly and release, what's the status for beta?
For more information, please visit auto_nag documentation.