1634259 - High Frequency PROCESS-CRASH | Main app process exited normally | application crashed [@ mozilla::CycleCollectedJSContext::RecursionDepth() const]

Status: RESOLVED FIXED

(Core :: DOM: Workers, defect, P2)

Description

[Treeherder Bug Filer]

Filed by: rmaries [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=300155975&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/KBxj2JD2RH-RWse-hPnvcg/runs/0/artifacts/public/logs/live_backing.log

---

[task 2020-04-30T06:09:37.830Z] 06:09:37 INFO - TEST-START | browser/components/uitour/test/browser_showMenu.js
[task 2020-04-30T06:09:54.102Z] 06:09:54 INFO - mozcrash Copy/paste: /builds/worker/fetches/minidump_stackwalk/minidump_stackwalk /tmp/tmpTFrZCQ.mozrunner/minidumps/4a84e97a-e105-81e3-7f03-785e8ee3ff44.dmp /builds/worker/workspace/build/symbols
[task 2020-04-30T06:09:59.301Z] 06:09:59 INFO - mozcrash Saved minidump as /builds/worker/workspace/build/blobber_upload_dir/4a84e97a-e105-81e3-7f03-785e8ee3ff44.dmp
[task 2020-04-30T06:09:59.379Z] 06:09:59 INFO - PROCESS-CRASH | Main app process exited normally | application crashed [@ mozilla::CycleCollectedJSContext::RecursionDepth() const]
[task 2020-04-30T06:09:59.380Z] 06:09:59 INFO - Crash dump filename: /tmp/tmpTFrZCQ.mozrunner/minidumps/4a84e97a-e105-81e3-7f03-785e8ee3ff44.dmp
[task 2020-04-30T06:09:59.381Z] 06:09:59 INFO - Operating system: Linux
[task 2020-04-30T06:09:59.382Z] 06:09:59 INFO - 0.0.0 Linux 4.4.0-1014-aws #14taskcluster1-Ubuntu SMP Tue Apr 3 10:27:00 UTC 2018 x86_64
[task 2020-04-30T06:09:59.382Z] 06:09:59 INFO - CPU: amd64
[task 2020-04-30T06:09:59.383Z] 06:09:59 INFO - family 6 model 85 stepping 7
[task 2020-04-30T06:09:59.383Z] 06:09:59 INFO - 2 CPUs
[task 2020-04-30T06:09:59.384Z] 06:09:59 INFO -
[task 2020-04-30T06:09:59.385Z] 06:09:59 INFO - GPU: UNKNOWN
[task 2020-04-30T06:09:59.385Z] 06:09:59 INFO -
[task 2020-04-30T06:09:59.386Z] 06:09:59 INFO - Crash reason: SIGSEGV /SEGV_MAPERR
[task 2020-04-30T06:09:59.386Z] 06:09:59 INFO - Crash address: 0x5b70
[task 2020-04-30T06:09:59.387Z] 06:09:59 INFO - Process uptime: not available
[task 2020-04-30T06:09:59.387Z] 06:09:59 INFO -
[task 2020-04-30T06:09:59.388Z] 06:09:59 INFO - Thread 27 (crashed)
[task 2020-04-30T06:09:59.388Z] 06:09:59 INFO - 0 libxul.so!mozilla::CycleCollectedJSContext::RecursionDepth() const [CycleCollectedJSContext.cpp:83beb87d9f6945ccddefe62341b5536398eb72a0 : 520 + 0x0]
[task 2020-04-30T06:09:59.389Z] 06:09:59 INFO - rax = 0x0000000000000000 rdx = 0x00007f39e108a5d0
[task 2020-04-30T06:09:59.389Z] 06:09:59 INFO - rcx = 0x00007f39fd1b9320 rbx = 0x0000000000000000
[task 2020-04-30T06:09:59.390Z] 06:09:59 INFO - rsi = 0x0000000000000001 rdi = 0x0000000000000000
[task 2020-04-30T06:09:59.391Z] 06:09:59 INFO - rbp = 0x00007f39de6fdd60 rsp = 0x00007f39de6fdd50
[task 2020-04-30T06:09:59.391Z] 06:09:59 INFO - r8 = 0x0000000000727736 r9 = 0x0000000000000003
[task 2020-04-30T06:09:59.392Z] 06:09:59 INFO - r10 = 0x0006b40bdf835ec1 r11 = 0x00000000ffffffff
[task 2020-04-30T06:09:59.392Z] 06:09:59 INFO - r12 = 0x0000000000000000 r13 = 0x00007f39de6fde80
[task 2020-04-30T06:09:59.393Z] 06:09:59 INFO - r14 = 0x0000000000000000 r15 = 0x00007f39e6699e00
[task 2020-04-30T06:09:59.393Z] 06:09:59 INFO - rip = 0x00007f39e90af19b
[task 2020-04-30T06:09:59.394Z] 06:09:59 INFO - Found by: given as instruction pointer in context
[task 2020-04-30T06:09:59.395Z] 06:09:59 INFO - 1 libxul.so!mozilla::dom::WorkerPrivate::OnProcessNextEvent() [WorkerPrivate.cpp:83beb87d9f6945ccddefe62341b5536398eb72a0 : 2988 + 0xd]
[task 2020-04-30T06:09:59.396Z] 06:09:59 INFO - rbx = 0x00007f39e0712000 rbp = 0x00007f39de6fdd80
[task 2020-04-30T06:09:59.396Z] 06:09:59 INFO - rsp = 0x00007f39de6fdd70 r12 = 0x0000000000000000
[task 2020-04-30T06:09:59.396Z] 06:09:59 INFO - r13 = 0x00007f39de6fde80 r14 = 0x0000000000000000
[task 2020-04-30T06:09:59.397Z] 06:09:59 INFO - r15 = 0x00007f39e6699e00 rip = 0x00007f39ebc2070d
[task 2020-04-30T06:09:59.397Z] 06:09:59 INFO - Found by: call frame info
[task 2020-04-30T06:09:59.397Z] 06:09:59 INFO - 2 libxul.so!mozilla::dom::WorkerThread::Observer::OnProcessNextEvent(nsIThreadInternal*, bool) [WorkerThread.cpp:83beb87d9f6945ccddefe62341b5536398eb72a0 : 356 + 0x9]
[task 2020-04-30T06:09:59.397Z] 06:09:59 INFO - rbx = 0x00007f39e6699e00 rbp = 0x00007f39de6fdda0
[task 2020-04-30T06:09:59.398Z] 06:09:59 INFO - rsp = 0x00007f39de6fdd90 r12 = 0x0000000000000000
[task 2020-04-30T06:09:59.398Z] 06:09:59 INFO - r13 = 0x00007f39de6fde80 r14 = 0x0000000000000000
[task 2020-04-30T06:09:59.398Z] 06:09:59 INFO - r15 = 0x00007f39e6699e00 rip = 0x00007f39ebc31379
[task 2020-04-30T06:09:59.398Z] 06:09:59 INFO - Found by: call frame info
[task 2020-04-30T06:09:59.398Z] 06:09:59 INFO - 3 libxul.so!nsThread::ProcessNextEvent(bool, bool*) [nsThread.cpp:83beb87d9f6945ccddefe62341b5536398eb72a0 : 1107 + 0x3b]
[task 2020-04-30T06:09:59.399Z] 06:09:59 INFO - rbx = 0x00007f39de6fdee0 rbp = 0x00007f39de6fe300
[task 2020-04-30T06:09:59.399Z] 06:09:59 INFO - rsp = 0x00007f39de6fddb0 r12 = 0x0000000000000000
[task 2020-04-30T06:09:59.399Z] 06:09:59 INFO - r13 = 0x00007f39de6fde80 r14 = 0x00007f39e10093c0
[task 2020-04-30T06:09:59.399Z] 06:09:59 INFO - r15 = 0x00007f39e6699e00 rip = 0x00007f39e9189894
[task 2020-04-30T06:09:59.400Z] 06:09:59 INFO - Found by: call frame info
[task 2020-04-30T06:09:59.400Z] 06:09:59 INFO - 4 libxul.so!NS_ProcessPendingEvents(nsIThread*, unsigned int) [nsThreadUtils.cpp:83beb87d9f6945ccddefe62341b5536398eb72a0 : 429 + 0xf]
[task 2020-04-30T06:09:59.400Z] 06:09:59 INFO - rbx = 0x0000000000000000 rbp = 0x00007f39de6fe350
[task 2020-04-30T06:09:59.400Z] 06:09:59 INFO - rsp = 0x00007f39de6fe310 r12 = 0x00007f39e10093c0
[task 2020-04-30T06:09:59.401Z] 06:09:59 INFO - r13 = 0x00007f39de6fe31f r14 = 0x00000000ffffffff
[task 2020-04-30T06:09:59.401Z] 06:09:59 INFO - r15 = 0x000000000102d69c rip = 0x00007f39e9187ce3
[task 2020-04-30T06:09:59.401Z] 06:09:59 INFO - Found by: call frame info
[task 2020-04-30T06:09:59.401Z] 06:09:59 INFO - 5 libxul.so!mozilla::dom::WorkerPrivate::ClearMainEventQueue(mozilla::dom::WorkerPrivate::WorkerRanOrNot) [WorkerPrivate.cpp:83beb87d9f6945ccddefe62341b5536398eb72a0 : 3518 + 0xd]
[task 2020-04-30T06:09:59.401Z] 06:09:59 INFO - rbx = 0x0000000000000000 rbp = 0x00007f39de6fe390
[task 2020-04-30T06:09:59.402Z] 06:09:59 INFO - rsp = 0x00007f39de6fe360 r12 = 0x0000000080004005
[task 2020-04-30T06:09:59.402Z] 06:09:59 INFO - r13 = 0x00001edb1e635b4e r14 = 0x0000000000000001
[task 2020-04-30T06:09:59.402Z] 06:09:59 INFO - r15 = 0x00007f39e0712000 rip = 0x00007f39ebc217cf
[task 2020-04-30T06:09:59.402Z] 06:09:59 INFO - Found by: call frame info
[task 2020-04-30T06:09:59.402Z] 06:09:59 INFO - 6 libxul.so!mozilla::dom::WorkerPrivate::ScheduleDeletion(mozilla::dom::WorkerPrivate::WorkerRanOrNot) [WorkerPrivate.cpp:83beb87d9f6945ccddefe62341b5536398eb72a0 : 3363 + 0xb]
[task 2020-04-30T06:09:59.402Z] 06:09:59 INFO - rbx = 0x00007f39e0712000 rbp = 0x00007f39de6fe3d0
[task 2020-04-30T06:09:59.403Z] 06:09:59 INFO - rsp = 0x00007f39de6fe3a0 r12 = 0x0000000080004005
[task 2020-04-30T06:09:59.403Z] 06:09:59 INFO - r13 = 0x00001edb1e635b4e r14 = 0x0000000000000001
[task 2020-04-30T06:09:59.403Z] 06:09:59 INFO - r15 = 0x00007f39e6571880 rip = 0x00007f39ebc19ab2
[task 2020-04-30T06:09:59.403Z] 06:09:59 INFO - Found by: call frame info
[task 2020-04-30T06:09:59.404Z] 06:09:59 INFO - 7 libxul.so!mozilla::dom::workerinternals::(anonymous namespace)::WorkerThreadPrimaryRunnable::Run() [RuntimeService.cpp:83beb87d9f6945ccddefe62341b5536398eb72a0 : 2334 + 0x1e]
[task 2020-04-30T06:09:59.404Z] 06:09:59 INFO - rbx = 0x00007f39de6fe620 rbp = 0x00007f39de6fe5e0
[task 2020-04-30T06:09:59.404Z] 06:09:59 INFO - rsp = 0x00007f39de6fe3e0 r12 = 0x0000000080004005
[task 2020-04-30T06:09:59.404Z] 06:09:59 INFO - r13 = 0x00001edb1e635b4e r14 = 0x00007f39e0712000
[task 2020-04-30T06:09:59.404Z] 06:09:59 INFO - r15 = 0x00007f39e6571880 rip = 0x00007f39ebc069e2
[task 2020-04-30T06:09:59.405Z] 06:09:59 INFO - Found by: call frame info
[task 2020-04-30T06:09:59.405Z] 06:09:59 INFO - 8 libxul.so!nsThread::ProcessNextEvent(bool, bool*) [nsThread.cpp:83beb87d9f6945ccddefe62341b5536398eb72a0 : 1200 + 0x11]
[task 2020-04-30T06:09:59.405Z] 06:09:59 INFO - rbx = 0x00007f39de6fe620 rbp = 0x00007f39de6feb40
[task 2020-04-30T06:09:59.405Z] 06:09:59 INFO - rsp = 0x00007f39de6fe5f0 r12 = 0x00007f39e1009488
[task 2020-04-30T06:09:59.406Z] 06:09:59 INFO - r13 = 0x00001edb1e635b4e r14 = 0x00007f39e10093c0
[task 2020-04-30T06:09:59.406Z] 06:09:59 INFO - r15 = 0x00007f39de6fe6c0 rip = 0x00007f39e9189f36
[task 2020-04-30T06:09:59.407Z] 06:09:59 INFO - Found by: call frame info
[task 2020-04-30T06:09:59.408Z] 06:09:59 INFO - 9 libxul.so!NS_ProcessNextEvent(nsIThread*, bool) [nsThreadUtils.cpp:83beb87d9f6945ccddefe62341b5536398eb72a0 : 481 + 0xc]
[task 2020-04-30T06:09:59.408Z] 06:09:59 INFO - rbx = 0x0000000000000000 rbp = 0x00007f39de6feb70
[task 2020-04-30T06:09:59.409Z] 06:09:59 INFO - rsp = 0x00007f39de6feb50 r12 = 0x00007f39de6feb88
[task 2020-04-30T06:09:59.409Z] 06:09:59 INFO - r13 = 0x00007f39e6571820 r14 = 0x00007f39de6feb57
[task 2020-04-30T06:09:59.409Z] 06:09:59 INFO - r15 = 0x00007f39e10093c0 rip = 0x00007f39e918db9f
[task 2020-04-30T06:09:59.410Z] 06:09:59 INFO - Found by: call frame info
[task 2020-04-30T06:09:59.410Z] 06:09:59 INFO - 10 libxul.so!mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) [MessagePump.cpp:83beb87d9f6945ccddefe62341b5536398eb72a0 : 302 + 0xa]
[task 2020-04-30T06:09:59.410Z] 06:09:59 INFO - rbx = 0x00007f39de6fec68 rbp = 0x00007f39de6febc0
[task 2020-04-30T06:09:59.410Z] 06:09:59 INFO - rsp = 0x00007f39de6feb80 r12 = 0x00007f39de6feb88
[task 2020-04-30T06:09:59.411Z] 06:09:59 INFO - r13 = 0x00007f39e6571820 r14 = 0x00007f39e6571800
[task 2020-04-30T06:09:59.411Z] 06:09:59 INFO - r15 = 0x00007f39e10093c0 rip = 0x00007f39e977fe2b
[task 2020-04-30T06:09:59.411Z] 06:09:59 INFO - Found by: call frame info
[task 2020-04-30T06:09:59.412Z] 06:09:59 INFO - 11 libxul.so!MessageLoop::RunInternal() [message_loop.cc:83beb87d9f6945ccddefe62341b5536398eb72a0 : 315 + 0x17]
[task 2020-04-30T06:09:59.412Z] 06:09:59 INFO - rbx = 0x00007f39de6fec68 rbp = 0x00007f39de6fec00
[task 2020-04-30T06:09:59.412Z] 06:09:59 INFO - rsp = 0x00007f39de6febd0 r12 = 0x00007f39de6fec60
[task 2020-04-30T06:09:59.413Z] 06:09:59 INFO - r13 = 0x00007f39e10093c0 r14 = 0x00007f39de6fec10
[task 2020-04-30T06:09:59.413Z] 06:09:59 INFO - r15 = 0x00007f39e1009400 rip = 0x00007f39e972db89
[task 2020-04-30T06:09:59.413Z] 06:09:59 INFO - Found by: call frame info
[task 2020-04-30T06:09:59.413Z] 06:09:59 INFO - 12 libxul.so!MessageLoop::Run() [message_loop.cc:83beb87d9f6945ccddefe62341b5536398eb72a0 : 290 + 0x8]
[task 2020-04-30T06:09:59.414Z] 06:09:59 INFO - rbx = 0x00007f39de6fec68 rbp = 0x00007f39de6fec40
[task 2020-04-30T06:09:59.414Z] 06:09:59 INFO - rsp = 0x00007f39de6fec10 r12 = 0x00007f39de6fec60
[task 2020-04-30T06:09:59.414Z] 06:09:59 INFO - r13 = 0x00007f39e10093c0 r14 = 0x00007f39de6fec10
[task 2020-04-30T06:09:59.415Z] 06:09:59 INFO - r15 = 0x00007f39e1009400 rip = 0x00007f39e972dae3
[task 2020-04-30T06:09:59.415Z] 06:09:59 INFO - Found by: call frame info
[task 2020-04-30T06:09:59.415Z] 06:09:59 INFO - 13 libxul.so!nsThread::ThreadFunc(void*) [nsThread.cpp:83beb87d9f6945ccddefe62341b5536398eb72a0 : 444 + 0x8]
[task 2020-04-30T06:09:59.415Z] 06:09:59 INFO - rbx = 0x00007f39de6fec68 rbp = 0x00007f39de6fee60
[task 2020-04-30T06:09:59.416Z] 06:09:59 INFO - rsp = 0x00007f39de6fec50 r12 = 0x00007f39de6fec60
[task 2020-04-30T06:09:59.416Z] 06:09:59 INFO - r13 = 0x00007f39e10093c0 r14 = 0x00007f39de6fec68
[task 2020-04-30T06:09:59.416Z] 06:09:59 INFO - r15 = 0x00007f39e1009400 rip = 0x00007f39e9187556
[task 2020-04-30T06:09:59.417Z] 06:09:59 INFO - Found by: call frame info
[task 2020-04-30T06:09:59.417Z] 06:09:59 INFO - 14 libnspr4.so!_pt_root [ptthread.c:83beb87d9f6945ccddefe62341b5536398eb72a0 : 201 + 0x7]
[task 2020-04-30T06:09:59.417Z] 06:09:59 INFO - rbx = 0x00007f39e078f040 rbp = 0x00007f39de6feeb0
[task 2020-04-30T06:09:59.418Z] 06:09:59 INFO - rsp = 0x00007f39de6fee70 r12 = 0x00007f39de6ff630
[task 2020-04-30T06:09:59.418Z] 06:09:59 INFO - r13 = 0x0000000000000000 r14 = 0x0000000000002804
[task 2020-04-30T06:09:59.418Z] 06:09:59 INFO - r15 = 0x00007f39de6ff700 rip = 0x00007f39fd38a897
[task 2020-04-30T06:09:59.418Z] 06:09:59 INFO - Found by: call frame info
[task 2020-04-30T06:09:59.419Z] 06:09:59 INFO - 15 libpthread.so.0 + 0x76db
[task 2020-04-30T06:09:59.419Z] 06:09:59 INFO - rbx = 0x0000000000000000 rbp = 0x0000000000000000
[task 2020-04-30T06:09:59.419Z] 06:09:59 INFO - rsp = 0x00007f39de6feec0 r12 = 0x00007f39de6fef80
[task 2020-04-30T06:09:59.419Z] 06:09:59 INFO - r13 = 0x0000000000000000 r14 = 0x00007f39e078f040
[task 2020-04-30T06:09:59.420Z] 06:09:59 INFO - r15 = 0x00007ffe08e125a0 rip = 0x00007f39fcfa66db
[task 2020-04-30T06:09:59.420Z] 06:09:59 INFO - Found by: call frame info
[task 2020-04-30T06:09:59.421Z] 06:09:59 INFO - 16 libc.so.6 + 0x12188f
[task 2020-04-30T06:09:59.421Z] 06:09:59 INFO - rsp = 0x00007f39de6fef80 rip = 0x00007f39fbf8488f
[task 2020-04-30T06:09:59.422Z] 06:09:59 INFO - Found by: stack scanning

Comment 1

[Natalia Csoregi [:nataliaCs]]

This has started spiking and the retrigger/backfills point to

• https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=Linux%2C18.04%2Cx64%2Cdebug%2CMochitests%2Cwith%2Cfission%2Cenabled%2Ctest-linux1804-64%2Fdebug-mochitest-browser-chrome-fis-e10s-1%2CM-fis%28bc1%29&tochange=83beb87d9f6945ccddefe62341b5536398eb72a0&fromchange=f5b6bbac55bc2d9e8f49e7fdc21dcfccd03e3952&selectedTaskRun=Pj6j0IO_TfadIKcy85rxjw-0
  Same for other chunk
• https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=linux%2C18.04%2Cx64%2Cdebug%2Cmochitests%2Cwith%2Cfission%2Cenabled%2Ctest-linux1804-64%2Fdebug-mochitest-browser-chrome-fis-e10s-15%2Cm-fis%28bc15%29&tochange=83beb87d9f6945ccddefe62341b5536398eb72a0&fromchange=876524e046ead3390f2f98cb8bd5065cfd4aa540&selectedTaskRun=Rxu_i0pmQFS5dAb9C_6B0w-0

As per the convo on Matrix, leaving the ni for further investigation.
Thank you.

Comment 2

[Intermittent Failures Robot]

44 failures in 1127 pushes (0.039 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 40
• mozilla-central: 2
• try: 2

Platform breakdown:

• linux1804-64: 44

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1634259&startday=2020-04-30&endday=2020-04-30&tree=all

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1594572

Comment 4

[Intermittent Failures Robot]

36 failures in 691 pushes (0.052 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 31
• mozilla-central: 4
• try: 1

Platform breakdown:

• linux1804-64: 36

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1634259&startday=2020-05-01&endday=2020-05-01&tree=all

Comment 5

[Intermittent Failures Robot]

100 failures in 5423 pushes (0.018 failures/push) were associated with this bug in the last 7 days.

This is the #11 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 87
• mozilla-central: 10
• try: 3

Platform breakdown:

• linux1804-64: 100

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1634259&startday=2020-04-27&endday=2020-05-03&tree=all

Comment 6

[Jens Stutte [:jstutte]]

This looks to be the same as bug 1634995 comment 2. Kris, do you prefer to keep this one?

Comment 7

[Intermittent Failures Robot]

29 failures in 929 pushes (0.031 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 23
• try: 1
• mozilla-central: 5

Platform breakdown:

• linux1804-64: 29

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1634259&startday=2020-05-04&endday=2020-05-04&tree=all

Comment 10

[Jens Stutte [:jstutte]]

(Copied from bug 1634995)

I see a missing nullptr check here: CycleCollectedJSContext::Get() can return nullptr but the caller does not check it and thus the next access on the data structure on mOwningThread may fail.

I see many places in WorkerPrivate.cpp, where CycleCollectedJSContext::Get() is used as if it always returns a healthy (raw!) pointer. Either we make sure, that this is the case or we should check all call sites for proper error handling.

Comment 11

[Andrew McCreight [:mccr8]]

In my opinion, the crash in bug looks different than the [@ mozilla::dom::WorkerPrivate::OnProcessNextEvent()] one. The one here looks like we're nesting event loops too deeply, whereas with the signature it is only nested once.

Comment 12

[Intermittent Failures Robot]

21 failures in 1176 pushes (0.018 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 16
• mozilla-central: 2
• try: 3

Platform breakdown:

• linux1804-64: 21

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1634259&startday=2020-05-05&endday=2020-05-05&tree=all

Comment 13

[Jens Stutte [:jstutte]]

(In reply to Andrew McCreight [:mccr8] from comment #11)

In my opinion, the crash in bug looks different than the [@ mozilla::dom::WorkerPrivate::OnProcessNextEvent()] one. The one here looks like we're nesting event loops too deeply, whereas with the signature it is only nested once.

Well, it is the same exact line of code that causes the nullptr. So there might be a common local fix to the crash but different consequences on other parts of the code when the underlying condition occurs. I would want to improve the local robustness first and then look for the fallout.

I see three local options:

1. If CycleCollectedJSContext::Get() is never supposed to fail, we should probably change its signature/behavior.
2. If it is only supposed to not fail while a WorkerPrivate is living, we might want to wrap it with a local function ensuring this invariant.
3. If it is supposed to possibly fail any time, we should improve error handling in all call sites AND look out for fallouts of this error handling.

:asuth, what do you think?

Comment 14

[Intermittent Failures Robot]

33 failures in 931 pushes (0.035 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 15
• mozilla-central: 13
• try: 5

Platform breakdown:

• linux1804-64: 33

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1634259&startday=2020-05-06&endday=2020-05-06&tree=all

Comment 16

[Andrew Sutherland [:asuth] (he/him)]

Taking bug per slack conversation with :kmag.

Comment 17

[Intermittent Failures Robot]

25 failures in 980 pushes (0.026 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 17
• try: 3
• mozilla-central: 5

Platform breakdown:

• linux1804-64: 25

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1634259&startday=2020-05-07&endday=2020-05-07&tree=all

Comment 18

[Intermittent Failures Robot]

29 failures in 869 pushes (0.033 failures/push) were associated with this bug yesterday.

Repository breakdown:

• try: 2
• autoland: 19
• mozilla-central: 8

Platform breakdown:

• linux1804-64: 26
• linux1804-64-shippable: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1634259&startday=2020-05-08&endday=2020-05-08&tree=all

Comment 19

[Intermittent Failures Robot]

21 failures in 260 pushes (0.081 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 10
• try: 8
• mozilla-central: 3

Platform breakdown:

• linux1804-64: 17
• linux1804-64-qr: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1634259&startday=2020-05-09&endday=2020-05-09&tree=all

Comment 20

[Andreea Pavel [:apavel]]

Hi Andrew, are there updates here?

I looked in about ~25 failure logs and the tests failing are browser_ext_windows_size.js and browser_readerMode_readingTime.js. Created the disabling patch for those 2 tests. Let us know if we can land the patch.

Comment 21

[Andreea Pavel [:apavel]]

Attachment: Bug 1634259 - disable browser_ext_windows_size.js and browser_readerMode_readingTime.js on linux18.04 debug r?#intermittent-reviewers

Comment 22

[Intermittent Failures Robot]

167 failures in 5276 pushes (0.032 failures/push) were associated with this bug in the last 7 days.

This is the #2 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 108
• try: 22
• mozilla-central: 37

Platform breakdown:

• linux1804-64: 159
• linux1804-64-shippable: 4
• linux1804-64-qr: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1634259&startday=2020-05-04&endday=2020-05-10&tree=all

Comment 23

[Firefox Bug Husbandry Bot]

Because this bug's Severity has not been changed from the default since it was filed, and it's Priority is -- (non,) indicating it has has not been previously triaged, the bug's Severity is being updated to -- (default, untriaged.)

Comment 24

[Intermittent Failures Robot]

20 failures in 997 pushes (0.02 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 16
• mozilla-central: 2
• try: 2

Platform breakdown:

• linux1804-64: 17
• linux1804-64-shippable: 2
• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1634259&startday=2020-05-11&endday=2020-05-11&tree=all

Comment 25

[Andreea Pavel [:apavel]]

Andrew can we land the disabling patch until there is a fix?

Comment 26

[Intermittent Failures Robot]

23 failures in 1049 pushes (0.022 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 19
• mozilla-central: 4

Platform breakdown:

• linux1804-64: 23

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1634259&startday=2020-05-12&endday=2020-05-12&tree=all

Comment 27

[Andrew Sutherland [:asuth] (he/him)]

Attachment: Bug 1634259 - Improve worker shutdown edge case handling. r=perry

Bug 1594572 attempted to fix a shutdown edge-case where a worker that was
started late enough would fail to create a PBackground connection and
early return. This early return, however, left the worker never scheduled
for deletion, resulting in a shutdown hang. See my restating block at
https://phabricator.services.mozilla.com/D73134#2224963 for more context.

The problem with this was that the attempt to clear the main event queue
ran afoul of pre/post event-processing hooks intended to ensure that control
runnables are processed if some non-worker code goes and spins a nested
event loop on the worker thread. But because we never got around to
creating a cycle collected JS runtime on the thread, the call to get it
would return null and when we tried to get the recursion depth of a null
pointer, we would crash a little.

This patch addresses the immediate regression by adding a helper that returns
a depth of 1 in this edge-case. It also fixes another problem with the fix
that is now more obvious having reviewed bug 1636147... we were failing to
mark the status of the worker as dead and drain any control runnables, which
could have resulted in the CrashIfHangingRunnable usage not working out right.
(Note that CrashIfHangingRunnable does handle Cancel() correctly, so the
fact that we end up canceling the runnable shouldn't break things.)

Comment 28

[Andrew Sutherland [:asuth] (he/him)]

I have a fix up for review and perry is available to review for today, so maybe let's just try the fix. Try push is at:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=607d6485f17418ccb448bed5a5cc1af1fcda0eb3

Comment 29

[Intermittent Failures Robot]

35 failures in 1048 pushes (0.033 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 24
• mozilla-central: 9
• try: 2

Platform breakdown:

• linux1804-64: 35

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1634259&startday=2020-05-13&endday=2020-05-13&tree=all

Comment 30

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/bd7c4e7e5994
Improve worker shutdown edge case handling. r=perry

Comment 31

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/bd7c4e7e5994

Comment 32

[Intermittent Failures Robot]

17 failures in 970 pushes (0.018 failures/push) were associated with this bug yesterday.

Repository breakdown:

• try: 3
• autoland: 9
• mozilla-central: 5

Platform breakdown:

• linux1804-64: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1634259&startday=2020-05-14&endday=2020-05-14&tree=all

Comment 33

[Intermittent Failures Robot]

96 failures in 5488 pushes (0.017 failures/push) were associated with this bug in the last 7 days.

This is the #8 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 69
• mozilla-central: 20
• try: 7

Platform breakdown:

• linux1804-64: 93
• linux1804-64-shippable: 2
• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1634259&startday=2020-05-11&endday=2020-05-17&tree=all

Comment 36

[Intermittent Failures Robot]

1 failures in 4727 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform breakdown:

• linux1804-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1634259&startday=2020-05-18&endday=2020-05-24&tree=all