1719481 - Intermittent ccov PROCESS-CRASH | Last test finished | application crashed [@ PR_NativeRunThread + 0x16a]

Status: NEW

(Testing :: Code Coverage, defect)

Description

[Treeherder Bug Filer]

Filed by: ncsoregi [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=344563965&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/LT5kLaJnR_e_kwAUmqJSFg/runs/0/artifacts/public/logs/live_backing.log

---

[task 2021-07-07T12:25:06.987Z] 12:25:06     INFO - PROCESS-CRASH | Last test finished | application crashed [@ PR_NativeRunThread + 0x16a]
[task 2021-07-07T12:25:06.987Z] 12:25:06     INFO - Mozilla crash reason: MOZ_CRASH(Shutdown hanging after all known phases and workers finished.)
[task 2021-07-07T12:25:06.987Z] 12:25:06     INFO - Crash dump filename: C:\Users\task_1625659277\AppData\Local\Temp\tmptkgwu0vx.mozrunner\minidumps\da69e453-e7a0-4138-8dc8-0760c21b39d3.dmp
[task 2021-07-07T12:25:06.987Z] 12:25:06     INFO - Operating system: Windows NT
[task 2021-07-07T12:25:06.987Z] 12:25:06     INFO -                   10.0.17134 
[task 2021-07-07T12:25:06.987Z] 12:25:06     INFO - CPU: amd64
[task 2021-07-07T12:25:06.987Z] 12:25:06     INFO -      family 6 model 94 stepping 3
[task 2021-07-07T12:25:06.988Z] 12:25:06     INFO -      8 CPUs
[task 2021-07-07T12:25:06.988Z] 12:25:06     INFO - 
[task 2021-07-07T12:25:06.988Z] 12:25:06     INFO - GPU: UNKNOWN
[task 2021-07-07T12:25:06.988Z] 12:25:06     INFO - 
[task 2021-07-07T12:25:06.988Z] 12:25:06     INFO - Crash reason:  EXCEPTION_BREAKPOINT
[task 2021-07-07T12:25:06.988Z] 12:25:06     INFO - Crash address: 0xba5eb1c7
[task 2021-07-07T12:25:06.988Z] 12:25:06     INFO - Process uptime: 73 seconds
[task 2021-07-07T12:25:06.988Z] 12:25:06     INFO - 
[task 2021-07-07T12:25:06.988Z] 12:25:06     INFO - Thread 19 (crashed) - Shutdown Hang Terminator 0  xul.dll!void mozilla::nsTerminator::AdvancePhase(enum mozilla::ShutdownPhase) + 0x447
[task 2021-07-07T12:25:06.988Z] 12:25:06     INFO -     rax = 0x00007ffec36033ee   rdx = 0x0000000000000000
[task 2021-07-07T12:25:06.988Z] 12:25:06     INFO -     rcx = 0x00007ffef1af4bb0   rbx = 0x00000219bbe59d30
[task 2021-07-07T12:25:06.988Z] 12:25:06     INFO -     rsi = 0x00007ffefeae3ca0   rdi = 0x0000000000000276
[task 2021-07-07T12:25:06.988Z] 12:25:06     INFO -     rbp = 0x0000000000000000   rsp = 0x000000d3e7fbfc50
[task 2021-07-07T12:25:06.988Z] 12:25:06     INFO -      r8 = 0x0000000000000008    r9 = 0x00007ffefd290cec
[task 2021-07-07T12:25:06.988Z] 12:25:06     INFO -     r10 = 0x00000fffdfa72be7   r11 = 0x0022008000000002
[task 2021-07-07T12:25:06.989Z] 12:25:06     INFO -     r12 = 0x00007ffedbaf7c20   r13 = 0x00000219ca46b138
[task 2021-07-07T12:25:06.989Z] 12:25:06     INFO -     r14 = 0x00007ffedbaceef4   r15 = 0x00007fff001f82c0
[task 2021-07-07T12:25:06.989Z] 12:25:06     INFO -     rip = 0x00007ffeba5eb1c7
[task 2021-07-07T12:25:06.989Z] 12:25:06     INFO -     Found by: given as instruction pointer in context
[task 2021-07-07T12:25:06.989Z] 12:25:06     INFO -  1  nss3.dll!PR_NativeRunThread + 0x16a
[task 2021-07-07T12:25:06.989Z] 12:25:06     INFO -     rbx = 0x00000219bbe59d30   rbp = 0x0000000000000000
[task 2021-07-07T12:25:06.989Z] 12:25:06     INFO -     rsp = 0x000000d3e7fbfca0   r12 = 0x00007ffedbaf7c20
[task 2021-07-07T12:25:06.989Z] 12:25:06     INFO -     r13 = 0x00000219ca46b138   r14 = 0x00007ffedbaceef4
[task 2021-07-07T12:25:06.989Z] 12:25:06     INFO -     r15 = 0x00007fff001f82c0   rip = 0x00007ffedb95186a
[task 2021-07-07T12:25:06.989Z] 12:25:06     INFO -     Found by: call frame info
[task 2021-07-07T12:25:06.989Z] 12:25:06     INFO -  2  nss3.dll!PR_MD_CREATE_THREAD + 0xd9
[task 2021-07-07T12:25:06.989Z] 12:25:06     INFO -     rbx = 0x00000219bbe59d30   rbp = 0x0000000000000000
[task 2021-07-07T12:25:06.989Z] 12:25:06     INFO -     rsp = 0x000000d3e7fbfd10   r12 = 0x00007ffedbaf7c20
[task 2021-07-07T12:25:06.989Z] 12:25:06     INFO -     r13 = 0x00000219ca46b138   r14 = 0x00007ffedbaceef4
[task 2021-07-07T12:25:06.990Z] 12:25:06     INFO -     r15 = 0x00007fff001f82c0   rip = 0x00007ffedb93dc69
[task 2021-07-07T12:25:06.990Z] 12:25:06     INFO -     Found by: call frame info
[task 2021-07-07T12:25:06.990Z] 12:25:06     INFO -  3  ucrtbase.dll!RtlpHpSegPageRangeShrink + 0xda
[task 2021-07-07T12:25:06.990Z] 12:25:06     INFO -     rbx = 0x00000219bbe59d30   rbp = 0x0000000000000000
[task 2021-07-07T12:25:06.990Z] 12:25:06     INFO -     rsp = 0x000000d3e7fbfd40   r12 = 0x00007ffedbaf7c20
[task 2021-07-07T12:25:06.990Z] 12:25:06     INFO -     r13 = 0x00000219ca46b138   r14 = 0x00007ffedbaceef4
[task 2021-07-07T12:25:06.990Z] 12:25:06     INFO -     r15 = 0x00007fff001f82c0   rip = 0x00007ffefd26c4be
[task 2021-07-07T12:25:06.990Z] 12:25:06     INFO -     Found by: call frame info
[task 2021-07-07T12:25:06.990Z] 12:25:06     INFO - 

Comment 1

[Intermittent Failures Robot]

8 failures in 2387 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 6
• autoland: 2

Platform and build breakdown:

• windows10-64-ccov-qr: 5
  • opt: 5
• windows10-64-qr: 3
  • debug: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1719481&startday=2021-07-05&endday=2021-07-11&tree=all

Comment 2

[Intermittent Failures Robot]

15 failures in 2224 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 8
• mozilla-central: 7

Platform and build breakdown:

• windows10-64-qr: 9
  • debug: 9
• windows10-64-ccov-qr: 5
  • opt: 5
• windows10-32-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1719481&startday=2021-07-12&endday=2021-07-18&tree=all

Comment 3

[Intermittent Failures Robot]

16 failures in 2940 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 16

Platform and build breakdown:

• windows10-64-ccov-qr: 16
  • opt: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1719481&startday=2021-07-19&endday=2021-07-25&tree=all

Comment 4

[Intermittent Failures Robot]

28 failures in 3131 pushes (0.009 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 28

Platform and build breakdown:

• windows10-64-ccov-qr: 17
  • opt: 17
• windows10-64-2004-ccov-qr: 11
  • opt: 11

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1719481&startday=2021-07-26&endday=2021-08-01&tree=all

Comment 5

[Intermittent Failures Robot]

21 failures in 580 pushes (0.036 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 21

Platform and build breakdown:

• windows10-64-ccov-qr: 6
  • opt: 6
• windows10-64-2004-ccov-qr: 15
  • opt: 15

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1719481&startday=2021-08-04&endday=2021-08-04&tree=all

Comment 6

[Iulian Moraru]

Update:
There have been 59 failures within the last 7 days:
• 44 failures on Windows 10 x64 2004 CCov WebRender opt
• 15 failures on Windows 10 x64 CCov WebRender opt
Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=347570272&repo=mozilla-central&lineNumber=4467

[task 2021-08-05T20:11:44.686Z] 20:11:44     INFO - REFTEST TEST-START | layout/reftests/svg/as-image/defer-unsupported-1.svg == layout/reftests/svg/as-image/defer-unsupported-1-ref.svg
[task 2021-08-05T20:11:44.687Z] 20:11:44     INFO - REFTEST TEST-LOAD | file:///Z:/task_162819332576699/build/tests/reftest/tests/layout/reftests/svg/as-image/defer-unsupported-1.svg | 140 / 141 (99%)
[task 2021-08-05T20:11:44.762Z] 20:11:44     INFO - REFTEST TEST-LOAD | file:///Z:/task_162819332576699/build/tests/reftest/tests/layout/reftests/svg/as-image/defer-unsupported-1-ref.svg | 140 / 141 (99%)
[task 2021-08-05T20:11:44.826Z] 20:11:44     INFO - REFTEST TEST-PASS | layout/reftests/svg/as-image/defer-unsupported-1.svg == layout/reftests/svg/as-image/defer-unsupported-1-ref.svg | image comparison, max difference: 0, number of differing pixels: 0
[task 2021-08-05T20:11:44.826Z] 20:11:44     INFO - REFTEST TEST-END | layout/reftests/svg/as-image/defer-unsupported-1.svg == layout/reftests/svg/as-image/defer-unsupported-1-ref.svg
[task 2021-08-05T20:11:44.841Z] 20:11:44     INFO - REFTEST INFO | Slowest test took 995ms (file:///Z:/task_162819332576699/build/tests/reftest/tests/layout/reftests/svg/as-image/img-novb-widthAndHeight-all-1-ref.html)
[task 2021-08-05T20:11:44.841Z] 20:11:44     INFO - REFTEST INFO | Total canvas count = 4
[task 2021-08-05T20:11:44.903Z] 20:11:44     INFO - 1628194304903	Marionette	TRACE	Received observer notification quit-application
[task 2021-08-05T20:11:44.905Z] 20:11:44     INFO - 1628194304903	Marionette	INFO	Stopped listening on port 2828
[task 2021-08-05T20:11:44.905Z] 20:11:44     INFO - 1628194304904	Marionette	DEBUG	Marionette stopped listening
[task 2021-08-05T20:11:44.975Z] 20:11:44     INFO - JavaScript error: resource:///modules/Interactions.jsm, line 230: NS_ERROR_FAILURE: Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIUserIdleService.removeIdleObserver]
[task 2021-08-05T20:11:44.983Z] 20:11:44     INFO - ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2021-08-05T20:11:45.393Z] 20:11:45     INFO - ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2021-08-05T20:11:45.394Z] 20:11:45     INFO - ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2021-08-05T20:11:45.394Z] 20:11:45     INFO - ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2021-08-05T20:11:46.795Z] 20:11:46     INFO - ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2021-08-05T20:12:52.809Z] 20:12:52    ERROR - TEST-UNEXPECTED-FAIL | Last test finished | application terminated with exit code 1
[task 2021-08-05T20:12:52.809Z] 20:12:52     INFO - REFTEST INFO | Downloading symbols from: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Fejj4_-TTPK135s8vNnWqw/artifacts/public/build/target.crashreporter-symbols.zip
[task 2021-08-05T20:12:55.929Z] 20:12:55     INFO - REFTEST INFO | Copy/paste: Z:/task_162819332576699/fetches\minidump_stackwalk\minidump_stackwalk.exe C:\Users\task_162819332576699\AppData\Local\Temp\tmpwosl7qvy.mozrunner\minidumps\e9aa6647-cb09-45dc-97c0-efa2fdb5d848.dmp C:\Users\task_162819332576699\AppData\Local\Temp\tmpohhk74m4
[task 2021-08-05T20:12:59.727Z] 20:12:59     INFO - REFTEST INFO | Saved minidump as Z:\task_162819332576699\build\blobber_upload_dir\e9aa6647-cb09-45dc-97c0-efa2fdb5d848.dmp
[task 2021-08-05T20:12:59.732Z] 20:12:59     INFO - REFTEST INFO | Saved app info as Z:\task_162819332576699\build\blobber_upload_dir\e9aa6647-cb09-45dc-97c0-efa2fdb5d848.extra
[task 2021-08-05T20:12:59.776Z] 20:12:59     INFO - REFTEST PROCESS-CRASH | Last test finished | application crashed [@ PR_NativeRunThread + 0x16a]
[task 2021-08-05T20:12:59.776Z] 20:12:59     INFO - Mozilla crash reason: MOZ_CRASH(Shutdown hanging after all known phases and workers finished.)
[task 2021-08-05T20:12:59.776Z] 20:12:59     INFO - Crash dump filename: C:\Users\task_162819332576699\AppData\Local\Temp\tmpwosl7qvy.mozrunner\minidumps\e9aa6647-cb09-45dc-97c0-efa2fdb5d848.dmp
[task 2021-08-05T20:12:59.776Z] 20:12:59     INFO - Operating system: Windows NT
[task 2021-08-05T20:12:59.776Z] 20:12:59     INFO -                   10.0.19041 
[task 2021-08-05T20:12:59.776Z] 20:12:59     INFO - CPU: amd64
[task 2021-08-05T20:12:59.776Z] 20:12:59     INFO -      family 6 model 63 stepping 2
[task 2021-08-05T20:12:59.776Z] 20:12:59     INFO -      6 CPUs
[task 2021-08-05T20:12:59.776Z] 20:12:59     INFO - 
[task 2021-08-05T20:12:59.776Z] 20:12:59     INFO - GPU: UNKNOWN
[task 2021-08-05T20:12:59.776Z] 20:12:59     INFO - 
[task 2021-08-05T20:12:59.776Z] 20:12:59     INFO - Crash reason:  EXCEPTION_BREAKPOINT
[task 2021-08-05T20:12:59.776Z] 20:12:59     INFO - Crash address: 0xacd98107
[task 2021-08-05T20:12:59.776Z] 20:12:59     INFO - Process uptime: 104 seconds
[task 2021-08-05T20:12:59.776Z] 20:12:59     INFO - 
[task 2021-08-05T20:12:59.776Z] 20:12:59     INFO - Thread 17 (crashed) - Shutdown Hang Terminator 0  xul.dll!void mozilla::nsTerminator::AdvancePhase(enum mozilla::ShutdownPhase) + 0x447
[task 2021-08-05T20:12:59.776Z] 20:12:59     INFO -     rax = 0x00007ffeb5f5be7b   rdx = 0x0000000000000000
[task 2021-08-05T20:12:59.776Z] 20:12:59     INFO -     rcx = 0x00007ffee46ba8e0   rbx = 0x000002208fd59b30
[task 2021-08-05T20:12:59.776Z] 20:12:59     INFO -     rsi = 0x00007ffeed0fada0   rdi = 0x0000000000000276
[task 2021-08-05T20:12:59.776Z] 20:12:59     INFO -     rbp = 0x0000000000000000   rsp = 0x00000037540cfd60
[task 2021-08-05T20:12:59.776Z] 20:12:59     INFO -      r8 = 0x0000000000000002    r9 = 0x00000037540cfc08
[task 2021-08-05T20:12:59.776Z] 20:12:59     INFO -     r10 = 0x0000000000000013   r11 = 0x2200000080082000
[task 2021-08-05T20:12:59.776Z] 20:12:59     INFO -     r12 = 0x00007ffed6607c20   r13 = 0x000002209a219538
[task 2021-08-05T20:12:59.777Z] 20:12:59     INFO -     r14 = 0x00007ffed65def24   r15 = 0x00007ffeeeb9faa0
[task 2021-08-05T20:12:59.777Z] 20:12:59     INFO -     rip = 0x00007ffeacd98107
[task 2021-08-05T20:12:59.777Z] 20:12:59     INFO -     Found by: given as instruction pointer in context
[task 2021-08-05T20:12:59.777Z] 20:12:59     INFO -  1  nss3.dll!PR_NativeRunThread + 0x16a
[task 2021-08-05T20:12:59.777Z] 20:12:59     INFO -     rbx = 0x000002208fd59b30   rbp = 0x0000000000000000
[task 2021-08-05T20:12:59.777Z] 20:12:59     INFO -     rsp = 0x00000037540cfdb0   r12 = 0x00007ffed6607c20
[task 2021-08-05T20:12:59.777Z] 20:12:59     INFO -     r13 = 0x000002209a219538   r14 = 0x00007ffed65def24
[task 2021-08-05T20:12:59.777Z] 20:12:59     INFO -     r15 = 0x00007ffeeeb9faa0   rip = 0x00007ffed646186a
[task 2021-08-05T20:12:59.777Z] 20:12:59     INFO -     Found by: call frame info
[task 2021-08-05T20:12:59.777Z] 20:12:59     INFO -  2  nss3.dll!PR_MD_CREATE_THREAD + 0xd9
[task 2021-08-05T20:12:59.777Z] 20:12:59     INFO -     rbx = 0x000002208fd59b30   rbp = 0x0000000000000000
[task 2021-08-05T20:12:59.777Z] 20:12:59     INFO -     rsp = 0x00000037540cfe20   r12 = 0x00007ffed6607c20
[task 2021-08-05T20:12:59.777Z] 20:12:59     INFO -     r13 = 0x000002209a219538   r14 = 0x00007ffed65def24
[task 2021-08-05T20:12:59.777Z] 20:12:59     INFO -     r15 = 0x00007ffeeeb9faa0   rip = 0x00007ffed644dc69
[task 2021-08-05T20:12:59.777Z] 20:12:59     INFO -     Found by: call frame info
[task 2021-08-05T20:12:59.777Z] 20:12:59     INFO -  3  ucrtbase.dll!RtlpReAllocateHeap + 0x1ce
[task 2021-08-05T20:12:59.777Z] 20:12:59     INFO -     rbx = 0x000002208fd59b30   rbp = 0x0000000000000000
[task 2021-08-05T20:12:59.777Z] 20:12:59     INFO -     rsp = 0x00000037540cfe50   r12 = 0x00007ffed6607c20
[task 2021-08-05T20:12:59.777Z] 20:12:59     INFO -     r13 = 0x000002209a219538   r14 = 0x00007ffed65def24
[task 2021-08-05T20:12:59.777Z] 20:12:59     INFO -     r15 = 0x00007ffeeeb9faa0   rip = 0x00007ffeecb01bb2
[task 2021-08-05T20:12:59.777Z] 20:12:59     INFO -     Found by: call frame info
[task 2021-08-05T20:12:59.777Z] 20:12:59     INFO - 

Comment 7

[Iulian Moraru]

Nika, as the owner of this component, could you help us assign it to someone?
Thank you.

Comment 8

[Andrew McCreight [:mccr8]]

FWIW, it looks like these are windows10-64 ccov WebRender shutdown hangs, spread across various test suites.

In a few logs I looked at, the main thread was in XPCOM shutdown, doing mozilla::ipc::BrowserProcessSubThread::`scalar deleting destructor'

Comment 9

[Intermittent Failures Robot]

48 failures in 2829 pushes (0.017 failures/push) were associated with this bug in the last 7 days.

This is the #24 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 48

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 33
  • opt: 33
• windows10-64-ccov-qr: 15
  • opt: 15

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1719481&startday=2021-08-02&endday=2021-08-08&tree=all

Comment 10

[Nika Layzell [:nika] (ni? for response)]

I managed to get a minidump open, though only for bug 1685204. I'm expecting that these two crashes have the same cause. We're hanging within ChildReaper::WillDestroyCurrentMessageLoop here: https://searchfox.org/mozilla-central/rev/162a8620f126c27de70766808708696260f2256a/ipc/chromium/src/chrome/common/process_watcher_win.cc#38 in response to the I/O thread trying to exit. The main thread is then hanging trying to join the I/O thread from BrowserProcessSubThread's destructor.

I'm guessing these are only appearing in debug & ccov builds because toolkit.shutdown.fastShutdownStage is disabled for these builds (https://searchfox.org/mozilla-central/rev/162a8620f126c27de70766808708696260f2256a/modules/libpref/init/StaticPrefList.yaml#11013-11017).

ni? :jld who I know has been touching the ChildReaper code recently, and might have changed something which is leading to these hangs.

Comment 11

[Intermittent Failures Robot]

7 failures in 3025 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 7

Platform and build breakdown:

• windows10-64-ccov-qr: 7
  • opt: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1719481&startday=2021-08-09&endday=2021-08-15&tree=all

Comment 13

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

I've been touching the code, but I haven't landed those patches, and they're only to the Unix side, and they don't change this behavior, which is intentional: On builds that need to do a full orderly shutdown because of instrumentation (leak checking, ccov, etc.), the parent process blocks at this point waiting for child processes to exit.

What would help is a way to get the stack for the child process in this case. Nika, you pointed out the other week that this should be converted to use nsIAsyncShutdown instead of just blocking in late shutdown like this, and I see it has a state property for exporting information into a hang crash… but in this case I think we'd want a way to say “do a paired minidump with pid N instead” (rather than trying to somehow get all the process's stacks in a useful format and encoded as nsIPropertyBags/nsIVariants).

Comment 14

[Intermittent Failures Robot]

15 failures in 2899 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 11
• autoland: 4

Platform and build breakdown:

• windows10-64-ccov-qr: 10
  • opt: 10
• windows10-64-2004-qr: 4
  • debug: 4
• windows10-32-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1719481&startday=2021-08-16&endday=2021-08-22&tree=all

Comment 15

[Intermittent Failures Robot]

26 failures in 2580 pushes (0.01 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 11
• mozilla-central: 15

Platform and build breakdown:

• windows10-64-2004-qr: 13
  • debug: 13
• windows10-32-qr: 1
  • debug: 1
• windows10-64-ccov-qr: 12
  • opt: 12

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1719481&startday=2021-08-23&endday=2021-08-29&tree=all

Comment 16

[Intermittent Failures Robot]

17 failures in 3084 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 10
• autoland: 7

Platform and build breakdown:

• windows10-64-ccov-qr: 7
  • opt: 7
• windows10-64-2004-qr: 8
  • debug: 8
• windows10-64-2004-ccov-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1719481&startday=2021-08-30&endday=2021-09-05&tree=all

Comment 18

[Intermittent Failures Robot]

18 failures in 3266 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 12
• autoland: 5
• mozilla-esr91: 1

Platform and build breakdown:

• windows10-64-2004-qr: 11
  • debug: 11
• windows10-64-ccov-qr: 4
  • opt: 4
• windows10-64-2004-ccov-qr: 2
  • opt: 2
• windows10-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1719481&startday=2021-09-06&endday=2021-09-12&tree=all

Comment 19

[Intermittent Failures Robot]

6 failures in 3375 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 6

Platform and build breakdown:

• windows10-64-ccov-qr: 5
  • opt: 5
• windows10-64-2004-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1719481&startday=2021-09-13&endday=2021-09-19&tree=all

Comment 20

[Intermittent Failures Robot]

7 failures in 2873 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 7

Platform and build breakdown:

• windows10-64-ccov-qr: 6
  • opt: 6
• windows10-64-2004-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1719481&startday=2021-09-20&endday=2021-09-26&tree=all

Comment 21

[Intermittent Failures Robot]

30 failures in 528 pushes (0.057 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 30

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 30
  • opt: 30

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1719481&startday=2021-09-30&endday=2021-09-30&tree=all

Comment 22

[Intermittent Failures Robot]

17 failures in 408 pushes (0.042 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 17

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 17
  • opt: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1719481&startday=2021-10-01&endday=2021-10-01&tree=all

Comment 23

[Intermittent Failures Robot]

20 failures in 165 pushes (0.121 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 20

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 20
  • opt: 20

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1719481&startday=2021-10-02&endday=2021-10-02&tree=all

Comment 24

[Intermittent Failures Robot]

82 failures in 3376 pushes (0.024 failures/push) were associated with this bug in the last 7 days.

This is the #9 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 82

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 82
  • opt: 82

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1719481&startday=2021-09-27&endday=2021-10-03&tree=all

Comment 25

[Marian-Vasile Laza]

There have been 109 total failures in the last 7 days, recent failure log.
Affected platforms are:

• windows10-64-2004-ccov-qr

Comment 26

[Intermittent Failures Robot]

18 failures in 862 pushes (0.021 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 18

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 18
  • opt: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1719481&startday=2021-10-07&endday=2021-10-07&tree=all

Comment 27

[Intermittent Failures Robot]

17 failures in 551 pushes (0.031 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 17

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 17
  • opt: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1719481&startday=2021-10-08&endday=2021-10-08&tree=all

Comment 28

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The crashes here are all code coverage build related. Maybe they stopped on Friday, or failing jobs haven't been classified over the weekend yet. Marco, could you have a look please?

Comment 29

[Intermittent Failures Robot]

71 failures in 3626 pushes (0.02 failures/push) were associated with this bug in the last 7 days.

This is the #13 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 71

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 71
  • opt: 71

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1719481&startday=2021-10-04&endday=2021-10-10&tree=all

Comment 30

[Marco Castelluccio [:marco]]

From the crash reason (MOZ_CRASH(Shutdown hanging after all known phases and workers finished.)), I'd say this was another instance where ccov instrumentation is slowing down shutdown too much, and with the move to Azure this got much worse.

It looks like this stopped on October 8th though, so maybe bug 1733611 was enough to fix this (or at least bring it back to be rarely happening).

Comment 31

[Andreea Pavel [:apavel]]

No failures on trunk since the 9th of October.

Comment 32

[Intermittent Failures Robot]

1 failures in 3315 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1719481&startday=2021-10-11&endday=2021-10-17&tree=all

Comment 33

[Intermittent Failures Robot]

1 failures in 3186 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1719481&startday=2021-10-18&endday=2021-10-24&tree=all

Comment 34

[Intermittent Failures Robot]

1 failures in 2723 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• linux1804-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1719481&startday=2021-11-08&endday=2021-11-14&tree=all

Comment 35

[Intermittent Failures Robot]

1 failures in 3259 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1719481&startday=2021-11-15&endday=2021-11-21&tree=all

Comment 36

[Intermittent Failures Robot]

1 failures in 3060 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows10-32-2004-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1719481&startday=2022-01-10&endday=2022-01-16&tree=all

Comment 37

[Intermittent Failures Robot]

1 failures in 2917 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1719481&startday=2022-05-30&endday=2022-06-05&tree=all

Comment 38

[Intermittent Failures Robot]

1 failures in 3075 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr91: 1

Platform and build breakdown:

• macosx1015-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1719481&startday=2022-06-20&endday=2022-06-26&tree=all

Comment 39

[BugBot [:suhaib / :marco/ :calixte]]

The severity field for this bug is set to S4. However, the following bug duplicate has higher severity:

• Bug 1685204: S3

:marco, could you consider increasing the severity of this bug to S3?

For more information, please visit auto_nag documentation.

Comment 40

[Marco Castelluccio [:marco]]

Very rare nowadays, let's keep it as S4.