1831236 - Intermittent Main app process exited normally | application crashed [@ DbgBreakPoint]

Status: ASSIGNED

(Core :: DLL Services, defect, P3)

Description

[Treeherder Bug Filer]

Filed by: imoraru [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=414552639&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/ZJslA1jqSHSN7cM28KK-Xg/runs/0/artifacts/public/logs/live_backing.log

---

[task 2023-05-03T22:22:51.183Z] 22:22:51     INFO - GECKO(9748) | [GPU 5624, Main Thread] WARNING: '!ipcActor->SendRecordDiscardedData(discardedData)', file /builds/worker/checkouts/gecko/toolkit/components/telemetry/core/ipc/TelemetryIPCAccumulator.cpp:298
[task 2023-05-03T22:22:58.476Z] 22:22:58     INFO - GECKO(9748) | [Parent 2084, IPC I/O Parent] WARNING: Process 5624 hanging at shutdown; attempting crash report (fatal error): file /builds/worker/checkouts/gecko/ipc/chromium/src/chrome/common/process_watcher_win.cc:164
[task 2023-05-03T22:22:58.634Z] 22:22:58     INFO - GECKO(9748) | [Parent 2084, Main Thread] WARNING: NS_ENSURE_TRUE(Preferences::InitStaticMembers()) failed: file /builds/worker/checkouts/gecko/modules/libpref/Preferences.cpp:4688
[task 2023-05-03T22:22:58.696Z] 22:22:58     INFO - TEST-INFO | Main app process: exit 0
[task 2023-05-03T22:22:58.697Z] 22:22:58     INFO - TEST-INFO | Confirming we saw 25 DOCSHELL created and 24 destroyed log strings.
[task 2023-05-03T22:22:58.697Z] 22:22:58     INFO - TEST-INFO | Confirming we saw 69 DOMWINDOW created and 66 destroyed log strings.
[task 2023-05-03T22:22:58.698Z] 22:22:58     INFO - TEST-INFO | browser/base/content/test/tabcrashed/browser_aboutRestartRequired_basic.js | This test created 1 hidden window(s)
[task 2023-05-03T22:22:58.699Z] 22:22:58     INFO - TEST-INFO | browser/base/content/test/tabcrashed/browser_aboutRestartRequired_basic.js | This test created 1 hidden docshell(s)
[task 2023-05-03T22:22:58.699Z] 22:22:58     INFO - TEST-INFO | browser/base/content/test/tabcrashed/browser_aboutRestartRequired_buildid_false-positive.js | This test created 1 hidden window(s)
[task 2023-05-03T22:22:58.699Z] 22:22:58     INFO - TEST-INFO | browser/base/content/test/tabcrashed/browser_aboutRestartRequired_buildid_false-positive.js | This test created 1 hidden docshell(s)
[task 2023-05-03T22:22:58.700Z] 22:22:58     INFO - TEST-INFO | browser/base/content/test/tabcrashed/browser_aboutRestartRequired_buildid_mismatch.js | This test created 1 hidden window(s)
[task 2023-05-03T22:22:58.700Z] 22:22:58     INFO - TEST-INFO | browser/base/content/test/tabcrashed/browser_aboutRestartRequired_buildid_mismatch.js | This test created 1 hidden docshell(s)
[task 2023-05-03T22:22:58.701Z] 22:22:58     INFO - TEST-INFO | browser/base/content/test/tabcrashed/browser_aboutRestartRequired_buildid_no-platform-ini.js | This test created 1 hidden window(s)
[task 2023-05-03T22:22:58.701Z] 22:22:58     INFO - TEST-INFO | browser/base/content/test/tabcrashed/browser_aboutRestartRequired_buildid_no-platform-ini.js | This test created 1 hidden docshell(s)
[task 2023-05-03T22:22:58.702Z] 22:22:58     INFO - runtests.py | Application ran for: 0:00:26.094867
[task 2023-05-03T22:22:58.702Z] 22:22:58     INFO - zombiecheck | Reading PID log: C:\Users\task_168314909515731\AppData\Local\Temp\tmp2obfzzcrpidlog
[task 2023-05-03T22:22:58.703Z] 22:22:58     INFO - ==> process 2084 launched child process 5624 ("Z:\task_168314909515731\build\application\firefox\firefox.exe" -contentproc --channel="2084.0.433379937\1367619092" -parentBuildID 20230503214103 -prefsHandle 2144 -prefMapHandle 2136 -prefsLen 21829 -prefMapSize 253246 -appDir "Z:\task_168314909515731\build\application\firefox\browser" - {79d77ef0-f8e8-4b60-ae55-94fc37740232} 2084 "\\.\pipe\gecko-crash-server-pipe.2084" 2224 22ad6fe1590 gpu)
[task 2023-05-03T22:22:58.704Z] 22:22:58     INFO - ==> process 2084 launched child process 9984 ("Z:\task_168314909515731\build\application\firefox\firefox.exe" -contentproc --channel="2084.1.1669533028\285533145" -childID 1 -isForBrowser -prefsHandle 2820 -prefMapHandle 2828 -prefsLen 21655 -prefMapSize 253246 -jsInitHandle 1416 -jsInitLen 294416 -parentBuildID 20230503214103 -win32kLockedDown -appDir "Z:\task_168314909515731\build\application\firefox\browser" - {271a5779-112a-4d6c-a0c0-b3347f1bfc50} 2084 "\\.\pipe\gecko-crash-server-pipe.2084" 2844 22adcb82c90 tab)
[task 2023-05-03T22:22:58.705Z] 22:22:58     INFO - ==> process 2084 launched child process 8256 ("Z:\task_168314909515731\build\application\firefox\firefox.exe" -contentproc --channel="2084.2.1208176676\2120736540" -childID 2 -isForBrowser -prefsHandle 3280 -prefMapHandle 3064 -prefsLen 23191 -prefMapSize 253246 -jsInitHandle 1416 -jsInitLen 294416 -parentBuildID 20230503214103 -win32kLockedDown -appDir "Z:\task_168314909515731\build\application\firefox\browser" - {d5099c97-a37f-44f3-97ac-9d8df32ac0bf} 2084 "\\.\pipe\gecko-crash-server-pipe.2084" 3120 22adb55b190 tab)
[task 2023-05-03T22:22:58.706Z] 22:22:58     INFO - ==> process 2084 launched child process 6040 ("Z:\task_168314909515731\build\application\firefox\firefox.exe" -contentproc --channel="2084.3.1713277974\1473730368" -parentBuildID 20230503214103 -sandboxingKind 0 -prefsHandle 4024 -prefMapHandle 4020 -prefsLen 30501 -prefMapSize 253246 -win32kLockedDown -appDir "Z:\task_168314909515731\build\application\firefox\browser" - {757bc750-ccb9-44b4-b861-acdeda82c820} 2084 "\\.\pipe\gecko-crash-server-pipe.2084" 3996 22ae1a7a190 utility)
[task 2023-05-03T22:22:58.708Z] 22:22:58     INFO - ==> process 2084 launched child process 8768 ("Z:\task_168314909515731\build\application\firefox\firefox.exe" -contentproc --channel="2084.4.343464681\1992970145" -childID 3 -isForBrowser -prefsHandle 4300 -prefMapHandle 4296 -prefsLen 30501 -prefMapSize 253246 -jsInitHandle 1416 -jsInitLen 294416 -parentBuildID 20230503214103 -win32kLockedDown -appDir "Z:\task_168314909515731\build\application\firefox\browser" - {6f70797b-985f-450e-9463-5acf18cc40de} 2084 "\\.\pipe\gecko-crash-server-pipe.2084" 4264 22ae1a7c290 tab)
[task 2023-05-03T22:22:58.709Z] 22:22:58     INFO - ==> process 2084 launched child process 1248 ("Z:\task_168314909515731\build\application\firefox\firefox.exe" -contentproc --channel="2084.5.916687376\1836848456" -childID 4 -isForBrowser -prefsHandle 4832 -prefMapHandle 4836 -prefsLen 28818 -prefMapSize 253246 -jsInitHandle 1416 -jsInitLen 294416 -parentBuildID 20230503214103 -win32kLockedDown -appDir "Z:\task_168314909515731\build\application\firefox\browser" - {c0923b49-08bb-447a-8d2e-68964ba00634} 2084 "\\.\pipe\gecko-crash-server-pipe.2084" 4864 22addf66b90 tab)
[task 2023-05-03T22:22:58.710Z] 22:22:58     INFO - ==> process 2084 launched child process 8424 ("Z:\task_168314909515731\build\application\firefox\firefox.exe" -contentproc --channel="2084.6.1853197\2112620572" -childID 5 -isForBrowser -prefsHandle 5012 -prefMapHandle 4548 -prefsLen 28914 -prefMapSize 253246 -jsInitHandle 1416 -jsInitLen 294416 -parentBuildID 20230503214103 -win32kLockedDown -appDir "Z:\task_168314909515731\build\application\firefox\browser" - {af75f03f-aae2-4afc-ad34-118c61c970e8} 2084 "\\.\pipe\gecko-crash-server-pipe.2084" 5024 22ae02fcf90 tab)
[task 2023-05-03T22:22:58.711Z] 22:22:58     INFO - ==> process 2084 launched child process 3328 ("Z:\task_168314909515731\build\application\firefox\firefox.exe" -contentproc --channel="2084.7.631767156\1788972505" -childID 6 -isForBrowser -prefsHandle 5260 -prefMapHandle 5256 -prefsLen 28914 -prefMapSize 253246 -jsInitHandle 1416 -jsInitLen 294416 -parentBuildID 20230503214103 -win32kLockedDown -appDir "Z:\task_168314909515731\build\application\firefox\browser" - {bb0b8280-c020-4b9c-8f61-c9f8a0e56ef4} 2084 "\\.\pipe\gecko-crash-server-pipe.2084" 5224 22ae02fc390 tab)
[task 2023-05-03T22:22:58.712Z] 22:22:58     INFO - ==> process 2084 launched child process 4824 ("Z:\task_168314909515731\build\application\firefox\firefox.exe" -contentproc --channel="2084.8.1716416259\541752871" -childID 7 -isForBrowser -prefsHandle 4296 -prefMapHandle 4300 -prefsLen 28914 -prefMapSize 253246 -jsInitHandle 1416 -jsInitLen 294416 -parentBuildID 20230503214103 -win32kLockedDown -appDir "Z:\task_168314909515731\build\application\firefox\browser" - {d6cc1061-6015-46f8-a47c-fc2a88e4312d} 2084 "\\.\pipe\gecko-crash-server-pipe.2084" 5020 22ae0b97290 tab)
[task 2023-05-03T22:22:58.712Z] 22:22:58     INFO - zombiecheck | Checking for orphan process with PID: 9984
[task 2023-05-03T22:22:58.712Z] 22:22:58     INFO - zombiecheck | Checking for orphan process with PID: 8256
[task 2023-05-03T22:22:58.713Z] 22:22:58     INFO - zombiecheck | Checking for orphan process with PID: 8768
[task 2023-05-03T22:22:58.713Z] 22:22:58     INFO - zombiecheck | Checking for orphan process with PID: 1248
[task 2023-05-03T22:22:58.713Z] 22:22:58     INFO - zombiecheck | Checking for orphan process with PID: 3328
[task 2023-05-03T22:22:58.714Z] 22:22:58     INFO - zombiecheck | Checking for orphan process with PID: 8424
[task 2023-05-03T22:22:58.714Z] 22:22:58     INFO - zombiecheck | Checking for orphan process with PID: 4824
[task 2023-05-03T22:22:58.715Z] 22:22:58     INFO - zombiecheck | Checking for orphan process with PID: 6040
[task 2023-05-03T22:22:58.715Z] 22:22:58     INFO - zombiecheck | Checking for orphan process with PID: 5624
[task 2023-05-03T22:22:58.716Z] 22:22:58     INFO - mozcrash Copy/paste: Z:/task_168314909515731/fetches\minidump-stackwalk\minidump-stackwalk.exe --symbols-url=https://symbols.mozilla.org/ --cyborg=C:\Users\task_168314909515731\AppData\Local\Temp\tmph26n3ssy\aa5d9a2f-e4e9-4383-ad71-9399a02745dd.trace C:\Users\task_168314909515731\AppData\Local\Temp\tmp15n8vkqj.mozrunner\minidumps\aa5d9a2f-e4e9-4383-ad71-9399a02745dd.dmp Z:\task_168314909515731\build\symbols
[task 2023-05-03T22:23:09.231Z] 22:23:09     INFO - mozcrash Saved minidump as Z:\task_168314909515731\build\blobber_upload_dir\aa5d9a2f-e4e9-4383-ad71-9399a02745dd.dmp
[task 2023-05-03T22:23:09.481Z] 22:23:09     INFO - PROCESS-CRASH | Main app process exited normally | application crashed [@ DbgBreakPoint]
[task 2023-05-03T22:23:09.481Z] 22:23:09     INFO - Crash dump filename: C:\Users\task_168314909515731\AppData\Local\Temp\tmp15n8vkqj.mozrunner\minidumps\aa5d9a2f-e4e9-4383-ad71-9399a02745dd.dmp
[task 2023-05-03T22:23:09.481Z] 22:23:09     INFO - Operating system: Windows NT
[task 2023-05-03T22:23:09.481Z] 22:23:09     INFO -                   10.0.22621
[task 2023-05-03T22:23:09.490Z] 22:23:09     INFO - CPU: amd64
[task 2023-05-03T22:23:09.490Z] 22:23:09     INFO -      family 6 model 106 stepping 6
[task 2023-05-03T22:23:09.490Z] 22:23:09     INFO -      8 CPUs
[task 2023-05-03T22:23:09.490Z] 22:23:09     INFO - 
[task 2023-05-03T22:23:09.490Z] 22:23:09     INFO - Crash reason:  EXCEPTION_BREAKPOINT
[task 2023-05-03T22:23:09.490Z] 22:23:09     INFO - Crash address: 0x7ffbbbb12af0
[task 2023-05-03T22:23:09.490Z] 22:23:09     INFO - Crashing instruction: `int 0x3`
[task 2023-05-03T22:23:09.490Z] 22:23:09     INFO - No memory accessed by instruction
[task 2023-05-03T22:23:09.490Z] 22:23:09     INFO - Process uptime: 25 seconds
[task 2023-05-03T22:23:09.491Z] 22:23:09     INFO - 
[task 2023-05-03T22:23:09.491Z] 22:23:09     INFO - Thread 30  (crashed)
[task 2023-05-03T22:23:09.491Z] 22:23:09     INFO -  0  ntdll.dll!DbgBreakPoint + 0x0
[task 2023-05-03T22:23:09.491Z] 22:23:09     INFO -      rax = 0x00007ffbbbb12af0    rdx = 0x00007ffbbbb12af0
[task 2023-05-03T22:23:09.491Z] 22:23:09     INFO -      rcx = 0x0043504900435049    rbx = 0x0000000000000000
[task 2023-05-03T22:23:09.491Z] 22:23:09     INFO -      rsi = 0x0043504900435049    rdi = 0x0000000000000000
[task 2023-05-03T22:23:09.491Z] 22:23:09     INFO -      rbp = 0x0000000000000000    rsp = 0x000000285f23fae8
[task 2023-05-03T22:23:09.491Z] 22:23:09     INFO -       r8 = 0x0043504900435049     r9 = 0x0000000000000000
[task 2023-05-03T22:23:09.491Z] 22:23:09     INFO -      r10 = 0x00000fff7776255e    r11 = 0x0101400140011111
[task 2023-05-03T22:23:09.491Z] 22:23:09     INFO -      r12 = 0x0000000000000000    r13 = 0x0000000000000000
[task 2023-05-03T22:23:09.491Z] 22:23:09     INFO -      r14 = 0x0000000000000000    r15 = 0x0000000000000000
[task 2023-05-03T22:23:09.491Z] 22:23:09     INFO -      rip = 0x00007ffbbbb12af0
[task 2023-05-03T22:23:09.491Z] 22:23:09     INFO -     Found by: given as instruction pointer in context
[task 2023-05-03T22:23:09.491Z] 22:23:09     INFO -  1  kernel32.dll!BaseThreadInitThunk + 0x1c
[task 2023-05-03T22:23:09.491Z] 22:23:09     INFO -      rbx = 0x0000000000000000    rbp = 0x0000000000000000
[task 2023-05-03T22:23:09.491Z] 22:23:09     INFO -      rsp = 0x000000285f23faf0    r12 = 0x0000000000000000
[task 2023-05-03T22:23:09.491Z] 22:23:09     INFO -      r13 = 0x0000000000000000    r14 = 0x0000000000000000
[task 2023-05-03T22:23:09.491Z] 22:23:09     INFO -      r15 = 0x0000000000000000    rip = 0x00007ffbbb1526bd
[task 2023-05-03T22:23:09.491Z] 22:23:09     INFO -     Found by: call frame info
[task 2023-05-03T22:23:09.491Z] 22:23:09     INFO -  2  mozglue.dll!mozilla::interceptor::FuncHook<mozilla::interceptor::WindowsDllInterceptor<mozilla::interceptor::VMSharingPolicyShared>,void (*)(int, void *, void *)>::operator()(int&, void*&, void*&) const [nsWindowsDllInterceptor.h:f99ee8082b684fbe6a257b8aba442dcc413a20f1 : 150]
[task 2023-05-03T22:23:09.491Z] 22:23:09     INFO -     Found by: inlining
[task 2023-05-03T22:23:09.492Z] 22:23:09     INFO -  3  mozglue.dll!patched_BaseThreadInitThunk(int, void*, void*) [WindowsDllBlocklist.cpp:f99ee8082b684fbe6a257b8aba442dcc413a20f1 : 616 + 0x14]
[task 2023-05-03T22:23:09.492Z] 22:23:09     INFO -      rbx = 0x00007ffbbbb12af0    rsi = 0x0043504900435049
[task 2023-05-03T22:23:09.492Z] 22:23:09     INFO -      rdi = 0x0000000000000000    rbp = 0x0000000000000000
[task 2023-05-03T22:23:09.492Z] 22:23:09     INFO -      rsp = 0x000000285f23fb20    r12 = 0x0000000000000000
[task 2023-05-03T22:23:09.492Z] 22:23:09     INFO -      r13 = 0x0000000000000000    r14 = 0x0000000000000000
[task 2023-05-03T22:23:09.492Z] 22:23:09     INFO -      r15 = 0x0000000000000000    rip = 0x00007ffbb09144e4
[task 2023-05-03T22:23:09.492Z] 22:23:09     INFO -     Found by: call frame info
[task 2023-05-03T22:23:09.492Z] 22:23:09     INFO -  4  ntdll.dll!RtlUserThreadStart + 0x27
[task 2023-05-03T22:23:09.492Z] 22:23:09     INFO -      rbx = 0x0000000000000000    rsi = 0x0000000000000000
[task 2023-05-03T22:23:09.492Z] 22:23:09     INFO -      rdi = 0x0000000000000000    rbp = 0x0000000000000000
[task 2023-05-03T22:23:09.492Z] 22:23:09     INFO -      rsp = 0x000000285f23fb90    r12 = 0x0000000000000000
[task 2023-05-03T22:23:09.492Z] 22:23:09     INFO -      r13 = 0x0000000000000000    r14 = 0x0000000000000000
[task 2023-05-03T22:23:09.492Z] 22:23:09     INFO -      r15 = 0x0000000000000000    rip = 0x00007ffbbbacdfb8
[task 2023-05-03T22:23:09.492Z] 22:23:09     INFO -     Found by: call frame info
[task 2023-05-03T22:23:09.492Z] 22:23:09     INFO - 

Comment 1

[Intermittent Failures Robot]

4 failures in 3411 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4

Platform and build breakdown:

• windows11-64-2009-qr: 1
  • debug: 1
• windows11-64-2009-ccov-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-05-01&endday=2023-05-07&tree=all

Comment 2

[Intermittent Failures Robot]

8 failures in 4077 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 5
• mozilla-beta: 2

Platform and build breakdown:

• windows11-64-2009-qr: 2
  • debug: 2
• windows11-64-2009-ccov-qr: 5
  • opt: 5
• windows11-32-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-05-08&endday=2023-05-14&tree=all

Comment 3

[Intermittent Failures Robot]

9 failures in 3578 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 4
• try: 1
• mozilla-beta: 3

Platform and build breakdown:

• windows11-64-2009-qr: 2
  • debug: 2
• windows11-64-2009-ccov-qr: 4
  • opt: 4
• windows11-32-2009-qr: 3
  • debug: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-05-15&endday=2023-05-21&tree=all

Comment 5

[Intermittent Failures Robot]

6 failures in 3946 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 5
• autoland: 1

Platform and build breakdown:

• windows11-64-2009-ccov-qr: 5
  • opt: 5
• windows11-32-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-05-22&endday=2023-05-28&tree=all

Comment 6

[Intermittent Failures Robot]

7 failures in 3471 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4
• autoland: 2
• try: 1

Platform and build breakdown:

• windows11-64-2009-ccov-qr: 4
  • opt: 4
• windows11-64-2009-qr: 2
  • debug: 2
• windows11-32-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-05-29&endday=2023-06-04&tree=all

Comment 7

[Intermittent Failures Robot]

3 failures in 3885 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3

Platform and build breakdown:

• windows11-64-2009-ccov-qr: 2
  • opt: 2
• windows11-32-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-06-05&endday=2023-06-11&tree=all

Comment 8

[Mike Conley (:mconley) (:⚙️)]

So, in a way, this isn't super surprising because the test directory that's running is for testing crashed tabs, so we're intentionally crashing tabs here. Normally, those crash reports are cleaned up and the test passes cleanly.

What's strange here is that the crash report isn't being cleaned up, and mak noticed that DllBlocklist_Initialize is on the stack at the time of the crash. There's another one where the mozilla::interceptor::WindowsDllInterceptor stuff is on the stack.

Going to tentatively move this to Core :: DLL Services, since it seems related to that.

Comment 9

[Intermittent Failures Robot]

9 failures in 4079 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 8
• mozilla-beta: 1

Platform and build breakdown:

• windows11-64-2009-ccov-qr: 6
  • opt: 6
• windows11-64-2009-qr: 3
  • debug: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-06-12&endday=2023-06-18&tree=all

Comment 10

[Intermittent Failures Robot]

8 failures in 3880 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 2
• mozilla-central: 4
• autoland: 2

Platform and build breakdown:

• windows11-64-2009-qr: 3
  • debug: 3
• windows11-64-2009-ccov-qr: 4
  • opt: 4
• windows11-32-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-06-19&endday=2023-06-25&tree=all

Comment 11

[Intermittent Failures Robot]

5 failures in 3594 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 4

Platform and build breakdown:

• windows11-64-2009-qr: 2
  • debug: 2
• windows11-64-2009-ccov-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-06-26&endday=2023-07-02&tree=all

Comment 12

[Intermittent Failures Robot]

7 failures in 2956 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• try: 1
• mozilla-central: 3

Platform and build breakdown:

• windows11-64-2009-qr: 3
  • debug: 3
• windows11-64-2009-ccov-qr: 3
  • opt: 3
• windows11-32-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-07-03&endday=2023-07-09&tree=all

Comment 13

[Intermittent Failures Robot]

6 failures in 3309 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3
• mozilla-beta: 1
• autoland: 2

Platform and build breakdown:

• windows11-32-2009-qr: 1
  • debug: 1
• windows11-64-2009-qr: 3
  • debug: 3
• windows11-64-2009-ccov-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-07-10&endday=2023-07-16&tree=all

Comment 14

[Intermittent Failures Robot]

6 failures in 3520 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 5

Platform and build breakdown:

• windows11-64-2009-qr: 2
  • debug: 2
• windows11-64-2009-ccov-qr: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-07-17&endday=2023-07-23&tree=all

Comment 15

[Intermittent Failures Robot]

15 failures in 3403 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 5
• mozilla-central: 7
• try: 3

Platform and build breakdown:

• windows11-32-2009-qr: 6
  • debug: 6
• windows11-64-2009-qr: 8
  • debug: 8
• windows11-64-2009-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-07-24&endday=2023-07-30&tree=all

Comment 16

[Intermittent Failures Robot]

21 failures in 3276 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 8
• mozilla-central: 6
• mozilla-beta: 7

Platform and build breakdown:

• windows11-32-2009-qr: 11
  • debug: 11
• windows11-64-2009-qr: 7
  • debug: 7
• windows11-64-2009-ccov-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-07-31&endday=2023-08-06&tree=all

Comment 17

[Intermittent Failures Robot]

8 failures in 3341 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• mozilla-central: 4
• try: 1

Platform and build breakdown:

• windows11-64-2009-qr: 2
  • debug: 2
• windows11-32-2009-mingwclang-qr: 1
  • debug: 1
• windows11-64-2009-ccov-qr: 3
  • opt: 3
• windows10-64-2009-qr: 1
  • debug: 1
• windows11-32-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-08-07&endday=2023-08-13&tree=all

Comment 18

[Intermittent Failures Robot]

7 failures in 3076 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3
• mozilla-beta: 1
• autoland: 2
• try: 1

Platform and build breakdown:

• windows11-64-2009-ccov-qr: 3
  • opt: 3
• windows11-64-2009-qr: 4
  • debug: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-08-14&endday=2023-08-20&tree=all

Comment 19

[Intermittent Failures Robot]

7 failures in 2218 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 7

Platform and build breakdown:

• windows11-64-2009-qr: 1
  • debug: 1
• windows11-64-2009-ccov-qr: 6
  • opt: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-08-21&endday=2023-08-27&tree=all

Comment 20

[Intermittent Failures Robot]

14 failures in 3019 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 9
• mozilla-central: 4
• mozilla-beta: 1

Platform and build breakdown:

• windows11-32-2009-qr: 7
  • debug: 7
• windows11-64-2009-ccov-qr: 1
  • opt: 1
• windows11-64-2009-mingwclang-qr: 1
  • debug: 1
• windows11-64-2009-qr: 5
  • debug: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-08-28&endday=2023-09-03&tree=all

Comment 21

[Intermittent Failures Robot]

48 failures in 3267 pushes (0.015 failures/push) were associated with this bug in the last 7 days.

This is the #24 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 17
• autoland: 24
• try: 4
• mozilla-beta: 3

Platform and build breakdown:

• windows11-32-2009-qr: 21
  • debug: 21
• windows11-64-2009-qr: 22
  • debug: 22
• windows11-64-2009-ccov-qr: 4
  • opt: 4
• windows11-64-2009-mingwclang-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-09-04&endday=2023-09-10&tree=all

Comment 22

[Intermittent Failures Robot]

42 failures in 3337 pushes (0.013 failures/push) were associated with this bug in the last 7 days.

This is the #30 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 8
• autoland: 23
• mozilla-beta: 5
• try: 5
• mozilla-esr115: 1

Platform and build breakdown:

• windows11-64-2009-ccov-qr: 3
  • opt: 3
• windows11-64-2009-qr: 26
  • debug: 26
• windows11-32-2009-qr: 13
  • debug: 13

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-09-11&endday=2023-09-17&tree=all

Comment 23

[Intermittent Failures Robot]

18 failures in 3850 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1
• autoland: 10
• mozilla-central: 7

Platform and build breakdown:

• windows11-64-2009-qr: 8
  • debug: 8
• windows11-32-2009-qr: 6
  • debug: 6
• windows11-64-2009-ccov-qr: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-09-18&endday=2023-09-24&tree=all

Comment 24

[Intermittent Failures Robot]

31 failures in 3726 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 10
• autoland: 13
• mozilla-beta: 6
• try: 2

Platform and build breakdown:

• windows11-64-2009-ccov-qr: 6
  • opt: 6
• windows11-32-2009-qr: 12
  • debug: 12
• windows11-64-2009-qr: 13
  • debug: 13

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-09-25&endday=2023-10-01&tree=all

Comment 25

[Intermittent Failures Robot]

36 failures in 4018 pushes (0.009 failures/push) were associated with this bug in the last 7 days.

This is the #47 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-beta: 5
• try: 3
• mozilla-central: 13
• autoland: 15

Platform and build breakdown:

• windows11-64-2009-qr: 13
  • debug: 13
• windows11-32-2009-qr: 17
  • debug: 17
• windows11-64-2009-ccov-qr: 5
  • opt: 5
• windows11-32-2009-mingwclang-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-10-02&endday=2023-10-08&tree=all

Comment 26

[Natalia Csoregi [:nataliaCs]]

Update

There have been 35 failures within the last 7 days:

• 1 failure on Windows 11 x86 22H2 MinGW WebRender debug
• 12 failures on Windows 11 x86 22H2 WebRender debug
• 6 failures on Windows 11 x64 22H2 CCov WebRender opt
• 16 failures on Windows 11 x64 22H2 WebRender debug

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=431980757&repo=autoland&lineNumber=3379

[task 2023-10-10T13:40:59.699Z] 13:40:59     INFO - PROCESS-CRASH | application crashed [@ DbgBreakPoint] | browser/base/content/test/alerts/browser.toml 
[task 2023-10-10T13:40:59.699Z] 13:40:59     INFO - Process type: unknown
[task 2023-10-10T13:40:59.699Z] 13:40:59     INFO - Process pid: 7512
[task 2023-10-10T13:40:59.701Z] 13:40:59     INFO - Crash dump filename: C:\Users\task_169694144824839\AppData\Local\Temp\tmpjz_z9v9t.mozrunner\minidumps\0f12ada8-209d-4023-a9db-d176acf70818.dmp
[task 2023-10-10T13:40:59.701Z] 13:40:59     INFO - Operating system: Windows NT
[task 2023-10-10T13:40:59.701Z] 13:40:59     INFO -                   10.0.22621
[task 2023-10-10T13:40:59.701Z] 13:40:59     INFO - CPU: amd64
[task 2023-10-10T13:40:59.701Z] 13:40:59     INFO -      family 6 model 106 stepping 6
[task 2023-10-10T13:40:59.701Z] 13:40:59     INFO -      8 CPUs
[task 2023-10-10T13:40:59.701Z] 13:40:59     INFO - 
[task 2023-10-10T13:40:59.701Z] 13:40:59     INFO - Crash reason:  EXCEPTION_BREAKPOINT
[task 2023-10-10T13:40:59.701Z] 13:40:59     INFO - Crash address: 0x7ffd67db2940
[task 2023-10-10T13:40:59.701Z] 13:40:59     INFO - Crashing instruction: `int 0x3`
[task 2023-10-10T13:40:59.701Z] 13:40:59     INFO - No memory accessed by instruction
[task 2023-10-10T13:40:59.701Z] 13:40:59     INFO - Process uptime: 49 seconds
[task 2023-10-10T13:40:59.701Z] 13:40:59     INFO - 
[task 2023-10-10T13:40:59.701Z] 13:40:59     INFO - Thread 3  (crashed)
[task 2023-10-10T13:40:59.701Z] 13:40:59     INFO -  0  ntdll.dll!DbgBreakPoint + 0x0
[task 2023-10-10T13:40:59.701Z] 13:40:59     INFO -      rax = 0x00007ffd67db2940    rdx = 0x00007ffd67db2940
[task 2023-10-10T13:40:59.701Z] 13:40:59     INFO -      rcx = 0x0043504900435049    rbx = 0x0000000000000000
[task 2023-10-10T13:40:59.701Z] 13:40:59     INFO -      rsi = 0x0043504900435049    rdi = 0x0000000000000000
[task 2023-10-10T13:40:59.701Z] 13:40:59     INFO -      rbp = 0x0000000000000000    rsp = 0x0000009370fff808
[task 2023-10-10T13:40:59.701Z] 13:40:59     INFO -       r8 = 0x0043504900435049     r9 = 0x0000000000000000
[task 2023-10-10T13:40:59.701Z] 13:40:59     INFO -      r10 = 0x00000fffacfb6528    r11 = 0x0500050004444444
[task 2023-10-10T13:40:59.701Z] 13:40:59     INFO -      r12 = 0x0000000000000000    r13 = 0x0000000000000000
[task 2023-10-10T13:40:59.702Z] 13:40:59     INFO -      r14 = 0x0000000000000000    r15 = 0x0000000000000000
[task 2023-10-10T13:40:59.702Z] 13:40:59     INFO -      rip = 0x00007ffd67db2940
[task 2023-10-10T13:40:59.702Z] 13:40:59     INFO -     Found by: given as instruction pointer in context
[task 2023-10-10T13:40:59.702Z] 13:40:59     INFO -  1  kernel32.dll!BaseThreadInitThunk + 0x1c
[task 2023-10-10T13:40:59.702Z] 13:40:59     INFO -      rbx = 0x0000000000000000    rbp = 0x0000000000000000
[task 2023-10-10T13:40:59.702Z] 13:40:59     INFO -      rsp = 0x0000009370fff810    r12 = 0x0000000000000000
[task 2023-10-10T13:40:59.702Z] 13:40:59     INFO -      r13 = 0x0000000000000000    r14 = 0x0000000000000000
[task 2023-10-10T13:40:59.702Z] 13:40:59     INFO -      r15 = 0x0000000000000000    rip = 0x00007ffd6607269d
[task 2023-10-10T13:40:59.702Z] 13:40:59     INFO -     Found by: call frame info
[task 2023-10-10T13:40:59.702Z] 13:40:59     INFO -  2  mozglue.dll!mozilla::interceptor::FuncHook<mozilla::interceptor::WindowsDllInterceptor<mozilla::interceptor::VMSharingPolicyShared>,void (*)(int, void *, void *)>::operator()(int&, void*&, void*&) const [nsWindowsDllInterceptor.h:7a4c861a2d10a6757c8844de9bd8f71c94cb6150 : 150]
[task 2023-10-10T13:40:59.702Z] 13:40:59     INFO -     Found by: inlining
[task 2023-10-10T13:40:59.702Z] 13:40:59     INFO -  3  mozglue.dll!patched_BaseThreadInitThunk(int, void*, void*) [WindowsDllBlocklist.cpp:7a4c861a2d10a6757c8844de9bd8f71c94cb6150 : 561 + 0x14]
[task 2023-10-10T13:40:59.702Z] 13:40:59     INFO -      rbx = 0x00007ffd67db2940    rsi = 0x0043504900435049
[task 2023-10-10T13:40:59.702Z] 13:40:59     INFO -      rdi = 0x0000000000000000    rbp = 0x0000000000000000
[task 2023-10-10T13:40:59.702Z] 13:40:59     INFO -      rsp = 0x0000009370fff840    r12 = 0x0000000000000000
[task 2023-10-10T13:40:59.702Z] 13:40:59     INFO -      r13 = 0x0000000000000000    r14 = 0x0000000000000000
[task 2023-10-10T13:40:59.702Z] 13:40:59     INFO -      r15 = 0x0000000000000000    rip = 0x00007ffd407cf2c4
[task 2023-10-10T13:40:59.702Z] 13:40:59     INFO -     Found by: call frame info
[task 2023-10-10T13:40:59.702Z] 13:40:59     INFO -  4  ntdll.dll!RtlUserThreadStart + 0x27
[task 2023-10-10T13:40:59.703Z] 13:40:59     INFO -      rbx = 0x0000000000000000    rsi = 0x0000000000000000
[task 2023-10-10T13:40:59.703Z] 13:40:59     INFO -      rdi = 0x0000000000000000    rbp = 0x0000000000000000
[task 2023-10-10T13:40:59.703Z] 13:40:59     INFO -      rsp = 0x0000009370fff8b0    r12 = 0x0000000000000000
[task 2023-10-10T13:40:59.703Z] 13:40:59     INFO -      r13 = 0x0000000000000000    r14 = 0x0000000000000000
[task 2023-10-10T13:40:59.703Z] 13:40:59     INFO -      r15 = 0x0000000000000000    rip = 0x00007ffd67d6a9f8
[task 2023-10-10T13:40:59.703Z] 13:40:59     INFO -     Found by: call frame info
[task 2023-10-10T13:40:59.703Z] 13:40:59     INFO - 

Greg, can you assign this to someone?
Thank you.

Comment 27

[Greg Stoll :gstoll]

The linked failure log has the error 0x241 in the Utility process - could this be related to bug 1851889?

Comment 28

[:gerard-majax]

(In reply to Greg Stoll from comment #27)

The linked failure log has the error 0x241 in the Utility process - could this be related to bug 1851889?

Indeed ERROR_INVALID_IMAGE_HASH. Unfortunately, not a lot I can do ... ?

Comment 29

[:gerard-majax]

only on the last one it seems, other failure not always report that, maybe it's just a coincidence and/or anotehr hint that "something" is wrong with the worker ?

Comment 30

[Intermittent Failures Robot]

35 failures in 3764 pushes (0.009 failures/push) were associated with this bug in the last 7 days.

This is the #50 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 13
• mozilla-central: 11
• mozilla-esr115: 2
• mozilla-beta: 5
• try: 4

Platform and build breakdown:

• windows11-64-2009-qr: 20
  • debug: 20
• windows11-32-2009-qr: 7
  • debug: 7
• windows11-64-2009-ccov-qr: 7
  • opt: 7
• windows11-32-2009-mingwclang-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-10-09&endday=2023-10-15&tree=all

Comment 31

[Greg Stoll :gstoll]

I'm not sure what's going on here. Looking at the minidump in this crash report, I can't tell what's going on in BaseThreadInitThunk that makes it call DbgBreakPoint. :yannis, would you mind taking a look?

Comment 32

[Yannis Juglaret [:yannis]]

I think that the intermittent failures that we observe with this signature were introduced by bug 1793525. They would occur when the main process kindly asked a child process to shut down but the child process is taking too long to do that. Thanks to bug 1793525, when that happens, we now get a dump of the child process, so we can have an idea of why it did not shut down in time. To do that, we should look at what the main thread of the child process was doing when we forced the crash (the logs contain stacks for the threads of the child process), and see if that explains why it is not processing events.

If we are seeing false positives where the child process was legitimately taking a lot of time for shutdown, we could consider increasing the constants that define how much time we wait. NI [:jld] for his opinion on about that.

Now, I looked at the child process dumps from comment 25 and comment 30, and grouped them based on common patterns of what was going on in the main thread of the child process. Here are the results of doing that (test categorization is not exact, please check the linked jobs directly if you need something more accurate):

• xul.dll!mozilla::ipc::UtilityProcessImpl::LoadLibraryOrCrash (24 failures in mochitest-browser-chrome/mochitest-plain/mochitest-a11y/mochitest-webgpu): 25-11, 25-12, 25-22, 25-25, 25-29, 25-30, 25-4, 25-6, 25-7, 25-8, 30-1, 30-14, 30-15, 30-18, 30-2, 30-21, 30-25, 30-27, 30-28, 30-31, 30-33, 30-34, 30-4, 30-5;
• xul.dll!mozilla::ShutdownXPCOM (12 failures in mochitest-browser-chrome/mochitest-plain/mochitest-media, 1 failure in xpcshell test): 25-0, 25-1, 25-14, 25-15, 25-19, 25-20, 25-31, 30-10, 30-13, 30-16, 30-19, 30-22, 30-6;
• xul.dll!mozilla::layers::CompositorThreadHolder::Shutdown (11 failures in mochitest-browser-media-wmfme): 25-18, 25-2, 25-23, 25-28, 25-3, 30-0, 30-20, 30-23, 30-29, 30-3, 30-9;
• xul.dll!mozilla::WMFDecoderModule::Init (8 failures in mochitest-browser-media-wmfme): 25-10, 25-16, 25-26, 25-32, 25-5, 30-11, 30-30, 30-32;
• xul.dll!CrashReporter::UnsetRemoteExceptionHandler (4 failures in mochitest-browser-chrome-msix, 3 failures in mochitest-media): 25-13, 25-24, 25-33, 30-12, 30-24, 30-26, 30-7;
• xul.dll!mozilla::Telemetry::Init (3 failures in mochitest-browser-chrome/mochitest-plain): 25-21, 25-35, 30-17;
• xul.dll!nsComponentManagerImpl::Init (2 failures in mochitest-plain): 25-17, 25-27;
• xul.dll!mozilla::ChildProfilerController::Init (1 failure in mochitest-browser-chrome): 25-34;
• xul.dll!mozilla::ipc::PUtilityProcessChild::OnChannelClose (1 failure in mochitest-browser-chrome-swr): 30-8;
• xul.dll!mozilla::net::SocketProcessChild::Init (1 failure in xpcshell-spi-nw): 25-9.

The failures in xul.dll!mozilla::ipc::UtilityProcessImpl::LoadLibraryOrCrash should indeed be considered the same as bug 1851889. Here, we crash while the child process is walking its own stack and printing it to stdout before crashing, as part of MOZ_ReportCrash. It is expected that this can take some time, as it requires matching memory addresses to code lines using PDB files, and xul.pdb is huge. If the child process was given enough time, these failures would end up in bug 1851889. Also, fixing the issue there would make the ones we have here disappear as well.

The other failures would require more analysis. Also, profiling these tests could give a more precise idea of what is really taking time for the failures that seem quite generic like xul.dll!mozilla::ShutdownXPCOM and xul.dll!mozilla::layers::CompositorThreadHolder::Shutdown. Here we only know what the process was doing when we reached the timeout, and we could reach wrong conclusions if we only look at that.

Comment 33

[Yannis Juglaret [:yannis]]

The failures in xul.dll!CrashReporter::UnsetRemoteExceptionHandler look like instances where the child process was just about to be done with shutdown when we triggered the crash. But because we wanted to trigger a crash, the remote thread that we created started running google_breakpad::ExceptionHandler::HandleException, so this thread is thus now holding ExceptionHandler::handler_stack_critical_section_. Then, while the exception handler was preparing for crashing in the remote thread, the main thread reached xul.dll!CrashReporter::UnsetRemoteExceptionHandler and got stuck there, because it cannot acquire ExceptionHandler::handler_stack_critical_section_. So, I would say that seeing failures in xul.dll!CrashReporter::UnsetRemoteExceptionHandler are false positives that suggest that we may be a bit too strict on the amount of time that we are giving to child processes.

Comment 34

[Intermittent Failures Robot]

16 failures in 3484 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 10
• autoland: 5
• try: 1

Platform and build breakdown:

• windows11-64-2009-ccov-qr: 5
  • opt: 5
• windows11-64-2009-qr: 6
  • debug: 6
• windows11-32-2009-qr: 5
  • debug: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-10-16&endday=2023-10-22&tree=all

Comment 35

[Jed Davis [:jld] ⟨⏰|UTC-8⟩ ⟦he/him⟧]

Yes, it looks like the timeout for regular debug builds should be increased. Maybe try 20s?

But, one thing I noticed: the hangs on ccov builds seem to all be in CompositorThreadHolder::Shutdown (and, conversely, everything with that signature seems to be a ccov build). I'm not sure what's going on there, but it looks like that's happening in the GPU process. I suppose that could be a false positive, but it seems strange that it's seemingly always in the same place and always the GPU process rather than content.

Comment 36

[Intermittent Failures Robot]

37 failures in 3657 pushes (0.01 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 19
• mozilla-central: 14
• elm: 2
• mozilla-beta: 2

Platform and build breakdown:

• windows11-64-2009-qr: 15
  • debug: 15
• windows11-64-2009-ccov-qr: 8
  • opt: 8
• windows11-32-2009-qr: 14
  • debug: 14

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-10-23&endday=2023-10-29&tree=all

Comment 37

[Yannis Juglaret [:yannis]]

Attachment: Bug 1831236 - Increase shutdown wait time for debug builds. r=jld

As we are seeing test failures that suggest that proper shutdown
sometimes requires extra time in debug builds, let's try to upgrade
kShutdownWaitMs from 8 to 20 seconds for those.

Comment 38

[Intermittent Failures Robot]

27 failures in 3356 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 7
• mozilla-central: 12
• autoland: 6
• try: 2

Platform and build breakdown:

• windows11-64-2009-qr: 10
  • debug: 10
• windows11-64-2009-ccov-qr: 9
  • opt: 9
• windows11-32-2009-qr: 8
  • debug: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-10-30&endday=2023-11-05&tree=all

Comment 39

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4cbecf0c0844
Increase shutdown wait time for debug builds. r=jld

Comment 40

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/4cbecf0c0844

Comment 41

[Natalia Csoregi [:nataliaCs]]

This is still happening on Windows 11 x86 22H2 WebRender debug: https://treeherder.mozilla.org/jobs?repo=autoland&revision=05f23c29b41cfcb7bfe0b081854260fa6d04388f&searchStr=windows%2C11%2Cx86%2C22h2%2Cwebrender%2Cdebug&group_state=expanded&selectedTaskRun=HmdCxCirSeuGRYHL85zZKQ.0

Comment 42

[Yannis Juglaret [:yannis]]

The patch from comment 39 only hopes to remove some false positives. The example from comment 41 is the Windows equivalent of the Linux failures in bug 1805613 that led to filing bug 1826222. We are waiting for cld-worker.js during mozilla::dom::workerinternals::RuntimeService::Cleanup(). The logs show:

[task 2023-11-07T08:31:11.349Z] 08:31:11     INFO - GECKO(4452) | [Child 6024: Main Thread]: D/WorkerShutdownDump Found ActiveWorker (dedicated): resource://gre/modules/translation/cld-worker.js
[task 2023-11-07T08:31:11.350Z] 08:31:11     INFO - GECKO(4452) | [Child 6024: Main Thread]: D/WorkerShutdownDump Principal: [System Principal]
[task 2023-11-07T08:31:11.350Z] 08:31:11     INFO - GECKO(4452) | [Child 6024: Main Thread]: D/WorkerShutdownDump LoadingPrincipal: [System Principal]
[task 2023-11-07T08:31:11.351Z] 08:31:11     INFO - GECKO(4452) | [Child 6024: Main Thread]: D/WorkerShutdownDump BusyCount: 4
[task 2023-11-07T08:31:11.352Z] 08:31:11     INFO - GECKO(4452) | [Child 6024: Main Thread]: D/WorkerShutdownDump CrashInfo: IsChromeWorker(false)|ScriptLoader|XMLHttpRequestWorker
...
[task 2023-11-07T08:32:22.032Z] 08:32:22     INFO - PROCESS-CRASH | application crashed [@ DbgBreakPoint] | dom/serviceworkers/test/mochitest-dFPI.ini:dom/serviceworkers/test/mochitest-common.ini 
[task 2023-11-07T08:32:22.032Z] 08:32:22     INFO - Process type: unknown
[task 2023-11-07T08:32:22.032Z] 08:32:22     INFO - Process pid: 6024
...
[task 2023-11-07T08:32:22.043Z] 08:32:22     INFO - Thread 0 MainThread
...
[task 2023-11-07T08:32:22.044Z] 08:32:22     INFO -  9  xul.dll!NS_ProcessNextEvent(nsIThread*, bool) [nsThreadUtils.cpp:05f23c29b41cfcb7bfe0b081854260fa6d04388f : 480 + 0x15]
...
[task 2023-11-07T08:32:22.044Z] 08:32:22     INFO - 10  xul.dll!mozilla::dom::workerinternals::RuntimeService::Cleanup() [RuntimeService.cpp:05f23c29b41cfcb7bfe0b081854260fa6d04388f : 1625 + 0x7]
...
ask 2023-11-07T08:31:14.067Z] 08:31:14     INFO - GECKO(4452) | [Child 10092: Main Thread]: D/WorkerShutdownDump Found ActiveWorker (dedicated): resource://gre/modules/translation/cld-worker.js
[task 2023-11-07T08:31:14.068Z] 08:31:14     INFO - GECKO(4452) | [Child 10092: Main Thread]: D/WorkerShutdownDump Principal: [System Principal]
[task 2023-11-07T08:31:14.068Z] 08:31:14     INFO - GECKO(4452) | [Child 10092: Main Thread]: D/WorkerShutdownDump LoadingPrincipal: [System Principal]
[task 2023-11-07T08:31:14.069Z] 08:31:14     INFO - GECKO(4452) | [Child 10092: Main Thread]: D/WorkerShutdownDump BusyCount: 4
[task 2023-11-07T08:31:14.069Z] 08:31:14     INFO - GECKO(4452) | [Child 10092: Main Thread]: D/WorkerShutdownDump CrashInfo: IsChromeWorker(false)|ScriptLoader|XMLHttpRequestWorker
...
[task 2023-11-07T08:32:30.816Z] 08:32:30     INFO - PROCESS-CRASH | application crashed [@ DbgBreakPoint] | dom/serviceworkers/test/mochitest-dFPI.ini:dom/serviceworkers/test/mochitest-common.ini 
[task 2023-11-07T08:32:30.816Z] 08:32:30     INFO - Process type: unknown
[task 2023-11-07T08:32:30.816Z] 08:32:30     INFO - Process pid: 10092
...
[task 2023-11-07T08:32:30.819Z] 08:32:30     INFO - Thread 0 MainThread
...
[task 2023-11-07T08:32:30.821Z] 08:32:30     INFO -  9  xul.dll!NS_ProcessNextEvent(nsIThread*, bool) [nsThreadUtils.cpp:05f23c29b41cfcb7bfe0b081854260fa6d04388f : 480 + 0x15]
...
[task 2023-11-07T08:32:30.821Z] 08:32:30     INFO - 10  xul.dll!mozilla::dom::workerinternals::RuntimeService::Cleanup() [RuntimeService.cpp:05f23c29b41cfcb7bfe0b081854260fa6d04388f : 1625 + 0x7]

Comment 43

[Intermittent Failures Robot]

52 failures in 552 pushes (0.094 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 50
• try: 1
• mozilla-beta: 1

Platform and build breakdown:

• windows11-32-2009-qr: 21
  • debug: 21
• windows11-64-2009-qr: 26
  • debug: 26
• windows10-64-2009-qr: 5
  • debug: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-11-07&endday=2023-11-07&tree=all

Comment 44

[Intermittent Failures Robot]

64 failures in 3288 pushes (0.019 failures/push) were associated with this bug in the last 7 days.

This is the #19 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 53
• try: 4
• mozilla-beta: 4
• mozilla-central: 3

Platform and build breakdown:

• windows11-32-2009-qr: 25
  • debug: 25
• windows11-64-2009-qr: 31
  • debug: 31
• windows10-64-2009-qr: 5
  • debug: 5
• windows11-64-2009-ccov-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-11-06&endday=2023-11-12&tree=all

Comment 45

[Intermittent Failures Robot]

11 failures in 3495 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• mozilla-central: 8

Platform and build breakdown:

• windows11-64-2009-qr: 4
  • debug: 4
• windows11-64-2009-ccov-qr: 6
  • opt: 6
• windows11-32-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-11-13&endday=2023-11-19&tree=all

Comment 46

[Intermittent Failures Robot]

18 failures in 3623 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• mozilla-central: 6
• mozilla-beta: 5
• autoland: 6

Platform and build breakdown:

• windows11-32-2009-qr: 7
  • debug: 7
• windows11-64-2009-ccov-qr: 5
  • opt: 5
• windows11-64-2009-qr: 6
  • debug: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-11-20&endday=2023-11-26&tree=all

Comment 47

[Intermittent Failures Robot]

19 failures in 4108 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4
• try: 3
• autoland: 6
• mozilla-beta: 6

Platform and build breakdown:

• windows11-64-2009-ccov-qr: 2
  • opt: 2
• windows11-64-2009-qr: 10
  • debug: 10
• windows11-32-2009-qr: 7
  • debug: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-11-27&endday=2023-12-03&tree=all

Comment 48

[Intermittent Failures Robot]

15 failures in 3671 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 4
• try: 2
• autoland: 2
• mozilla-central: 7

Platform and build breakdown:

• windows11-64-2009-qr: 7
  • debug: 7
• windows11-32-2009-qr: 2
  • debug: 2
• windows11-64-2009-ccov-qr: 6
  • opt: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-12-04&endday=2023-12-10&tree=all

Comment 49

[Intermittent Failures Robot]

9 failures in 4224 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 9

Platform and build breakdown:

• windows11-64-2009-qr: 2
  • debug: 2
• windows11-64-2009-ccov-qr: 6
  • opt: 6
• windows11-32-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-12-11&endday=2023-12-17&tree=all

Comment 50

[Intermittent Failures Robot]

18 failures in 3349 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 7
• mozilla-central: 8
• mozilla-beta: 3

Platform and build breakdown:

• windows10-64-2009-qr: 1
  • debug: 1
• windows11-64-2009-ccov-qr: 7
  • opt: 7
• windows11-64-2009-qr: 5
  • debug: 5
• windows11-32-2009-qr: 5
  • debug: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-12-18&endday=2023-12-24&tree=all

Comment 51

[Intermittent Failures Robot]

13 failures in 1162 pushes (0.011 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-beta: 3
• mozilla-central: 6
• try: 2

Platform and build breakdown:

• windows11-32-2009-qr: 6
  • debug: 6
• windows11-64-2009-ccov-qr: 4
  • opt: 4
• windows11-64-2009-qr: 3
  • debug: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2023-12-25&endday=2023-12-31&tree=all

Comment 52

[Intermittent Failures Robot]

15 failures in 2274 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 7
• autoland: 3
• try: 2
• mozilla-beta: 3

Platform and build breakdown:

• windows11-64-2009-qr: 8
  • debug: 8
• windows11-64-2009-ccov-qr: 5
  • opt: 5
• windows11-32-2009-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-01-01&endday=2024-01-07&tree=all

Comment 53

[Intermittent Failures Robot]

19 failures in 3217 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 8
• try: 2
• autoland: 8
• mozilla-beta: 1

Platform and build breakdown:

• windows11-64-2009-ccov-qr: 6
  • opt: 6
• windows11-32-2009-qr: 6
  • debug: 6
• windows11-64-2009-qr: 7
  • debug: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-01-08&endday=2024-01-14&tree=all

Comment 54

[Intermittent Failures Robot]

14 failures in 3599 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 6
• autoland: 8

Platform and build breakdown:

• windows11-64-2009-ccov-qr: 2
  • opt: 2
• windows11-32-2009-qr: 6
  • debug: 6
• windows11-32-2009-mingwclang-qr: 1
  • debug: 1
• windows11-64-2009-qr: 5
  • debug: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-01-15&endday=2024-01-21&tree=all

Comment 55

[Intermittent Failures Robot]

19 failures in 4034 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 5
• mozilla-central: 7
• try: 6
• mozilla-beta: 1

Platform and build breakdown:

• windows11-64-2009-qr: 10
  • debug: 10
• windows11-32-2009-qr: 5
  • debug: 5
• windows11-64-2009-ccov-qr: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-01-22&endday=2024-01-28&tree=all

Comment 56

[Intermittent Failures Robot]

15 failures in 3952 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 9
• mozilla-beta: 6

Platform and build breakdown:

• windows11-64-2009-mingwclang-qr: 1
  • debug: 1
• windows11-64-2009-ccov-qr: 7
  • opt: 7
• windows11-32-2009-qr: 3
  • debug: 3
• windows11-64-2009-qr: 4
  • debug: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-01-29&endday=2024-02-04&tree=all

Comment 57

[Intermittent Failures Robot]

20 failures in 4009 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4
• mozilla-esr115: 2
• autoland: 12
• mozilla-beta: 2

Platform and build breakdown:

• windows11-64-2009-ccov-qr: 2
  • opt: 2
• windows11-64-2009-qr: 9
  • debug: 9
• windows11-32-2009-qr: 9
  • debug: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-02-05&endday=2024-02-11&tree=all

Comment 58

[Intermittent Failures Robot]

5 failures in 3125 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 5

Platform and build breakdown:

• windows11-32-2009-mingwclang-qr: 1
  • debug: 1
• windows11-64-2009-qr: 1
  • debug: 1
• windows11-64-2009-ccov-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-02-12&endday=2024-02-18&tree=all

Comment 59

[Intermittent Failures Robot]

24 failures in 3382 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 3
• mozilla-central: 11
• comm-central: 1
• pine: 1
• autoland: 8

Platform and build breakdown:

• windows11-64-2009-qr: 9
  • debug: 9
• windows11-64-2009-ccov-qr: 8
  • opt: 8
• windows11-32-2009-qr: 7
  • debug: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-02-19&endday=2024-02-25&tree=all

Comment 60

[Intermittent Failures Robot]

24 failures in 4052 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 6
• mozilla-esr115: 1
• mozilla-central: 15
• autoland: 2

Platform and build breakdown:

• windows11-32-2009-qr: 6
  • debug: 6
• windows11-64-2009-qr: 7
  • debug: 7
• windows11-64-2009-ccov-qr: 9
  • opt: 9
• windows11-64-2009-mingwclang-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-02-26&endday=2024-03-03&tree=all

Comment 61

[Intermittent Failures Robot]

16 failures in 4198 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 8
• try: 4
• autoland: 3
• mozilla-beta: 1

Platform and build breakdown:

• windows11-64-2009-ccov-qr: 6
  • opt: 6
• windows11-64-2009-qr: 7
  • debug: 7
• windows11-32-2009-qr: 3
  • debug: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-03-04&endday=2024-03-10&tree=all

Comment 62

[Intermittent Failures Robot]

17 failures in 3849 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 12
• autoland: 4
• try: 1

Platform and build breakdown:

• windows11-64-2009-qr: 5
  • debug: 5
• windows11-64-2009-ccov-qr: 11
  • opt: 11
• windows11-32-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-03-11&endday=2024-03-17&tree=all

Comment 63

[Intermittent Failures Robot]

25 failures in 3791 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 15
• autoland: 3
• try: 3
• mozilla-beta: 4

Platform and build breakdown:

• windows11-64-2009-ccov-qr: 14
  • opt: 14
• windows11-64-2009-qr: 4
  • debug: 4
• windows11-32-2009-qr: 6
  • debug: 6
• windows10-64-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-03-18&endday=2024-03-24&tree=all

Comment 64

[Natalia Csoregi [:nataliaCs]]

Update

There have been 30 failures within the last 7 days:

• 9 failures on Windows 11 x86 22H2 WebRender debug
• 18 failures on Windows 11 x64 22H2 CCov WebRender opt
• 3 failures on Windows 11 x64 22H2 WebRender debug

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=452951153&repo=mozilla-central&lineNumber=1610

[task 2024-03-30T22:43:35.979Z] 22:43:35     INFO - PROCESS-CRASH | application crashed [@ DbgBreakPoint] | dom/media/test/browser/wmfme/browser.toml 
[task 2024-03-30T22:43:35.979Z] 22:43:35     INFO - Process type: unknown
[task 2024-03-30T22:43:35.979Z] 22:43:35     INFO - Process pid: 7480
[task 2024-03-30T22:43:35.979Z] 22:43:35     INFO - Crash dump filename: C:\Users\task_171183824317314\AppData\Local\Temp\tmpc335tzuo.mozrunner\minidumps\90538d94-a950-452a-8b33-dca0a7ac5908.dmp
[task 2024-03-30T22:43:35.979Z] 22:43:35     INFO - Operating system: Windows NT
[task 2024-03-30T22:43:35.981Z] 22:43:35     INFO -                   10.0.22621
[task 2024-03-30T22:43:35.981Z] 22:43:35     INFO - CPU: amd64
[task 2024-03-30T22:43:35.981Z] 22:43:35     INFO -      family 6 model 85 stepping 7
[task 2024-03-30T22:43:35.981Z] 22:43:35     INFO -      8 CPUs
[task 2024-03-30T22:43:35.981Z] 22:43:35     INFO - 
[task 2024-03-30T22:43:35.981Z] 22:43:35     INFO - Crash reason:  EXCEPTION_BREAKPOINT
[task 2024-03-30T22:43:35.981Z] 22:43:35     INFO - Crash address: 0x00007ffa2f473060
[task 2024-03-30T22:43:35.981Z] 22:43:35     INFO - Crashing instruction: `int 0x3`
[task 2024-03-30T22:43:35.981Z] 22:43:35     INFO - No memory accessed by instruction
[task 2024-03-30T22:43:35.981Z] 22:43:35     INFO - Process uptime: 115 seconds
[task 2024-03-30T22:43:35.981Z] 22:43:35     INFO - 
[task 2024-03-30T22:43:35.981Z] 22:43:35     INFO - Thread 35  (crashed)
[task 2024-03-30T22:43:35.981Z] 22:43:35     INFO -  0  ntdll.dll!DbgBreakPoint + 0x0
[task 2024-03-30T22:43:35.981Z] 22:43:35     INFO -      rax = 0x00007ffa2f473060    rdx = 0x00007ffa2f473060
[task 2024-03-30T22:43:35.981Z] 22:43:35     INFO -      rcx = 0x0043504900435049    rbx = 0x0000000000000000
[task 2024-03-30T22:43:35.981Z] 22:43:35     INFO -      rsi = 0x0043504900435049    rdi = 0x0000000000000000
[task 2024-03-30T22:43:35.981Z] 22:43:35     INFO -      rbp = 0x0000000000000000    rsp = 0x0000008c3ffff8c8
[task 2024-03-30T22:43:35.981Z] 22:43:35     INFO -       r8 = 0x0043504900435049     r9 = 0x0000000000000000
[task 2024-03-30T22:43:35.981Z] 22:43:35     INFO -      r10 = 0x00000fff45e8e60c    r11 = 0x0010004050005000
[task 2024-03-30T22:43:35.981Z] 22:43:35     INFO -      r12 = 0x0000000000000000    r13 = 0x0000000000000000
[task 2024-03-30T22:43:35.981Z] 22:43:35     INFO -      r14 = 0x0000000000000000    r15 = 0x0000000000000000
[task 2024-03-30T22:43:35.981Z] 22:43:35     INFO -      rip = 0x00007ffa2f473060
[task 2024-03-30T22:43:35.981Z] 22:43:35     INFO -     Found by: given as instruction pointer in context
[task 2024-03-30T22:43:35.981Z] 22:43:35     INFO -  1  kernel32.dll!BaseThreadInitThunk + 0x1c
[task 2024-03-30T22:43:35.982Z] 22:43:35     INFO -      rbx = 0x0000000000000000    rbp = 0x0000000000000000
[task 2024-03-30T22:43:35.982Z] 22:43:35     INFO -      rsp = 0x0000008c3ffff8d0    r12 = 0x0000000000000000
[task 2024-03-30T22:43:35.982Z] 22:43:35     INFO -      r13 = 0x0000000000000000    r14 = 0x0000000000000000
[task 2024-03-30T22:43:35.982Z] 22:43:35     INFO -      r15 = 0x0000000000000000    rip = 0x00007ffa2e27257d
[task 2024-03-30T22:43:35.982Z] 22:43:35     INFO -     Found by: call frame info
[task 2024-03-30T22:43:35.982Z] 22:43:35     INFO -  2  mozglue.dll!DllBlocklist_Initialize(unsigned int) + 0x3c9
[task 2024-03-30T22:43:35.982Z] 22:43:35     INFO -      rbx = 0x00007ffa2f473060    rsi = 0x0043504900435049
[task 2024-03-30T22:43:35.982Z] 22:43:35     INFO -      rdi = 0x0000000000000000    rbp = 0x0000000000000000
[task 2024-03-30T22:43:35.982Z] 22:43:35     INFO -      rsp = 0x0000008c3ffff900    r12 = 0x0000000000000000
[task 2024-03-30T22:43:35.982Z] 22:43:35     INFO -      r13 = 0x0000000000000000    r14 = 0x0000000000000000
[task 2024-03-30T22:43:35.982Z] 22:43:35     INFO -      r15 = 0x0000000000000000    rip = 0x00007ff9f90b3f2a
[task 2024-03-30T22:43:35.982Z] 22:43:35     INFO -     Found by: call frame info
[task 2024-03-30T22:43:35.982Z] 22:43:35     INFO -  3  ntdll.dll!RtlUserThreadStart + 0x27
[task 2024-03-30T22:43:35.982Z] 22:43:35     INFO -      rbx = 0x0000000000000000    rsi = 0x0000000000000000
[task 2024-03-30T22:43:35.982Z] 22:43:35     INFO -      rdi = 0x0000000000000000    rbp = 0x0000000000000000
[task 2024-03-30T22:43:35.982Z] 22:43:35     INFO -      rsp = 0x0000008c3ffff970    r12 = 0x0000000000000000
[task 2024-03-30T22:43:35.982Z] 22:43:35     INFO -      r13 = 0x0000000000000000    r14 = 0x0000000000000000
[task 2024-03-30T22:43:35.982Z] 22:43:35     INFO -      r15 = 0x0000000000000000    rip = 0x00007ffa2f42aa58
[task 2024-03-30T22:43:35.982Z] 22:43:35     INFO -     Found by: call frame info
[task 2024-03-30T22:43:35.982Z] 22:43:35     INFO - 

Comment 65

[Intermittent Failures Robot]

34 failures in 3590 pushes (0.009 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 20
• try: 1
• mozilla-beta: 4
• autoland: 9

Platform and build breakdown:

• windows11-32-2009-qr: 13
  • debug: 13
• windows11-64-2009-ccov-qr: 17
  • opt: 17
• windows11-64-2009-qr: 4
  • debug: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-03-25&endday=2024-03-31&tree=all

Comment 66

[Intermittent Failures Robot]

30 failures in 3805 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 19
• mozilla-beta: 7
• autoland: 2
• try: 2

Platform and build breakdown:

• windows11-32-2009-qr: 10
  • debug: 10
• windows11-64-2009-ccov-qr: 14
  • opt: 14
• windows11-64-2009-qr: 5
  • debug: 5
• windows11-64-2009-mingwclang-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-04-01&endday=2024-04-07&tree=all

Comment 67

[Intermittent Failures Robot]

24 failures in 3235 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

This is the #46 most frequent failure this week.

Repository breakdown:

• autoland: 2
• mozilla-central: 22

Platform and build breakdown:

• windows11-64-2009-qr: 4
  • debug: 4
• windows11-32-2009-mingwclang-qr: 1
  • debug: 1
• windows11-64-2009-ccov-qr: 17
  • opt: 17
• windows11-32-2009-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-04-08&endday=2024-04-14&tree=all

Comment 68

[Intermittent Failures Robot]

15 failures in 4252 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 14
• try: 1

Platform and build breakdown:

• windows11-64-2009-ccov-qr: 14
  • opt: 14
• windows10-64-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-04-15&endday=2024-04-21&tree=all

Comment 69

[Intermittent Failures Robot]

9 failures in 4336 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 5
• autoland: 2
• try: 1
• mozilla-beta: 1

Platform and build breakdown:

• windows11-64-2009-ccov-qr: 5
  • opt: 5
• windows11-64-2009-qr: 2
  • debug: 2
• windows11-32-2009-qr: 1
  • debug: 1
• windows10-64-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-04-22&endday=2024-04-28&tree=all

Comment 70

[Intermittent Failures Robot]

6 failures in 4059 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1
• mozilla-central: 3
• autoland: 1
• try: 1

Platform and build breakdown:

• windows11-64-2009-qr: 4
  • debug: 4
• windows11-32-2009-qr: 1
  • debug: 1
• windows11-64-2009-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-04-29&endday=2024-05-05&tree=all

Comment 71

[Yannis Juglaret [:yannis]]

The landing of bug 1851889 and more recently bug 1879375 should have removed the biggest causes for the intermittent failures listed in this bug. Failures can still happen for other reasons though.

Comment 72

[Intermittent Failures Robot]

7 failures in 3649 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 5
• autoland: 1
• try: 1

Platform and build breakdown:

• windows11-64-2009-ccov-qr: 3
  • opt: 3
• windows11-64-2009-qr: 3
  • debug: 3
• windows11-32-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-05-06&endday=2024-05-12&tree=all

Comment 73

[Intermittent Failures Robot]

8 failures in 4547 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• mozilla-beta: 3
• mozilla-central: 1
• try: 1

Platform and build breakdown:

• windows11-32-2009-qr: 4
  • debug: 4
• windows11-64-2009-qr: 4
  • debug: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-05-13&endday=2024-05-19&tree=all

Comment 74

[Intermittent Failures Robot]

14 failures in 3995 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4
• autoland: 6
• mozilla-beta: 3
• try: 1

Platform and build breakdown:

• windows11-32-2009-mingwclang-qr: 1
  • debug: 1
• windows11-32-2009-qr: 7
  • debug: 7
• windows11-64-2009-mingwclang-qr: 2
  • debug: 2
• windows10-64-2009-qr: 1
  • debug: 1
• windows11-64-2009-qr: 3
  • debug: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-05-20&endday=2024-05-26&tree=all

Comment 75

[Intermittent Failures Robot]

4 failures in 4202 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-central: 1
• mozilla-beta: 1

Platform and build breakdown:

• windows11-64-2009-qr: 3
  • debug: 3
• windows11-32-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-05-27&endday=2024-06-02&tree=all

Comment 76

[Intermittent Failures Robot]

5 failures in 3649 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4
• mozilla-central: 1

Platform and build breakdown:

• windows11-64-2009-qr: 3
  • debug: 3
• windows11-64-2009-ccov-qr: 1
  • opt: 1
• windows11-32-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-06-03&endday=2024-06-09&tree=all

Comment 77

[Intermittent Failures Robot]

6 failures in 4625 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 2
• autoland: 2
• mozilla-central: 2

Platform and build breakdown:

• windows11-32-2009-qr: 1
  • debug: 1
• windows11-64-2009-qr: 3
  • debug: 3
• windows11-64-2009-ccov-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-06-10&endday=2024-06-16&tree=all

Comment 78

[Intermittent Failures Robot]

11 failures in 4577 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 6
• mozilla-central: 2
• mozilla-esr115: 1
• autoland: 2

Platform and build breakdown:

• windows11-32-2009-qr: 4
  • debug: 4
• windows11-64-2009-qr: 6
  • debug: 6
• windows11-64-2009-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-06-17&endday=2024-06-23&tree=all

Comment 79

[Intermittent Failures Robot]

5 failures in 4116 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-central: 3

Platform and build breakdown:

• windows11-32-2009-qr: 2
  • debug: 2
• windows11-64-2009-qr: 2
  • debug: 2
• windows11-64-2009-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-06-24&endday=2024-06-30&tree=all

Comment 80

[Intermittent Failures Robot]

1 failures in 3199 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• windows11-64-2009-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-07-01&endday=2024-07-07&tree=all

Comment 81

[Intermittent Failures Robot]

3 failures in 3780 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• mozilla-beta: 1

Platform and build breakdown:

• windows11-32-2009-qr: 1
  • debug: 1
• windows11-64-2009-qr: 1
  • debug: 1
• windows11-64-2009-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-07-08&endday=2024-07-14&tree=all

Comment 82

[Intermittent Failures Robot]

6 failures in 3799 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• mozilla-beta: 1
• mozilla-esr128: 2
• autoland: 2

Platform and build breakdown:

• windows11-64-2009-qr: 4
  • debug: 4
• windows11-32-2009-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-07-15&endday=2024-07-21&tree=all

Comment 83

[Intermittent Failures Robot]

3 failures in 4124 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 1
• mozilla-beta: 1

Platform and build breakdown:

• windows11-64-2009-qr: 2
  • debug: 2
• windows11-64-2009-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-07-22&endday=2024-07-28&tree=all

Comment 84

[Intermittent Failures Robot]

3 failures in 3839 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• autoland: 1

Platform and build breakdown:

• windows11-32-2009-qr: 1
  • debug: 1
• windows11-64-2009-ccov-qr: 1
  • opt: 1
• windows11-64-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-07-29&endday=2024-08-04&tree=all

Comment 85

[Intermittent Failures Robot]

5 failures in 3459 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• mozilla-central: 2

Platform and build breakdown:

• windows11-64-2009-qr: 5
  • debug: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-08-05&endday=2024-08-11&tree=all

Comment 86

[Intermittent Failures Robot]

5 failures in 2154 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• autoland: 2
• mozilla-beta: 2

Platform and build breakdown:

• windows11-64-2009-qr: 4
  • debug: 4
• windows11-32-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-08-12&endday=2024-08-18&tree=all

Comment 87

[Intermittent Failures Robot]

9 failures in 3218 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 3
• try: 3
• autoland: 2
• mozilla-central: 1

Platform and build breakdown:

• windows11-64-2009-qr: 7
  • debug: 7
• windows11-64-2009-ccov-qr: 1
  • opt: 1
• windows11-32-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-08-19&endday=2024-08-25&tree=all

Comment 88

[Intermittent Failures Robot]

2 failures in 3668 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1
• mozilla-central: 1

Platform and build breakdown:

• windows11-64-2009-qr: 1
  • debug: 1
• windows11-32-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-09-02&endday=2024-09-08&tree=all

Comment 89

[Intermittent Failures Robot]

2 failures in 4448 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr128: 1
• mozilla-beta: 1

Platform and build breakdown:

• windows11-64-2009-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-09-09&endday=2024-09-15&tree=all

Comment 90

[Intermittent Failures Robot]

8 failures in 4860 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 2
• mozilla-beta: 1
• autoland: 1
• mozilla-esr128: 4

Platform and build breakdown:

• windows11-64-2009-qr: 4
  • debug: 4
• windows10-64-2009-qr: 3
  • debug: 3
• windows11-32-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-09-16&endday=2024-09-22&tree=all

Comment 91

[Intermittent Failures Robot]

2 failures in 4889 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• try: 1

Platform and build breakdown:

• windows11-64-2009-qr: 1
  • debug: 1
• windows10-64-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-09-23&endday=2024-09-29&tree=all

Comment 92

[Intermittent Failures Robot]

9 failures in 3574 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr128: 6
• mozilla-central: 1
• autoland: 1
• mozilla-beta: 1

Platform and build breakdown:

• windows10-64-2009-qr: 7
  • debug: 7
• windows11-64-2009-ccov-qr: 1
  • opt: 1
• windows11-64-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-09-30&endday=2024-10-06&tree=all

Comment 93

[Intermittent Failures Robot]

22 failures in 4679 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 10
• try: 3
• mozilla-esr128: 6
• mozilla-beta: 1
• mozilla-central: 2

Platform and build breakdown:

• windows10-64-2009-qr: 16
  • debug: 16
• windows11-64-2009-qr: 3
  • debug: 3
• windows11-32-2009-qr: 2
  • debug: 2
• windows11-64-2009-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-10-07&endday=2024-10-13&tree=all

Comment 94

[Intermittent Failures Robot]

49 failures in 4579 pushes (0.011 failures/push) were associated with this bug in the last 7 days.

This is the #48 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 8
• autoland: 14
• mozilla-beta: 10
• try: 4
• mozilla-esr128: 11
• mozilla-esr115: 2

Platform and build breakdown:

• windows10-64-2009-qr: 42
  • debug: 42
• windows11-64-2009-qr: 4
  • debug: 4
• windows11-64-2009-ccov-qr: 2
  • opt: 2
• windows11-32-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-10-14&endday=2024-10-20&tree=all

Comment 95

[Intermittent Failures Robot]

26 failures in 4456 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 4
• mozilla-central: 4
• autoland: 14
• mozilla-beta: 3
• mozilla-esr128: 1

Platform and build breakdown:

• windows10-64-2009-qr: 19
  • debug: 19
• windows11-32-2009-qr: 1
  • debug: 1
• windows11-64-2009-qr: 5
  • debug: 5
• linux1804-64-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-10-21&endday=2024-10-27&tree=all

Comment 96

[Intermittent Failures Robot]

26 failures in 4456 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 4
• mozilla-central: 4
• autoland: 14
• mozilla-beta: 3
• mozilla-esr128: 1

Platform and build breakdown:

• windows10-64-2009-qr: 19
  • debug: 19
• windows11-32-2009-qr: 1
  • debug: 1
• windows11-64-2009-qr: 5
  • debug: 5
• linux1804-64-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-10-21&endday=2024-10-27&tree=all

Comment 97

[Intermittent Failures Robot]

48 failures in 4361 pushes (0.011 failures/push) were associated with this bug in the last 7 days.

This is the #47 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 12
• autoland: 17
• try: 5
• mozilla-esr128: 4
• mozilla-beta: 10

Platform and build breakdown:

• windows10-64-2009-qr: 45
  • debug: 45
• windows11-64-2009-ccov-qr: 1
  • opt: 1
• windows11-64-2009-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-10-28&endday=2024-11-03&tree=all

Comment 98

[Intermittent Failures Robot]

54 failures in 4515 pushes (0.012 failures/push) were associated with this bug in the last 7 days.

This is the #44 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 18
• mozilla-beta: 22
• mozilla-esr128: 1
• try: 3
• mozilla-central: 10

Platform and build breakdown:

• windows10-64-2009-qr: 51
  • debug: 51
• windows11-64-2009-ccov-qr: 2
  • opt: 2
• windows11-64-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-11-04&endday=2024-11-10&tree=all

Comment 99

[Intermittent Failures Robot]

15 failures in 1008 pushes (0.015 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-beta: 3
• mozilla-central: 2
• try: 7
• mozilla-esr128: 2
• autoland: 1

Platform and build breakdown:

• windows10-64-2009-qr: 14
  • debug: 14
• linux2204-64-wayland-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-11-14&endday=2024-11-14&tree=all

Comment 100

[Intermittent Failures Robot]

47 failures in 4227 pushes (0.011 failures/push) were associated with this bug in the last 7 days.

This is the #37 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-beta: 9
• mozilla-central: 10
• try: 11
• autoland: 15
• mozilla-esr128: 2

Platform and build breakdown:

• windows10-64-2009-qr: 45
  • debug: 45
• linux2204-64-wayland-shippable: 1
  • opt: 1
• windows11-64-2009-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-11-11&endday=2024-11-17&tree=all

Comment 101

[Intermittent Failures Robot]

47 failures in 4227 pushes (0.011 failures/push) were associated with this bug in the last 7 days.

This is the #37 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-beta: 9
• mozilla-central: 10
• try: 11
• autoland: 15
• mozilla-esr128: 2

Platform and build breakdown:

• windows10-64-2009-qr: 45
  • debug: 45
• linux2204-64-wayland-shippable: 1
  • opt: 1
• windows11-64-2009-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-11-11&endday=2024-11-17&tree=all

Comment 102

[Intermittent Failures Robot]

35 failures in 3002 pushes (0.012 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 6
• autoland: 21
• try: 6
• mozilla-beta: 1
• mozilla-esr115: 1

Platform and build breakdown:

• windows10-64-2009-qr: 32
  • debug: 32
• windows11-64-2009-qr: 3
  • debug: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-11-18&endday=2024-11-24&tree=all

Comment 103

[Intermittent Failures Robot]

42 failures in 3953 pushes (0.011 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-beta: 15
• autoland: 13
• try: 6
• mozilla-central: 6
• mozilla-esr128: 2

Platform and build breakdown:

• windows10-64-2009-qr: 38
  • debug: 38
• windows11-64-2009-qr: 1
  • debug: 1
• windows11-32-2009-qr: 2
  • debug: 2
• windows11-64-2009-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-11-25&endday=2024-12-01&tree=all

Comment 104

[Intermittent Failures Robot]

38 failures in 4722 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 4
• mozilla-esr128: 5
• try: 5
• mozilla-beta: 14
• autoland: 9
• mozilla-esr115: 1

Platform and build breakdown:

• windows10-64-2009-qr: 37
  • debug: 37
• windows11-64-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-12-02&endday=2024-12-08&tree=all

Comment 105

[Intermittent Failures Robot]

16 failures in 829 pushes (0.019 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-beta: 5
• mozilla-esr115: 2
• mozilla-esr128: 3
• try: 2
• mozilla-central: 2
• autoland: 2

Platform and build breakdown:

• windows10-64-2009-qr: 16
  • debug: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-12-10&endday=2024-12-10&tree=all

Comment 106

[Intermittent Failures Robot]

Summary

47 failures in 4615 pushes (0.01 failures/push) were associated with this bug in the last 7 days.

This failure happened more than 30 times this week! Resolving this bug is a high priority.

Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled.

Repository breakdown:

• autoland: 14
• try: 5
• mozilla-central: 8
• mozilla-beta: 8
• mozilla-esr115: 2
• mozilla-esr128: 10

Platform and build breakdown:

• windows10-64-2009-qr: 46
  • debug: 46
• windows11-64-2009-qr: 1
  • debug: 1

Table

	1proc	1proc-s	1proc-swr	a11y-checks	a11y-checks-s	a11y-checks-swr	a11y-checks-swr-s	condprof	condprof-s	fis	fis-hv	fis-hv-s	fis-s	headless	headless-s	http2	http3	mda-gpu	mda-gpu-s	msix	msix-s	no_variant	nofis	nofis-aab	nofis-s	nofis-ship	nofis-ship-s	nofis-spi	nofis-swr	nofis-swr-s	nogpu	nogpu-s	s	spi	spi-nw	spi-nw-1proc	spi-nw-nofis	spi-nw-nofis-s	spi-nw-s	spi-s	swr	swr-s	wmfme	wr-dc1-p	xorig	xorig-s
windows10-64-2009-qr/debug																						46
windows11-64-2009-qr/debug																																			1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-12-09&endday=2024-12-15&tree=all

Comment 107

[Intermittent Failures Robot]

Summary

23 failures in 3774 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 8
• try: 3
• mozilla-esr115: 1
• mozilla-beta: 5
• mozilla-central: 5
• mozilla-esr128: 1

Platform and build breakdown:

• windows10-64-2009-qr: 20
  • debug: 20
• windows11-64-2009-ccov-qr: 1
  • opt: 1
• windows11-64-2009-qr: 2
  • debug: 2

Table

	1proc	1proc-s	1proc-swr	a11y-checks	a11y-checks-s	a11y-checks-swr	a11y-checks-swr-s	condprof-s	fis	fis-hv	fis-hv-s	fis-s	headless	headless-s	http2	http3	mda-gpu	mda-gpu-s	msix	msix-s	no_variant	nofis	nofis-aab	nofis-s	nofis-ship	nofis-ship-s	nofis-spi	nofis-swr	nofis-swr-s	nogpu	nogpu-s	s	spi	spi-nw	spi-nw-1proc	spi-nw-nofis	spi-nw-nofis-s	spi-nw-s	spi-s	swr	swr-s	vt	vt-s	wmfme	wr-dc1-p	wr-dc2-o	wr-dc3-c	wr-dc3-c-s	xorig	xorig-s
windows10-64-2009-qr/debug																					20
windows11-64-2009-ccov-qr/opt																					1
windows11-64-2009-qr/debug																			2

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-12-16&endday=2024-12-22&tree=all

Comment 108

[Intermittent Failures Robot]

Summary

4 failures in 900 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1
• mozilla-central: 3

Platform and build breakdown:

• windows11-32-2009-qr: 2
  • debug: 2
• windows11-64-2009-ccov-qr: 2
  • opt: 2

Table

	1proc	1proc-s	1proc-swr	a11y-checks	a11y-checks-s	a11y-checks-swr	a11y-checks-swr-s	cf	condprof-s	fis	fis-hv	fis-hv-s	fis-s	headless	headless-s	http2	http3	mda-gpu	mda-gpu-s	msix	msix-s	no_variant	nofis	nofis-aab	nofis-s	nofis-ship	nofis-ship-s	nofis-spi	nofis-swr	nogpu	nogpu-s	s	spi	spi-nw	spi-nw-1proc	spi-nw-cf	spi-nw-nofis	spi-nw-s	spi-s	swr	swr-s	vt	vt-s	wmfme	wmfme-s	wr-dc0	wr-dc2-o	wr-dc3-c	xorig	xorig-s
windows11-32-2009-qr/debug																						2
windows11-64-2009-ccov-qr/opt																						2

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-12-23&endday=2024-12-29&tree=all

Comment 109

[Intermittent Failures Robot]

Summary

2 failures in 1319 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• try: 1

Platform and build breakdown:

• windows11-64-2009-qr: 2
  • debug: 2

Table

	msix
windows11-64-2009-qr/debug	2

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2024-12-30&endday=2025-01-05&tree=all

Comment 110

[Intermittent Failures Robot]

Summary

7 failures in 3682 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1
• mozilla-esr128: 1
• mozilla-central: 5

Platform and build breakdown:

• windows10-64-2009-qr: 1
  • debug: 1
• windows11-32-2009-qr: 1
  • debug: 1
• windows11-64-2009-ccov-qr: 5
  • opt: 5

Table

	no_variant
windows10-64-2009-qr/debug	1
windows11-32-2009-qr/debug	1
windows11-64-2009-ccov-qr/opt	5

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2025-01-06&endday=2025-01-12&tree=all

Comment 111

[Intermittent Failures Robot]

Summary

7 failures in 4508 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4
• mozilla-beta: 2
• autoland: 1

Platform and build breakdown:

• windows11-64-2009-ccov-qr: 3
  • opt: 3
• windows11-64-2009-qr: 4
  • debug: 4

Table

	msix	no_variant
windows11-64-2009-ccov-qr/-/opt		3
windows11-64-2009-qr/-/debug	1	3

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2025-01-13&endday=2025-01-19&tree=all

Comment 112

[Intermittent Failures Robot]

Summary

3 failures in 4035 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• mozilla-esr128: 1

Platform and build breakdown:

• windows11-64-2009-ccov-qr: 1
  • opt: 1
• windows11-64-2009-qr: 1
  • debug: 1
• windows11-64-24h2-ccov: 1
  • opt: 1

Table

	no_variant
windows11-64-2009-ccov-qr--/opt	1
windows11-64-2009-qr--/debug	1
windows11-64-24h2-ccov--/opt	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1831236&startday=2025-01-20&endday=2025-01-26&tree=all