1265637 - Intermittent Assertion failure: isEmpty() (failing this assertion means this LinkedList's creator is buggy: it should have removed all this list's elements before the list's destruction), at LinkedList.h:332

Status: RESOLVED FIXED

(Core :: JavaScript Engine, defect)

Description

[Carsten Book [:Tomcat]]

In bughunter we see more and more :
Assertion failure: isEmpty() (failing this assertion means this LinkedList's creator is buggy: it should have removed all this list's elements before the list's destruction), at c:\builds\moz2_slave\m-cen-w32-d-000000000000000000\build\src\obj-firefox\dist\include\mozilla/LinkedList.h:332

the problem on this is that this is making crash detection more complicated and we might miss regressions because of this

Waldo, froydnj , seems you were working on this, can you take a look ? thanks!

Comment 1

[Nathan Froyd [:froydnj]]

(In reply to Carsten Book [:Tomcat] from comment #0)
> In bughunter we see more and more :
> Assertion failure: isEmpty() (failing this assertion means this LinkedList's
> creator is buggy: it should have removed all this list's elements before the
> list's destruction), at
> c:\builds\moz2_slave\m-cen-w32-d-000000000000000000\build\src\obj-
> firefox\dist\include\mozilla/LinkedList.h:332
> 
> the problem on this is that this is making crash detection more complicated
> and we might miss regressions because of this
> 
> Waldo, froydnj , seems you were working on this, can you take a look ?
> thanks!

Do we not have stacks to tell us what LinkedList is not getting cleared?  That's the important piece of information here, and should be reflected in the call stack.

Comment 2

[Carsten Book [:Tomcat]]

bc, do you have a stack? the log i saw didn't generated one, but you might have more insight to get a stack

Comment 3

[Bob Clary [:bc] (inactive)]

The stacks are available from the crash reports. There are a variety of different stacks with this assertion so there may be more than one bug involved here.

Comment 4

[Jeff Walden [:Waldo]]

This assertion indicates a problem with the caller, not with mfbt code.  mfbt has nothing to do with anything that hits this assertion -- it's all on the callers.

When filing bugs for this stuff, you should set needinfo corresponding to the location of the stack frame that *triggered* this assertion.  Not against mfbt, or against froydnj or me.  :-)

Comment 5

[Bob Clary [:bc] (inactive)]

Attachment: stack

steps to reproduce are to start the browser from the command line with -silent, e.g. firefox-debug/dist/bin/firefox -silent -profile /tmp/foobar

Comment 6

[Kartikaya Gupta (email:[e-mail])]

I'm seeing this locally on OS X a lot, I get this assertion failure during shutdown.

Stack looks similar to the one Bob posted in comment 5, it looks like a JS compartment is holding the linked list?


Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   XUL                           	0x000000010e34dd42 mozilla::LinkedList<js::UnboxedLayout>::~LinkedList() + 178 (LinkedList.h:329)
1   XUL                           	0x000000010e3200e0 JSCompartment::~JSCompartment() + 944 (ArrayBufferObject.h:497)
2   XUL                           	0x000000010e368587 JS::Zone::sweepCompartments(js::FreeOp*, bool, bool) + 583 (Utility.h:249)
3   XUL                           	0x000000010e3688bd js::gc::GCRuntime::sweepZones(js::FreeOp*, bool) + 381 (jsgc.cpp:3898)
4   XUL                           	0x000000010e375987 js::gc::GCRuntime::incrementalCollectSlice(js::SliceBudget&, JS::gcreason::Reason) + 1623 (jsgc.cpp:6258)
5   XUL                           	0x000000010e3765bc js::gc::GCRuntime::gcCycle(bool, js::SliceBudget&, JS::gcreason::Reason) + 460 (jsgc.cpp:6448)
6   XUL                           	0x000000010e376f35 js::gc::GCRuntime::collect(bool, js::SliceBudget, JS::gcreason::Reason) + 741 (jsgc.cpp:6553)
7   XUL                           	0x000000010e366456 js::gc::GCRuntime::gc(JSGCInvocationKind, JS::gcreason::Reason) + 86 (jsgc.cpp:6614)
8   XUL                           	0x000000010e5da23c JSRuntime::~JSRuntime() + 844 (Runtime.cpp:426)
9   XUL                           	0x000000010e2a2746 JS_DestroyRuntime(JSRuntime*) + 22 (Utility.h:249)
10  XUL                           	0x0000000109c7bb52 mozilla::CycleCollectedJSRuntime::~CycleCollectedJSRuntime() + 226 (CycleCollectedJSRuntime.cpp:475)
11  XUL                           	0x000000010a69e8be XPCJSRuntime::~XPCJSRuntime() + 14 (mozalloc.h:210)
12  XUL                           	0x000000010a6e8b0a nsXPConnect::~nsXPConnect() + 138 (nsXPConnect.cpp:107)
13  XUL                           	0x000000010a6e8b4e nsXPConnect::~nsXPConnect() + 14 (mozalloc.h:210)
14  XUL                           	0x000000010a6e89b1 nsXPConnect::Release() + 97 (nsXPConnect.cpp:42)
15  XUL                           	0x000000010a6acc49 xpcModuleDtor() + 9 (XPCJSID.cpp:267)
16  XUL                           	0x0000000109d0798c nsTArray_Impl<nsAutoPtr<nsComponentManagerImpl::KnownModule>, nsTArrayInfallibleAllocator>::RemoveElementsAt(unsigned long, unsigned long) + 124 (nsCOMPtr.h:403)
17  XUL                           	0x0000000109d02cc0 nsComponentManagerImpl::Shutdown() + 192 (nsComponentManager.cpp:912)
18  XUL                           	0x0000000109d40b75 mozilla::ShutdownXPCOM(nsIServiceManager*) + 1477 (XPCOMInit.cpp:991)
19  XUL                           	0x000000010d55e90a ScopedXPCOMStartup::~ScopedXPCOMStartup() + 186 (nsAppRunner.cpp:1473)
20  XUL                           	0x000000010d5661b7 XREMain::XRE_main(int, char**, nsXREAppData const*) + 1175 (mozalloc.h:210)
21  XUL                           	0x000000010d56645e XRE_main + 238 (nsAppRunner.cpp:4559)
22  org.mozilla.nightlydebug      	0x0000000109219114 main + 2212 (nsBrowserApp.cpp:220)
23  org.mozilla.nightlydebug      	0x0000000109218534 start + 52

Comment 7

[Jonathan Watt [:jwatt]]

An easy fix for this would be to add:

  unboxedLayouts.clear();
  wasmModuleWeakList.clear();

to the JSCompartment dtor. However, it's unclear to me whether that would just be hiding a bug elsewhere that should be fixed.

Comment 8

[Terrence Cole [:terrence]]

(In reply to Jonathan Watt [:jwatt] from comment #7)
> An easy fix for this would be to add:
> 
>   unboxedLayouts.clear();
>   wasmModuleWeakList.clear();
> 
> to the JSCompartment dtor. However, it's unclear to me whether that would
> just be hiding a bug elsewhere that should be fixed.

That would be hiding a bug elsewhere, and would probably just move the assertion later as the heap would stilll not be empty.

I just checked in bug 1268992, which should print the leaking edges before we get to this crash. That should allow us to use the CC leak logs to quickly track down the bug in gecko or chrome that is holding things live past shutdown.

Comment 9

[Kartikaya Gupta (email:[e-mail])]

I got the assertion on a try job, not sure if that's useful to you: https://treeherder.mozilla.org/logviewer.html#?job_id=21209253&repo=try#L10466

Comment 12

[Milan Sreckovic [:milan] (needinfo for best results)]

I just got it on Windows on shutdown as well; changing the platform to all.

Comment 14

[Intermittent Failures Robot]

1 failures in 632 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* osx-10-10: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2018-05-21&endday=2018-05-27&tree=trunk

Comment 15

[Intermittent Failures Robot]

1 failures in 2740 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2018-10-08&endday=2018-10-14&tree=all

Comment 16

[Intermittent Failures Robot]

1 failures in 2387 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 1

Platform breakdown:
* windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2018-11-19&endday=2018-11-25&tree=all

Comment 18

[Intermittent Failures Robot]

2 failures in 2808 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 2

Platform breakdown:
* linux64-qr: 1
* windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2018-11-26&endday=2018-12-02&tree=all

Comment 19

[Intermittent Failures Robot]

2 failures in 2485 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 2

Platform breakdown:
* windows10-64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2018-12-10&endday=2018-12-16&tree=all

Comment 20

[Intermittent Failures Robot]

1 failures in 2627 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-inbound: 1

Platform breakdown:

• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-01-07&endday=2019-01-13&tree=all

Comment 21

[Intermittent Failures Robot]

1 failures in 2628 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• windows10-64-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-01-28&endday=2019-02-03&tree=all

Comment 22

[Intermittent Failures Robot]

1 failures in 2658 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform breakdown:

• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-02-04&endday=2019-02-10&tree=all

Comment 23

[Intermittent Failures Robot]

9 failures in 2832 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• mozilla-inbound: 2
• mozilla-beta: 1
• mozilla-central: 3

Platform breakdown:

• windows10-64: 6
• windows10-64-ccov: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-02-11&endday=2019-02-17&tree=all

Comment 24

[Intermittent Failures Robot]

1 failures in 2602 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-02-18&endday=2019-02-24&tree=all

Comment 25

[Intermittent Failures Robot]

17 failures in 608 pushes (0.028 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-inbound: 5
• mozilla-central: 1
• autoland: 10
• try: 1

Platform breakdown:

• linux64: 5
• windows10-64-ccov: 1
• osx-10-10: 10
• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-03-01&endday=2019-03-01&tree=all

Comment 26

[Andreea Pavel [:apavel]]

There are 20 total failures in the last 7 days, on linux64, osx and windows.

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=231359838&repo=autoland&lineNumber=6655

22:38:42 INFO - TEST-START | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_contentScripts_register.js
22:38:43 INFO - rmtree() failed for "('c:\users\task_1551478786\appdata\local\temp\xpc-profile-ksp3rq',)". Reason: The process cannot access the file because it is being used by another process (13). Retrying...
22:38:44 INFO - mozcrash Saved minidump as Z:\task_1551478786\build\blobber_upload_dir\c5e91ecd-3e9c-4eb7-8468-ec0346799c71.dmp
22:38:44 INFO - mozcrash Saved app info as Z:\task_1551478786\build\blobber_upload_dir\c5e91ecd-3e9c-4eb7-8468-ec0346799c71.extra
22:38:44 WARNING - PROCESS-CRASH | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_proxy_socks.js | application crashed [@ static void ??__FsList@?1??ThreadList@nsThread@@KAAEAV?$LinkedList@VnsThread@@@mozilla@@XZ@YAXXZ()]
22:38:44 INFO - Crash dump filename: c:\users\task_1551478786\appdata\local\temp\xpc-other-d6j_ek\c5e91ecd-3e9c-4eb7-8468-ec0346799c71.dmp
22:38:44 INFO - Operating system: Windows NT
22:38:44 INFO - 10.0.17134
22:38:44 INFO - CPU: amd64
22:38:44 INFO - family 6 model 85 stepping 4
22:38:44 INFO - 8 CPUs
22:38:44 INFO - GPU: UNKNOWN
22:38:44 INFO - Crash reason: EXCEPTION_BREAKPOINT
22:38:44 INFO - Crash address: 0x7ffd8f5828bc
22:38:44 INFO - Assertion: Unknown assertion type 0x00000000
22:38:44 INFO - Process uptime: 3 seconds
22:38:44 INFO - Thread 0 (crashed)
22:38:44 INFO - 0 xul.dll!static void ??__FsList@?1??ThreadList@nsThread@@KAAEAV?$LinkedList@VnsThread@@@mozilla@@XZ@YAXXZ() [Unified_cpp_xpcom_threads1.cpp:1712a35cd239049c186f7da80cdc759885ae81ae : 367 + 0x9c]
22:38:44 INFO - rax = 0x00007ffd967ef727 rdx = 0x00007ffdd5eda640
22:38:44 INFO - rcx = 0x00007ffdcd1afae0 rbx = 0x000002455feb21f8
22:38:44 INFO - rsi = 0x000002455feb1c40 rdi = 0x0000a3dfda9fe39f
22:38:44 INFO - rbp = 0x000002455feb22a0 rsp = 0x0000008fb9dff7d0
22:38:44 INFO - r8 = 0x0000008fb9df9928 r9 = 0x0000008fb9dfaf40
22:38:44 INFO - r10 = 0x0000000000000000 r11 = 0x0000008fb9dfae50
22:38:44 INFO - r12 = 0x0000000000000000 r13 = 0x0000000000000040
22:38:44 INFO - r14 = 0x0000008fb9dff8d8 r15 = 0x000002455feb1c40
22:38:44 INFO - rip = 0x00007ffd8f5828bc
22:38:44 INFO - Found by: given as instruction pointer in context
22:38:44 INFO - 1 ucrtbase.dll!<lambda_f03950bc5685219e0bcd2087efbe011e>::operator() + 0xc3
22:38:44 INFO - rbx = 0x000002455feb21f8 rbp = 0x000002455feb22a0
22:38:44 INFO - rsp = 0x0000008fb9dff800 r12 = 0x0000000000000000
22:38:44 INFO - r13 = 0x0000000000000040 r14 = 0x0000008fb9dff8d8
22:38:44 INFO - r15 = 0x000002455feb1c40 rip = 0x00007ffdd5e01243
22:38:44 INFO - Found by: call frame info
22:38:44 INFO - 2 ucrtbase.dll!__crt_seh_guarded_call<int>::operator()<<lambda_7777bce6b2f8c936911f934f8298dc43>,<lambda_f03950bc5685219e0bcd2087efbe011e> & ptr64,<lambda_3883c3dff614d5e0c5f61bb1ac94921c> > + 0x3b
22:38:44 INFO - rbx = 0x000002455feb21f8 rbp = 0x000002455feb22a0
22:38:44 INFO - rsp = 0x0000008fb9dff860 r12 = 0x0000000000000000
22:38:44 INFO - r13 = 0x0000000000000040 r14 = 0x0000008fb9dff8d8
22:38:44 INFO - r15 = 0x000002455feb1c40 rip = 0x00007ffdd5e01017
22:38:44 INFO - Found by: call frame info
22:38:44 INFO - 3 ucrtbase.dll!execute_onexit_table + 0x34
22:38:44 INFO - rbx = 0x000002455feb21f8 rbp = 0x000002455feb22a0
22:38:44 INFO - rsp = 0x0000008fb9dff890 r12 = 0x0000000000000000
22:38:44 INFO - r13 = 0x0000000000000040 r14 = 0x0000008fb9dff8d8
22:38:44 INFO - r15 = 0x000002455feb1c40 rip = 0x00007ffdd5e00fd4
22:38:44 INFO - Found by: call frame info
22:38:44 INFO - 4 xul.dll!static int dllmain_crt_process_detach(const bool) [dll_dllmain.cpp : 106 + 0x5]
22:38:44 INFO - rbx = 0x000002455feb21f8 rbp = 0x000002455feb22a0
22:38:44 INFO - rsp = 0x0000008fb9dff8c0 r12 = 0x0000000000000000
22:38:44 INFO - r13 = 0x0000000000000040 r14 = 0x0000008fb9dff8d8
22:38:44 INFO - r15 = 0x000002455feb1c40 rip = 0x00007ffd967bd4be
22:38:44 INFO - Found by: call frame info
22:38:44 INFO - 5 xul.dll!static int dllmain_dispatch(struct HINSTANCE *, const unsigned long, void *) [dll_dllmain.cpp : 212 + 0xd]
22:38:44 INFO - rbx = 0x000002455feb21f8 rbp = 0x000002455feb22a0
22:38:44 INFO - rsp = 0x0000008fb9dff8f0 r12 = 0x0000000000000000
22:38:44 INFO - r13 = 0x0000000000000040 r14 = 0x0000008fb9dff8d8
22:38:44 INFO - r15 = 0x000002455feb1c40 rip = 0x00007ffd967bd65c
22:38:44 INFO - Found by: call frame info
22:38:44 INFO - 6 ntdll.dll!remainderf + 0x233
22:38:44 INFO - rbx = 0x000002455feb21f8 rbp = 0x000002455feb22a0
22:38:44 INFO - rsp = 0x0000008fb9dff950 r12 = 0x0000000000000000
22:38:44 INFO - r13 = 0x0000000000000040 r14 = 0x0000008fb9dff8d8
22:38:44 INFO - r15 = 0x000002455feb1c40 rip = 0x00007ffdd8f34053
22:38:44 INFO - Found by: call frame info
22:38:44 INFO - 7 0x2455fea3560
22:38:44 INFO - rbx = 0x000002455feb21f8 rbp = 0x000002455feb22a0
22:38:44 INFO - rsp = 0x0000008fb9dff9b0 r12 = 0x0000000000000000
22:38:44 INFO - r13 = 0x0000000000000040 r14 = 0x0000008fb9dff8d8
22:38:44 INFO - r15 = 0x000002455feb1c40 rip = 0x000002455fea3560
22:38:44 INFO - Found by: call frame info
22:38:44 INFO - 8 xul.dll + 0x738d6b0
22:38:44 INFO - rbp = 0x000002455feb22a0 rsp = 0x0000008fb9dff9b8
22:38:44 INFO - rip = 0x00007ffd967bd6b0
22:38:44 INFO - Found by: stack scanning
22:38:44 INFO - 9 ntdll.dll!_uncaught_exceptions + 0x15
22:38:44 INFO - rbp = 0x000002455feb22a0 rsp = 0x0000008fb9dff9c0
22:38:44 INFO - rip = 0x00007ffdd8f40a05
22:38:44 INFO - Found by: stack scanning
22:38:44 INFO - Loaded modules:
22:38:44 INFO - 0x7ff7beae0000 - 0x7ff7beb61fff plugin-container.exe 67.0.0.6999 (main)
22:38:44 INFO - 0x7ffd8f430000 - 0x7ffd98eb0fff xul.dll 67.0.0.6999
22:38:44 INFO - 0x7ffdaacf0000 - 0x7ffdaad09fff pnrpnsp.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdad7b0000 - 0x7ffdad8dbfff AudioSes.dll 10.0.17134.137
22:38:44 INFO - 0x7ffdb0d10000 - 0x7ffdb0d1dfff winrnr.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdb25c0000 - 0x7ffdb2667fff mscms.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdbc460000 - 0x7ffdbc475fff NapiNSP.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdbf9a0000 - 0x7ffdbfc8cfff nss3.dll 67.0.0.6999
22:38:44 INFO - 0x7ffdc1a70000 - 0x7ffdc1a88fff usp10.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdc2cd0000 - 0x7ffdc2cdffff ColorAdapterClient.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdc5ae0000 - 0x7ffdc5b7afff msvcp140.dll 14.15.26706.0
22:38:44 INFO - 0x7ffdc75c0000 - 0x7ffdc78dbfff DWrite.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdc7ca0000 - 0x7ffdc7e68fff dbghelp.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdcb2d0000 - 0x7ffdcb2d8fff wsock32.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdcb600000 - 0x7ffdcb610fff credui.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdcbf20000 - 0x7ffdcbf30fff lgpllibs.dll 67.0.0.6999
22:38:44 INFO - 0x7ffdcc8d0000 - 0x7ffdcc948fff InputHost.dll ???
22:38:44 INFO - 0x7ffdcc950000 - 0x7ffdcc9e6fff TextInputFramework.dll 10.0.17134.191
22:38:44 INFO - 0x7ffdccd20000 - 0x7ffdcce33fff Windows.UI.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdcd140000 - 0x7ffdcd1b7fff mozglue.dll 67.0.0.6999
22:38:44 INFO - 0x7ffdcdab0000 - 0x7ffdcdac5fff VCRUNTIME140.dll 14.15.26706.0
22:38:44 INFO - 0x7ffdce130000 - 0x7ffdce1a5fff MMDevAPI.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdcf350000 - 0x7ffdcf359fff version.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdcfae0000 - 0x7ffdcfaf9fff dhcpcsvc.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdd0100000 - 0x7ffdd041dfff CoreUIComponents.dll 10.0.17134.112
22:38:44 INFO - 0x7ffdd06c0000 - 0x7ffdd06c9fff avrt.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdd0790000 - 0x7ffdd08dcfff WinTypes.dll 10.0.17134.112
22:38:44 INFO - 0x7ffdd18a0000 - 0x7ffdd1a53fff propsys.dll 7.0.17134.112
22:38:44 INFO - 0x7ffdd1c70000 - 0x7ffdd1c82fff wtsapi32.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdd1cb0000 - 0x7ffdd1cc8fff nlaapi.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdd24b0000 - 0x7ffdd27bafff d3d11.dll 10.0.17134.112
22:38:44 INFO - 0x7ffdd3270000 - 0x7ffdd3349fff CoreMessaging.dll 10.0.17134.285
22:38:44 INFO - 0x7ffdd3350000 - 0x7ffdd3379fff WINMMBASE.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdd3470000 - 0x7ffdd3492fff winmm.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdd3750000 - 0x7ffdd37e7fff uxtheme.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdd39a0000 - 0x7ffdd39c8fff dwmapi.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdd3fd0000 - 0x7ffdd408afff dxgi.dll 10.0.17134.112
22:38:44 INFO - 0x7ffdd4300000 - 0x7ffdd4330fff ntmarta.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdd47b0000 - 0x7ffdd47e7fff IPHLPAPI.DLL 10.0.17134.1
22:38:44 INFO - 0x7ffdd47f0000 - 0x7ffdd48adfff dnsapi.dll 10.0.17134.165
22:38:44 INFO - 0x7ffdd4a30000 - 0x7ffdd4a95fff mswsock.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdd4c00000 - 0x7ffdd4c0afff CRYPTBASE.DLL 10.0.17134.1
22:38:44 INFO - 0x7ffdd4d10000 - 0x7ffdd4d34fff bcrypt.dll 10.0.17134.112
22:38:44 INFO - 0x7ffdd4fc0000 - 0x7ffdd4fe6fff devobj.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdd5110000 - 0x7ffdd5137fff userenv.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdd5210000 - 0x7ffdd522efff profapi.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdd5230000 - 0x7ffdd5240fff kernel.appcore.dll 10.0.17134.112
22:38:44 INFO - 0x7ffdd5250000 - 0x7ffdd529bfff powrprof.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdd52a0000 - 0x7ffdd52b1fff msasn1.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdd52c0000 - 0x7ffdd52c9fff fltLib.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdd52d0000 - 0x7ffdd5318fff cfgmgr32.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdd5320000 - 0x7ffdd533ffff win32u.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdd5340000 - 0x7ffdd55b2fff KERNELBASE.dll 10.0.17134.165
22:38:44 INFO - 0x7ffdd55c0000 - 0x7ffdd5cccfff windows.storage.dll 10.0.17134.285
22:38:44 INFO - 0x7ffdd5cd0000 - 0x7ffdd5d6efff msvcp_win.dll 10.0.17134.137
22:38:44 INFO - 0x7ffdd5d70000 - 0x7ffdd5de9fff bcryptPrimitives.dll 10.0.17134.285
22:38:44 INFO - 0x7ffdd5df0000 - 0x7ffdd5ee9fff ucrtbase.dll 10.0.17134.254
22:38:44 INFO - 0x7ffdd5ef0000 - 0x7ffdd6081fff gdi32full.dll 10.0.17134.285
22:38:44 INFO - 0x7ffdd6140000 - 0x7ffdd6196fff wintrust.dll 10.0.17134.81
22:38:44 INFO - 0x7ffdd61a0000 - 0x7ffdd6381fff crypt32.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdd6550000 - 0x7ffdd65a0fff shlwapi.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdd6630000 - 0x7ffdd67bffff user32.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdd67c0000 - 0x7ffdd681afff sechost.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdd6830000 - 0x7ffdd6b52fff combase.dll 10.0.17134.112
22:38:44 INFO - 0x7ffdd6c50000 - 0x7ffdd6c7cfff imm32.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdd6ce0000 - 0x7ffdd6d7dfff msvcrt.dll 7.0.17134.1
22:38:44 INFO - 0x7ffdd6d80000 - 0x7ffdd71cafff setupapi.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdd71d0000 - 0x7ffdd723bfff ws2_32.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdd7240000 - 0x7ffdd72e8fff SHCore.dll 10.0.17134.112
22:38:44 INFO - 0x7ffdd7350000 - 0x7ffdd73effff clbcatq.dll 2001.12.10941.16384
22:38:44 INFO - 0x7ffdd73f0000 - 0x7ffdd7540fff ole32.dll 10.0.17134.137
22:38:44 INFO - 0x7ffdd7550000 - 0x7ffdd7611fff oleaut32.dll 10.0.17134.48
22:38:44 INFO - 0x7ffdd77a0000 - 0x7ffdd7851fff kernel32.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdd7870000 - 0x7ffdd7910fff advapi32.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdd7920000 - 0x7ffdd8d5ffff shell32.dll 10.0.17134.228
22:38:44 INFO - 0x7ffdd8d60000 - 0x7ffdd8e83fff rpcrt4.dll 10.0.17134.112
22:38:44 INFO - 0x7ffdd8e90000 - 0x7ffdd8e97fff nsi.dll 10.0.17134.1
22:38:44 INFO - 0x7ffdd8ea0000 - 0x7ffdd8ec7fff gdi32.dll 10.0.17134.285
22:38:44 INFO - 0x7ffdd8f00000 - 0x7ffdd90e0fff ntdll.dll 10.0.17134.254
22:38:44 INFO - >>>>>>>
22:38:44 INFO - PID 15724 | Unable to load \untrusted-startup-test-dll.dll; LoadLibraryW failed: 126[15724, Main Thread] WARNING: Failed to get directory to cache.: file z:/build/build/src/security/sandbox/win/src/sandboxbroker/sandboxBroker.cpp, line 81

The assertion always appears during devtools/client/debugger/new/test/mochitest/browser_dbg-worker-scopes.js, before this test fails with time out.

Jan can you take a look at this?

Comment 27

[Intermittent Failures Robot]

20 failures in 2981 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-inbound: 8
• autoland: 10
• mozilla-central: 1
• try: 1

Platform breakdown:

• osx-10-10: 12
• linux64: 6
• windows10-64-ccov: 1
• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-02-25&endday=2019-03-03&tree=all

Comment 28

[Intermittent Failures Robot]

1 failures in 2818 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-inbound: 1

Platform breakdown:

• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-03-04&endday=2019-03-10&tree=all

Comment 29

[Intermittent Failures Robot]

1 failures in 2771 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-03-11&endday=2019-03-17&tree=all

Comment 30

[Intermittent Failures Robot]

1 failures in 2843 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-04-01&endday=2019-04-07&tree=all

Comment 31

[Jan de Mooij [:jandem]]

Clearing the NI because this mostly stopped being an issue. Unboxed object removal and leak fixes might have helped (I think we got this with the unboxedLayouts list when Gecko leaked GC things on shutdown).

Comment 32

[Intermittent Failures Robot]

22 failures in 354 pushes (0.062 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 22

Platform breakdown:

• linux64: 21
• osx-10-10: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-04-19&endday=2019-04-19&tree=all

Comment 33

[Intermittent Failures Robot]

20 failures in 151 pushes (0.132 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 8
• mozilla-inbound: 9
• mozilla-central: 3

Platform breakdown:

• linux64: 15
• linux64-shippable: 1
• osx-10-10: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-04-20&endday=2019-04-20&tree=all

Comment 34

[Intermittent Failures Robot]

62 failures in 2673 pushes (0.023 failures/push) were associated with this bug in the last 7 days.

This is the #35 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 37
• mozilla-inbound: 15
• mozilla-central: 9
• try: 1

Platform breakdown:

• linux64: 52
• osx-10-10: 7
• linux64-shippable: 1
• windows10-64-ccov: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-04-15&endday=2019-04-21&tree=all

Comment 36

[Intermittent Failures Robot]

26 failures in 307 pushes (0.085 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 12
• mozilla-inbound: 10
• mozilla-central: 3
• try: 1

Platform breakdown:

• linux64: 21
• osx-10-10: 4
• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-04-22&endday=2019-04-22&tree=all

Comment 37

[Intermittent Failures Robot]

19 failures in 499 pushes (0.038 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 13
• mozilla-inbound: 4
• mozilla-central: 1
• try: 1

Platform breakdown:

• linux64: 14
• osx-10-10: 3
• windows10-64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-04-23&endday=2019-04-23&tree=all

Comment 38

[Intermittent Failures Robot]

51 failures in 3031 pushes (0.017 failures/push) were associated with this bug in the last 7 days.

This is the #45 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 25
• mozilla-inbound: 14
• mozilla-central: 10
• try: 2

Platform breakdown:

• linux64: 35
• osx-10-10: 7
• windows10-64: 3
• windows10-64-ccov: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-04-22&endday=2019-04-28&tree=all

Comment 39

[Intermittent Failures Robot]

22 failures in 4131 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 9
• mozilla-central: 3
• mozilla-inbound: 7
• try: 3

Platform breakdown:

• linux64: 21
• osx-10-10: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-04-29&endday=2019-05-05&tree=all

Comment 40

[Intermittent Failures Robot]

19 failures in 4705 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3
• autoland: 13
• mozilla-inbound: 3

Platform breakdown:

• osx-10-10: 18
• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-05-06&endday=2019-05-12&tree=all

Comment 41

[Intermittent Failures Robot]

22 failures in 4658 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4
• autoland: 11
• mozilla-inbound: 6
• try: 1

Platform breakdown:

• osx-10-10: 6
• macosx1010-64: 15
• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-05-13&endday=2019-05-19&tree=all

Comment 42

[Intermittent Failures Robot]

16 failures in 4730 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 8
• mozilla-inbound: 3
• mozilla-central: 2
• try: 2
• mozilla-beta: 1

Platform breakdown:

• macosx1010-64: 15
• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-05-20&endday=2019-05-26&tree=all

Comment 43

[Intermittent Failures Robot]

18 failures in 4337 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 11
• mozilla-inbound: 5
• mozilla-beta: 2

Platform breakdown:

• macosx1010-64: 15
• windows10-64: 1
• linux64-qr: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-05-27&endday=2019-06-02&tree=all

Comment 44

[Intermittent Failures Robot]

12 failures in 5197 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 6
• mozilla-inbound: 4
• mozilla-beta: 2

Platform breakdown:

• macosx1010-64: 12

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-06-03&endday=2019-06-09&tree=all

Comment 45

[Intermittent Failures Robot]

8 failures in 4753 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• autoland: 2
• mozilla-beta: 3
• mozilla-inbound: 1
• mozilla-central: 1

Platform breakdown:

• macosx1010-64: 3
• windows10-64: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-06-10&endday=2019-06-16&tree=all

Comment 46

[Intermittent Failures Robot]

8 failures in 2581 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 6
• mozilla-central: 1
• try: 1

Platform breakdown:

• macosx1010-64: 7
• linux64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-06-17&endday=2019-06-23&tree=all

Comment 47

[Intermittent Failures Robot]

1 failures in 4409 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform breakdown:

• macosx1010-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-06-24&endday=2019-06-30&tree=all

Comment 48

[Intermittent Failures Robot]

3 failures in 3501 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 2
• mozilla-release: 1

Platform breakdown:

• macosx1010-64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-07-01&endday=2019-07-07&tree=all

Comment 49

[Intermittent Failures Robot]

3 failures in 4710 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 1
• autoland: 2

Platform breakdown:

• macosx1010-64: 1
• macosx1014-64: 1
• linux64-asan-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-07-08&endday=2019-07-14&tree=all

Comment 50

[Intermittent Failures Robot]

10 failures in 4298 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 1
• autoland: 6
• mozilla-release: 1
• mozilla-inbound: 1
• mozilla-central: 1

Platform breakdown:

• macosx1010-64: 2
• macosx1014-64: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-07-15&endday=2019-07-21&tree=all

Comment 51

[Intermittent Failures Robot]

3 failures in 4608 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-central: 1

Platform breakdown:

• windows10-64: 1
• macosx1014-64: 1
• linux64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-07-22&endday=2019-07-28&tree=all

Comment 52

[Intermittent Failures Robot]

6 failures in 4439 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• mozilla-inbound: 2
• autoland: 3

Platform breakdown:

• windows10-64: 4
• macosx1014-64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-07-29&endday=2019-08-04&tree=all

Comment 53

[Intermittent Failures Robot]

2 failures in 4666 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 1

Platform breakdown:

• windows10-64: 1
• linux64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-08-05&endday=2019-08-11&tree=all

Comment 54

[Intermittent Failures Robot]

8 failures in 4839 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 6
• mozilla-central: 2

Platform breakdown:

• macosx1014-64: 7
• linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-08-12&endday=2019-08-18&tree=all

Comment 56

[Intermittent Failures Robot]

8 failures in 5073 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 5
• mozilla-central: 2
• mozilla-beta: 1

Platform breakdown:

• windows10-64: 3
• linux64-ccov: 1
• windows10-64-ccov: 1
• windows7-32: 1
• linux64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-08-19&endday=2019-08-25&tree=all

Comment 57

[Intermittent Failures Robot]

5 failures in 5047 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4
• autoland: 1

Platform breakdown:

• windows10-64-ccov: 3
• linux64: 1
• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-08-26&endday=2019-09-01&tree=all

Comment 58

[Intermittent Failures Robot]

9 failures in 4408 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 8
• autoland: 1

Platform breakdown:

• windows10-64-ccov: 8
• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-09-02&endday=2019-09-08&tree=all

Comment 59

[Intermittent Failures Robot]

5 failures in 4350 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 4

Platform breakdown:

• windows10-64: 1
• windows10-64-ccov: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-09-16&endday=2019-09-22&tree=all

Comment 60

[Intermittent Failures Robot]

8 failures in 4530 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 7
• mozilla-inbound: 1

Platform breakdown:

• linux64: 1
• windows10-64-ccov: 3
• linux64-ccov: 1
• windows10-64: 1
• windows7-32: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-09-23&endday=2019-09-29&tree=all

Comment 61

[Intermittent Failures Robot]

8 failures in 4292 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-central: 6

Platform breakdown:

• linux64: 2
• windows10-64-ccov: 4
• linux64-ccov: 1
• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-09-30&endday=2019-10-06&tree=all

Comment 62

[Intermittent Failures Robot]

11 failures in 3801 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 8
• autoland: 3

Platform breakdown:

• windows10-64-ccov: 5
• linux64-ccov: 2
• linux64: 3
• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-10-07&endday=2019-10-13&tree=all

Comment 63

[Intermittent Failures Robot]

11 failures in 3964 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 6
• mozilla-central: 5

Platform breakdown:

• windows10-64: 3
• linux64: 3
• windows10-64-ccov: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-10-14&endday=2019-10-20&tree=all

Comment 64

[Intermittent Failures Robot]

10 failures in 4182 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 8
• autoland: 2

Platform breakdown:

• linux64: 3
• windows10-64-ccov: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-10-21&endday=2019-10-27&tree=all

Comment 65

[Andreea Pavel [:apavel]]

There are 10 failures in the last 7 days and 43 in the last 30.
Since this doesn't seem to be going away, Steven can you assign someone to take a look?

Comment 66

[Steven DeTar [:sdetar]]

Jan, do you have any idea what to do with this bug?

Comment 67

[Jan de Mooij [:jandem]]

(In reply to Steven DeTar [:sdetar] from comment #66)

Jan, do you have any idea what to do with this bug?

The logs show that we fail to finish an off-thread parse task on the browser side because it's too late during shutdown. Then during JS_Shutdown we fail because there are still parse tasks in the list... I'll poke a bit more.

Comment 68

[Intermittent Failures Robot]

10 failures in 4314 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 5
• autoland: 5

Platform breakdown:

• linux64: 7
• windows10-64-ccov: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-10-28&endday=2019-11-03&tree=all

Comment 69

[Intermittent Failures Robot]

13 failures in 4850 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 7
• mozilla-inbound: 1
• autoland: 5

Platform breakdown:

• windows10-64-ccov: 7
• linux64: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-11-04&endday=2019-11-10&tree=all

Comment 70

[Intermittent Failures Robot]

13 failures in 4207 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 1
• autoland: 5
• mozilla-central: 7

Platform breakdown:

• windows10-64: 1
• linux64: 5
• windows10-64-ccov: 6
• linux64-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-11-11&endday=2019-11-17&tree=all

Comment 71

[Jan de Mooij [:jandem]]

What I see in the logs is things like this:

INFO - GECKO(3598) | [Child 7149, JS Helper] WARNING: Called GetMainThread but there isn't a main thread and we're not the main thread.: file /builds/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 579
INFO - GECKO(3598) | [Child 7149, JS Helper] WARNING: 'NS_FAILED(rv)', file /builds/worker/workspace/build/src/xpcom/threads/nsThreadUtils.cpp, line 241
INFO - GECKO(3598) | [Child 7149, JS Helper] ###!!! ASSERTION: Failed NS_DispatchToMainThread() in shutdown; leaking: 'false', file /builds/worker/workspace/build/src/xpcom/threads/nsThreadUtils.cpp, line 243
... snip ...
INFO - GECKO(3598) | WARNING: YOU ARE LEAKING THE WORLD (at least one JSRuntime and everything alive inside it, that is) AT JS_ShutDown TIME.  FIX THIS!
INFO - GECKO(3598) | Assertion failure: isEmpty() (failing this assertion means this LinkedList's creator is buggy: it should have removed all this list's elements before the list's destruction), at /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/LinkedList.h:433

Maybe we're leaking a JSRuntime, then when we try to finish a parse task we fail (too late in shutdown) and in the end we also assert when destroying the process-wide HelperThreadState because the parse task linked list is not empty.

Does that seem plausible?

(I also noticed many of these logs mention devtools jsonview code or tests..)

Comment 72

[Andrew McCreight [:mccr8]]

Yes, that is plausible. If helper threads are going to try to dispatch things to the main thread, they need to do that earlier. For regular browser stuff,
I think we shut down threads with the xpcom-shutdown-threads event. (Generally I'm not a fan of these various assertions that turn leaks into crashes.)

Comment 73

[Intermittent Failures Robot]

14 failures in 5444 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 11
• mozilla-central: 3

Platform breakdown:

• linux64: 11
• windows10-64-ccov: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-11-18&endday=2019-11-24&tree=all

Comment 74

[Intermittent Failures Robot]

16 failures in 4331 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 10
• mozilla-central: 6

Platform breakdown:

• linux64: 9
• windows10-64-ccov: 6
• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-11-25&endday=2019-12-01&tree=all

Comment 75

[Intermittent Failures Robot]

12 failures in 5144 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 10
• mozilla-central: 2

Platform breakdown:

• linux64: 9
• windows10-64-ccov: 2
• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-12-02&endday=2019-12-08&tree=all

Comment 76

[Intermittent Failures Robot]

7 failures in 4867 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• mozilla-central: 4

Platform breakdown:

• linux64: 4
• windows10-64-ccov: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-12-09&endday=2019-12-15&tree=all

Comment 77

[Intermittent Failures Robot]

5 failures in 4582 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4
• mozilla-beta: 1

Platform breakdown:

• windows10-64: 3
• linux64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-12-16&endday=2019-12-22&tree=all

Comment 78

[Intermittent Failures Robot]

3 failures in 1365 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3

Platform breakdown:

• linux64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2019-12-23&endday=2019-12-29&tree=all

Comment 79

[Intermittent Failures Robot]

6 failures in 4472 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 6

Platform breakdown:

• linux64: 3
• windows10-64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-01-06&endday=2020-01-12&tree=all

Comment 80

[Intermittent Failures Robot]

1 failures in 3773 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-01-13&endday=2020-01-19&tree=all

Comment 81

[Intermittent Failures Robot]

1 failures in 4004 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-01-20&endday=2020-01-26&tree=all

Comment 82

[Intermittent Failures Robot]

1 failures in 2190 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-01-27&endday=2020-02-02&tree=all

Comment 83

[Intermittent Failures Robot]

1 failures in 4690 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• linux1804-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-02-10&endday=2020-02-16&tree=all

Comment 84

[Intermittent Failures Robot]

3 failures in 4045 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• try: 1

Platform breakdown:

• windows10-64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-02-17&endday=2020-02-23&tree=all

Comment 85

[Intermittent Failures Robot]

1 failures in 5233 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• linux1804-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-02-24&endday=2020-03-01&tree=all

Comment 86

[Intermittent Failures Robot]

2 failures in 5483 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform breakdown:

• linux1804-64: 1
• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-03-02&endday=2020-03-08&tree=all

Comment 87

[Intermittent Failures Robot]

2 failures in 5336 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform breakdown:

• windows10-64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-03-09&endday=2020-03-15&tree=all

Comment 88

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug is a regression, but please revert this change in case of error.

Comment 89

[Intermittent Failures Robot]

5 failures in 4955 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4
• mozilla-central: 1

Platform breakdown:

• windows10-64: 3
• linux1804-64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-03-16&endday=2020-03-22&tree=all

Comment 90

[Intermittent Failures Robot]

2 failures in 4773 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform breakdown:

• linux1804-64: 1
• windows7-32-mingwclang: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-03-23&endday=2020-03-29&tree=all

Comment 91

[Intermittent Failures Robot]

2 failures in 5015 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-release: 1

Platform breakdown:

• windows7-32-mingwclang: 1
• linux1804-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-03-30&endday=2020-04-05&tree=all

Comment 92

[Intermittent Failures Robot]

5 failures in 4823 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4
• mozilla-central: 1

Platform breakdown:

• linux1804-64: 2
• linux1804-64-qr: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-04-06&endday=2020-04-12&tree=all

Comment 93

[Intermittent Failures Robot]

3 failures in 4528 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3

Platform breakdown:

• linux1804-64-qr: 2
• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-04-13&endday=2020-04-19&tree=all

Comment 94

[Intermittent Failures Robot]

7 failures in 5582 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 7

Platform breakdown:

• linux1804-64: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-04-20&endday=2020-04-26&tree=all

Comment 95

[Intermittent Failures Robot]

3 failures in 5276 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-central: 1

Platform breakdown:

• linux1804-64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-05-04&endday=2020-05-10&tree=all

Comment 96

[Intermittent Failures Robot]

3 failures in 5488 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3

Platform breakdown:

• linux1804-64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-05-11&endday=2020-05-17&tree=all

Comment 97

[Intermittent Failures Robot]

19 failures in 959 pushes (0.02 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 13
• mozilla-central: 3
• comm-central: 1
• try: 2

Platform breakdown:

• windows10-64-qr: 6
• windows7-32: 3
• linux32: 1
• linux1804-64: 3
• windows10-64: 4
• linux1804-64-qr: 1
• macosx1014-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-05-20&endday=2020-05-20&tree=all

Comment 98

[Intermittent Failures Robot]

17 failures in 759 pushes (0.022 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 10
• mozilla-central: 3
• comm-central: 4

Platform breakdown:

• windows10-64: 4
• linux1804-64: 2
• linux32: 1
• macosx1014-64: 3
• linux64: 2
• linux1804-64-qr: 2
• windows10-64-qr: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-05-21&endday=2020-05-21&tree=all

Comment 99

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

The recent increase (started May 19th?) ran dom/security/ test before.

Comment 100

[Intermittent Failures Robot]

69 failures in 4727 pushes (0.015 failures/push) were associated with this bug in the last 7 days.

This is the #13 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• try: 3
• autoland: 53
• mozilla-central: 8
• comm-central: 5

Platform breakdown:

• linux1804-64: 15
• windows10-64: 23
• windows10-64-qr: 12
• windows7-32: 4
• linux32: 2
• linux1804-64-qr: 3
• macosx1014-64: 8
• linux64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-05-18&endday=2020-05-24&tree=all

Comment 101

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

The recent occurrences after the dom/security/ mochitests ran start with bug 1629866 according to retriggers.

Comment 102

[Intermittent Failures Robot]

21 failures in 1051 pushes (0.02 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 18
• mozilla-central: 2
• comm-central: 1

Platform breakdown:

• linux1804-64: 6
• windows10-64-qr: 6
• windows10-64: 5
• macosx1014-64: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-05-29&endday=2020-05-29&tree=all

Comment 103

[Alexandru Michis [:malexandru]]

James,
This failure has a high frequency on wpt jobs on osx debug since the wpt-sync from the 29th of May:
https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&searchStr=os%2Cx%2C10.14%2Cdebug%2Cweb%2Cplatform%2Ctests%2Ctest-macosx1014-64%2Fdebug-web-platform-tests-e10s&fromchange=aa90b2d3ef10d3184df361806c7432c00c69a085&tochange=086d3ca5e98e220db1dc2ed96e7fe951e5327f6d&selectedTaskRun=KjERaEv1T72rD1N6cuA-YA-0

Could you please take a look?

Comment 104

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Comment 103 is about a leak reported during /html/browsers/offline/introduction-4/event_checking.https.html.

Comment 105

[Intermittent Failures Robot]

70 failures in 5458 pushes (0.013 failures/push) were associated with this bug in the last 7 days.

This is the #12 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 5
• autoland: 52
• try: 8
• comm-central: 5

Platform breakdown:

• windows10-64: 20
• linux1804-64: 12
• windows10-64-qr: 16
• windows7-32-mingwclang: 2
• linux1804-64-qr: 1
• windows7-32: 4
• macosx1014-64: 12
• linux32: 1
• linux64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-05-25&endday=2020-05-31&tree=all

Comment 107

[Magnus Melin [:mkmelin]]

From the dupe we have in comm-central (where this is a perma fail on debug) - bug 1639446 - it started May 20 with this push:
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=8f68705097b4bf88cd61b43b14401cde98ac75b6&tochange=855249e545c361516a65bcba8f5bc6b423e2d131

Comment 108

[Intermittent Failures Robot]

55 failures in 4949 pushes (0.011 failures/push) were associated with this bug in the last 7 days.

This is the #29 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 26
• comm-central: 23
• mozilla-central: 3
• try: 3

Platform breakdown:

• linux1804-64-qr: 2
• windows10-64: 15
• linux1804-64: 9
• linux32: 1
• linux64: 7
• windows10-64-qr: 5
• macosx1014-64: 4
• windows7-32: 12

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-06-01&endday=2020-06-07&tree=all

Comment 109

[Intermittent Failures Robot]

32 failures in 890 pushes (0.036 failures/push) were associated with this bug yesterday.

Repository breakdown:

• comm-central: 21
• try: 2
• autoland: 3
• mozilla-beta: 4
• mozilla-central: 2

Platform breakdown:

• windows10-64: 7
• linux64: 6
• windows7-32: 10
• macosx1014-64: 4
• linux1804-64: 1
• windows10-64-qr: 3
• linux1804-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-06-08&endday=2020-06-08&tree=all

Comment 110

[Intermittent Failures Robot]

44 failures in 1067 pushes (0.041 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 16
• try: 2
• mozilla-central: 2
• comm-central: 24

Platform breakdown:

• windows10-64: 14
• linux1804-64: 6
• windows7-32: 11
• macosx1014-64: 5
• windows10-64-qr: 2
• linux64: 5
• linux1804-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-06-10&endday=2020-06-10&tree=all

Comment 111

[Intermittent Failures Robot]

35 failures in 1000 pushes (0.035 failures/push) were associated with this bug yesterday.

Repository breakdown:

• comm-central: 20
• autoland: 9
• mozilla-beta: 4
• try: 2

Platform breakdown:

• linux64: 5
• macosx1014-64: 4
• windows7-32: 8
• windows10-64: 8
• linux1804-64: 4
• linux1804-64-qr: 2
• windows10-64-qr: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-06-11&endday=2020-06-11&tree=all

Comment 112

[Intermittent Failures Robot]

26 failures in 979 pushes (0.027 failures/push) were associated with this bug yesterday.

Repository breakdown:

• comm-central: 10
• autoland: 9
• try: 3
• mozilla-central: 4

Platform breakdown:

• linux64: 4
• macosx1014-64: 2
• windows10-64-qr: 3
• windows7-32: 3
• windows10-64: 4
• linux1804-64: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-06-12&endday=2020-06-12&tree=all

Comment 113

[Andrew McCreight [:mccr8]]

[Tracking Requested - why for this release]: High frequency leak regression (from bug 1629866) according to comment 101.

Comment 114

[Intermittent Failures Robot]

190 failures in 5211 pushes (0.036 failures/push) were associated with this bug in the last 7 days.

This is the #2 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• comm-central: 97
• try: 10
• autoland: 61
• mozilla-beta: 8
• mozilla-central: 14

Platform breakdown:

• windows10-64: 46
• linux64: 27
• windows7-32: 40
• macosx1014-64: 22
• linux1804-64: 31
• windows10-64-qr: 18
• linux1804-64-qr: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-06-08&endday=2020-06-14&tree=all

Comment 115

[Andrew McCreight [:mccr8]]

While they are lower frequency than the failures in bug 1634641, the failures here in mochitest-browser-chrome-e10s-5 are happening while running browser/components/downloads/test/browser/, so I think they are the debug equivalent of the regression from bug 1606652. This is a debug build, so maybe there's a little more information in the log?

For instance, I see this shortly before the failure:

[Child 3900, JS Helper] WARNING: Called GetMainThread but there isn't a main thread and we're not the main thread.: file /builds/worker/checkouts/gecko/xpcom/threads/nsThreadManager.cpp, line 651
[Child 3900, JS Helper] WARNING: 'NS_FAILED(rv)', file /builds/worker/checkouts/gecko/xpcom/threads/nsThreadUtils.cpp, line 254
[Child 3900, JS Helper] ###!!! ASSERTION: Failed NS_DispatchToMainThread() in shutdown; leaking: 'false', file /builds/worker/checkouts/gecko/xpcom/threads/nsThreadUtils.cpp, line 256
GECKO(3167) | #01: mozilla::SchedulerGroup::InternalUnlabeledDispatch(mozilla::TaskCategory, already_AddRefed<mozilla::SchedulerGroup::Runnable>&&) [xpcom/threads/SchedulerGroup.cpp:98]
GECKO(3167) | #02: mozilla::SchedulerGroup::LabeledDispatch(mozilla::TaskCategory, already_AddRefed<nsIRunnable>&&, mozilla::dom::DocGroup*) [xpcom/threads/SchedulerGroup.cpp:83]
GECKO(3167) | #03: mozilla::dom::OffThreadScriptLoaderCallback(JS::OffThreadToken*, void*) [dom/script/ScriptLoader.cpp:2218]
GECKO(3167) | #04: js::HelperThread::handleParseWorkload(js::AutoLockHelperThreadState&) [js/src/vm/HelperThreads.cpp:2257]

Comment 116

[Intermittent Failures Robot]

27 failures in 784 pushes (0.034 failures/push) were associated with this bug yesterday.

Repository breakdown:

• comm-central: 14
• autoland: 10
• try: 3

Platform breakdown:

• linux64: 4
• windows10-64: 7
• windows7-32: 4
• macosx1014-64: 2
• linux1804-64: 9
• windows10-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-06-15&endday=2020-06-15&tree=all

Comment 117

[Intermittent Failures Robot]

43 failures in 874 pushes (0.049 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 25
• mozilla-central: 6
• try: 5
• comm-central: 7

Platform breakdown:

• windows10-64-qr: 4
• windows7-32: 8
• linux1804-64: 17
• macosx1014-64: 8
• windows10-64: 4
• linux64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-06-16&endday=2020-06-16&tree=all

Comment 118

[Denis Palmeiro [:denispal]]

Although this is quite hard to reproduce, it does indeed seem to be a duplicate of bug 1634641 as I can also generate this assert in the other bug.

Comment 119

[Intermittent Failures Robot]

36 failures in 942 pushes (0.038 failures/push) were associated with this bug yesterday.

Repository breakdown:

• comm-central: 14
• autoland: 15
• mozilla-central: 5
• try: 2

Platform breakdown:

• windows7-32: 9
• linux64: 4
• windows10-64: 4
• macosx1014-64: 5
• windows10-64-qr: 2
• linux1804-64: 10
• windows7-32-mingwclang: 1
• linux1804-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-06-17&endday=2020-06-17&tree=all

Comment 120

[Intermittent Failures Robot]

59 failures in 872 pushes (0.068 failures/push) were associated with this bug yesterday.

Repository breakdown:

• comm-central: 37
• autoland: 19
• mozilla-central: 1
• try: 2

Platform breakdown:

• windows7-32: 15
• linux64: 11
• macosx1014-64: 7
• windows10-64: 13
• linux1804-64: 11
• windows10-64-qr: 1
• linux1804-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-06-18&endday=2020-06-18&tree=all

Comment 121

[Intermittent Failures Robot]

18 failures in 339 pushes (0.053 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 9
• comm-central: 5
• mozilla-central: 4

Platform breakdown:

• linux1804-64: 8
• windows10-64: 3
• windows7-32: 3
• macosx1014-64: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-06-19&endday=2020-06-19&tree=all

Comment 122

[Intermittent Failures Robot]

216 failures in 4035 pushes (0.054 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 87
• comm-central: 95
• try: 15
• mozilla-central: 19

Platform breakdown:

• windows10-64: 36
• linux64: 27
• windows7-32: 44
• macosx1014-64: 33
• linux1804-64: 63
• windows10-64-qr: 9
• windows7-32-mingwclang: 1
• linux1804-64-qr: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-06-15&endday=2020-06-21&tree=all

Comment 123

[Intermittent Failures Robot]

69 failures in 843 pushes (0.082 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-esr78: 3
• mozilla-beta: 4
• autoland: 20
• mozilla-central: 6
• comm-central: 28
• try: 5
• mozilla-release: 1
• try-comm-central: 2

Platform breakdown:

• windows10-64: 14
• windows10-64-qr: 5
• linux1804-64: 17
• windows7-32: 14
• macosx1014-64: 9
• linux64: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-06-22&endday=2020-06-22&tree=all

Comment 124

[Intermittent Failures Robot]

61 failures in 1020 pushes (0.06 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 20
• mozilla-esr78: 1
• comm-central: 35
• mozilla-central: 3
• try: 2

Platform breakdown:

• windows10-64: 16
• linux1804-64: 13
• windows7-32: 12
• linux64: 10
• macosx1014-64: 8
• windows10-64-qr: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-06-23&endday=2020-06-23&tree=all

Comment 125

[Intermittent Failures Robot]

78 failures in 1046 pushes (0.075 failures/push) were associated with this bug yesterday.

Repository breakdown:

• try: 13
• autoland: 35
• mozilla-central: 12
• comm-central: 18

Platform breakdown:

• linux1804-64-qr: 11
• linux1804-64: 24
• windows7-32: 13
• windows10-64: 11
• macosx1014-64: 13
• windows10-64-qr: 1
• linux64: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-06-24&endday=2020-06-24&tree=all

Comment 126

[Intermittent Failures Robot]

66 failures in 1151 pushes (0.057 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 44
• mozilla-central: 5
• comm-central: 11
• try: 5
• mozilla-release: 1

Platform breakdown:

• linux1804-64: 24
• macosx1014-64: 14
• windows10-64-qr: 6
• windows10-64: 7
• windows7-32: 7
• linux64: 4
• linux1804-64-qr: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-06-25&endday=2020-06-25&tree=all

Comment 127

[Intermittent Failures Robot]

33 failures in 806 pushes (0.041 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 21
• mozilla-central: 1
• comm-central: 3
• try: 8

Platform breakdown:

• linux1804-64: 12
• windows10-64: 6
• windows7-32: 3
• linux64: 1
• windows10-64-qr: 6
• macosx1014-64: 4
• linux1804-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-06-26&endday=2020-06-26&tree=all

Comment 128

[Narcis Beleuzu [:NarcisB]]

There are 187 failures in the last 7 days.
All failures are on debug:
Linux1804-64 -> 95 failures
Macosx1014-64 -> 36 failures
Windows10-64 -> 39 failures
Windows7-32 - 17 failures

Steven, can you please take a look?

Comment 129

[Intermittent Failures Robot]

15 failures in 247 pushes (0.061 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 10
• try: 4
• mozilla-central: 1

Platform breakdown:

• linux1804-64: 6
• windows7-32: 3
• windows10-64-qr: 3
• macosx1014-64: 1
• windows10-64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-06-27&endday=2020-06-27&tree=all

Comment 130

[Intermittent Failures Robot]

336 failures in 5252 pushes (0.064 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-esr78: 4
• mozilla-beta: 4
• autoland: 153
• mozilla-central: 30
• comm-central: 102
• try: 39
• mozilla-release: 2
• try-comm-central: 2

Platform breakdown:

• windows10-64: 60
• windows10-64-qr: 24
• linux1804-64: 98
• windows7-32: 55
• macosx1014-64: 51
• linux64: 32
• linux1804-64-qr: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-06-22&endday=2020-06-28&tree=all

Comment 131

[Jason Orendorff [:jorendorff]]

Denis is currently working on this.

Comment 132

[Intermittent Failures Robot]

55 failures in 769 pushes (0.072 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 19
• try: 1
• mozilla-release: 2
• mozilla-beta: 3
• comm-central: 27
• mozilla-central: 3

Platform breakdown:

• windows10-64: 12
• linux1804-64: 13
• macosx1014-64: 8
• windows7-32: 12
• windows10-64-qr: 2
• linux64: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-06-29&endday=2020-06-29&tree=all

Comment 133

[Peter Van der Beken [:peterv]]

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #101)

The recent occurrences after the dom/security/ mochitests ran start with bug 1629866 according to retriggers.

This was an existing leak, where we were leaking a ton of BrowsingContexts. Bug 1629866 made the BrowsingContext hold an additional object (ChildSHistory), and that made the leak worse, but it doesn't cause it. Somebody needs to figure out the existing leak.

Comment 134

[Intermittent Failures Robot]

70 failures in 954 pushes (0.073 failures/push) were associated with this bug yesterday.

Repository breakdown:

• comm-central: 28
• autoland: 31
• try: 3
• mozilla-central: 7
• mozilla-beta: 1

Platform breakdown:

• windows10-64: 13
• linux64: 8
• macosx1014-64: 16
• windows7-32: 19
• linux1804-64: 10
• windows10-64-qr: 3
• linux1804-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-06-30&endday=2020-06-30&tree=all

Comment 135

[Intermittent Failures Robot]

52 failures in 834 pushes (0.062 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 28
• mozilla-central: 5
• comm-central: 10
• try: 9

Platform breakdown:

• macosx1014-64: 15
• windows10-64: 6
• linux1804-64-qr: 7
• linux1804-64: 11
• windows7-32: 5
• linux64: 4
• windows10-64-qr: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-07-01&endday=2020-07-01&tree=all

Comment 136

[Intermittent Failures Robot]

54 failures in 992 pushes (0.054 failures/push) were associated with this bug yesterday.

Repository breakdown:

• comm-central: 28
• autoland: 21
• try: 2
• mozilla-central: 2
• mozilla-release: 1

Platform breakdown:

• windows7-32: 10
• linux64: 8
• macosx1014-64: 8
• windows10-64: 9
• linux1804-64: 15
• linux1804-64-qr: 2
• windows10-64-qr: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-07-02&endday=2020-07-02&tree=all

Comment 137

[Intermittent Failures Robot]

42 failures in 692 pushes (0.061 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 19
• mozilla-central: 4
• comm-central: 10
• try: 7
• mozilla-beta: 2

Platform breakdown:

• macosx1014-64: 6
• linux1804-64: 20
• windows10-64-qr: 3
• windows7-32: 5
• windows10-64: 2
• linux64: 4
• linux1804-64-qr: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-07-03&endday=2020-07-03&tree=all

Comment 138

[Andreea Pavel [:apavel]]

Denis are there updates here?

Comment 139

[Denis Palmeiro [:denispal]]

I have been mostly focused on bug 1634641, hoping it will address both issues. However if browser_pdfjs_preview.js is being disabled, then I can shift focus back onto this and come back to 1634641 afterwards.

Comment 140

[Intermittent Failures Robot]

306 failures in 4516 pushes (0.068 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 128
• try: 24
• mozilla-release: 3
• mozilla-beta: 6
• comm-central: 121
• mozilla-central: 24

Platform breakdown:

• windows10-64: 48
• linux1804-64: 78
• macosx1014-64: 58
• windows7-32: 59
• windows10-64-qr: 14
• linux64: 36
• linux1804-64-qr: 13

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-06-29&endday=2020-07-05&tree=all

Comment 141

[Denis Palmeiro [:denispal]]

What looks like may be happening here is that we initiate an off thread script parse from the Scriptloader and while this is in the middle of compiling, a ShutdownXPCom is issued. Scriptloader tries to cancel these off thread parses, but since the tokens used to cancel these scripts are only made available when the parse is finished, they are never actually properly cleaned up and just linger in the parseFinishedList from the helper thread which leads to this assert. I believe this was always an existing problem, but since bug 1606652 we are now issuing many more scripts to be off thread compiled so it has a higher chance to fail now.

This also explains the race condition and may explain the leaks in bug 1634641. Trying to verify this with some changes at the moment and will work on a fix next.

Comment 142

[Intermittent Failures Robot]

45 failures in 849 pushes (0.053 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 16
• comm-central: 21
• mozilla-central: 4
• try: 1
• mozilla-release: 1
• mozilla-esr78: 1
• mozilla-beta: 1

Platform breakdown:

• windows7-32: 9
• windows10-64: 7
• linux64: 6
• macosx1014-64: 8
• linux1804-64: 14
• linux1804-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-07-06&endday=2020-07-06&tree=all

Comment 143

[Intermittent Failures Robot]

38 failures in 928 pushes (0.041 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 5
• comm-central: 11
• try: 5
• autoland: 17

Platform breakdown:

• windows7-32: 6
• macosx1014-64: 6
• windows10-64: 7
• linux64: 3
• linux1804-64-qr: 1
• linux1804-64: 12
• windows10-64-qr: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-07-07&endday=2020-07-07&tree=all

Comment 144

[Denis Palmeiro [:denispal]]

bug 1606652 is only enabled for nightly, so this should not affect 79.

Comment 145

[Geoff Lankow (:darktrojan)]

Thunderbird has had permanently failing tests due to this since at least 78. We'd appreciate any fix being uplifted.

Comment 146

[Intermittent Failures Robot]

51 failures in 1037 pushes (0.049 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-beta: 3
• autoland: 27
• try: 8
• mozilla-central: 5
• mozilla-esr78: 1
• comm-central: 7

Platform breakdown:

• windows10-64-qr: 4
• linux1804-64: 24
• windows10-64: 4
• windows7-32: 8
• macosx1014-64: 7
• linux1804-64-qr: 2
• linux64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-07-08&endday=2020-07-08&tree=all

Comment 147

[Intermittent Failures Robot]

59 failures in 993 pushes (0.059 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 36
• comm-central: 14
• mozilla-central: 6
• try: 3

Platform breakdown:

• linux1804-64: 28
• windows10-64-qr: 2
• linux64: 4
• windows7-32: 10
• windows10-64: 7
• macosx1014-64: 4
• linux1804-64-qr: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-07-09&endday=2020-07-09&tree=all

Comment 148

[Intermittent Failures Robot]

47 failures in 1064 pushes (0.044 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 35
• try: 3
• mozilla-central: 6
• mozilla-esr78: 1
• mozilla-beta: 2

Platform breakdown:

• linux1804-64: 32
• linux1804-64-qr: 3
• windows10-64-qr: 1
• windows10-64: 6
• windows7-32: 3
• macosx1014-64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-07-10&endday=2020-07-10&tree=all

Comment 149

[Intermittent Failures Robot]

25 failures in 236 pushes (0.106 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-esr78: 1
• autoland: 10
• try: 6
• mozilla-central: 8

Platform breakdown:

• linux1804-64: 9
• windows10-64-qr: 6
• windows10-64: 3
• linux1804-64-qr: 3
• windows7-32: 1
• macosx1014-64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-07-11&endday=2020-07-11&tree=all

Comment 150

[Intermittent Failures Robot]

285 failures in 5265 pushes (0.054 failures/push) were associated with this bug in the last 7 days.

This is the #2 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 151
• comm-central: 53
• mozilla-central: 36
• try: 34
• mozilla-release: 1
• mozilla-esr78: 4
• mozilla-beta: 6

Platform breakdown:

• windows7-32: 40
• windows10-64: 37
• linux64: 15
• macosx1014-64: 32
• linux1804-64: 125
• linux1804-64-qr: 17
• windows10-64-qr: 19

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-07-06&endday=2020-07-12&tree=all

Comment 151

[Intermittent Failures Robot]

77 failures in 750 pushes (0.103 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 37
• comm-central: 28
• mozilla-central: 9
• mozilla-beta: 2
• mozilla-esr78: 1

Platform breakdown:

• macosx1014-64: 9
• windows10-64-qr: 6
• windows7-32: 15
• linux1804-64-qr: 4
• linux1804-64: 23
• linux64: 8
• windows10-64: 12

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-07-13&endday=2020-07-13&tree=all

Comment 152

[Andreea Pavel [:apavel]]

Hi Denis are there updates regarding the fix?

Comment 153

[Intermittent Failures Robot]

50 failures in 768 pushes (0.065 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 23
• mozilla-central: 5
• comm-central: 13
• try: 8
• mozilla-beta: 1

Platform breakdown:

• linux1804-64: 23
• linux1804-64-qr: 3
• windows10-64: 9
• linux64: 4
• windows7-32: 5
• macosx1014-64: 5
• windows10-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-07-14&endday=2020-07-14&tree=all

Comment 154

[Denis Palmeiro [:denispal]]

Yes, I believe I have a fix for this in bug 1652126. I'm just trying to see if there's a better way to do it before I post a patch for review.

Comment 155

[Intermittent Failures Robot]

61 failures in 636 pushes (0.096 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 31
• comm-central: 20
• mozilla-central: 8
• try: 2

Platform breakdown:

• linux1804-64: 22
• macosx1014-64: 8
• windows7-32: 10
• windows10-64: 11
• linux64: 6
• linux1804-64-qr: 2
• windows10-64-qr: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-07-15&endday=2020-07-15&tree=all

Comment 156

[Intermittent Failures Robot]

61 failures in 702 pushes (0.087 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 25
• mozilla-central: 5
• comm-central: 28
• mozilla-beta: 2
• try: 1

Platform breakdown:

• linux1804-64: 24
• linux1804-64-qr: 3
• linux64: 8
• windows7-32: 9
• macosx1014-64: 9
• windows10-64: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-07-16&endday=2020-07-16&tree=all

Comment 157

[Intermittent Failures Robot]

30 failures in 867 pushes (0.035 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-beta: 2
• autoland: 16
• comm-central: 7
• try: 2
• mozilla-central: 3

Platform breakdown:

• linux1804-64-qr: 5
• linux1804-64: 14
• windows10-64: 4
• linux64: 2
• windows7-32: 2
• macosx1014-64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-07-17&endday=2020-07-17&tree=all

Comment 158

[Intermittent Failures Robot]

296 failures in 4014 pushes (0.074 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 144
• comm-central: 96
• try: 18
• mozilla-central: 30
• mozilla-beta: 7
• mozilla-esr78: 1

Platform breakdown:

• macosx1014-64: 35
• windows10-64-qr: 9
• windows7-32: 41
• linux1804-64-qr: 18
• linux1804-64: 121
• linux64: 28
• windows10-64: 44

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-07-13&endday=2020-07-19&tree=all

Comment 159

[Intermittent Failures Robot]

37 failures in 725 pushes (0.051 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 29
• mozilla-central: 3
• comm-central: 4
• mozilla-beta: 1

Platform breakdown:

• linux1804-64: 26
• windows10-64-qr: 3
• windows10-64: 1
• macosx1014-64: 1
• linux64: 4
• linux1804-64-qr: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-07-20&endday=2020-07-20&tree=all

Comment 160

[Intermittent Failures Robot]

31 failures in 948 pushes (0.033 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 14
• try: 7
• mozilla-central: 6
• comm-central: 3
• mozilla-esr78: 1

Platform breakdown:

• linux1804-64-qr: 3
• linux1804-64: 23
• linux64: 2
• macosx1014-64: 1
• windows7-32: 1
• windows10-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-07-21&endday=2020-07-21&tree=all

Comment 161

[Intermittent Failures Robot]

30 failures in 893 pushes (0.034 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 21
• mozilla-central: 4
• try: 5

Platform breakdown:

• linux1804-64: 17
• linux1804-64-qr: 9
• windows10-64: 2
• macosx1014-64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-07-22&endday=2020-07-22&tree=all

Comment 162

[Intermittent Failures Robot]

20 failures in 707 pushes (0.028 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 12
• mozilla-central: 4
• comm-central: 3
• mozilla-release: 1

Platform breakdown:

• linux1804-64: 7
• linux1804-64-qr: 4
• windows10-64: 3
• windows10-64-qr: 1
• linux64: 2
• macosx1014-64: 1
• windows7-32: 1
• macosx1014-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-07-23&endday=2020-07-23&tree=all

Comment 163

[Bogdan Tara[:bogdan_tara | bogdant]]

In the last 7 days there have been 115 occurrences on linux1804-64, macosx1014-64, windows10-64, build type debug.

Recent failure: https://treeherder.mozilla.org/logviewer.html#?job_id=311057848&repo=mozilla-central

Comment 164

[Cristina Coroiu [:ccoroiu]]

The fix from https://bugzilla.mozilla.org/show_bug.cgi?id=1652126#c11 doesn't seems to work.

Since https://bugzilla.mozilla.org/show_bug.cgi?id=1654357#c2 landed on 21st of July there are still many failures as seen on Intermittent Failure View: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2020-07-19&endday=2020-07-26&tree=trunk&bug=1265637

Denis, please take a look at this. Thank you

Comment 165

[Denis Palmeiro [:denispal]]

(In reply to Cristina Coroiu [:ccoroiu] from comment #164)

The fix from https://bugzilla.mozilla.org/show_bug.cgi?id=1652126#c11 doesn't seems to work.

Since https://bugzilla.mozilla.org/show_bug.cgi?id=1654357#c2 landed on 21st of July there are still many failures as seen on Intermittent Failure View: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2020-07-19&endday=2020-07-26&tree=trunk&bug=1265637

Denis, please take a look at this. Thank you

The tracebacks look different now. I don't see any in handleParseWorkload like before. In any case, Joel can you please help back out bug 1606652 to see if this fixes the problem?

Comment 166

[Alexandru Michis [:malexandru]]

I did a try push with Bug 1606652 backed out:
https://treeherder.mozilla.org/#/jobs?repo=try&selectedTaskRun=JRBQSSVcQSC9AjHqIcEsaw.0&resultStatus=testfailed%2Cbusted%2Cexception%2Csuccess%2Cusercancel%2Crunning%2Cpending%2Crunnable&revision=a103d9148c440ec3977476c2eb830104026bee81

It contains browser-chrome tests on which the assertion failure discussed here has been seen recently, I don't know if the
failure rate seen there can warrant a backout, but "handleParseWorkload" can be seen in the trace.

Comment 167

[Denis Palmeiro [:denispal]]

Thanks Alexandru. I assume then that bug 1606652 is not the offending regressor here? This would seem to explain why turning the feature off also had no effect.

Comment 168

[Intermittent Failures Robot]

158 failures in 4310 pushes (0.037 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 86
• mozilla-central: 25
• comm-central: 27
• mozilla-beta: 1
• try: 15
• mozilla-esr78: 1
• mozilla-release: 1
• try-comm-central: 2

Platform breakdown:

• linux1804-64: 79
• windows10-64-qr: 9
• windows10-64: 12
• macosx1014-64: 12
• linux64: 16
• linux1804-64-qr: 21
• windows7-32: 8
• macosx1014-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-07-20&endday=2020-07-26&tree=all

Comment 169

[Joel Maher ( :jmaher ) (UTC -8)]

I assume based on :malexandru's try push this is not needed.

Comment 170

[Intermittent Failures Robot]

32 failures in 662 pushes (0.048 failures/push) were associated with this bug yesterday.

Repository breakdown:

• comm-central: 21
• try: 4
• autoland: 3
• mozilla-central: 4

Platform breakdown:

• linux64: 6
• macosx1014-64: 4
• windows10-64: 7
• windows7-32: 6
• linux1804-64-qr: 1
• linux1804-64: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-07-27&endday=2020-07-27&tree=all

Comment 171

[Intermittent Failures Robot]

38 failures in 864 pushes (0.044 failures/push) were associated with this bug yesterday.

Repository breakdown:

• try: 6
• autoland: 14
• mozilla-central: 4
• comm-central: 14

Platform breakdown:

• linux1804-64: 14
• macosx1014-64: 6
• windows10-64: 6
• linux1804-64-qr: 3
• macosx1014-64-qr: 1
• linux64: 4
• windows7-32: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-07-28&endday=2020-07-28&tree=all

Comment 172

[Intermittent Failures Robot]

24 failures in 846 pushes (0.028 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 17
• try: 4
• mozilla-central: 3

Platform breakdown:

• windows10-64-qr: 2
• linux1804-64-qr: 3
• linux1804-64: 13
• macosx1014-64: 2
• windows7-32: 1
• windows10-64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-07-29&endday=2020-07-29&tree=all

Comment 173

[Intermittent Failures Robot]

31 failures in 801 pushes (0.039 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 25
• mozilla-central: 5
• try: 1

Platform breakdown:

• linux1804-64-qr: 11
• windows10-64: 1
• linux1804-64: 15
• macosx1014-64: 2
• windows7-32: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-07-30&endday=2020-07-30&tree=all

Comment 174

[Intermittent Failures Robot]

35 failures in 832 pushes (0.042 failures/push) were associated with this bug yesterday.

Repository breakdown:

• try: 5
• autoland: 28
• mozilla-central: 2

Platform breakdown:

• linux1804-64-qr: 5
• linux1804-64: 19
• windows10-64: 3
• windows7-32: 7
• macosx1014-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-07-31&endday=2020-07-31&tree=all

Comment 175

[Intermittent Failures Robot]

186 failures in 4289 pushes (0.043 failures/push) were associated with this bug in the last 7 days.

This is the #2 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• comm-central: 35
• try: 25
• autoland: 101
• mozilla-central: 25

Platform breakdown:

• linux64: 10
• macosx1014-64: 17
• windows10-64: 24
• windows7-32: 21
• linux1804-64-qr: 25
• linux1804-64: 86
• macosx1014-64-qr: 1
• windows10-64-qr: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-07-27&endday=2020-08-02&tree=all

Comment 176

[Intermittent Failures Robot]

19 failures in 628 pushes (0.03 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 12
• try: 3
• mozilla-beta: 1
• mozilla-central: 2
• mozilla-esr78: 1

Platform breakdown:

• linux1804-64-qr: 4
• linux1804-64: 13
• windows10-64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-08-03&endday=2020-08-03&tree=all

Comment 177

[Intermittent Failures Robot]

29 failures in 884 pushes (0.033 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 4
• autoland: 20
• try: 4
• mozilla-beta: 1

Platform breakdown:

• linux1804-64: 23
• windows10-64: 1
• macosx1014-64: 1
• linux1804-64-qr: 2
• windows10-64-qr: 1
• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-08-04&endday=2020-08-04&tree=all

Comment 178

[Andreea Pavel [:apavel]]

Denis are you still working on this considering it's not from bug 1606652 ?

Comment 179

[Intermittent Failures Robot]

18 failures in 948 pushes (0.019 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 14
• try: 2
• mozilla-central: 2

Platform breakdown:

• linux1804-64: 15
• windows10-64: 2
• windows10-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-08-05&endday=2020-08-05&tree=all

Comment 180

[Denis Palmeiro [:denispal]]

(In reply to Andreea Pavel [:apavel] from comment #178)

Denis are you still working on this considering it's not from bug 1606652 ?

No, I am not. I am mostly working on bug 1652126 to help fix bug 1634641.

Comment 181

[Intermittent Failures Robot]

21 failures in 958 pushes (0.022 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 12
• mozilla-beta: 2
• mozilla-central: 6
• try: 1

Platform breakdown:

• linux1804-64: 17
• linux1804-64-qr: 3
• macosx1014-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-08-06&endday=2020-08-06&tree=all

Comment 182

[Andreea Pavel [:apavel]]

Steven can you assign someone to take a look here?

Comment 183

[Steven DeTar [:sdetar]]

Denis has been looking at this, though but via one other bugs in Comment 180. This bug just seems to be more of meta bug based on the history.

Comment 184

[Intermittent Failures Robot]

111 failures in 4618 pushes (0.024 failures/push) were associated with this bug in the last 7 days.

This is the #4 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 71
• try: 14
• mozilla-beta: 4
• mozilla-central: 20
• mozilla-esr78: 2

Platform breakdown:

• linux1804-64-qr: 12
• linux1804-64: 81
• windows10-64: 5
• macosx1014-64: 5
• windows10-64-qr: 4
• windows7-32: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-08-03&endday=2020-08-09&tree=all

Comment 185

[Andreea Pavel [:apavel]]

Hi Joel, can you please take a look at the above comments and tell us what to do next here? Do we make this a meta bug?

Comment 186

[Joel Maher ( :jmaher ) (UTC -8)]

this bug has been around for a long time, I suspect there are many causes and fixes required to get rid of this. Unfortunately this is across many OS and test suites, I cannot think of a way to narrow this down much more other than debugging. This maybe could become a meta bug, or we could just have a bunch of sub bugs to fix cases as they are debugged.

Comment 187

[Kartikaya Gupta (email:[e-mail])]

I wrote a patch that I think may help split this bug into sub-bugs based on the different users of the LinkedList code. Patch is here, but so far the try push hasn't turned up any of this failure so I haven't yet verified the error message has been changed in the way I intend. I'll keep retriggering periodically.

Comment 188

[Andrew McCreight [:mccr8]]

My theory is that some major leaks (those that include a lot of JS and DOM stuff) turn into this assertion. I would expect that they are all for the same class. Honestly, my preference would be to stop asserting for this and let the leak checker do what it does best. Although large JS/DOM leaks tend to also turn into a single bucket.

Comment 189

[Kartikaya Gupta (email:[e-mail])]

I'm happy to write a patch to remove the assertion if you think that's the way to go! IMO assertions should only be used for things that are relatively rare and are actually going to be addressed, and this doesn't seem to qualify given how long and how frequently this failure has been occurring.

Comment 190

[Andrew McCreight [:mccr8]]

As a leak person, I think that would be an improvement on the current situation. MFBT people may disagree.

Comment 191

[Kartikaya Gupta (email:[e-mail])]

Nathan, as an MFBT person, what do you think? (see comment 185 onwards)

Comment 192

[Intermittent Failures Robot]

64 failures in 3601 pushes (0.018 failures/push) were associated with this bug in the last 7 days.

This is the #8 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 34
• try: 10
• mozilla-central: 18
• mozilla-release: 1
• mozilla-esr78: 1

Platform breakdown:

• linux1804-64: 46
• macosx1014-64: 3
• windows10-64: 5
• linux1804-64-qr: 5
• windows10-64-qr: 4
• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-08-10&endday=2020-08-16&tree=all

Comment 193

[Intermittent Failures Robot]

21 failures in 712 pushes (0.029 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 13
• mozilla-central: 6
• mozilla-esr78: 1
• try: 1

Platform breakdown:

• windows10-64: 2
• linux1804-64-qr: 1
• windows10-64-qr: 1
• linux1804-64: 14
• macosx1014-64: 2
• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-08-17&endday=2020-08-17&tree=all

Comment 194

[Intermittent Failures Robot]

30 failures in 811 pushes (0.037 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 20
• mozilla-central: 8
• try: 2

Platform breakdown:

• linux1804-64: 25
• windows10-64: 3
• linux1804-64-qr: 1
• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-08-18&endday=2020-08-18&tree=all

Comment 195

[Intermittent Failures Robot]

35 failures in 934 pushes (0.037 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 26
• mozilla-central: 5
• try: 4

Platform breakdown:

• linux1804-64: 26
• linux1804-64-qr: 3
• windows10-64: 3
• macosx1014-64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-08-19&endday=2020-08-19&tree=all

Comment 196

[Intermittent Failures Robot]

119 failures in 687 pushes (0.173 failures/push) were associated with this bug yesterday.

Repository breakdown:

• try: 82
• autoland: 30
• mozilla-central: 7

Platform breakdown:

• linux1804-64: 67
• linux1804-64-qr: 11
• macosx1014-64-qr: 1
• android-em-7-0-x86_64: 2
• windows10-64: 14
• windows7-32: 8
• macosx1014-64: 4
• windows10-64-qr: 9
• linux1804-64-asan: 2
• linux1804-64-tsan: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-08-20&endday=2020-08-20&tree=all

Comment 197

[Narcis Beleuzu [:NarcisB]]

There are 145 failures in the last 7 days.
Most of them (122 failures) are on linux1804-64/debug

:sdetar, can you please take a look?

Comment 198

[Intermittent Failures Robot]

44 failures in 643 pushes (0.068 failures/push) were associated with this bug yesterday.

Repository breakdown:

• try: 28
• mozilla-central: 8
• autoland: 8

Platform breakdown:

• linux1804-64: 26
• windows10-64-qr: 6
• windows7-32: 2
• linux1804-64-tsan: 1
• macosx1014-64-qr: 1
• macosx1014-64: 3
• windows10-64: 2
• linux1804-64-asan-qr: 1
• android-em-7-0-x86_64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-08-21&endday=2020-08-21&tree=all

Comment 199

[Cosmin Sabou [:CosminS]]

Jeff, could you have a look at what Kartikaya was proposing in comment 189? This bug has the second highest failure rate with:

• 139 total failures in the last 7 days: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2020-08-15&endday=2020-08-22&tree=trunk&bug=1265637
• 437 total failures in the last 30 days: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2020-07-23&endday=2020-08-22&tree=trunk&bug=1265637

Here's how the changes look: https://hg.mozilla.org/try/rev/c1c688e3603dc850ca7ebd813295c966ac9f6b55 and the try push which has no such failures on it. Thank you.

Comment 200

[Intermittent Failures Robot]

17 failures in 225 pushes (0.076 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 12
• try: 2
• mozilla-central: 3

Platform breakdown:

• linux1804-64: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-08-22&endday=2020-08-22&tree=all

Comment 201

[Intermittent Failures Robot]

273 failures in 4100 pushes (0.067 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 114
• mozilla-central: 38
• mozilla-esr78: 1
• try: 120

Platform breakdown:

• windows10-64: 24
• linux1804-64-qr: 16
• windows10-64-qr: 16
• linux1804-64: 182
• macosx1014-64: 12
• windows7-32: 12
• macosx1014-64-qr: 2
• android-em-7-0-x86_64: 4
• linux1804-64-asan: 2
• linux1804-64-tsan: 2
• linux1804-64-asan-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-08-17&endday=2020-08-23&tree=all

Comment 202

[Nathan Froyd [:froydnj]]

(In reply to Kartikaya Gupta (email:[e-mail]) from comment #191)

Nathan, as an MFBT person, what do you think? (see comment 185 onwards)

Comment 4 still stands, IMHO. But I guess if getting better information from the leak checker is desirable, we could go ahead and disable this assertion for a cycle or so.

Comment 203

[Kartikaya Gupta (email:[e-mail])]

I don't think disabling it for a cycle will help much. The question is more about whether the assertion provides any value in its current form at all.

Consider the case where there are callers who want to use the LinkedList code and legitimately don't care about the leaks - right now the only real option for those callers is to use a different LinkedList implementation. So from an API point of view, it seems unnecessarily limiting for the LinkedList code to have this assertion. It's unclear to me why the LinkedList code has this assertion in the first place - it can help detect leaks, but we have other mechanisms to do that as well.

(In reply to Andrew McCreight [:mccr8] from comment #188)

My theory is that some major leaks (those that include a lot of JS and DOM stuff) turn into this assertion. I would expect that they are all for the same class.

Do you know which class? If not then I think my patch may provide at least some useful information (assuming it works as I intended).

Honestly, my preference would be to stop asserting for this and let the leak checker do what it does best. Although large JS/DOM leaks tend to also turn into a single bucket.

I guess I don't fully understand this. Can you elaborate a bit? If we remove the assertion, is there information the leak checker will provide that will facilitate fixing this bug? That's what I was assuming, since the assertion causes a process crash and presuambly we don't run the leak checking diagnostic stuff after that.

Comment 204

[Andrew McCreight [:mccr8]]

(In reply to Kartikaya Gupta (email:[e-mail]) from comment #203)

Do you know which class? If not then I think my patch may provide at least some useful information (assuming it works as I intended).

I have no idea. I have some vague recollection it was some JS engine thing, but I could be wrong.

I guess I don't fully understand this. Can you elaborate a bit? If we remove the assertion, is there information the leak checker will provide that will facilitate fixing this bug? That's what I was assuming, since the assertion causes a process crash and presuambly we don't run the leak checking diagnostic stuff after that.

It is hard to say. It probably won't help. Different leak that leak the entire DOM of a web page tend to look the same to the leak checker. Sometimes we can get the URL of an individual page that is leaking, and that can help.

Comment 205

[Bob Clary [:bc] (inactive)]

This certainly seems to qualify for conversion to a non-fatal NS_ASSERTION instead of a fatal MOZ_ASSERT at least until the situation is clearer. We may have to adjust tests which check assertion counts, but it is better than a fatal assertion I think.

Comment 206

[Intermittent Failures Robot]

31 failures in 652 pushes (0.048 failures/push) were associated with this bug yesterday.

Repository breakdown:

• try: 6
• autoland: 16
• mozilla-central: 8
• mozilla-esr78: 1

Platform breakdown:

• linux1804-64: 27
• windows10-64-qr: 1
• windows10-64: 2
• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-08-24&endday=2020-08-24&tree=all

Comment 207

[Kartikaya Gupta (email:[e-mail])]

(In reply to Bob Clary [:bc] from comment #205)

This certainly seems to qualify for conversion to a non-fatal NS_ASSERTION instead of a fatal MOZ_ASSERT at least until the situation is clearer. We may have to adjust tests which check assertion counts, but it is better than a fatal assertion I think.

My understanding is that we can't use NS_ASSERTION in mfbt code, because NS_ASSERTION is defined in xpcom and there's some dependency violation if we use it in mfbt.

Comment 208

[Nathan Froyd [:froydnj]]

(In reply to Kartikaya Gupta (email:[e-mail]) from comment #207)

(In reply to Bob Clary [:bc] from comment #205)

This certainly seems to qualify for conversion to a non-fatal NS_ASSERTION instead of a fatal MOZ_ASSERT at least until the situation is clearer. We may have to adjust tests which check assertion counts, but it is better than a fatal assertion I think.

My understanding is that we can't use NS_ASSERTION in mfbt code, because NS_ASSERTION is defined in xpcom and there's some dependency violation if we use it in mfbt.

You can use it if MOZILLA_INTERNAL_API; see examples elsewhere in mfbt/.

Comment 209

[Intermittent Failures Robot]

27 failures in 778 pushes (0.035 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 24
• mozilla-central: 3

Platform breakdown:

• linux1804-64: 22
• linux1804-64-qr: 1
• windows7-32: 2
• windows10-64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-08-25&endday=2020-08-25&tree=all

Comment 210

[Intermittent Failures Robot]

18 failures in 825 pushes (0.022 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 10
• try: 6
• mozilla-central: 2

Platform breakdown:

• linux1804-64: 17
• linux1804-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-08-26&endday=2020-08-26&tree=all

Comment 211

[Kartikaya Gupta (email:[e-mail])]

My patch from comment 187 causes a crash and I don't know why. But it shows that at least in some instances the LinkedList is holding instances of type js::ParseTask. https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=b09bb71cc35359769a76a5dbc9e3f04bf31ab435

Comment 212

[Nathan Froyd [:froydnj]]

(In reply to Kartikaya Gupta (email:[e-mail]) from comment #203)

I don't think disabling it for a cycle will help much. The question is more about whether the assertion provides any value in its current form at all.

I meant insofar as we might be able to get more precise information about what's getting leaked, fix the leaks, and then turn it back on.

I suppose I'd r+ a patch to turn it off, but I should think we'd still have the leaks, and we might not have as much information about them. I see the patch in comment 187, but I don't understand "it causes a crash and I don't know why" in comment 211 -- it looks like it caused one crash in the try push, exactly as it was supposed to.

So I'd support one of two things:

1. Change the code according to the patch in comment 187 so we can get more information.
2. Remove the assert, but only if we actually get useful information from the leak checker or equivalent -- otherwise, we're getting rid of this problem and not actually moving forward to a solution.

Comment 213

[Kartikaya Gupta (email:[e-mail])]

Yeah, I think my patch (comment 187) is the way to go here, so we can split this bug by caller and file separate bugs in the buggy users.

(In reply to Nathan Froyd [:froydnj] from comment #212)

I see the patch in comment 187, but I don't understand "it causes a crash and I don't know why" in comment 211 -- it looks like it caused one crash in the try push, exactly as it was supposed to.

Indeed. My confusion was that I wasn't seeing the "<whatever> has a buggy user..." message in the treeherder summary and so I thought it was crashing while evaluating my printf string, rather than after emitting the string, which is what I wanted. But I see now that if I look at the raw job log the string is there, it just doesn't show up in the part of the log treeherder highlights as relevant. So (I guess) the patch is working fine.

Anyway, the important part is that the error message surfaced by TreeHerder does have the class being put into the LinkedList, so the patch as-is should be sufficient to accomplish what we want here, which is to allow sheriffs to classify the crashes by user code.

I'll put the patch up for review on a dependent bug.

Comment 214

[Intermittent Failures Robot]

97 failures in 4046 pushes (0.024 failures/push) were associated with this bug in the last 7 days.

This is the #3 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• try: 21
• autoland: 59
• mozilla-central: 16
• mozilla-esr78: 1

Platform breakdown:

• linux1804-64: 83
• windows10-64-qr: 1
• windows10-64: 5
• windows7-32: 3
• linux1804-64-qr: 4
• macosx1014-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-08-24&endday=2020-08-30&tree=all

Comment 215

[Intermittent Failures Robot]

5 failures in 2984 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• try: 3
• mozilla-beta: 1
• mozilla-release: 1

Platform breakdown:

• linux1804-64: 4
• macosx1014-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-08-31&endday=2020-09-06&tree=all

Comment 216

[Andreea Pavel [:apavel]]

There are no failures here since bug 1661798 got fixed.
Kartikaya, can this be closed as fixed too?

Comment 217

[Kartikaya Gupta (email:[e-mail])]

Yeah I guess so, there are now individual bugs being filled for the specific call sites that are leaking.

Comment 218

[Intermittent Failures Robot]

1 failures in 3410 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr78: 1

Platform breakdown:

• linux1804-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-09-07&endday=2020-09-13&tree=all

Comment 219

[Intermittent Failures Robot]

3 failures in 4019 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr78: 3

Platform breakdown:

• windows10-64: 2
• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-09-21&endday=2020-09-27&tree=all

Comment 220

[Intermittent Failures Robot]

1 failures in 4195 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr78: 1

Platform breakdown:

• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-09-28&endday=2020-10-04&tree=all

Comment 221

[Intermittent Failures Robot]

2 failures in 3694 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr78: 2

Platform breakdown:

• windows7-32: 1
• linux1804-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-10-12&endday=2020-10-18&tree=all

Comment 222

[Intermittent Failures Robot]

1 failures in 4275 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr78: 1

Platform breakdown:

• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-10-26&endday=2020-11-01&tree=all

Comment 223

[Intermittent Failures Robot]

1 failures in 4128 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr78: 1

Platform breakdown:

• linux1804-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1265637&startday=2020-11-02&endday=2020-11-08&tree=all

Comment 224

[Intermittent Failures Robot]

1 failures in 4040 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr78: 1

Platform breakdown:

• linux1804-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1265637&startday=2020-11-09&endday=2020-11-15&tree=all

Comment 225

[Intermittent Failures Robot]

1 failures in 4810 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr78: 1

Platform breakdown:

• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1265637&startday=2020-11-16&endday=2020-11-22&tree=all

Comment 226

[Intermittent Failures Robot]

2 failures in 3712 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr78: 2

Platform breakdown:

• windows10-64: 1
• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1265637&startday=2020-11-30&endday=2020-12-06&tree=all

Comment 227

[Intermittent Failures Robot]

1 failures in 4060 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr78: 1

Platform breakdown:

• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1265637&startday=2020-12-07&endday=2020-12-13&tree=all

Comment 228

[Intermittent Failures Robot]

1 failures in 3421 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr78: 1

Platform breakdown:

• linux1804-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1265637&startday=2020-12-14&endday=2020-12-20&tree=all

Comment 229

[Andreea Pavel [:apavel]]

There are some recent failures here for esr-78, does this need to be reopened?

Comment 230

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

No, because it is known to affect Firefox 78 but won't fixed for it (firefox-esr78: wontfix).

Comment 231

[Intermittent Failures Robot]

1 failures in 2956 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr78: 1

Platform breakdown:

• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1265637&startday=2021-01-04&endday=2021-01-10&tree=all

Comment 232

[Intermittent Failures Robot]

2 failures in 3810 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr78: 2

Platform breakdown:

• windows10-64: 1
• linux1804-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1265637&startday=2021-01-11&endday=2021-01-17&tree=all

Comment 233

[Intermittent Failures Robot]

2 failures in 3891 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr78: 2

Platform and build breakdown:

• linux1804-64: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1265637&startday=2021-01-25&endday=2021-01-31&tree=all

Comment 234

[Intermittent Failures Robot]

2 failures in 3966 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr78: 2

Platform and build breakdown:

• windows10-64: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1265637&startday=2021-02-01&endday=2021-02-07&tree=all

Comment 235

[Intermittent Failures Robot]

2 failures in 4322 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr78: 2

Platform and build breakdown:

• windows7-32: 1
  • debug: 1
• linux1804-64: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1265637&startday=2021-02-08&endday=2021-02-14&tree=all

Comment 236

[Intermittent Failures Robot]

5 failures in 4327 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr78: 5

Platform and build breakdown:

• windows10-64: 2
  • debug: 2
• linux1804-64: 3
  • debug: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1265637&startday=2021-02-22&endday=2021-02-28&tree=all

Comment 237

[Intermittent Failures Robot]

3 failures in 3683 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr78: 3

Platform and build breakdown:

• windows7-32: 1
  • debug: 1
• linux1804-64: 1
  • debug: 1
• windows10-64: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1265637&startday=2021-03-01&endday=2021-03-07&tree=all