1486101 - Intermittent Assertion failure: m_refCount == 1, at /builds/worker/workspace/build/src/js/src/jit/ExecutableAllocator.cpp:51

Status: RESOLVED FIXED

(Core :: Networking: HTTP, defect, P2)

Description

[Treeherder Bug Filer]

Filed by: apavel [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=195774502&repo=autoland

https://queue.taskcluster.net/v1/task/K6fmvHDiQFCFNgqhq58RLw/runs/0/artifacts/public/logs/live_backing.log

https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/K6fmvHDiQFCFNgqhq58RLw/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1&only_show_unexpected=1

[task 2018-08-24T17:05:43.748Z] 17:05:43     INFO - ERROR: GC found live SharedScriptData 0x7f7c65a19e40 with ref count 2 at shutdown
[task 2018-08-24T17:05:43.749Z] 17:05:43     INFO - ERROR: GC found live SharedScriptData 0x7f7c65abd000 with ref count 2 at shutdown
[task 2018-08-24T17:05:43.750Z] 17:05:43     INFO - ERROR: GC found live SharedScriptData 0x7f7c65e83100 with ref count 2 at shutdown
[task 2018-08-24T17:05:43.752Z] 17:05:43     INFO - ERROR: GC found live SharedScriptData 0x7f7c65ae1e20 with ref count 2 at shutdown
[task 2018-08-24T17:05:43.753Z] 17:05:43     INFO - ERROR: GC found live SharedScriptData 0x7f7c65bb4dc0 with ref count 2 at shutdown
[task 2018-08-24T17:05:43.754Z] 17:05:43     INFO - Assertion failure: m_refCount == 1, at /builds/worker/workspace/build/src/js/src/jit/ExecutableAllocator.cpp:51
[task 2018-08-24T17:05:43.755Z] 17:05:43     INFO - [Parent 1357, IPDL Background] WARNING: 'QuotaClient::IsShuttingDownOnBackgroundThread()', file /builds/worker/workspace/build/src/dom/indexedDB/ActorsParent.cpp, line 21965
[task 2018-08-24T17:05:43.756Z] 17:05:43     INFO - [Parent 1357, IPDL Background] WARNING: 'NS_FAILED(rv)', file /builds/worker/workspace/build/src/dom/indexedDB/ActorsParent.cpp, line 21200
[task 2018-08-24T17:05:43.757Z] 17:05:43     INFO - [Parent 1357, QuotaManager IO] WARNING: 'QuotaClient::IsShuttingDownOnNonBackgroundThread()', file /builds/worker/workspace/build/src/dom/indexedDB/ActorsParent.cpp, line 21336
[task 2018-08-24T17:05:43.759Z] 17:05:43     INFO - [Parent 1357, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /builds/worker/workspace/build/src/dom/indexedDB/ActorsParent.cpp, line 21200
[task 2018-08-24T17:05:43.760Z] 17:05:43     INFO - 1535130343531	[e-mail]	WARN	Loading extension '[e-mail]': storage.local data migration cancelled, unable to open IDB connection: The operation failed for reasons unrelated to the database itself and not covered by any other error code.::
[task 2018-08-24T17:05:43.761Z] 17:05:43     INFO - 1535130343532	[e-mail]	WARN	Loading extension '[e-mail]': JSONFile backend is being kept enabled by an unexpected IDBBackend failure: The operation failed for reasons unrelated to the database itself and not covered by any other error code.::
[task 2018-08-24T17:05:43.762Z] 17:05:43     INFO - [Parent 1357, Main Thread] WARNING: 'aRv.Failed()', file /builds/worker/workspace/build/src/dom/indexedDB/IDBDatabase.cpp, line 586
[task 2018-08-24T17:05:44.063Z] 17:05:44     INFO - 1535130344058	Marionette	DEBUG	Received observer notification xpcom-will-shutdown

[task 2018-08-24T17:05:55.009Z] 17:05:55     INFO - REFTEST PROCESS-CRASH | file:///builds/worker/workspace/build/tests/reftest/tests/layout/reftests/svg/smil/repeat/init-repeat-1.svg | application crashed [@ js::jit::ExecutablePool::release(bool)]
[task 2018-08-24T17:05:55.010Z] 17:05:55     INFO - Crash dump filename: /tmp/tmpgRWMue.mozrunner/minidumps/0c1146fb-0f3e-8620-aa34-93734a398711.dmp
[task 2018-08-24T17:05:55.011Z] 17:05:55     INFO - Operating system: Linux
[task 2018-08-24T17:05:55.011Z] 17:05:55     INFO -                   0.0.0 Linux 4.4.0-1014-aws #14taskcluster1-Ubuntu SMP Tue Apr 3 10:27:00 UTC 2018 x86_64
[task 2018-08-24T17:05:55.012Z] 17:05:55     INFO - CPU: amd64
[task 2018-08-24T17:05:55.012Z] 17:05:55     INFO -      family 6 model 62 stepping 4
[task 2018-08-24T17:05:55.013Z] 17:05:55     INFO -      2 CPUs
[task 2018-08-24T17:05:55.014Z] 17:05:55     INFO - 
[task 2018-08-24T17:05:55.017Z] 17:05:55     INFO - GPU: UNKNOWN
[task 2018-08-24T17:05:55.018Z] 17:05:55     INFO - 
[task 2018-08-24T17:05:55.018Z] 17:05:55     INFO - Crash reason:  SIGSEGV
[task 2018-08-24T17:05:55.019Z] 17:05:55     INFO - Crash address: 0x0
[task 2018-08-24T17:05:55.020Z] 17:05:55     INFO - Process uptime: not available
[task 2018-08-24T17:05:55.020Z] 17:05:55     INFO - 
[task 2018-08-24T17:05:55.020Z] 17:05:55     INFO - Thread 0 (crashed)
[task 2018-08-24T17:05:55.023Z] 17:05:55     INFO -  0  libxul.so!js::jit::ExecutablePool::release(bool) [ExecutableAllocator.cpp:9f1b27308bd6dfaa17c75720660b5bf40cd4ccab : 51 + 0x0]
[task 2018-08-24T17:05:55.023Z] 17:05:55     INFO -     rax = 0x0000000000000000   rdx = 0x0000000000000000
[task 2018-08-24T17:05:55.024Z] 17:05:55     INFO -     rcx = 0x00007f7c7e4172dd   rbx = 0x00007f7c63237330
[task 2018-08-24T17:05:55.025Z] 17:05:55     INFO -     rsi = 0x00007f7c7e6e6770   rdi = 0x00007f7c7e6e5540
[task 2018-08-24T17:05:55.025Z] 17:05:55     INFO -     rbp = 0x00007ffc190b6eb0   rsp = 0x00007ffc190b6ea0
[task 2018-08-24T17:05:55.029Z] 17:05:55     INFO -      r8 = 0x00007f7c7e6e6770    r9 = 0x00007f7c7f9c9740
[task 2018-08-24T17:05:55.030Z] 17:05:55     INFO -     r10 = 0x0000000000000000   r11 = 0x0000000000000000
[task 2018-08-24T17:05:55.030Z] 17:05:55     INFO -     r12 = 0x00007f7c66714000   r13 = 0x00007f7c6689dde0
[task 2018-08-24T17:05:55.030Z] 17:05:55     INFO -     r14 = 0x00007f7c66704800   r15 = 0x00007f7c66714688
[task 2018-08-24T17:05:55.030Z] 17:05:55     INFO -     rip = 0x00007f7c6fbdcd8e
[task 2018-08-24T17:05:55.030Z] 17:05:55     INFO -     Found by: given as instruction pointer in context
[task 2018-08-24T17:05:55.030Z] 17:05:55     INFO -  1  libxul.so!js::jit::ExecutableAllocator::~ExecutableAllocator() [ExecutableAllocator.cpp:9f1b27308bd6dfaa17c75720660b5bf40cd4ccab : 99 + 0xa]
[task 2018-08-24T17:05:55.032Z] 17:05:55     INFO -     rbx = 0x0000000000000000   rbp = 0x00007ffc190b6ed0
[task 2018-08-24T17:05:55.032Z] 17:05:55     INFO -     rsp = 0x00007ffc190b6ec0   r12 = 0x00007f7c66714000
[task 2018-08-24T17:05:55.032Z] 17:05:55     INFO -     r13 = 0x00007f7c6689dde0   r14 = 0x00007f7c66704800
[task 2018-08-24T17:05:55.032Z] 17:05:55     INFO -     r15 = 0x00007f7c66714688   rip = 0x00007f7c6fbdcebc
[task 2018-08-24T17:05:55.032Z] 17:05:55     INFO -     Found by: call frame info
[task 2018-08-24T17:05:55.032Z] 17:05:55     INFO -  2  libxul.so!JSRuntime::destroyRuntime() [Utility.h:9f1b27308bd6dfaa17c75720660b5bf40cd4ccab : 561 + 0x8]
[task 2018-08-24T17:05:55.032Z] 17:05:55     INFO -     rbx = 0x00007f7c66704800   rbp = 0x00007ffc190b6f20
[task 2018-08-24T17:05:55.032Z] 17:05:55     INFO -     rsp = 0x00007ffc190b6ee0   r12 = 0x00007f7c66714000
[task 2018-08-24T17:05:55.032Z] 17:05:55     INFO -     r13 = 0x00007f7c6689dde0   r14 = 0x00007f7c66722000
[task 2018-08-24T17:05:55.032Z] 17:05:55     INFO -     r15 = 0x00007f7c66714688   rip = 0x00007f7c700ef0c6
[task 2018-08-24T17:05:55.032Z] 17:05:55     INFO -     Found by: call frame info
[task 2018-08-24T17:05:55.032Z] 17:05:55     INFO -  3  libxul.so!js::DestroyContext(JSContext*) [JSContext.cpp:9f1b27308bd6dfaa17c75720660b5bf40cd4ccab : 213 + 0x5]
[task 2018-08-24T17:05:55.032Z] 17:05:55     INFO -     rbx = 0x00007f7c66899000   rbp = 0x00007ffc190b6f60
[task 2018-08-24T17:05:55.032Z] 17:05:55     INFO -     rsp = 0x00007ffc190b6f30   r12 = 0x00007f7c6689dd98
[task 2018-08-24T17:05:55.033Z] 17:05:55     INFO -     r13 = 0x00007f7c6689dde0   r14 = 0x00007f7c6689dd80
[task 2018-08-24T17:05:55.033Z] 17:05:55     INFO -     r15 = 0x00007f7c66722000   rip = 0x00007f7c700412ad
[task 2018-08-24T17:05:55.033Z] 17:05:55     INFO -     Found by: call frame info
[task 2018-08-24T17:05:55.034Z] 17:05:55     INFO -  4  libxul.so!mozilla::CycleCollectedJSContext::~CycleCollectedJSContext() [CycleCollectedJSContext.cpp:9f1b27308bd6dfaa17c75720660b5bf40cd4ccab : 103 + 0x5]
[task 2018-08-24T17:05:55.035Z] 17:05:55     INFO -     rbx = 0x00007f7c66899000   rbp = 0x00007ffc190b6fa0
[task 2018-08-24T17:05:55.037Z] 17:05:55     INFO -     rsp = 0x00007ffc190b6f70   r12 = 0x00007f7c6689dd98
[task 2018-08-24T17:05:55.037Z] 17:05:55     INFO -     r13 = 0x00007f7c6689dde0   r14 = 0x00007f7c6689dd80
[task 2018-08-24T17:05:55.038Z] 17:05:55     INFO -     r15 = 0x00007f7c6689de48   rip = 0x00007f7c6c6a916f
[task 2018-08-24T17:05:55.038Z] 17:05:55     INFO -     Found by: call frame info
[task 2018-08-24T17:05:55.039Z] 17:05:55     INFO -  5  libxul.so!XPCJSContext::~XPCJSContext() [XPCJSContext.cpp:9f1b27308bd6dfaa17c75720660b5bf40cd4ccab : 919 + 0x14]
[task 2018-08-24T17:05:55.040Z] 17:05:55     INFO -     rbx = 0x00007f7c66899000   rbp = 0x00007ffc190b6fd0
[task 2018-08-24T17:05:55.040Z] 17:05:55     INFO -     rsp = 0x00007ffc190b6fb0   r12 = 0x00007ffc190b7128
[task 2018-08-24T17:05:55.041Z] 17:05:55     INFO -     r13 = 0x00007ffc190b7130   r14 = 0x0000000000000000
[task 2018-08-24T17:05:55.041Z] 17:05:55     INFO -     r15 = 0x00007f7c66851d00   rip = 0x00007f7c6ced27bb
[task 2018-08-24T17:05:55.042Z] 17:05:55     INFO -     Found by: call frame info

Comment 1

[Intermittent Failures Robot]

6 failures in 2568 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 1
* mozilla-inbound: 1
* autoland: 4

Platform breakdown:
* linux64: 4
* linux32: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2018-08-20&endday=2018-08-26&tree=all

Comment 2

[Eliza Balazs [:ebalazs_]]

In the last 7 days, there are 31 failures associated with this bug.

They occur mostly on linux64 (debug), linux 32 (debug) and some on windows10-64-ccov (debug).

Recent failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=196328748&repo=mozilla-central&lineNumber=60046

:sdetar, Can you have a look into this?

Comment 3

[Jan de Mooij [:jandem]]

This is a symptom of leaking GC things at shutdown, the log has many lines like:

[task 2018-08-24T17:05:42.018Z] 17:05:42     INFO - ERROR: GC found live Cell 0x7f7c63d07040 of kind FUNCTION at shutdown
[task 2018-08-24T17:05:42.019Z] 17:05:42     INFO - ERROR: GC found live Cell 0x7f7c63d07080 of kind FUNCTION at shutdown

So what's going on here is that scripts keep JIT code alive and the executable code allocator then fails an assertion because there are still references to code pools when it's destroyed.

Jon, is this one of these "needs a shutdownCollectedEverything() call" bugs? It worries me a bit that this seems to have gotten worse recently :/

Comment 4

[Jan de Mooij [:jandem]]

Maybe we should wait for bug 1485985 comment 6.

Comment 5

[Jon Coppeard (:jonco)]

(In reply to Jan de Mooij [:jandem] from comment #3)
> Jon, is this one of these "needs a shutdownCollectedEverything() call" bugs?

Yes it is.

> It worries me a bit that this seems to have gotten worse recently :/

Yeah we need to find out what is leaking GC things, and probably make this a more serious error.

Comment 6

[Intermittent Failures Robot]

36 failures in 2545 pushes (0.014 failures/push) were associated with this bug in the last 7 days.

This is the #48 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* try: 2
* mozilla-central: 10
* mozilla-inbound: 14
* autoland: 10

Platform breakdown:
* windows7-32: 1
* linux64-qr: 5
* linux32: 18
* linux64: 7
* windows10-64-ccov: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2018-08-27&endday=2018-09-02&tree=all

Comment 7

[Intermittent Failures Robot]

22 failures in 2667 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 9
* mozilla-inbound: 6
* autoland: 7

Platform breakdown:
* windows7-32: 1
* linux32: 5
* linux64-qr: 3
* windows10-64-ccov: 5
* linux64: 4
* windows10-64: 1
* osx-10-10: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2018-09-03&endday=2018-09-09&tree=all

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Jon Coppeard (:jonco) from comment #5)
> > It worries me a bit that this seems to have gotten worse recently :/
> 
> Yeah we need to find out what is leaking GC things, and probably make this a
> more serious error.

Jon, can you or someone else please have a look at it? The ranking is quite high in Orange Factor for this bug and bug 1485983. Thanks.

Comment 9

[Jan de Mooij [:jandem]]

See also bug 1485985 comment 10...

Comment 10

[Jon Coppeard (:jonco)]

From comment 9 it appears this specific problem is caused by bug 1484496.

For the general problem of the embedding causing us to leak GC things at shutdown, there are some things we could do to help pinpoint the source:
 - dump more information about what the leaked GC things are 
 - make this show up as a build failure like embedding leaks (bug 1407593)
 - dump the GC heap when this happens
 - maybe dump the CC graph too

Comment 11

[Intermittent Failures Robot]

29 failures in 2684 pushes (0.011 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 4
* mozilla-inbound: 6
* try: 2
* mozilla-central: 17

Platform breakdown:
* linux32: 7
* windows10-64-ccov: 15
* linux64-qr: 1
* linux64: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2018-09-10&endday=2018-09-16&tree=all

Comment 12

[Intermittent Failures Robot]

23 failures in 538 pushes (0.043 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 4
* autoland: 7
* mozilla-central: 12

Platform breakdown:
* osx-10-10: 1
* linux64-qr: 2
* linux32: 10
* linux64: 7
* windows10-64-ccov: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2018-09-21&endday=2018-09-21&tree=all

Comment 13

[Andreea Pavel [:apavel]]

There is a total of 43 failures in the last 7 days. 

Jon, this still has a high occurrence rate. I saw your comment above, are you working on it or should this be assigned to someone else? 

Thanks.

Comment 14

[Intermittent Failures Robot]

42 failures in 2902 pushes (0.014 failures/push) were associated with this bug in the last 7 days.

This is the #35 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* autoland: 13
* mozilla-inbound: 9
* try: 1
* mozilla-central: 19

Platform breakdown:
* linux64-ccov: 1
* linux64-qr: 3
* windows10-64: 1
* osx-10-10: 1
* linux64: 11
* windows7-32: 1
* linux32: 18
* windows10-64-ccov: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2018-09-17&endday=2018-09-23&tree=all

Comment 15

[Jan de Mooij [:jandem]]

(In reply to Andreea Pavel [:apavel] from comment #13)
> Jon, this still has a high occurrence rate. I saw your comment above, are
> you working on it or should this be assigned to someone else? 

Jon is on PTO, but AFAIK this is still blocked on the NI from kmag in bug 1485985.

Comment 16

[Andreea Pavel [:apavel]]

Jan, thanks. 

Waiting on bug 1485985 to be fixed here.

Comment 17

[Intermittent Failures Robot]

19 failures in 2773 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 4
* mozilla-beta: 2
* mozilla-inbound: 6
* mozilla-central: 7

Platform breakdown:
* linux32: 2
* windows10-64-ccov: 3
* windows7-32: 4
* linux64: 8
* linux64-qr: 1
* windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2018-09-24&endday=2018-09-30&tree=all

Comment 18

[Intermittent Failures Robot]

16 failures in 2708 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 1
* mozilla-central: 13
* try: 1
* mozilla-inbound: 1

Platform breakdown:
* windows10-64: 1
* linux64: 1
* windows10-64-ccov: 13
* windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2018-10-01&endday=2018-10-07&tree=all

Comment 19

[Jon Coppeard (:jonco)]

(In reply to Andreea Pavel [:apavel] from comment #13)
Sorry for the slow reply.  I'm not actively working on this.

Comment 21

[Andreea Pavel [:apavel]]

(In reply to Jon Coppeard (:jonco) from comment #19)
> (In reply to Andreea Pavel [:apavel] from comment #13)
> Sorry for the slow reply.  I'm not actively working on this.

No problem, already followed progress here in comment #16

Comment 22

[Intermittent Failures Robot]

12 failures in 2740 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* try: 1
* autoland: 1
* mozilla-beta: 1
* mozilla-central: 9

Platform breakdown:
* osx-10-10: 1
* windows10-64-ccov: 9
* windows7-32: 1
* linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2018-10-08&endday=2018-10-14&tree=all

Comment 23

[Intermittent Failures Robot]

5 failures in 2759 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 5

Platform breakdown:
* windows10-64: 1
* windows10-64-ccov: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2018-10-15&endday=2018-10-21&tree=all

Comment 24

[Intermittent Failures Robot]

2 failures in 2584 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 1
* mozilla-inbound: 1

Platform breakdown:
* windows10-64: 1
* osx-10-10: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2018-10-22&endday=2018-10-28&tree=all

Comment 25

[Intermittent Failures Robot]

4 failures in 2978 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 1
* autoland: 2
* mozilla-central: 1

Platform breakdown:
* windows7-32: 2
* osx-10-10: 1
* windows10-64-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2018-10-29&endday=2018-11-04&tree=all

Comment 26

[Intermittent Failures Robot]

1 failures in 3007 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 1

Platform breakdown:
* windows10-64-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2018-11-05&endday=2018-11-11&tree=all

Comment 28

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Taking over the ni? flag for Kris from bug 1485985.

Comment 29

[Intermittent Failures Robot]

5 failures in 2917 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 5

Platform breakdown:
* windows10-64-ccov: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2018-11-12&endday=2018-11-18&tree=all

Comment 30

[Intermittent Failures Robot]

14 failures in 2387 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 1
* mozilla-central: 13

Platform breakdown:
* windows7-32: 1
* windows10-64-ccov: 13

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2018-11-19&endday=2018-11-25&tree=all

Comment 31

[Andreea Pavel [:apavel]]

Henrik are you working on this?

Comment 32

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Andreea Pavel [:apavel] from comment #31)
> Henrik are you working on this?

No, this is not my area. As you can see from my last comment we are waiting for feedback from Kris.

Comment 33

[Intermittent Failures Robot]

33 failures in 2808 pushes (0.012 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* mozilla-inbound: 1
* autoland: 7
* mozilla-central: 24
* try: 1

Platform breakdown:
* linux32: 2
* windows10-64-ccov: 21
* windows7-32: 4
* linux64: 1
* windows10-64: 4
* linux64-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2018-11-26&endday=2018-12-02&tree=all

Comment 34

[Intermittent Failures Robot]

38 failures in 1862 pushes (0.02 failures/push) were associated with this bug in the last 7 days.

This is the #22 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* try: 2
* mozilla-inbound: 9
* mozilla-central: 21
* autoland: 6

Platform breakdown:
* linux32: 4
* windows10-64-ccov: 16
* windows10-64-qr: 1
* windows10-64: 2
* osx-10-10: 2
* windows7-32: 4
* linux64: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2018-12-03&endday=2018-12-09&tree=all

Comment 35

[Stefan Hindli [:stefan_hindli]]

Over the last 7 days there are 42 failures on this bug. These happen on linux32, linux64, osx-10-10, windows10-64, windows10-64-ccov, windows7-32.

Here is the most recent log example: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=216333338&repo=mozilla-central&lineNumber=25475

Comment 36

[Intermittent Failures Robot]

17 failures in 445 pushes (0.038 failures/push) were associated with this bug yesterday.

Repository breakdown:
* autoland: 4
* mozilla-inbound: 6
* mozilla-central: 5
* try: 2

Platform breakdown:
* windows7-32: 1
* linux64: 5
* windows10-64: 2
* linux64-ccov: 1
* linux32: 1
* osx-10-10: 5
* windows10-64-ccov: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2018-12-11&endday=2018-12-11&tree=all

Comment 37

[Intermittent Failures Robot]

21 failures in 484 pushes (0.043 failures/push) were associated with this bug yesterday.

Repository breakdown:
* autoland: 12
* mozilla-inbound: 5
* try: 1
* mozilla-central: 3

Platform breakdown:
* linux64-ccov: 1
* linux32: 3
* osx-10-10: 6
* windows7-32: 3
* linux64: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2018-12-12&endday=2018-12-12&tree=all

Comment 38

[Intermittent Failures Robot]

27 failures in 454 pushes (0.059 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 9
* try: 1
* autoland: 11
* mozilla-central: 6

Platform breakdown:
* linux32: 3
* windows10-64: 2
* windows10-64-ccov: 2
* osx-10-10: 6
* linux64: 7
* linux64-ccov: 2
* windows10-64-qr: 1
* windows7-32: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2018-12-13&endday=2018-12-13&tree=all

Comment 39

[Intermittent Failures Robot]

24 failures in 504 pushes (0.048 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. ** 

Repository breakdown:
* mozilla-inbound: 6
* autoland: 11
* mozilla-central: 7

Platform breakdown:
* linux32: 2
* windows10-64-ccov: 1
* windows7-32: 3
* linux64: 4
* windows10-64: 1
* osx-10-10: 11
* linux64-ccov: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2018-12-14&endday=2018-12-14&tree=all

Comment 40

[Intermittent Failures Robot]

18 failures in 156 pushes (0.115 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 2
* mozilla-central: 8
* autoland: 8

Platform breakdown:
* windows10-64-ccov: 2
* linux64-ccov: 1
* linux64: 11
* linux32: 2
* osx-10-10: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2018-12-15&endday=2018-12-15&tree=all

Comment 41

[Intermittent Failures Robot]

128 failures in 2485 pushes (0.052 failures/push) were associated with this bug in the last 7 days.

This is the #7 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. ** 

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. ** 

Repository breakdown:
* try: 4
* autoland: 53
* mozilla-inbound: 32
* mozilla-central: 39

Platform breakdown:
* linux32: 13
* linux64-ccov: 9
* windows10-64: 7
* windows7-32: 13
* windows10-64-ccov: 13
* linux64: 38
* windows10-64-qr: 1
* osx-10-10: 34

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2018-12-10&endday=2018-12-16&tree=all

Comment 42

[Intermittent Failures Robot]

22 failures in 476 pushes (0.046 failures/push) were associated with this bug yesterday.

Repository breakdown:
* autoland: 12
* mozilla-central: 2
* mozilla-inbound: 5
* try: 3

Platform breakdown:
* osx-10-10: 7
* linux32: 5
* windows7-32: 2
* linux64: 7
* windows10-64-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2018-12-17&endday=2018-12-17&tree=all

Comment 43

[Intermittent Failures Robot]

30 failures in 505 pushes (0.059 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-central: 6
* try: 2
* autoland: 14
* mozilla-inbound: 8

Platform breakdown:
* windows7-32: 3
* windows10-64-ccov: 1
* linux32: 3
* windows10-64: 1
* linux64-ccov: 2
* linux64: 7
* osx-10-10: 13

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2018-12-18&endday=2018-12-18&tree=all

Comment 44

[Andrei Ciure[:aciure]]

There are 154 failures asociated to this bug in the last 7 days. These are occurring on all platforms, on debug build.

:sdetar can you take a look at this?

Comment 45

[Steven DeTar [:sdetar]]

It seems from the comments this is still waiting for Kris

Comment 46

[Intermittent Failures Robot]

35 failures in 479 pushes (0.073 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 9
* autoland: 12
* mozilla-central: 14

Platform breakdown:
* windows10-64-ccov: 9
* linux64-ccov: 3
* linux32: 2
* osx-10-10: 11
* linux64: 8
* windows7-32: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2018-12-19&endday=2018-12-19&tree=all

Comment 47

[Intermittent Failures Robot]

30 failures in 441 pushes (0.068 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-central: 15
* mozilla-inbound: 5
* autoland: 10

Platform breakdown:
* linux64: 4
* windows10-64: 1
* windows7-32: 2
* linux64-ccov: 4
* osx-10-10: 7
* linux32: 5
* windows10-64-ccov: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2018-12-20&endday=2018-12-20&tree=all

Comment 48

[Raul Gurzau (:RaulG)]

Hi Kris, there is a high failure rate here, can you please take a look at this?

Comment 49

[Kris Maglione [:kmag]]

It looks like this is a leak in Firefox code, not a GC issue, but I don't have anything to do with the tests that are triggering the leak.

Comment 50

[Intermittent Failures Robot]

135 failures in 2648 pushes (0.051 failures/push) were associated with this bug in the last 7 days.

This is the #8 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. ** 

Repository breakdown:
* mozilla-central: 51
* try: 6
* mozilla-inbound: 28
* autoland: 50

Platform breakdown:
* windows10-64: 2
* linux64-ccov: 9
* windows10-64-ccov: 31
* linux32: 17
* linux64: 29
* osx-10-10: 38
* windows7-32: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2018-12-17&endday=2018-12-23&tree=all

Comment 51

[Andreea Pavel [:apavel]]

Justin, can you take a look at this?

Comment 52

[Andreea Pavel [:apavel]]

There are 112 total failures in the last 7 days on linux, osx and windows, all debug build types.

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=218724268&repo=mozilla-central&lineNumber=62465

Comment 53

[Andrew McCreight [:mccr8]]

I think the basic problem here is an assertion in the JS engine is causing leaks that we normally ignore to turn into a failure. Of the 113 failures that started in the last week, maybe 1/4 are in firefox-ui-functional-local-e10s, whatever that is, and the rest are in marionette tests.

Based on bug 1485985 comment 10, this is a regression from bug 1484373.

Comment 54

[Andreea Pavel [:apavel]]

Andrew, from the fix you mean? Kris just mentioned above that he doesn't have anything to do with this.

Comment 55

[Kris Maglione [:kmag]]

It looks like this is just the side-effect of a messy shutdown. The log is full of things like:

23:46:10     INFO -  JavaScript error: resource:///actors/AboutReaderChild.jsm, line 80: NS_ERROR_UNEXPECTED:
23:46:10     INFO -  JavaScript error: chrome://global/content/process-content.js, line 74: NS_ERROR_UNEXPECTED:
23:46:10     INFO -  [Child 6612, Main Thread] WARNING: NS_ENSURE_TRUE(maybeContext) failed: file z:/build/build/src/xpcom/threads/nsThread.cpp, line 870
23:46:10     INFO -  [Child 1800, Main Thread] WARNING: '!gThread', file z:/build/build/src/xpcom/threads/nsTimerImpl.cpp, line 299
23:46:11     INFO -  [Child 7732, Main Thread] WARNING: '!mMainThread', file z:/build/build/src/xpcom/threads/nsThreadManager.cpp, line 457

After which, it's not super surprising that something doesn't get fully cleaned up.

At least for the Windows failures, it seems like this is probably a side-effect of the way we're shutting down in response to OS shutdown. We should maybe just hard kill the content processes in those cases.

Comment 56

[Intermittent Failures Robot]

17 failures in 1008 pushes (0.017 failures/push) were associated with this bug in the last 7 days.

This is the #44 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. ** 

Repository breakdown:
* mozilla-inbound: 3
* mozilla-beta: 2
* autoland: 3
* mozilla-central: 8
* try: 1

Platform breakdown:
* linux64: 2
* windows10-64-qr: 1
* windows10-64: 5
* windows10-64-ccov: 8
* windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2018-12-24&endday=2018-12-30&tree=all

Comment 57

[Cosmin Sabou [:CosminS]]

Update: this has failed 19 times in the last 7 days, only on debug builds, most of them on firefox-ui-functional tests: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2018-12-28&endday=2019-01-04&tree=all&bug=1486101

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=219924275&repo=mozilla-central&lineNumber=67368

06:18:57     INFO -  ERROR: GC found live SharedScriptData 000001CFB3869B80 with ref count 2 at shutdown
06:18:57     INFO -  ERROR: GC found live SharedScriptData 000001CFB59F6D30 with ref count 2 at shutdown
06:18:57     INFO -  ERROR: GC found live SharedScriptData 000001CFB196F580 with ref count 2 at shutdown
06:18:57     INFO -  Assertion failure: m_refCount == 1, at z:/build/build/src/js/src/jit/ExecutableAllocator.cpp:49
06:20:05     INFO - TEST-UNEXPECTED-ERROR | testing\firefox-ui\tests\functional\sessionstore\test_restore_windows_after_windows_shutdown.py TestWindowsShutdownRegisterRestart.test_manual_restart | IOError: Process still running 70s after quit request
06:20:05     INFO - Traceback (most recent call last):
06:20:05     INFO -   File "Z:\task_1546580829\build\venv\lib\site-packages\marionette_harness\marionette_test\testcases.py", line 159, in run
06:20:05     INFO -     testMethod()
06:20:05     INFO -   File "Z:\task_1546580829\build\tests\firefox-ui\tests\testing\firefox-ui\tests\functional\sessionstore\test_restore_windows_after_windows_shutdown.py", line 44, in test_manual_restart
06:20:05     INFO -     self.windows_shutdown_with_variety(restart_by_os=False, expect_restore=True)
06:20:05     INFO -   File "Z:\task_1546580829\build\tests\firefox-ui\tests\testing\firefox-ui\tests\functional\sessionstore\session_store_test_case.py", line 310, in windows_shutdown_with_variety
06:20:05     INFO -     self.marionette.quit(in_app=True, callback=lambda: self.simulate_os_shutdown())
06:20:05     INFO -   File "Z:\task_1546580829\build\venv\lib\site-packages\marionette_driver\decorators.py", line 26, in _
06:20:05     INFO -     return func(*args, **kwargs)
06:20:05     INFO -   File "Z:\task_1546580829\build\venv\lib\site-packages\marionette_driver\marionette.py", line 1128, in quit
06:20:05     INFO -     raise IOError(message.format(self.shutdown_timeout))
06:20:05     INFO - TEST-INFO took 123836ms
06:20:05    ERROR - test_end for testing\firefox-ui\tests\functional\sessionstore\test_restore_windows_after_windows_shutdown.py TestWindowsShutdownRegisterRestart.test_manual_restart logged while not in progress. Logged with data: {"status": "ERROR", "extra": {"class_name": "test_restore_windows_after_windows_shutdown.TestWindowsShutdownRegisterRestart", "method_name": "test_manual_restart"}, "expected": "PASS", "test": "testing\\firefox-ui\\tests\\functional\\sessionstore\\test_restore_windows_after_windows_shutdown.py TestWindowsShutdownRegisterRestart.test_manual_restart", "message": "InvalidSessionIdException: Please start a session", "stack": "Traceback (most recent call last):\n  File \"Z:\\task_1546580829\\build\\venv\\lib\\site-packages\\marionette_harness\\marionette_test\\testcases.py\", line 190, in run\n    self.tearDown()\n  File \"Z:\\task_1546580829\\build\\tests\\firefox-ui\\tests\\testing\\firefox-ui\\tests\\functional\\sessionstore\\session_store_test_case.py\", line 65, in tearDown\n    super(SessionStoreTestCase, self).tearDown()\n  File \"Z:\\task_1546580829\\build\\venv\\lib\\site-packages\\firefox_puppeteer\\mixins.py\", line 97, in tearDown\n    self.marionette.set_context('chrome')\n  File \"Z:\\task_1546580829\\build\\venv\\lib\\site-packages\\marionette_driver\\marionette.py\", line 1461, in set_context\n    {\"value\": context})\n  File \"Z:\\task_1546580829\\build\\venv\\lib\\site-packages\\marionette_driver\\decorators.py\", line 26, in _\n    return func(*args, **kwargs)\n  File \"Z:\\task_1546580829\\build\\venv\\lib\\site-packages\\marionette_driver\\marionette.py\", line 753, in _send_message\n    raise errors.InvalidSessionIdException(\"Please start a session\")\n"}
06:20:05     INFO - 
06:20:05     INFO - SUMMARY
06:20:05     INFO - -------
06:20:05     INFO - passed: 63
06:20:05     INFO - failed: 2
06:20:05     INFO - todo: 2
06:20:05     INFO - 
06:20:05     INFO - FAILED TESTS
06:20:05     INFO - -------
06:20:05     INFO - test_restore_windows_after_windows_shutdown.py test_restore_windows_after_windows_shutdown.TestWindowsShutdownRegisterRestart.test_manual_restart
06:20:05     INFO - test_restore_windows_after_windows_shutdown.py test_restore_windows_after_windows_shutdown.TestWindowsShutdownRegisterRestart.test_manual_restart
06:20:05     INFO - SUITE-END | took 1249s
06:20:06    ERROR - Return code: 10
06:20:06    ERROR - # TBPL FAILURE #
06:20:06  WARNING - setting return code to 2

Andrew, care to have a crack at this? Justin, could  you take a look over this or assign it to someone? Thank you

Comment 58

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Also note that the background hang monitor doesn't kick in and kills the process after around 60 seconds. But maybe that is not possible because it is blocked itself? But from the testing side we hard kill Firefox after that amount of time in Marionette tests, and here mostly session restore tests seem to cause it.

Comment 59

[Intermittent Failures Robot]

23 failures in 1668 pushes (0.014 failures/push) were associated with this bug in the last 7 days.

This is the #37 most frequent failure this week.

Repository breakdown:
* try: 1
* mozilla-inbound: 4
* autoland: 6
* mozilla-central: 12

Platform breakdown:
* windows7-32: 1
* windows10-64-qr: 1
* linux64: 4
* windows10-64: 3
* linux32: 1
* osx-10-10: 2
* windows10-64-ccov: 11

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2018-12-31&endday=2019-01-06&tree=all

Comment 60

[Andreea Pavel [:apavel]]

The failure rate here is still high, 30 total failures in the last 7 days and 304 total failures in the last 30 days.

Andrew/Justin do you have an update here?

Comment 61

[Andrew McCreight [:mccr8]]

I don't know anything about this test suite, and I'm not sure why this leak is hitting an assertion when we usually don't hit it.

Comment 62

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Andrew McCreight [:mccr8] from comment #61)

I don't know anything about this test suite, and I'm not sure why this leak is hitting an assertion when we usually don't hit it.

It's not only Firefox ui tests which are causing this assertion and crash. But also lots of mochitests on Linux triggering it. Please see the OrangeFactor history.

06:20:05 INFO - FAILED TESTS
06:20:05 INFO - -------
06:20:05 INFO - test_restore_windows_after_windows_shutdown.py test_restore_windows_after_windows_shutdown.TestWindowsShutdownRegisterRestart.test_manual_restart
06:20:05 INFO - SUITE-END | took 1249s

This test failure is specific for Firefox UI tests and is only failing on Windows 10 64bit and that for ccov debug builds. Maybe something is wrong here with the shutdown for such builds? Marco, is there something you can help with?

Comment 63

[Intermittent Failures Robot]

17 failures in 495 pushes (0.034 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-inbound: 2
• autoland: 11
• mozilla-central: 4

Platform breakdown:

• windows10-64: 1
• osx-10-10: 4
• linux64: 3
• windows7-32: 3
• linux32: 4
• windows10-64-ccov: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2019-01-10&endday=2019-01-10&tree=all

Comment 64

[Intermittent Failures Robot]

42 failures in 432 pushes (0.097 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 11
• mozilla-beta: 1
• try: 2
• mozilla-inbound: 11
• autoland: 17

Platform breakdown:

• windows10-64-ccov: 3
• linux32: 5
• windows7-32: 7
• linux64: 14
• osx-10-10: 8
• windows10-64: 3
• linux64-ccov: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2019-01-11&endday=2019-01-11&tree=all

Comment 65

[Intermittent Failures Robot]

104 failures in 2627 pushes (0.04 failures/push) were associated with this bug in the last 7 days.

This is the #6 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 43
• mozilla-beta: 1
• mozilla-central: 34
• try: 4
• mozilla-inbound: 22

Platform breakdown:

• windows10-64-qr: 2
• linux32: 19
• windows10-64: 6
• osx-10-10: 15
• linux64-ccov: 4
• linux64: 29
• windows7-32: 12
• linux64-qr: 1
• windows10-64-ccov: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2019-01-07&endday=2019-01-13&tree=all

Comment 66

[Intermittent Failures Robot]

33 failures in 381 pushes (0.087 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-inbound: 8
• autoland: 18
• mozilla-central: 6
• try: 1

Platform breakdown:

• osx-10-10: 4
• linux32: 8
• macosx64-ccov: 1
• linux64-ccov: 2
• windows10-64-ccov: 1
• windows7-32: 2
• linux64: 15

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2019-01-14&endday=2019-01-14&tree=all

Comment 67

[Intermittent Failures Robot]

30 failures in 473 pushes (0.063 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 15
• mozilla-beta: 1
• mozilla-central: 5
• mozilla-inbound: 9

Platform breakdown:

• linux64: 7
• linux32: 8
• linux64-ccov: 2
• windows10-64: 2
• windows10-64-qr: 1
• osx-10-10: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2019-01-15&endday=2019-01-15&tree=all

Comment 68

[Andreea Pavel [:apavel]]

Marco, can you please take a look at the above comment?

Comment 69

[Marco Castelluccio [:marco]]

The only thing I can think of is that shutdown for ccov builds is very slow, as the coverage counters are written out on shutdown. So the timing of the shutdown can be different in ccov builds. I wonder if the test fails in test-verify mode too.

Comment 70

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Marco Castelluccio [:marco] from comment #69)

The only thing I can think of is that shutdown for ccov builds is very slow, as the coverage counters are written out on shutdown. So the timing of the shutdown can be different in ccov builds. I wonder if the test fails in test-verify mode too.

At least the Marionette and firefox-ui tests aren't supported yet in test-verify mode. I cannot say something for mochitests.

Also the ccov failures are only a small set compared to normal debug builds which fail way more often.

Comment 71

[Intermittent Failures Robot]

37 failures in 442 pushes (0.084 failures/push) were associated with this bug yesterday.

Repository breakdown:

• try: 1
• mozilla-central: 11
• mozilla-inbound: 15
• autoland: 10

Platform breakdown:

• linux64-ccov: 2
• linux32: 7
• windows10-64: 3
• linux64: 11
• windows7-32: 3
• windows10-64-ccov: 2
• osx-10-10: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2019-01-16&endday=2019-01-16&tree=all

Comment 72

[Intermittent Failures Robot]

50 failures in 476 pushes (0.105 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 34
• try: 2
• mozilla-inbound: 6
• mozilla-central: 8

Platform breakdown:

• linux64-ccov: 2
• osx-10-10: 17
• windows10-64: 2
• windows10-64-qr: 1
• linux64: 16
• windows10-64-ccov: 2
• linux32: 5
• windows7-32: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2019-01-17&endday=2019-01-17&tree=all

Comment 73

[Intermittent Failures Robot]

20 failures in 369 pushes (0.054 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 8
• mozilla-inbound: 6
• try: 3
• mozilla-central: 3

Platform breakdown:

• linux64-qr: 1
• windows10-64: 1
• linux32: 5
• windows7-32: 1
• linux64: 8
• windows10-64-ccov: 1
• osx-10-10: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2019-01-18&endday=2019-01-18&tree=all

Comment 74

[Intermittent Failures Robot]

19 failures in 144 pushes (0.132 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-inbound: 7
• autoland: 7
• mozilla-central: 5

Platform breakdown:

• linux64: 9
• windows10-64-ccov: 2
• windows7-32: 1
• linux32: 3
• linux64-ccov: 1
• osx-10-10: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2019-01-19&endday=2019-01-19&tree=all

Comment 75

[Intermittent Failures Robot]

206 failures in 2378 pushes (0.087 failures/push) were associated with this bug in the last 7 days.

This is the #2 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 95
• try: 9
• mozilla-beta: 1
• mozilla-inbound: 60
• mozilla-central: 41

Platform breakdown:

• macosx64-ccov: 1
• windows10-64-ccov: 9
• linux32: 40
• osx-10-10: 49
• linux64-ccov: 9
• linux64-qr: 1
• windows7-32: 12
• windows10-64: 8
• windows10-64-qr: 2
• linux64: 75

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2019-01-14&endday=2019-01-20&tree=all

Comment 76

[Intermittent Failures Robot]

30 failures in 295 pushes (0.102 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 15
• mozilla-inbound: 8
• mozilla-central: 7

Platform breakdown:

• osx-10-10: 11
• linux64-qr: 1
• windows10-64-ccov: 1
• windows10-64-qr: 1
• linux64: 8
• linux64-ccov: 2
• linux32: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2019-01-21&endday=2019-01-21&tree=all

Comment 77

[Intermittent Failures Robot]

17 failures in 382 pushes (0.045 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 4
• mozilla-central: 4
• try: 2
• mozilla-inbound: 7

Platform breakdown:

• windows10-64-qr: 1
• osx-10-10: 3
• linux64: 8
• linux32: 2
• windows10-64-ccov: 2
• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2019-01-22&endday=2019-01-22&tree=all

Comment 78

[Andrei Ciure[:aciure]]

There are 139 failures associated to this bug in the last 7 days. These are hitting all platforms but only the debug builds.

:Dolske any updates on this?

Comment 79

[Intermittent Failures Robot]

27 failures in 545 pushes (0.05 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 7
• try: 4
• mozilla-inbound: 11
• autoland: 5

Platform breakdown:

• linux64: 14
• linux32: 6
• windows10-64-ccov: 2
• linux64-ccov: 1
• osx-10-10: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2019-01-24&endday=2019-01-24&tree=all

Comment 80

[Intermittent Failures Robot]

28 failures in 407 pushes (0.069 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 6
• mozilla-inbound: 7
• autoland: 15

Platform breakdown:

• windows7-32: 3
• linux64: 10
• osx-10-10: 9
• linux32: 4
• windows10-64-ccov: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2019-01-25&endday=2019-01-25&tree=all

Comment 81

[Intermittent Failures Robot]

138 failures in 2383 pushes (0.058 failures/push) were associated with this bug in the last 7 days.

This is the #4 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• try: 18
• mozilla-central: 31
• mozilla-inbound: 39
• autoland: 50

Platform breakdown:

• windows10-64: 6
• linux32: 21
• windows7-32: 10
• windows10-64-ccov: 13
• linux64: 45
• linux64-ccov: 3
• linux64-qr: 1
• windows10-64-qr: 3
• osx-10-10: 36

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2019-01-21&endday=2019-01-27&tree=all

Comment 82

[Andreea Pavel [:apavel]]

There are 118 total failures in the last 7 days, Justin can you assign someone to work on this?

We cannot disable it since this fails in different tests.

Comment 83

[Dave Townsend [:mossop]]

(In reply to Andrew McCreight [:mccr8] from comment #53)

I think the basic problem here is an assertion in the JS engine is causing
leaks that we normally ignore to turn into a failure. Of the 113 failures
that started in the last week, maybe 1/4 are in
firefox-ui-functional-local-e10s, whatever that is, and the rest are in
marionette tests.

firefox-ui tests are also run under marionette. Could this be a webdriver bug?

Comment 84

[Intermittent Failures Robot]

28 failures in 389 pushes (0.072 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 11
• autoland: 12
• mozilla-inbound: 5

Platform breakdown:

• linux64-ccov: 3
• windows10-64: 3
• linux64: 6
• osx-10-10: 7
• windows7-32: 3
• windows10-64-ccov: 3
• linux32: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2019-01-28&endday=2019-01-28&tree=all

Comment 85

[Intermittent Failures Robot]

45 failures in 478 pushes (0.094 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-inbound: 11
• autoland: 21
• mozilla-central: 13

Platform breakdown:

• linux64: 17
• osx-10-10: 12
• windows10-64-qr: 2
• linux64-ccov: 2
• linux32: 5
• windows10-64: 1
• windows10-64-ccov: 4
• windows7-32: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2019-01-29&endday=2019-01-29&tree=all

Comment 86

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: minimized Marionette test

So I was able to trace down the problem to create a minimized testcase which can be run with Marionette. It seems to be a situation when in a new tab a page load is started, but it is getting closed before the page load has been finished.

I can reproduce this crash kinda easily on MacOS when running the following command with the testcase as attached:

mach marionette test -vv --gecko-log - %testcase.py% --run-until-failure

Firefox should crash after a couple of iterations of this test. If not, repeat the command or additionally use --repeat 100 or such to run more than 30 iterations.

I hope that helps to nail down this problem. Who could actually investigate that?

Comment 87

[Intermittent Failures Robot]

38 failures in 481 pushes (0.079 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 21
• cedar: 2
• mozilla-inbound: 1
• try: 1
• mozilla-central: 13

Platform breakdown:

• windows10-64-ccov: 6
• linux32: 4
• osx-10-10: 10
• linux64-ccov: 1
• windows7-32: 3
• windows10-64: 2
• linux64: 12

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2019-01-30&endday=2019-01-30&tree=all

Comment 89

[Intermittent Failures Robot]

43 failures in 517 pushes (0.083 failures/push) were associated with this bug yesterday.

Repository breakdown:

• try: 2
• autoland: 20
• mozilla-inbound: 11
• cedar: 1
• mozilla-central: 9

Platform breakdown:

• windows7-32: 1
• linux64: 16
• linux32: 6
• linux64-ccov: 1
• osx-10-10: 17
• windows10-64-ccov: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2019-01-31&endday=2019-01-31&tree=all

Comment 90

[Justin Dolske [:Dolske]]

I hate to keep bouncing this around, but tracking down an ill-defined leak is going to be difficult for the front-end team to assist with.

Andrew, does the reduced test case in comment 86 help? I'm not sure who is best positioned to look at leaks.

Comment 91

[Intermittent Failures Robot]

45 failures in 481 pushes (0.094 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-inbound: 8
• autoland: 18
• mozilla-central: 17
• try: 2

Platform breakdown:

• osx-10-10: 11
• linux32: 4
• windows10-64: 1
• linux64-qr: 1
• linux64: 12
• windows10-64-ccov: 10
• windows7-32: 3
• linux64-ccov: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2019-02-01&endday=2019-02-01&tree=all

Comment 92

[Intermittent Failures Robot]

20 failures in 176 pushes (0.114 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-inbound: 2
• mozilla-central: 5
• autoland: 13

Platform breakdown:

• linux64: 7
• windows7-32: 3
• osx-10-10: 1
• windows10-64-ccov: 2
• windows10-64: 2
• linux32: 4
• linux64-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2019-02-02&endday=2019-02-02&tree=all

Comment 93

[Intermittent Failures Robot]

235 failures in 2628 pushes (0.089 failures/push) were associated with this bug in the last 7 days.

This is the #3 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 112
• try: 6
• mozilla-inbound: 40
• cedar: 4
• mozilla-central: 73

Platform breakdown:

• windows10-64: 12
• linux32: 29
• osx-10-10: 60
• windows10-64-ccov: 30
• windows10-64-qr: 2
• linux64: 74
• linux64-qr: 1
• windows7-32: 16
• linux64-ccov: 11

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2019-01-28&endday=2019-02-03&tree=all

Comment 94

[Intermittent Failures Robot]

63 failures in 379 pushes (0.166 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 13
• autoland: 35
• mozilla-inbound: 14
• try: 1

Platform breakdown:

• linux64: 22
• windows10-64-ccov: 6
• windows7-32: 6
• linux32: 11
• windows10-64: 4
• osx-10-10: 10
• windows10-64-qr: 2
• linux64-ccov: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2019-02-04&endday=2019-02-04&tree=all

Comment 95

[Andrew McCreight [:mccr8]]

I'll try to debug this leak.

Comment 96

[Intermittent Failures Robot]

52 failures in 442 pushes (0.118 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-inbound: 9
• autoland: 36
• mozilla-central: 7

Platform breakdown:

• windows10-64-ccov: 3
• windows10-64: 1
• linux32: 12
• windows7-32: 6
• linux64: 19
• osx-10-10: 11

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2019-02-05&endday=2019-02-05&tree=all

Comment 97

[Intermittent Failures Robot]

81 failures in 478 pushes (0.169 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-release: 1
• mozilla-central: 6
• autoland: 67
• mozilla-inbound: 7

Platform breakdown:

• windows7-32: 3
• windows10-64: 9
• linux64: 32
• windows10-64-qr: 1
• windows10-64-ccov: 1
• osx-10-10: 10
• linux32: 23
• linux64-ccov: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2019-02-06&endday=2019-02-06&tree=all

Comment 98

[Intermittent Failures Robot]

65 failures in 486 pushes (0.134 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 36
• mozilla-beta: 2
• try: 1
• mozilla-inbound: 18
• mozilla-central: 8

Platform breakdown:

• osx-10-10: 14
• windows7-32: 7
• windows10-64-ccov: 3
• windows10-64: 2
• linux64-qr: 1
• linux32: 14
• linux64-ccov: 1
• linux64: 23

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2019-02-07&endday=2019-02-07&tree=all

Comment 99

[Andrew McCreight [:mccr8]]

Attachment: Make it easier to investigate this leak

This modifies test_title.py to be whimboo's minimized test case.

It also comments out a bunch of GC assertions so that we don't assert when there's a leak.

It also makes the bloat log fatally assert after dumping out the log.

With these combination of patches, you can run the following command to make it run until there's a leak, and you'll get some bloat log info:

MOZ_DISABLE_CONTENT_SANDBOX=t XPCOM_MEM_BLOAT_LOG=1 ./mach marionette test -vv --gecko-log - testing/marionette/harness/marionette_harness/tests/unit/test_title.py --run-until-failure

You can set more env vars to get CC logs, etc.

Comment 100

[Andrew McCreight [:mccr8]]

With this patch, I was able to get a CC log of the case when we leaked. There were 3 non-garbage items in the final CC log. A nsWebBrowser, a nsXPCWrappedJS (nsIStreamListener), and a nsXPCWrappedJS (nsIInterfaceRequestor). The two XPCWJS were in the same object chain. I looked at the JS object being used to implement the WJS and I figured out that it must be a FaviconLoad from FaviconLoader.jsm. It looks like it is being used as some kind of networking listener thing. The bloat log also includes an HttpChannelChild.

On a hunch, I applied the patches from bug 1520062 (which was landed and backed out) and with that I am no longer able to reproduce the leak, so I think the leak is from hitting some error case.

Comment 101

[Andrew McCreight [:mccr8]]

This bug also seems to have had zero instances on January 23, when bug 1520062 was in the tree, so that's consistent with my theory.

Comment 102

[Intermittent Failures Robot]

55 failures in 550 pushes (0.1 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 35
• mozilla-inbound: 9
• try: 2
• mozilla-central: 9

Platform breakdown:

• osx-10-10: 9
• linux64: 20
• windows10-64-ccov: 5
• windows7-32: 5
• windows10-64: 3
• linux64-ccov: 2
• linux32: 11

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2019-02-08&endday=2019-02-08&tree=all

Comment 103

[Intermittent Failures Robot]

340 failures in 2658 pushes (0.128 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• try: 5
• mozilla-inbound: 61
• autoland: 218
• mozilla-beta: 2
• mozilla-release: 1
• mozilla-central: 53

Platform breakdown:

• windows10-64: 19
• windows10-64-qr: 3
• linux64: 122
• linux64-qr: 1
• windows7-32: 29
• linux64-ccov: 7
• windows10-64-ccov: 27
• linux32: 76
• osx-10-10: 56

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2019-02-04&endday=2019-02-10&tree=all

Comment 104

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Thanks a lot Andrew! That sounds promising.

Comment 105

[Honza Bambas (:mayhemer)]

Andrew, thanks. I will duplicate it then to get this off our triage list.

Comment 106

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The test failures are still present and as such we should allow sheriffs to easily classify failures. Otherwise we might also see spreading failures across different bugs. If you don't mind I would like to keep this open, given that it is such a high frequent failure.

Note that we can remove the P1 which should also already help to get it off your triage list.

Comment 107

[Honza Bambas (:mayhemer)]

Oh, I didn't realize, sorry for that. Marking as P2 (to actually remove it from our triage list) which is the same priority as the blocker.

Comment 108

[Andrew McCreight [:mccr8]]

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+1] from comment #104)

Thanks a lot Andrew! That sounds promising.

Thanks for the minimized test case. It would have been much harder to investigate without it.

Comment 110

[Intermittent Failures Robot]

60 failures in 2832 pushes (0.021 failures/push) were associated with this bug in the last 7 days.

This is the #24 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 27
• mozilla-central: 10
• try: 4
• mozilla-esr60: 1
• mozilla-inbound: 18

Platform breakdown:

• linux64: 6
• windows7-32: 7
• windows10-64-qr: 8
• windows10-64: 7
• linux32: 23
• osx-10-10: 8
• windows10-64-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2019-02-11&endday=2019-02-17&tree=all

Comment 111

[Andreea Pavel [:apavel]]

There are 24 total failures in the last 7 days and 702 total failures in the last 30 days on all platforms, only debug build types.

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=229899897&repo=autoland&lineNumber=14338

[task 2019-02-22T14:18:25.055Z] 14:18:25 INFO - TEST-START | toolkit/components/extensions/test/mochitest/test-oop-extensions/test_ext_idle.html
[task 2019-02-22T14:18:25.074Z] 14:18:25 INFO - GECKO(3526) | ERROR: GC found live Cell 0xf74575e7040 of kind FUNCTION at shutdown
[task 2019-02-22T14:18:25.074Z] 14:18:25 INFO - GECKO(3526) | ERROR: GC found live Cell 0xf74575e7080 of kind FUNCTION at shutdown
[task 2019-02-22T14:18:25.074Z] 14:18:25 INFO - GECKO(3526) | ERROR: GC found live Cell 0xf74575e70c0 of kind FUNCTION at shutdown
[task 2019-02-22T14:18:25.074Z] 14:18:25 INFO - GECKO(3526) | ERROR: GC found live Cell 0xf74575e7100 of kind FUNCTION at shutdown
[task 2019-02-22T14:18:26.485Z] 14:18:26 INFO - GECKO(3526) | ERROR: GC found live SharedScriptData 0x7efc2a670900 with ref count 2 at shutdown
[task 2019-02-22T14:18:26.486Z] 14:18:26 INFO - GECKO(3526) | Assertion failure: m_refCount == 1, at /builds/worker/workspace/build/src/js/src/jit/ExecutableAllocator.cpp:49
[task 2019-02-22T14:18:26.487Z] 14:18:26 INFO - GECKO(3526) | [Parent 3526, Main Thread] WARNING: Cannot set transparency mode on non-popup windows.: file /builds/worker/workspace/build/src/widget/gtk/nsWindow.cpp, line 4059
[task 2019-02-22T14:18:26.487Z] 14:18:26 INFO - GECKO(3526) | [Parent 3526, Main Thread] WARNING: Cannot set transparency mode on non-popup windows.: file /builds/worker/workspace/build/src/widget/gtk/nsWindow.cpp, line 4059
[task 2019-02-22T14:18:26.488Z] 14:18:26 INFO - GECKO(3526) | ++DOMWINDOW == 9 (0x7efc65285000) [pid = 3639] [serial = 128] [outer = 0x7efc7fecf800]
[task 2019-02-22T14:18:26.651Z] 14:18:26 INFO - GECKO(3526) | Console message: [JavaScript Warning: "The character encoding of a framed document was not declared. The document may appear different if viewed without the document framing it." {file: "http://mochi.test:8888/tests/toolkit/components/extensions/test/mochitest/test-oop-extensions/test_ext_idle.html" line: 0}]
[task 2019-02-22T14:18:26.652Z] 14:18:26 INFO - GECKO(3526) | Console message: Warning: attempting to write 5552 bytes to preference extensions.webextensions.uuids. This is bad for general performance and memory usage. Such an amount of data should rather be written to an external file. This preference will not be sent to any content processes.
[task 2019-02-22T14:18:26.693Z] 14:18:26 INFO - GECKO(3526) | [Parent 3526, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80040111: file /builds/worker/workspace/build/src/netwerk/protocol/res/SubstitutingProtocolHandler.cpp, line 314
[task 2019-02-22T14:18:26.757Z] 14:18:26 INFO - GECKO(3526) | [Parent 3526, Main Thread] WARNING: Need TabChild to get the nativeWindow from!: file /builds/worker/workspace/build/src/widget/PuppetWidget.cpp, line 1078
[task 2019-02-22T14:18:26.761Z] 14:18:26 INFO - GECKO(3526) | ++DOCSHELL 0x7f98535a5000 == 18 [pid = 3619] [id = {79c099e3-272e-4e07-a27a-fc6c2cf57e7d}]
[task 2019-02-22T14:18:26.763Z] 14:18:26 INFO - GECKO(3526) | ++DOMWINDOW == 55 (0x7f985358e400) [pid = 3619] [serial = 176] [outer = (nil)]
[task 2019-02-22T14:18:26.765Z] 14:18:26 INFO - GECKO(3526) | [Child 3619, Main Thread] WARNING: Fallback to BasicLayerManager: file /builds/worker/workspace/build/src/dom/ipc/TabChild.cpp, line 2586
task 2019-02-22T14:18:37.338Z] 14:18:37 INFO - GECKO(3526) | MEMORY STAT | vsize 1497MB | residentFast 139MB | heapAllocated 14MB
[task 2019-02-22T14:18:37.357Z] 14:18:37 INFO - Buffered messages logged at 14:18:26
[task 2019-02-22T14:18:37.357Z] 14:18:37 INFO - AddTask.js | Entering test testWithRealIdleService
[task 2019-02-22T14:18:37.357Z] 14:18:37 INFO - Extension loaded
[task 2019-02-22T14:18:37.357Z] 14:18:37 INFO - Setting interval to 152
[task 2019-02-22T14:18:37.357Z] 14:18:37 INFO - TEST-PASS | toolkit/components/extensions/test/mochitest/test-oop-extensions/test_ext_idle.html | Idle status is active - Expected: active, Actual: active
[task 2019-02-22T14:18:37.359Z] 14:18:37 INFO - Buffered messages logged at 14:18:27
[task 2019-02-22T14:18:37.359Z] 14:18:37 INFO - Listener added
[task 2019-02-22T14:18:37.359Z] 14:18:37 INFO - Buffered messages logged at 14:18:37
[task 2019-02-22T14:18:37.360Z] 14:18:37 INFO - TEST-PASS | toolkit/components/extensions/test/mochitest/test-oop-extensions/test_ext_idle.html | listener fired with the expected state - Expected: idle, Actual: idle
[task 2019-02-22T14:18:37.360Z] 14:18:37 INFO - Listener fired
[task 2019-02-22T14:18:37.363Z] 14:18:37 INFO - TEST-PASS | toolkit/components/extensions/test/mochitest/test-oop-extensions/test_ext_idle.html | Idle status is idle - Expected: idle, Actual: idle
[task 2019-02-22T14:18:37.365Z] 14:18:37 INFO - TEST-PASS | toolkit/components/extensions/test/mochitest/test-oop-extensions/test_ext_idle.html | idle
[task 2019-02-22T14:18:37.367Z] 14:18:37 INFO - TEST-PASS | toolkit/components/extensions/test/mochitest/test-oop-extensions/test_ext_idle.html | test result correct
[task 2019-02-22T14:18:37.373Z] 14:18:37 INFO - AddTask.js | Leaving test testWithRealIdleService
[task 2019-02-22T14:18:37.373Z] 14:18:37 INFO - Buffered messages finished
[task 2019-02-22T14:18:37.374Z] 14:18:37 INFO - TEST-UNEXPECTED-ERROR | toolkit/components/extensions/test/mochitest/test-oop-extensions/test_ext_idle.html | unexpected-crash-dump-found - This test left crash dumps behind, but we weren't expecting it to!
[task 2019-02-22T14:18:37.376Z] 14:18:37 INFO - Found unexpected crash dump file /tmp/tmp9g3QJx.mozrunner/minidumps/051c3f0d-70e0-f358-553b-891fb40e584a.dmp.
[task 2019-02-22T14:18:37.377Z] 14:18:37 INFO - Found unexpected crash dump file /tmp/tmp9g3QJx.mozrunner/minidumps/051c3f0d-70e0-f358-553b-891fb40e584a.extra.
[task 2019-02-22T14:18:37.377Z] 14:18:37 INFO - TEST-UNEXPECTED-CRASH | toolkit/components/extensions/test/mochitest/test-oop-extensions/test_ext_idle.html | Finished in 12302ms
[task 2019-02-22T14:18:37.378Z] 14:18:37 INFO - {u'runtime': 12302}
[task 2019-02-22T14:18:37.378Z] 14:18:37 INFO - TEST-INFO took 12302ms
[task 2019-02-22T14:18:37.420Z] 14:18:37 INFO - GECKO(3526) | ++DOMWINDOW == 7 (0x7efc65281800) [pid = 3639] [serial = 129] [outer = 0x7efc7fecf800]
[task 2019-02-22T14:18:37.436Z] 14:18:37 INFO - GECKO(3526) | [Parent 3526, Main Thread] WARNING: Cannot set transparency mode on non-popup windows.: file /builds/worker/workspace/build/src/widget/gtk/nsWindow.cpp, line 4059

Andrew are there updates here? Failure rate seems to have decreased starting with ~18th of February.

Comment 112

[Intermittent Failures Robot]

19 failures in 2602 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 8
• mozilla-central: 4
• mozilla-inbound: 5
• try: 1
• mozilla-beta: 1

Platform breakdown:

• windows10-64-qr: 4
• windows10-64-ccov: 1
• windows7-32: 4
• windows10-64: 5
• linux32: 2
• linux64: 2
• linux64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2019-02-18&endday=2019-02-24&tree=all

Comment 113

[Andrew McCreight [:mccr8]]

See comment 100. Bug 1520062 landed and there have been no failures more recently than 2/22.

Comment 114

[Andreea Pavel [:apavel]]

(In reply to Andrew McCreight [:mccr8] from comment #113)

See comment 100. Bug 1520062 landed and there have been no failures more recently than 2/22.

Yes, that is correct, I posted the comment on the 22nd.

Comment 115

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Closing as fixed given that no more failures are present in the last 3 days on trunk. Beta is still affected but has a very low number of intermittents, so maybe a wontfix.

Comment 116

[Intermittent Failures Robot]

3 failures in 2981 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 3

Platform breakdown:

• windows10-64: 1
• windows7-32: 1
• windows10-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2019-02-25&endday=2019-03-03&tree=all

Comment 117

[Intermittent Failures Robot]

3 failures in 2818 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 3

Platform breakdown:

• linux64-qr: 1
• windows10-64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1486101&startday=2019-03-04&endday=2019-03-10&tree=all

Comment 118

[Mats Palmgren (inactive)]

Fwiw, I still see this assertion fairly often on Try, e.g.:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=83a6688560d512a4095a7497a412cf58c52109e0&selectedJob=243543274

Comment 119

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Mats Palmgren (:mats) from comment #118)

Fwiw, I still see this assertion fairly often on Try, e.g.:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=83a6688560d512a4095a7497a412cf58c52109e0&selectedJob=243543274

Can you please file a new bug, and then also classify the failure? That way we could see how often it actually fails on try. Thanks.