1401883 - Intermittent leakcheck | default process: 176 bytes leaked (Mutex, nsBaseAppShell)

Status: RESOLVED FIXED

(Core :: Widget, defect, P3)

Description

[Treeherder Bug Filer]

Filed by: archaeopteryx [at] coole-files.de

https://treeherder.mozilla.org/logviewer.html#?job_id=132405319&repo=autoland

https://queue.taskcluster.net/v1/task/alkWkeAoRoqKt4aUNZi1Hg/runs/0/artifacts/public/logs/live_backing.log

06:58:46     INFO -  365 INFO SimpleTest START
06:58:46     INFO -  366 INFO TEST-START | layout/mathml/tests/test_disabled_chrome.html
06:58:46     INFO -  GECKO(10188) | ++DOMWINDOW == 20 (000001CECBC32000) [pid = 10188] [serial = 20] [outer = 000001CECBC19800]
06:58:47     INFO -  GECKO(10188) | ++DOMWINDOW == 21 (000001CEC5B18800) [pid = 10188] [serial = 21] [outer = 000001CECBC19800]
06:58:47     INFO -  GECKO(10188) | ++DOCSHELL 000001CEC5B20000 == 8 [pid = 10188] [id = {3e43cb67-9bdd-48c0-bbed-a203c015ec46}]
06:58:47     INFO -  GECKO(10188) | ++DOMWINDOW == 22 (000001CEC5B20800) [pid = 10188] [serial = 22] [outer = 0000000000000000]
06:58:47     INFO -  GECKO(10188) | ++DOMWINDOW == 23 (000001CEC5B21000) [pid = 10188] [serial = 23] [outer = 000001CEC5B20800]
06:58:47     INFO -  GECKO(10188) | ++DOCSHELL 000001CEC5B2C000 == 9 [pid = 10188] [id = {684bcc33-2689-4d3b-95d1-a6f856936448}]
06:58:47     INFO -  GECKO(10188) | ++DOMWINDOW == 24 (000001CEC5B2C800) [pid = 10188] [serial = 24] [outer = 0000000000000000]
06:58:47     INFO -  GECKO(10188) | ++DOMWINDOW == 25 (000001CEC5B2E800) [pid = 10188] [serial = 25] [outer = 000001CEC5B2C800]
06:58:47     INFO -  GECKO(10188) | [10188, Main Thread] WARNING: NS_ENSURE_TRUE(aSecondURI) failed: file z:/build/build/src/dom/base/ThirdPartyUtil.cpp, line 98
06:58:48     INFO -  GECKO(10188) | ++DOMWINDOW == 26 (000001CEC5E65000) [pid = 10188] [serial = 26] [outer = 000001CEC5B2C800]
06:58:48     INFO -  GECKO(10188) | MEMORY STAT | vsize 1813MB | vsizeMaxContiguous 132289253MB | residentFast 294MB | heapAllocated 123MB
06:58:48     INFO -  367 INFO TEST-OK | layout/mathml/tests/test_disabled_chrome.html | took 1206ms
06:58:48     INFO -  368 INFO None369 INFO TEST-START | Shutdown
06:58:48     INFO -  370 INFO Passed:  3
06:58:48     INFO -  371 INFO Failed:  0
06:58:48     INFO -  372 INFO Todo:    0
06:58:48     INFO -  373 INFO Mode:    non-e10s
06:58:48     INFO -  374 INFO Slowest: 1206ms - chrome://mochitests/content/chrome/layout/mathml/tests/test_disabled_chrome.html
06:58:48     INFO -  375 INFO SimpleTest FINISHED
06:58:48     INFO -  376 INFO TEST-INFO | Ran 1 Loops
06:58:48     INFO -  377 INFO SimpleTest FINISHED
06:58:48     INFO -  GECKO(10188) | --DOCSHELL 000001CEC17E0000 == 8 [pid = 10188] [id = {4bf49a39-dcd7-4ec5-aa0d-53cf4ff6ceaf}]
06:58:48     INFO -  GECKO(10188) | *** UTM:SVC TimerManager:registerTimer called after profile-before-change notification. Ignoring timer registration for id: telemetry_modules_ping
06:58:49     INFO -  GECKO(10188) | [10188, Main Thread] WARNING: NS_ENSURE_TRUE(maybeContext) failed: file z:/build/build/src/xpcom/threads/nsThread.cpp, line 797
06:58:49     INFO -  GECKO(10188) | --DOCSHELL 000001CEBD8D2800 == 7 [pid = 10188] [id = {d1d3a87d-3d81-46c0-ad9a-8f9f60f519d1}]
06:58:49     INFO -  GECKO(10188) | --DOCSHELL 000001CEC100C000 == 6 [pid = 10188] [id = {4b741d05-b6dc-4d23-a5f0-b2d0ed54d0ea}]
06:58:49     INFO -  GECKO(10188) | --DOCSHELL 000001CEC2705800 == 5 [pid = 10188] [id = {75b79781-89b2-494d-8491-17986e13e5d6}]
06:58:49     INFO -  GECKO(10188) | --DOCSHELL 000001CECB74A800 == 4 [pid = 10188] [id = {b53f2ea2-ba5a-4326-bbb4-41c3272c1e15}]
06:58:49     INFO -  GECKO(10188) | --DOCSHELL 000001CEC5B2C000 == 3 [pid = 10188] [id = {684bcc33-2689-4d3b-95d1-a6f856936448}]
06:58:49     INFO -  GECKO(10188) | --DOCSHELL 000001CEC1260000 == 2 [pid = 10188] [id = {f6f356a5-bcfe-44f3-9389-d4325d468de3}]
06:58:49     INFO -  GECKO(10188) | --DOCSHELL 000001CEC5B20000 == 1 [pid = 10188] [id = {3e43cb67-9bdd-48c0-bbed-a203c015ec46}]
06:58:49     INFO -  GECKO(10188) | --DOCSHELL 000001CEC3AB3800 == 0 [pid = 10188] [id = {d84f3041-290a-49ba-b0d5-a4f7a62d012f}]
06:58:50     INFO -  GECKO(10188) | --DOMWINDOW == 25 (000001CEC1261800) [pid = 10188] [serial = 6] [outer = 0000000000000000] [url = about:blank]
06:58:50     INFO -  GECKO(10188) | --DOMWINDOW == 24 (000001CEC5E65000) [pid = 10188] [serial = 26] [outer = 0000000000000000] [url = http://mochi.test:8888/chrome/layout/mathml/tests/mathml_example_test.html]
06:58:50     INFO -  GECKO(10188) | --DOMWINDOW == 23 (000001CEC5B2E800) [pid = 10188] [serial = 25] [outer = 0000000000000000] [url = about:blank]
06:58:50     INFO -  GECKO(10188) | --DOMWINDOW == 22 (000001CEC5B2C800) [pid = 10188] [serial = 24] [outer = 0000000000000000] [url = http://mochi.test:8888/chrome/layout/mathml/tests/mathml_example_test.html]
06:58:50     INFO -  GECKO(10188) | --DOMWINDOW == 21 (000001CEBD8D3000) [pid = 10188] [serial = 1] [outer = 0000000000000000] [url = chrome://gfxsanity/content/sanityparent.html]
06:58:50     INFO -  GECKO(10188) | --DOMWINDOW == 20 (000001CEBD8D4000) [pid = 10188] [serial = 2] [outer = 0000000000000000] [url = about:blank]
06:58:50     INFO -  GECKO(10188) | --DOMWINDOW == 19 (000001CEC4A32000) [pid = 10188] [serial = 15] [outer = 0000000000000000] [url = about:blank]
06:58:50     INFO -  GECKO(10188) | --DOMWINDOW == 18 (000001CEC49DA000) [pid = 10188] [serial = 14] [outer = 0000000000000000] [url = about:blank]
06:58:50     INFO -  GECKO(10188) | --DOMWINDOW == 17 (000001CEC5B21000) [pid = 10188] [serial = 23] [outer = 0000000000000000] [url = chrome://mochitests/content/chrome/layout/mathml/tests/mathml_example_test.html]
06:58:50     INFO -  GECKO(10188) | --DOMWINDOW == 16 (000001CEC5B20800) [pid = 10188] [serial = 22] [outer = 0000000000000000] [url = chrome://mochitests/content/chrome/layout/mathml/tests/mathml_example_test.html]
06:58:50     INFO -  GECKO(10188) | --DOMWINDOW == 15 (000001CEC100C800) [pid = 10188] [serial = 3] [outer = 0000000000000000] [url = resource://gre-resources/hiddenWindow.html]
06:58:50     INFO -  GECKO(10188) | --DOMWINDOW == 14 (000001CEC100D800) [pid = 10188] [serial = 4] [outer = 0000000000000000] [url = about:blank]
06:58:50     INFO -  GECKO(10188) | --DOMWINDOW == 13 (000001CEC5B18800) [pid = 10188] [serial = 21] [outer = 0000000000000000] [url = chrome://mochitests/content/chrome/layout/mathml/tests/test_disabled_chrome.html]
06:58:50     INFO -  GECKO(10188) | --DOMWINDOW == 12 (000001CEBDC59000) [pid = 10188] [serial = 11] [outer = 0000000000000000] [url = chrome://gfxsanity/content/sanitytest.html]
06:58:50     INFO -  GECKO(10188) | --DOMWINDOW == 11 (000001CECBC32000) [pid = 10188] [serial = 20] [outer = 0000000000000000] [url = about:blank]
06:58:50     INFO -  GECKO(10188) | --DOMWINDOW == 10 (000001CEC1260800) [pid = 10188] [serial = 5] [outer = 0000000000000000] [url = chrome://browser/content/browser.xul]
06:58:50     INFO -  GECKO(10188) | --DOMWINDOW == 9 (000001CECBC1C800) [pid = 10188] [serial = 19] [outer = 0000000000000000] [url = about:blank]
06:58:50     INFO -  GECKO(10188) | --DOMWINDOW == 8 (000001CECBC19800) [pid = 10188] [serial = 18] [outer = 0000000000000000] [url = chrome://mochitests/content/chrome/layout/mathml/tests/test_disabled_chrome.html]
06:58:50     INFO -  GECKO(10188) | --DOMWINDOW == 7 (000001CEC271C800) [pid = 10188] [serial = 10] [outer = 0000000000000000] [url = about:blank]
06:58:50     INFO -  GECKO(10188) | --DOMWINDOW == 6 (000001CEC2706800) [pid = 10188] [serial = 9] [outer = 0000000000000000] [url = chrome://gfxsanity/content/sanitytest.html]
06:58:50     INFO -  GECKO(10188) | [10188, Main Thread] WARNING: '!mMainThread', file z:/build/build/src/xpcom/threads/nsThreadManager.cpp, line 396
06:58:50     INFO -  GECKO(10188) | [10188, Main Thread] WARNING: '!mMainThread', file z:/build/build/src/xpcom/threads/nsThreadManager.cpp, line 396
06:58:50     INFO -  GECKO(10188) | [10188, Main Thread] WARNING: '!mMainThread', file z:/build/build/src/xpcom/threads/nsThreadManager.cpp, line 396
06:58:50     INFO -  GECKO(10188) | [10188, Main Thread] WARNING: '!mMainThread', file z:/build/build/src/xpcom/threads/nsThreadManager.cpp, line 396
06:58:50     INFO -  GECKO(10188) | [10188, Main Thread] WARNING: '!mMainThread', file z:/build/build/src/xpcom/threads/nsThreadManager.cpp, line 396
06:58:50     INFO -  GECKO(10188) | [10188, Main Thread] WARNING: '!mMainThread', file z:/build/build/src/xpcom/threads/nsThreadManager.cpp, line 396
06:58:50     INFO -  GECKO(10188) | [10188, Main Thread] WARNING: '!mMainThread', file z:/build/build/src/xpcom/threads/nsThreadManager.cpp, line 396
06:58:50     INFO -  GECKO(10188) | [10188, Main Thread] WARNING: '!mMainThread', file z:/build/build/src/xpcom/threads/nsThreadManager.cpp, line 396
06:58:50     INFO -  GECKO(10188) | --DOMWINDOW == 5 (000001CEC3ABE800) [pid = 10188] [serial = 13] [outer = 0000000000000000] [url = about:blank]
06:58:50     INFO -  GECKO(10188) | --DOMWINDOW == 4 (000001CECA3E0800) [pid = 10188] [serial = 17] [outer = 0000000000000000] [url = chrome://mochikit/content/harness.xul]
06:58:50     INFO -  GECKO(10188) | --DOMWINDOW == 3 (000001CEC3AB4800) [pid = 10188] [serial = 12] [outer = 0000000000000000] [url = chrome://mochikit/content/harness.xul]
06:58:50     INFO -  GECKO(10188) | [10188, Main Thread] WARNING: '!mMainThread', file z:/build/build/src/xpcom/threads/nsThreadManager.cpp, line 396
06:58:50     INFO -  GECKO(10188) | [10188, Main Thread] WARNING: '!mMainThread', file z:/build/build/src/xpcom/threads/nsThreadManager.cpp, line 396
06:58:50     INFO -  GECKO(10188) | [10188, Main Thread] WARNING: '!mMainThread', file z:/build/build/src/xpcom/threads/nsThreadManager.cpp, line 396
06:58:50     INFO -  GECKO(10188) | [10188, Main Thread] WARNING: '!mMainThread', file z:/build/build/src/xpcom/threads/nsThreadManager.cpp, line 396
06:58:50     INFO -  GECKO(10188) | [10188, Main Thread] WARNING: '!mMainThread', file z:/build/build/src/xpcom/threads/nsThreadManager.cpp, line 396
06:58:50     INFO -  GECKO(10188) | [10188, Main Thread] WARNING: '!mMainThread', file z:/build/build/src/xpcom/threads/nsThreadManager.cpp, line 396
06:58:50     INFO -  GECKO(10188) | [10188, Main Thread] WARNING: '!mMainThread', file z:/build/build/src/xpcom/threads/nsThreadManager.cpp, line 396
06:58:50     INFO -  GECKO(10188) | [10188, Main Thread] WARNING: '!mMainThread', file z:/build/build/src/xpcom/threads/nsThreadManager.cpp, line 396
06:58:50     INFO -  GECKO(10188) | --DOMWINDOW == 2 (000001CEC17E0800) [pid = 10188] [serial = 8] [outer = 0000000000000000] [url = about:blank]
06:58:50     INFO -  GECKO(10188) | --DOMWINDOW == 1 (000001CEC8E4C800) [pid = 10188] [serial = 16] [outer = 0000000000000000] [url = http://mochi.test:8888/redirect.html]
06:58:50     INFO -  GECKO(10188) | --DOMWINDOW == 0 (000001CEC1567000) [pid = 10188] [serial = 7] [outer = 0000000000000000] [url = resource://gre-resources/hiddenWindow.html]
06:58:50     INFO -  GECKO(10188) | [10188, Main Thread] WARNING: OOPDeinit() without successful OOPInit(): file z:/build/build/src/toolkit/crashreporter/nsExceptionHandler.cpp, line 3562
06:58:50     INFO -  GECKO(10188) | nsStringStats
06:58:50     INFO -  GECKO(10188) |  => mAllocCount:          81528
06:58:50     INFO -  GECKO(10188) |  => mReallocCount:         6717
06:58:50     INFO -  GECKO(10188) |  => mFreeCount:           81528
06:58:50     INFO -  GECKO(10188) |  => mShareCount:         104859
06:58:50     INFO -  GECKO(10188) |  => mAdoptCount:           5495
06:58:50     INFO -  GECKO(10188) |  => mAdoptFreeCount:       5495
06:58:50     INFO -  GECKO(10188) |  => Process ID: 10188, Thread ID: 928
06:58:50     INFO -  TEST-INFO | Main app process: exit 0
06:58:50     INFO -  runtests.py | Application ran for: 0:00:08.954000
06:58:50     INFO -  zombiecheck | Reading PID log: c:\users\genericworker\appdata\local\temp\tmpmpcq2zpidlog
06:58:50     INFO -  Stopping web server
06:58:50     INFO -  Stopping web socket server
06:58:50     INFO -  Stopping ssltunnel
06:58:50     INFO -  TEST-INFO | leakcheck | default process: leak threshold set at 0 bytes
06:58:50     INFO -  TEST-INFO | leakcheck | plugin process: leak threshold set at 0 bytes
06:58:50     INFO -  TEST-INFO | leakcheck | tab process: leak threshold set at 10000 bytes
06:58:50     INFO -  TEST-INFO | leakcheck | geckomediaplugin process: leak threshold set at 20000 bytes
06:58:50     INFO -  TEST-INFO | leakcheck | gpu process: leak threshold set at 0 bytes
06:58:50     INFO -  == BloatView: ALL (cumulative) LEAK AND BLOAT STATISTICS, default process 10188
06:58:50     INFO -       |<----------------Class--------------->|<-----Bytes------>|<----Objects---->|
06:58:50     INFO -       |                                      | Per-Inst   Leaked|   Total      Rem|
06:58:50     INFO -     0 |TOTAL                                 |       33      176| 1163016        2|
06:58:50     INFO -   446 |Mutex                                 |       80       80|    1748        1|
06:58:50     INFO -   913 |nsBaseAppShell                        |       96       96|       1        1|
06:58:50     INFO -  nsTraceRefcnt::DumpStatistics: 1464 entries
06:58:50     INFO -  TEST-INFO | leakcheck | default process: leaked 1 Mutex
06:58:50     INFO -  TEST-INFO | leakcheck | default process: leaked 1 nsBaseAppShell
06:58:50    ERROR -  378 ERROR TEST-UNEXPECTED-FAIL | leakcheck | default process: 176 bytes leaked (Mutex, nsBaseAppShell)

Comment 1

[Intermittent Failures Robot]

11 failures in 943 pushes (0.012 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-beta: 5
* autoland: 5
* mozilla-inbound: 1

Platform breakdown:
* windows10-64-stylo-disabled: 6
* windows10-64: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1401883&startday=2017-09-18&endday=2017-09-24&tree=all

Comment 2

[Intermittent Failures Robot]

22 failures in 885 pushes (0.025 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 13
* mozilla-inbound: 5
* try: 2
* mozilla-beta: 2

Platform breakdown:
* windows10-64-stylo-disabled: 17
* windows10-64: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1401883&startday=2017-09-25&endday=2017-10-01&tree=all

Comment 3

[Intermittent Failures Robot]

33 failures in 824 pushes (0.04 failures/push) were associated with this bug in the last 7 days.   

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 17
* mozilla-inbound: 12
* mozilla-central: 3
* try: 1

Platform breakdown:
* windows10-64-stylo-disabled: 25
* windows10-64: 8

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1401883&startday=2017-10-02&endday=2017-10-08&tree=all

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

this is in mochitest-chrome and appears to be in many different directories- this makes it harder to track the leak down.

:gbrown, can you help track this down?

Comment 5

[Geoff Brown [:gbrown]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=72fa9760204dcb1f3255db15c02f809ee9ac5aec

The nsBaseAppShell dtor would normally be called like so:

22:15:48     INFO -  GECKO(7484) | #01: nsBaseAppShell::Release() [widget/nsBaseAppShell.cpp:22]
22:15:48     INFO -  GECKO(7484) | #02: nsAppStartup::~nsAppStartup() [toolkit/components/startup/nsAppStartup.h:46]
22:15:48     INFO -  GECKO(7484) | #03: nsAppStartup::Release() [toolkit/components/startup/nsAppStartup.cpp:230]
22:15:48     INFO -  GECKO(7484) | #04: ScopedXPCOMStartup::~ScopedXPCOMStartup() [toolkit/xre/nsAppRunner.cpp:1524]
22:15:48     INFO -  GECKO(7484) | #05: XREMain::XRE_main(int,char * * const,mozilla::BootstrapConfig const &) [toolkit/xre/nsAppRunner.cpp:4884]

In the leak case

https://public-artifacts.taskcluster.net/fTjvtO82TeqU_37kCWoImw/0/public/logs/live_backing.log

nsBaseAppShell is created:

22:26:03     INFO -  GECKO(8748) | #01: nsAppShellInit [widget/nsAppShellSingleton.h:39]
22:26:03     INFO -  GECKO(8748) | #02: nsComponentManagerImpl::KnownModule::Load() [xpcom/components/nsComponentManager.cpp:764]
22:26:03     INFO -  GECKO(8748) | #03: nsFactoryEntry::GetFactory() [xpcom/components/nsComponentManager.cpp:1785]
22:26:03     INFO -  GECKO(8748) | #04: nsComponentManagerImpl::CreateInstance(nsID const &,nsISupports *,nsID const &,void * *) [xpcom/components/nsComponentManager.cpp:1001]
22:26:03     INFO -  GECKO(8748) | #05: nsComponentManagerImpl::GetService(nsID const &,nsID const &,void * *) [xpcom/components/nsComponentManager.cpp:1246]
22:26:03     INFO -  GECKO(8748) | #06: nsGetServiceByCIDWithError::operator()(nsID const &,void * *) [xpcom/components/nsComponentManagerUtils.cpp:266]
22:26:03     INFO -  GECKO(8748) | #07: nsCOMPtr<nsIAppShell>::assign_from_gs_cid_with_error(nsGetServiceByCIDWithError const &,nsID const &) [xpcom/base/nsCOMPtr.h:1203]
22:26:03     INFO -  GECKO(8748) | #08: nsAppStartup::Init() [toolkit/components/startup/nsAppStartup.cpp:164]
22:26:03     INFO -  GECKO(8748) | #09: nsAppStartupConstructor [toolkit/components/build/nsToolkitCompsModule.cpp:66]
22:26:03     INFO -  GECKO(8748) | #10: nsComponentManagerImpl::CreateInstanceByContractID(char const *,nsISupports *,nsID const &,void * *) [xpcom/components/nsComponentManager.cpp:1087]
22:26:03     INFO -  GECKO(8748) | #11: nsComponentManagerImpl::GetServiceByContractID(char const *,nsID const &,void * *) [xpcom/components/nsComponentManager.cpp:1446]
22:26:03     INFO -  GECKO(8748) | #12: nsGetServiceByContractID::operator()(nsID const &,void * *) [xpcom/components/nsComponentManagerUtils.cpp:281]
22:26:03     INFO -  GECKO(8748) | #13: nsCOMPtr<nsIWindowCreator>::assign_from_gs_contractid(nsGetServiceByContractID,nsID const &) [xpcom/base/nsCOMPtr.h:1215]
22:26:03     INFO -  GECKO(8748) | #14: ScopedXPCOMStartup::SetWindowCreator(nsINativeAppSupport *) [toolkit/xre/nsAppRunner.cpp:1657]
22:26:03     INFO -  GECKO(8748) | #15: XREMain::XRE_mainRun() [toolkit/xre/nsAppRunner.cpp:4360]

but the dtor is never called. I wonder why...

Comment 6

[Intermittent Failures Robot]

25 failures in 188 pushes (0.133 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* mozilla-inbound: 23
* try: 1
* autoland: 1

Platform breakdown:
* windows10-64-stylo-disabled: 22
* windows10-64: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1401883&startday=2017-10-11&endday=2017-10-11&tree=all

Comment 7

[Geoff Brown [:gbrown]]

Sorry, I'm not finding much time for this. In the leak case, I am seeing ScopedXPCOMStartup::~ScopedXPCOMStartup() called (whew!) but not nsAppStartup::Release().

Comment 8

[Intermittent Failures Robot]

52 failures in 947 pushes (0.055 failures/push) were associated with this bug in the last 7 days. 

This is the #41 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* mozilla-inbound: 37
* autoland: 12
* mozilla-central: 2
* try: 1

Platform breakdown:
* windows10-64-stylo-disabled: 48
* windows10-64: 4

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1401883&startday=2017-10-09&endday=2017-10-15&tree=all

Comment 9

[Junior [inactive]]

Bug 870460 makes the leakage occur frequently.
However, I have no idea about the relationship between cookie and AppShell.

Comment 10

[Junior [inactive]]

The leakage depends on the life-cycle of thread.
I do some experiment in cookie thread.

The intermittent rate would be:
Using StreamTransportService (short lifecycle thread): 2% [1]
Shutdown on profile-before-change: 8% [2]
Shutdown on xpcom-shutdown: ~30% [3]

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=0bfd99ad67846e6b54f24ad1f6107d0a03176063
[2] https://treeherder.mozilla.org/#/jobs?repo=try&revision=97aa32bc8e72153da74d4018b894beae446cee73
[3] https://treeherder.mozilla.org/#/jobs?repo=try&revision=b6963a183d2032a456d6f58f031251e89601f5d0

Comment 11

[Intermittent Failures Robot]

30 failures in 864 pushes (0.035 failures/push) were associated with this bug in the last 7 days.   

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 13
* mozilla-inbound: 11
* mozilla-central: 5
* try: 1

Platform breakdown:
* windows10-64-stylo-disabled: 23
* windows10-64: 7

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1401883&startday=2017-10-16&endday=2017-10-22&tree=all

Comment 12

[Junior [inactive]]

I don't find any thread which shutdowns after xpcom-shutdown for the leakage case I get
https://treeherder.mozilla.org/logviewer.html#?job_id=138198756&repo=try&lineNumber=44467

Comment 13

[Geoff Brown [:gbrown]]

Sorry, I'm not getting anywhere with this.

I note that all such leaks appear to be Windows-10 only and mochitest-chrome only. There may be fewer leaks in chunk 2, but all chunks are affected and I don't see a close correlation with any small subset of mochitest-chrome tests.

This leak is fairly easy to reproduce on try: -p win64 -u mochitest-chrome --rebuild 10

I tried to apply https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Refcount_tracing_and_balancing on try, but that introduced a lot of failures:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=67b9a022b0948eec857f66e923a1869551162c0e


:junior's observations are interesting (thanks!), but I don't know what else to suggest.

Comment 14

[Intermittent Failures Robot]

42 failures in 912 pushes (0.046 failures/push) were associated with this bug in the last 7 days. 

This is the #36 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 18
* mozilla-inbound: 16
* mozilla-central: 5
* try: 2
* mozilla-beta: 1

Platform breakdown:
* windows10-64-stylo-disabled: 25
* windows10-64: 17

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1401883&startday=2017-10-23&endday=2017-10-29&tree=all

Comment 15

[Bogdan Tara[:bogdan_tara | bogdant]]

:gbrown , :junior , do you think we can find anyone who could work on this?

Comment 16

[Geoff Brown [:gbrown]]

:mccr8, can you help here?

Comment 17

[Andrew McCreight [:mccr8]]

I think the problem with your previous attempt to get refcount logs was that it was attempting to create a file in the child process, to do refcount logging, but the sandbox is enabled there. I could disable the sandbox, but because the leak here is only in the parent process anyways, I disabled refcount logging in the child process instead. Hopefully that will work.
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=4cff28eb159c3d35c38332cbfd4e4fa3e741bdbd

Comment 18

[Andrew McCreight [:mccr8]]

I'll move this to widget for now.

There are a lot of places that use this base app shell, so I think a refcount log is the way to go. One problem will be that the refcount log will be unsymbolicated. I can get a leak log and filter out everything but the leaked object, but somebody with a Windows machine will have to do the symbolification. Assuming that it works on Windows like it works on Linux and OSX.

Comment 19

[Andrew McCreight [:mccr8]]

I finally got a run that worked correctly. I still need to actually hit the leak to see a log. I've hit too other intermittent leaks, though...
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=96c6077c741103b3e36068a203fd7a992804c373

Comment 20

[Intermittent Failures Robot]

45 failures in 857 pushes (0.053 failures/push) were associated with this bug in the last 7 days. 

This is the #39 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 25
* mozilla-inbound: 15
* try: 2
* mozilla-central: 2
* mozilla-release: 1

Platform breakdown:
* windows10-64-stylo-disabled: 29
* windows10-64: 16

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1401883&startday=2017-10-30&endday=2017-11-05&tree=all

Comment 21

[Cristian Brindusan [:cbrindusan]]

There are 52 failures on the last week for this bug.

:mccr8 did you had any time to look further into this?

Comment 22

[Intermittent Failures Robot]

54 failures in 849 pushes (0.064 failures/push) were associated with this bug in the last 7 days. 

This is the #32 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 21
* mozilla-inbound: 20
* mozilla-central: 7
* try: 6

Platform breakdown:
* windows10-64-stylo-disabled: 40
* windows10-64: 14

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1401883&startday=2017-11-06&endday=2017-11-12&tree=all

Comment 23

[Andrew McCreight [:mccr8]]

Attachment: Refcount log for nsBaseAppShell. NOT FOR LANDING

Comment 24

[Andrew McCreight [:mccr8]]

I came up with a patch to refcount log nsBaseAppShell that seemed to actually work, but after I retriggered around 200 times I wasn't able to reproduce the leak: https://treeherder.mozilla.org/#/jobs?repo=try&revision=96c6077c741103b3e36068a203fd7a992804c373

Possibly the slowdown from logging keeps the leak from happening.

Another idea I had was to allocate a C++ wrapper for the nsBaseAppShell every time something requests a new reference, that forwards all calls. Then we could only log object creation and possibly still figure out what was leaking, with less overhead. I don't know if that will help.

Jim, maybe you have some ideas how this widget leak could be investigated?

Comment 25

[Intermittent Failures Robot]

36 failures in 762 pushes (0.047 failures/push) were associated with this bug in the last 7 days. 

This is the #32 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 20
* mozilla-inbound: 11
* mozilla-central: 3
* try: 1
* mozilla-beta: 1

Platform breakdown:
* windows10-64-stylo-disabled: 20
* windows10-64: 16

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1401883&startday=2017-11-13&endday=2017-11-19&tree=all

Comment 26

[Intermittent Failures Robot]

7 failures in 744 pushes (0.009 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-beta: 4
* autoland: 3

Platform breakdown:
* windows10-64: 6
* windows10-64-stylo-disabled: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1401883&startday=2017-11-20&endday=2017-11-26&tree=all

Comment 27

[Intermittent Failures Robot]

5 failures in 792 pushes (0.006 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-beta: 3
* mozilla-inbound: 1
* autoland: 1

Platform breakdown:
* windows10-64: 4
* windows10-64-stylo-disabled: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1401883&startday=2017-11-27&endday=2017-12-03&tree=all

Comment 28

[Intermittent Failures Robot]

10 failures in 889 pushes (0.011 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-beta: 3
* autoland: 3
* mozilla-central: 2
* try: 1
* mozilla-inbound: 1

Platform breakdown:
* windows10-64: 7
* windows10-64-stylo-disabled: 2
* windows10-64-ccov: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1401883&startday=2017-12-04&endday=2017-12-10&tree=all

Comment 29

[Intermittent Failures Robot]

6 failures in 423 pushes (0.014 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 3
* autoland: 2
* mozilla-inbound: 1

Platform breakdown:
* windows10-64: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1401883&startday=2017-12-11&endday=2017-12-17&tree=all

Comment 30

[Intermittent Failures Robot]

17 failures in 590 pushes (0.029 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-beta: 5
* autoland: 5
* mozilla-inbound: 3
* mozilla-central: 3
* try: 1

Platform breakdown:
* windows10-64: 12
* windows10-64-stylo-disabled: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1401883&startday=2017-12-18&endday=2017-12-24&tree=all

Comment 31

[Intermittent Failures Robot]

3 failures in 147 pushes (0.02 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-release: 1
* mozilla-central: 1
* autoland: 1

Platform breakdown:
* windows10-64: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1401883&startday=2017-12-25&endday=2017-12-31&tree=all

Comment 32

[Intermittent Failures Robot]

3 failures in 462 pushes (0.006 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* try: 1
* mozilla-inbound: 1
* mozilla-beta: 1

Platform breakdown:
* windows10-64: 2
* windows10-64-stylo-disabled: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1401883&startday=2018-01-01&endday=2018-01-07&tree=all

Comment 33

[Intermittent Failures Robot]

15 failures in 788 pushes (0.019 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-beta: 5
* autoland: 5
* mozilla-inbound: 4
* mozilla-central: 1

Platform breakdown:
* windows10-64: 13
* windows10-64-stylo-disabled: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1401883&startday=2018-01-08&endday=2018-01-14&tree=all

Comment 34

[Intermittent Failures Robot]

13 failures in 657 pushes (0.02 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 5
* mozilla-release: 2
* mozilla-inbound: 2
* mozilla-beta: 2
* autoland: 2

Platform breakdown:
* windows10-64: 9
* windows10-64-stylo-disabled: 3
* windows10-64-ccov: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1401883&startday=2018-01-15&endday=2018-01-21&tree=all

Comment 35

[Intermittent Failures Robot]

13 failures in 701 pushes (0.019 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* try: 4
* mozilla-beta: 3
* mozilla-inbound: 2
* mozilla-central: 2
* autoland: 2

Platform breakdown:
* windows10-64: 11
* windows10-64-stylo-disabled: 1
* windows10-64-ccov: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1401883&startday=2018-01-22&endday=2018-01-28&tree=all

Comment 36

[Intermittent Failures Robot]

7 failures in 735 pushes (0.01 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* try: 2
* mozilla-beta: 2
* mozilla-inbound: 1
* mozilla-central: 1
* autoland: 1

Platform breakdown:
* windows10-64: 7

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1401883&startday=2018-01-29&endday=2018-02-04&tree=all

Comment 37

[Intermittent Failures Robot]

27 failures in 154 pushes (0.175 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* autoland: 11
* mozilla-inbound: 7
* mozilla-central: 7
* try: 1
* mozilla-beta: 1

Platform breakdown:
* windows10-64: 27

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1401883&startday=2018-02-07&endday=2018-02-07&tree=all

Comment 38

[Tiberius Oros[:tiberius_oros]]

There have been 32 failures in the last week.
It seems the number of failures increased starting with February 5.

This only occurs on windows10-64 / debug.

Here is a recent log file and a snippet with the failure:
https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=161017372&lineNumber=2265

18:00:42     INFO - TEST-PASS | browser/components/search/test/browser_oneOffHeader.js | Search header string is correct when a search term has been entered - 
18:00:43     INFO - Buffered messages finished
18:00:43     INFO - TEST-UNEXPECTED-FAIL | browser/components/search/test/browser_oneOffHeader.js | Header has the correct index selected when a search engine has been selected - Got 1, expected 2
18:00:43     INFO - Stack trace:
18:00:43     INFO - chrome://mochikit/content/browser-test.js:test_is:1271
18:00:43     INFO - chrome://mochitests/content/browser/browser/components/search/test/browser_oneOffHeader.js:test_text:121
18:00:43     INFO - Not taking screenshot here: see the one that was previously logged

:jimm, have you had the chance to look over this?

Comment 39

[Intermittent Failures Robot]

28 failures in 125 pushes (0.224 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* autoland: 20
* mozilla-central: 6
* mozilla-inbound: 1
* mozilla-beta: 1

Platform breakdown:
* windows10-64: 26
* windows10-64-ccov: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1401883&startday=2018-02-08&endday=2018-02-08&tree=all

Comment 40

[Intermittent Failures Robot]

62 failures in 702 pushes (0.088 failures/push) were associated with this bug in the last 7 days. 

This is the #15 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 34
* mozilla-central: 13
* mozilla-inbound: 10
* try: 2
* mozilla-beta: 2
* mozilla-release: 1

Platform breakdown:
* windows10-64: 61
* windows10-64-ccov: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1401883&startday=2018-02-05&endday=2018-02-11&tree=all

Comment 41

[Intermittent Failures Robot]

5 failures in 685 pushes (0.007 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-beta: 2
* mozilla-release: 1
* mozilla-inbound: 1
* mozilla-central: 1

Platform breakdown:
* windows10-64: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1401883&startday=2018-02-12&endday=2018-02-18&tree=all

Comment 42

[Intermittent Failures Robot]

5 failures in 675 pushes (0.007 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 3
* mozilla-beta: 2

Platform breakdown:
* windows10-64: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1401883&startday=2018-02-19&endday=2018-02-25&tree=all

Comment 43

[Intermittent Failures Robot]

13 failures in 831 pushes (0.016 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 4
* mozilla-inbound: 3
* autoland: 3
* try: 1
* mozilla-release: 1
* mozilla-beta: 1

Platform breakdown:
* windows10-64: 13

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1401883&startday=2018-02-26&endday=2018-03-04&tree=all

Comment 44

[Intermittent Failures Robot]

11 failures in 747 pushes (0.015 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-release: 4
* autoland: 3
* mozilla-beta: 2
* try: 1
* mozilla-inbound: 1

Platform breakdown:
* windows10-64: 11

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1401883&startday=2018-03-05&endday=2018-03-11&tree=all

Comment 45

[Intermittent Failures Robot]

6 failures in 814 pushes (0.007 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 2
* try: 1
* mozilla-inbound: 1
* mozilla-beta: 1
* autoland: 1

Platform breakdown:
* windows10-64: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1401883&startday=2018-03-12&endday=2018-03-18&tree=all

Comment 46

[Intermittent Failures Robot]

37 failures in 149 pushes (0.248 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* autoland: 31
* mozilla-inbound: 5
* mozilla-central: 1

Platform breakdown:
* windows10-64: 37

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1401883&startday=2018-03-20&endday=2018-03-20&tree=all

Comment 47

[Tiberius Oros[:tiberius_oros]]

Update: 
There have been 56 total failures in the last 7 days.
Almost all the failures occur on windows10-64 / debug, with 2 exceptions for windows10-64-ccov / debug.

Here is a recent relevant log file and a snippet with the failure:
https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=169402099&lineNumber=16022

10:58:25     INFO -  TEST-INFO | leakcheck | gpu process: leak threshold set at 0 bytes
10:58:25     INFO -  == BloatView: ALL (cumulative) LEAK AND BLOAT STATISTICS, default process 6056
10:58:25     INFO -       |<----------------Class--------------->|<-----Bytes------>|<----Objects---->|
10:58:25     INFO -       |                                      | Per-Inst   Leaked|   Total      Rem|
10:58:25     INFO -     0 |TOTAL                                 |       34      176| 7560403        2|
10:58:25     INFO -   540 |Mutex                                 |       80       80|    3446        1|
10:58:25     INFO -  1239 |nsBaseAppShell                        |       96       96|       1        1|
10:58:25     INFO -  nsTraceRefcnt::DumpStatistics: 1854 entries
10:58:25     INFO -  TEST-INFO | leakcheck | default process: leaked 1 Mutex
10:58:25     INFO -  TEST-INFO | leakcheck | default process: leaked 1 nsBaseAppShell
10:58:25    ERROR -  1236 ERROR TEST-UNEXPECTED-FAIL | leakcheck | default process: 176 bytes leaked (Mutex, nsBaseAppShell)
10:58:25     INFO -  runtests.py | Running tests: end.
10:58:25     INFO -  Buffered messages finished

Comment 48

[Intermittent Failures Robot]

93 failures in 822 pushes (0.113 failures/push) were associated with this bug in the last 7 days. 

This is the #8 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **   

Repository breakdown:
* autoland: 50
* mozilla-central: 22
* mozilla-inbound: 10
* try: 8
* mozilla-beta: 3

Platform breakdown:
* windows10-64: 80
* windows10-64-ccov: 13

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1401883&startday=2018-03-20&endday=2018-03-26&tree=trunk

Comment 49

[Intermittent Failures Robot]

39 failures in 168 pushes (0.232 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* mozilla-central: 14
* autoland: 14
* mozilla-inbound: 11

Platform breakdown:
* windows10-64: 37
* windows10-64-ccov: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1401883&startday=2018-03-27&endday=2018-03-27&tree=trunk

Comment 50

[Intermittent Failures Robot]

56 failures in 161 pushes (0.348 failures/push) were associated with this bug yesterday.    

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. ** 

Repository breakdown:
* mozilla-inbound: 29
* autoland: 17
* try: 7
* mozilla-central: 3

Platform breakdown:
* windows10-64: 56

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1401883&startday=2018-03-28&endday=2018-03-28&tree=trunk

Comment 51

[Tiberius Oros[:tiberius_oros]]

There have been 127 failures in the last week.
It fails on windows10-64 / debug.

Recent log file and snippet with the failure:
https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=170955498&lineNumber=10683

03:05:50     INFO -  nsTraceRefcnt::DumpStatistics: 1849 entries
03:05:50     INFO -  TEST-INFO | leakcheck | default process: leaked 1 Mutex
03:05:50     INFO -  TEST-INFO | leakcheck | default process: leaked 1 nsBaseAppShell
03:05:50    ERROR -  658 ERROR TEST-UNEXPECTED-FAIL | leakcheck | default process: 176 bytes leaked (Mutex, nsBaseAppShell)
03:05:50     INFO -  runtests.py | Running tests: end.
03:05:50     INFO -  Buffered messages finished
03:05:50     INFO -  Running manifest: toolkit\components\windowcreator\test\chrome.ini
03:05:50     INFO -  Z:\task_1522291025\build\tests\bin\pk12util.exe: PKCS12 IMPORT SUCCESSFUL

Comment 52

[Andreea Pavel [:apavel]]

:mccr8 hi. this failure seems to have an increasing rate on all trees, noticed on inbound though, can you please take a look?

Also this bug and bug 1447440 seem to be related. 

Treeherder links: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=windows%2010%20x64%20debug%20mochitests%20test-windows10-64%2Fdebug-mochitest-chrome-1%20m(c1)&fromchange=cf2f8604cb5ee52c143c91f56d465a8486ff8f1b&group_state=expanded&selectedJob=171107409 
and
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=windows%2010%20x64%20debug%20mochitests%20test-windows10-64%2Fdebug-mochitest-chrome-3%20m(c3)&fromchange=034e1d7593701112b0c92fb1b40e0e69663b0ca9&group_state=expanded&selectedJob=171138576


Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=171138576&repo=mozilla-inbound&lineNumber=4455

Thanks

Comment 53

[Intermittent Failures Robot]

63 failures in 120 pushes (0.525 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* mozilla-inbound: 36
* autoland: 16
* mozilla-central: 7
* try: 4

Platform breakdown:
* windows10-64: 63

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1401883&startday=2018-03-29&endday=2018-03-29&tree=trunk

Comment 54

[Andrew McCreight [:mccr8]]

I already looked at this back in November and I was unable to make any progress.

Comment 55

[Intermittent Failures Robot]

47 failures in 94 pushes (0.5 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* autoland: 24
* mozilla-inbound: 12
* mozilla-central: 9
* try: 2

Platform breakdown:
* windows10-64: 46
* linux64-jsdcov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1401883&startday=2018-03-30&endday=2018-03-30&tree=trunk

Comment 56

[Intermittent Failures Robot]

17 failures in 28 pushes (0.607 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* mozilla-inbound: 6
* mozilla-central: 5
* try: 4
* autoland: 2

Platform breakdown:
* windows10-64: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1401883&startday=2018-03-31&endday=2018-03-31&tree=trunk

Comment 57

[Intermittent Failures Robot]

237 failures in 718 pushes (0.33 failures/push) were associated with this bug in the last 7 days. 

This is the #2 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **   

Repository breakdown:
* mozilla-inbound: 101
* autoland: 73
* mozilla-central: 40
* try: 22
* mozilla-beta: 1

Platform breakdown:
* windows10-64: 233
* windows10-64-ccov: 3
* linux64-jsdcov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1401883&startday=2018-03-26&endday=2018-04-01&tree=trunk

Comment 58

[Intermittent Failures Robot]

41 failures in 82 pushes (0.5 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* mozilla-inbound: 19
* autoland: 12
* mozilla-central: 7
* try: 3

Platform breakdown:
* windows10-64: 41

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1401883&startday=2018-04-02&endday=2018-04-02&tree=trunk

Comment 59

[Joel Maher ( :jmaher ) (UTC -8)]

this looks to be in the category of perma failing- we will look to disable something soon.

Comment 60

[Andreea Pavel [:apavel]]

Attachment: Bug 1401883 - disabled on Win64

Comment 61

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8964632 [details]
Bug 1401883 - disabled on Win64

one reason for the r- is that on my try pushes:
https://treeherder.mozilla.org/#/jobs?repo=try&author=[e-mail]&fromchange=58a4fcd84bc875067bed6bf4f36e73520bd8d8da

you can see that we have the same rate of failure before/after this patch (which is ~40%).

To me we just need to disable these tests on windows, or we need to fix this issue.  40% failure rate is not acceptable.  Unfortunately C1/C2/C3 all fail and it seems to be not just a couple tests or a couple directories, it seems to be wider spread.

Comment 62

[Joel Maher ( :jmaher ) (UTC -8)]

I did some retriggers on treeherder for march 20th:
https://treeherder.mozilla.org/#/jobs?repo=autoland&filter-searchStr=windows%20x64%20mochitest-chrome&tochange=fae7d981440374495e890a5ecda4bb698a84f95b&fromchange=28088e11bfc458d12e809b79628d4c2dd942bd83

not every orange job is the same failure, but the majority are the 176 bytes leaked.

It looks like bug 1439383 introduced something to cause this to fail quite frequently:
https://hg.mozilla.org/integration/autoland/rev/5be07e86738e8993022e1c507b5b09e602872a33

:keeler, I see you authored the patch in bug 1439383- this bug is now the #1 intermittent failure on the whole tree- can you take a look at fixing this leak?

Comment 63

[Joel Maher ( :jmaher ) (UTC -8)]

I did a small hack to ignore this specific leak and it is super green:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=74f18525744adff607cc38f61800b3c4c0e83d8e

Comment 64

[Andrew McCreight [:mccr8]]

We should back out bug 1439383 before we land a workaround like that. (Thanks so much for figuring out what caused the regression!)

Comment 65

[Intermittent Failures Robot]

58 failures in 165 pushes (0.352 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* autoland: 22
* mozilla-inbound: 21
* mozilla-central: 10
* try: 5

Platform breakdown:
* windows10-64: 58

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1401883&startday=2018-04-03&endday=2018-04-03&tree=trunk

Comment 66

[Dana Keeler (she/her) (use needinfo) [:keeler]]

I tried to figure out what from bug 1439383 could have made this more frequent, but since it's just shutting down a thread and doesn't have anything to do with nsBaseAppShell, I couldn't come up with any explanations. Bug 1439383 just got backed out anyway, so this should go back to pre-March 21 levels (which, of course, doesn't solve the intermittent, but at least we have an idea of potentially associated code).

Comment 67

[Intermittent Failures Robot]

19 failures in 158 pushes (0.12 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* mozilla-inbound: 8
* autoland: 5
* mozilla-central: 4
* try: 1
* mozilla-beta: 1

Platform breakdown:
* windows10-64: 17
* windows10-64-ccov: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1401883&startday=2018-04-04&endday=2018-04-04&tree=trunk

Comment 68

[Nathan Froyd [:froydnj]]

It is super annoying that this caused bug 1439383 to be backed out.  Holding up memory wins for silly intermittent memory leaks is unfortunate.

I took mccr8's patch from comment 24 and wrote a wrapper for the appshell constructor to hand out:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=d575a6b0635a6769708cf5f9040396e2a3a27944

I haven't looked closely at the refcount logs, but I believe what the leak checker is saying is that we didn't leak any wrapper objects for nsBaseAppShell being used *as an nsIAppShell*.  If we were leaking nsBaseAppShell as an nsIAppShell, then the wrapper objects would have leaked too (I think).  If that's the case, the next step is to get the wrapper's QI to hand out different objects for nsIThreadObserver and nsIObserver, the other two interfaces implemented by nsBaseAppShell.  One of those wrappers should leak, and then we should have a better idea of where to start diagnosing the leak.

It's possible that idea doesn't work, in which case I think that means that somebody is casting ns{Base,}AppShell objects directly to the interface superclasses without going through QI...which also should narrow down the potential space of places to check?

Comment 69

[(Away)]

FWIW my clang-cl try pushes hit this intermittent a *lot*. I don't know if the compiler change is tickling something or if simply everyone else is hitting this a lot too. (Sad to say, I saw the disable-recommended keyword and am quietly hoping that this bug just goes away before I need to worry about landing.)

Comment 70

[Andrew McCreight [:mccr8]]

(In reply to David Major [:dmajor] from comment #69)
> FWIW my clang-cl try pushes hit this intermittent a *lot*. I don't know if
> the compiler change is tickling something or if simply everyone else is
> hitting this a lot too.

This was the top intermittent failure, so you aren't alone. Please let us know if you are still seeing it a lot after bug 1439383 got backed out.

Comment 71

[Joel Maher ( :jmaher ) (UTC -8)]

you can see the failure rate dropped significantly 2 days ago:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2018-03-30&endday=2018-04-06&tree=trunk&bug=1401883

The backout (or something else around the same time) did resolve the large majority of these failures.

Comment 72

[Intermittent Failures Robot]

136 failures in 765 pushes (0.178 failures/push) were associated with this bug in the last 7 days. 

This is the #7 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **   

Repository breakdown:
* mozilla-inbound: 53
* autoland: 43
* mozilla-central: 29
* try: 10
* mozilla-beta: 1

Platform breakdown:
* windows10-64: 128
* windows10-64-ccov: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1401883&startday=2018-04-02&endday=2018-04-08&tree=trunk

Comment 73

[Nathan Froyd [:froydnj]]

OK, I don't understand what's going on here.  Try push:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=2d35da556cb039b07e62873960aa7e8ed9a8f548

To make it clear, this push implements:

* A wrapper returned for constructing nsIAppShell instances.  The wrapper QIs itself to nsIAppShell and nsISupports.  This *should* mean that when we leak nsBaseAppShell, we're also leaking this object.  It's possible that the wrapper gets QI'd to nsI{Thread,}Observer instead, and we lose tracking of the wrapper, but this does not seem to be the case.  (nsBaseAppShell sets itself up as a thread observer and registers itself with the observer service directly, and it seems kind of unusual for something else to QI to the observer classes directly.  Previous try pushes intercepted QI to the observer classes and didn't see anything unusual.  See also below.)
* Wrappers are used for registering nsI{Thread,}Observer instances from inside nsBaseAppShell.  Thus, if we were leaking through those references, we'd leak those wrappers as well.

But the leaks (which are numerous, as you can see) never leak the wrapper objects; they continue to only leak the Mutex and the nsBaseAppShell.  Which makes no sense to me!

There are also a fair number of crashes in the nsTraceRefcnt code, which I'm puzzled about too.  Usually when that happens, it means that two different classes are using the same name for their nsTraceRefcnt key...but that doesn't appear to be happening in this case: grepping around the tree doesn't produce any hits for the newly added class names besides the newly added classes themselves.

Going to try asserting on peculiar QI requests and see if that turns up anything unusual.

Comment 74

[Nathan Froyd [:froydnj]]

OK, asserting on peculiar QI requests doesn't turn up anything unusual.  That means that all the references code are getting are coming through XPCOM construction mechanisms.  And those should all go through the wrapper handed out in the constructor...but the wrappers themselves are not leaking!

I think this means that the only reference that can leak is the one held by nsAppShellSingleton.h itself...but that reference *should* go away by the time the final bit of the leak checker runs to check for not-freed stuff.

Bug 1439383 suggests that an extra event causes this to occur more frequently, so maybe there's some runnable that's holding a reference, and the runnable is unceremoniously dropped on the floor at some point?  But then the runnable would leak too, so that doesn't seem tenable.

Oh!  Here's the problem.  Runnables get dispatched, and when they get dispatched, the thread observer (i.e. the appshell) is notified:

https://searchfox.org/mozilla-central/source/widget/nsBaseAppShell.cpp#219

which calls the widget-specific ScheduleNativeEventCallback:

https://searchfox.org/mozilla-central/source/widget/windows/nsAppShell.cpp#460

On Windows, that scheduling takes an extra reference to the appshell, and posts a message to the native Windows event loop.  The corresponding handler will release the reference:

https://searchfox.org/mozilla-central/source/widget/windows/nsAppShell.cpp#189

But what's probably happening in this bug is that ScheduleNativeEventCallback is called, and then we shut down processing of the native Windows event loop before calling EventWindowProc, thereby leaking the reference.  *Those* refcounting operations are run on the actual object, and that's why the wrappers weren't being leaked.

That is, this is just the reincarnation of bug 1220517.  More tomorrow when I can think.

Comment 75

[Intermittent Failures Robot]

53 failures in 792 pushes (0.067 failures/push) were associated with this bug in the last 7 days. 

This is the #27 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* mozilla-central: 25
* mozilla-inbound: 13
* autoland: 12
* try: 3

Platform breakdown:
* windows10-64: 29
* windows10-64-ccov: 24

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1401883&startday=2018-04-09&endday=2018-04-15&tree=trunk

Comment 76

[Tiberius Oros[:tiberius_oros]]

Update:

There have been 42 failures in the last 7 days.
All of the failures occur on debug.
Platforms affected:

windows10-64: 28
windows10-64-ccov: 14

Here is a recent log file: 
https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=174251092&lineNumber=1942 

Summary: Intermittent leakcheck | default process: 176 bytes leaked (Mutex, nsBaseAppShell)

:froydnj, did you manage to have another look at this?

Comment 77

[Intermittent Failures Robot]

41 failures in 782 pushes (0.052 failures/push) were associated with this bug in the last 7 days. 

This is the #41 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* mozilla-central: 15
* mozilla-inbound: 12
* autoland: 10
* try: 3
* mozilla-beta: 1

Platform breakdown:
* windows10-64: 33
* windows10-64-ccov: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1401883&startday=2018-04-16&endday=2018-04-22&tree=trunk

Comment 78

[Andreea Pavel [:apavel]]

Checking the logs we found that this leak is not on a specific directory.

Comment 79

[Intermittent Failures Robot]

34 failures in 780 pushes (0.044 failures/push) were associated with this bug in the last 7 days. 

This is the #50 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* mozilla-inbound: 14
* mozilla-central: 9
* autoland: 8
* try: 2
* mozilla-beta: 1

Platform breakdown:
* windows10-64: 24
* windows10-64-ccov: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1401883&startday=2018-04-23&endday=2018-04-29&tree=trunk

Comment 80

[Intermittent Failures Robot]

16 failures in 551 pushes (0.029 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 10
* mozilla-inbound: 3
* try: 1
* mozilla-esr60: 1
* autoland: 1

Platform breakdown:
* windows10-64-ccov: 10
* windows10-64: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1401883&startday=2018-04-30&endday=2018-05-06&tree=trunk

Comment 81

[Intermittent Failures Robot]

12 failures in 660 pushes (0.018 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 11
* mozilla-inbound: 1

Platform breakdown:
* windows10-64-ccov: 11
* windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1401883&startday=2018-05-07&endday=2018-05-13&tree=trunk

Comment 82

[Joel Maher ( :jmaher ) (UTC -8)]

the ccov failures are on this directory:
toolkit/components/url-classifier/tests/mochitest

and the normal windows10 failures are on:
browser/base/content/test/chrome


we should be able to bisect this down and skip some tests- luckily the failure rate is pretty low these days.

Comment 83

[Joel Maher ( :jmaher ) (UTC -8)]

14 tests:
https://searchfox.org/mozilla-central/source/toolkit/components/url-classifier/tests/mochitest/chrome.ini


2 tests:
https://searchfox.org/mozilla-central/source/browser/base/content/test/chrome/chrome.ini

Comment 84

[Joel Maher ( :jmaher ) (UTC -8)]

bisection didn't work so well for me, seemed that I couldn't repro or everything reproduced it.  The faillure rate is pretty low where we could consider disabling the url-classifier tests for win10 ccov builds if needed- but I would think we will stay <20 failures/week until something triggers this again.

Comment 85

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Joel - did you try with the patch from bug 1439383? It seemed to make this much more common - might make it easier to narrow it down.

Comment 86

[Intermittent Failures Robot]

29 failures in 639 pushes (0.045 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 11
* autoland: 9
* try: 4
* mozilla-inbound: 4
* mozilla-beta: 1

Platform breakdown:
* windows10-64: 15
* windows10-64-ccov: 11
* android-4-3-armv7-api16: 2
* windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1401883&startday=2018-05-14&endday=2018-05-20&tree=trunk

Comment 87

[Intermittent Failures Robot]

18 failures in 86 pushes (0.209 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* try: 15
* mozilla-central: 2
* autoland: 1

Platform breakdown:
* windows10-64: 7
* windows7-32: 5
* windows10-64-ccov: 5
* android-4-3-armv7-api16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1401883&startday=2018-05-21&endday=2018-05-21&tree=trunk

Comment 88

[Joel Maher ( :jmaher ) (UTC -8)]

why did these get annotated on try :(

Comment 89

[Joel Maher ( :jmaher ) (UTC -8)]

I tried applying the patch from bug 1439383 and did a bisection again:
https://treeherder.mozilla.org/#/jobs?repo=try&author=[e-mail]&fromchange=f49e9d82ae4c49327947b6f4e1ec1189e7bdab8e&filter-searchStr=c3&tochange=0cf3bc8396ce0ab411e899f3fd377052982e5969&selectedJob=179893480

I didn't see any on the ccov builds, and a few instances on the debug builds, although so few it is hard to know if there is a pattern or a fix.

Comment 90

[Intermittent Failures Robot]

65 failures in 632 pushes (0.103 failures/push) were associated with this bug in the last 7 days. 

This is the #28 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* try: 21
* mozilla-central: 18
* mozilla-inbound: 11
* autoland: 10
* mozilla-esr60: 4
* mozilla-beta: 1

Platform breakdown:
* windows10-64: 42
* windows10-64-ccov: 17
* windows7-32: 5
* android-4-3-armv7-api16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1401883&startday=2018-05-21&endday=2018-05-27&tree=trunk

Comment 91

[Nathan Froyd [:froydnj]]

Attachment: don't hold unnecessary references to the Windows app shell

When runnables are posted to the main thread's event loop, the event
loop notifies any thread observers that this has been done.  The app
shell registers itself as just such a runnable, and posts messages to
the native event loop that processing native events needs to be done.

On Windows, this posting takes an extra reference to the app shell, to
keep it alive until the message is processed by the native event loop,
since app shell code needs to be invoked during that processing.  The
processing releases this extra reference, so everything stays balanced.

Except that it's possible for messages to be posted to the native event
loop, and then browser shutdown happens.  Those messages are not
processed and the associated references taken are not released.  This
imbalance means that in debug builds, we appear to be leaking the app
shell, and that leaking results in intermittent oranges.

This intermittent orange has manifested itself in a variety of ways over
the years, depending on how big the app shell itself was (since that
changes the number of bytes leaked) and how many leak-checked things the
app shell was holding on to.  This bug is merely the latest
manifestation; the last serious work on analyzing the phenomenon and
fixing it was done in bug 1220517.

The solution proposed in that bug was that we simply stop the extra
reference counting; when the app shell is destroyed normally, we
shouldn't be processing the native event loop any more anyway.  So even
if the native event loop is holding (freed) pointers to the app shell,
we'd never execute the callback and perform a use-after-free.  Reading
through the code suggests that this *ought* to be the case, but the
potential for shooting ourselves in the foot seems awfully high.

In any event, this is not a problem unique to Windows; we have seen this
same sort of thing happen on OS X.  See nsAppShell::ProcessGeckoEvents
in widget/cocoa/nsAppShell.mm.

Here we propose a slightly different solution: we keep track of the
number of native event callbacks we have scheduled, incrementing when we
schedule, and decrementing when we actually run one.  When the app shell
is destroyed, we simply set the number of outstanding events to zero,
and we prohibit the callback from accessing the app shell if there are
no outstanding events.  This solution is analogous to dropping the extra
reference counting, but avoids potential badness if we do wind up
processing native events after the app shell is destroyed.

This patch is really Ting-Yu's patch from bug 1220517, modified by me to be
slightly safer.  You can see some of the discussion in bug 1220517 comment 31
and following.  bsmedberg and I were skeptical that the proposed fix of just
dropping the refcounting was bulletproof, but it did fix the problem.  This fix
should work just as well, and be slightly less sketchy.

Comment 92

[Intermittent Failures Robot]

42 failures in 655 pushes (0.064 failures/push) were associated with this bug in the last 7 days. 

This is the #34 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* mozilla-central: 14
* mozilla-inbound: 10
* try: 9
* autoland: 6
* mozilla-esr60: 2
* mozilla-beta: 1

Platform breakdown:
* windows10-64: 26
* windows10-64-ccov: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1401883&startday=2018-05-28&endday=2018-06-03&tree=trunk

Comment 93

[Jim Mathies [:jimm]]

Comment on attachment 8982625 [details] [diff] [review]
don't hold unnecessary references to the Windows app shell

Review of attachment 8982625 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks for the detail in the patch. Seems like a safe change.

Comment 94

[Intermittent Failures Robot]

5 failures in 109 pushes (0.046 failures/push) were associated with this bug yesterday.    

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. ** 

Repository breakdown:
* mozilla-central: 3
* autoland: 2

Platform breakdown:
* windows10-64: 4
* windows10-64-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1401883&startday=2018-06-05&endday=2018-06-05&tree=trunk

Comment 95

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/5fd3144d5d00
don't hold unnecessary references to the Windows app shell; r=jimm

Comment 96

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/5fd3144d5d00

Comment 97

[Intermittent Failures Robot]

27 failures in 669 pushes (0.04 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 11
* autoland: 8
* try: 4
* mozilla-inbound: 3
* mozilla-esr60: 1

Platform breakdown:
* windows10-64: 20
* windows10-64-ccov: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1401883&startday=2018-06-04&endday=2018-06-10&tree=trunk

Comment 99

[Ryan VanderMeulen [:RyanVM]]

No reports of this since this patch landed on the 7th. Is this something we could consider uplifting to Beta/ESR60?

Comment 100

[Nathan Froyd [:froydnj]]

Comment on attachment 8982625 [details] [diff] [review]
don't hold unnecessary references to the Windows app shell

(In reply to Ryan VanderMeulen [:RyanVM] from comment #99)
> No reports of this since this patch landed on the 7th. Is this something we
> could consider uplifting to Beta/ESR60?

Sure, would basically only matter for tests, though.

[Approval Request Comment]
If this is not a sec:{high,crit} bug, please state case for ESR consideration: Makes sheriffs' lives easier, perhaps?
User impact if declined: None.
Fix Landed on Version: 62
Risk to taking this patch (and alternatives if risky): Very low risk.
String or UUID changes made by this patch: None

Approval Request Comment
[Feature/Bug causing the regression]: Something lost to the sands of time.
[User impact if declined]: None.
[Is this code covered by automated tests?]: Yes.
[Has the fix been verified in Nightly?]: Yes.
[Needs manual test from QE? If yes, steps to reproduce]: No.
[List of other uplifts needed for the feature/fix]: None.
[Is the change risky?]: Not really.
[String changes made/needed]: None.

Comment 101

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 8982625 [details] [diff] [review]
don't hold unnecessary references to the Windows app shell

Makes Windows a bit less leaky in CI. Approved for 61.0b14 and ESR 60.1.

Comment 102

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/8a24c581a182

Comment 103

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr60/rev/14a7aa94151d