Intermittent leakcheck | default process: 15870 bytes leaked (CacheEntry, CacheEntryHandle, CacheFile, CacheFileMetadata, CacheFileOutputStream, ...)

NEW
Unassigned

Status

()

defect
P3
normal
3 years ago
10 months ago

People

(Reporter: bkelly, Unassigned)

Tracking

(Blocks 1 bug, {intermittent-failure, leave-open, memory-leak})

Trunk
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(e10s+, firefox47 wontfix, firefox48 wontfix, firefox49 wontfix, firefox50 wontfix, firefox51 wontfix, firefox52 wontfix, firefox53 wontfix, firefox54 wontfix, firefox55 disabled, firefox56 disabled)

Details

(Whiteboard: [MemShrink:P2][stockwell unknown])

Attachments

(1 attachment)

+++ This bug was initially created as a clone of Bug #1238707 +++

After fixing bug 1238707 another leak was revealed:

https://treeherder.mozilla.org/logviewer.html#?job_id=18936889&repo=try

I ran another push with leak stack walking enabled, and the leaked CacheEntry has the following stack:
19:00:51     INFO -  Serial Numbers of Leaked Objects:
19:00:51     INFO -  123 @1D2FB9D0 (1 references)
19:00:51     INFO -  allocation stack:
19:01:06     INFO -  #00: mozilla::net::CacheEntry::CacheEntry(nsACString_internal const &,nsIURI *,nsACString_internal const &,bool,bool,bool) [netwerk/cache2/CacheEntry.cpp:225]
19:01:06     INFO -  #01: mozilla::net::CacheStorageService::AddStorageEntry(nsACString_internal const &,nsIURI *,nsACString_internal const &,bool,bool,bool,bool,mozilla::net::CacheEntryHandle * *) [netwerk/cache2/CacheStorageService.cpp:1493]
19:01:06     INFO -  #02: mozilla::net::CacheStorageService::AddStorageEntry(mozilla::net::CacheStorage const *,nsIURI *,nsACString_internal const &,bool,mozilla::net::CacheEntryHandle * *) [netwerk/cache2/CacheStorageService.cpp:1411]
19:01:06     INFO -  #03: mozilla::net::CacheStorage::OpenTruncate(nsIURI *,nsACString_internal const &,nsICacheEntry * *) [netwerk/cache2/CacheStorage.cpp:131]
19:01:06     INFO -  #04: mozilla::net::nsHttpChannel::OpenCacheEntry(bool) [netwerk/protocol/http/nsHttpChannel.cpp:3110]
19:01:06     INFO -  #05: mozilla::net::nsHttpChannel::Connect() [netwerk/protocol/http/nsHttpChannel.cpp:380]
19:01:06     INFO -  #06: mozilla::net::nsHttpChannel::ContinueBeginConnectWithResult() [netwerk/protocol/http/nsHttpChannel.cpp:5560]
19:01:06     INFO -  #07: mozilla::net::nsHttpChannel::BeginConnect() [netwerk/protocol/http/nsHttpChannel.cpp:5463]
19:01:06     INFO -  #08: mozilla::net::nsHttpChannel::OnProxyAvailable(nsICancelable *,nsIChannel *,nsIProxyInfo *,nsresult) [netwerk/protocol/http/nsHttpChannel.cpp:5630]
19:01:06     INFO -  #09: nsAsyncResolveRequest::DoCallback() [netwerk/base/nsProtocolProxyService.cpp:253]
19:01:06     INFO -  #10: nsAsyncResolveRequest::OnQueryComplete(nsresult,nsCString const &,nsCString const &) [netwerk/base/nsProtocolProxyService.cpp:214]
19:01:06     INFO -  #11: ExecuteCallback::Run() [netwerk/base/nsPACMan.cpp:87]
19:01:06     INFO -  #12: nsThread::ProcessNextEvent(bool,bool *) [xpcom/threads/nsThread.cpp:994]
19:01:06     INFO -  #13: NS_ProcessNextEvent(nsIThread *,bool) [xpcom/glue/nsThreadUtils.cpp:297]
19:01:06     INFO -  #14: mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate *) [ipc/glue/MessagePump.cpp:98]
19:01:06     INFO -  #15: MessageLoop::RunInternal() [ipc/chromium/src/base/message_loop.cc:236]
19:01:06     INFO -  #16: MessageLoop::RunHandler() [ipc/chromium/src/base/message_loop.cc:230]
19:01:06     INFO -  #17: MessageLoop::Run() [ipc/chromium/src/base/message_loop.cc:204]
19:01:06     INFO -  #18: nsBaseAppShell::Run() [widget/nsBaseAppShell.cpp:158]
19:01:06     INFO -  #19: nsAppShell::Run() [widget/windows/nsAppShell.cpp:264]
19:01:06     INFO -  #20: nsAppStartup::Run() [toolkit/components/startup/nsAppStartup.cpp:282]
19:01:06     INFO -  #21: XREMain::XRE_mainRun() [toolkit/xre/nsAppRunner.cpp:4342]
19:01:06     INFO -  #22: XREMain::XRE_main(int,char * * const,nsXREAppData const *) [toolkit/xre/nsAppRunner.cpp:4439]
19:01:06     INFO -  #23: XRE_main [toolkit/xre/nsAppRunner.cpp:4545]
19:01:06     INFO -  #24: do_main [browser/app/nsBrowserApp.cpp:220]
19:01:06     INFO -  #25: NS_internal_main(int,char * *,char * *) [browser/app/nsBrowserApp.cpp:360]
19:01:06     INFO -  #26: wmain [toolkit/xre/nsWindowsWMain.cpp:138]
19:01:06     INFO -  #27: __scrt_common_main_seh [f:/dd/vctools/crt/vcstartup/src/startup/exe_common.inl:264]
19:01:06     INFO -  #28: kernel32.dll + 0x53c45
19:01:06     INFO -  #29: ntdll.dll + 0x637f5
19:01:06     INFO -  #30: ntdll.dll + 0x637c8

19:00:51     INFO -  Leaked URLs:
19:00:51     INFO -    http://mochi.test:8888/tests/dom/workers/test/serviceworkers/fetch/synthesized-redirect-twice-real-file.txt
19:00:51     INFO -    http://mochi.test:8888/tests/dom/workers/test/serviceworkers/fetch/index.html
19:00:51     INFO -    http://mochi.test:8888/tests/dom/workers/test/serviceworkers/fetch/fetch_worker_script.js
19:00:51     INFO -    http://mochi.test:8888/tests/dom/workers/test/serviceworkers/fetch/fetch_worker_script.js
19:00:51     INFO -    http://mochi.test:8888/tests/dom/workers/test/serviceworkers/synthesized-redirect-real-file.txt
19:00:51     INFO -    http://mochi.test:8888/tests/dom/workers/test/serviceworkers/fetch/synthesized-redirect-twice-real-file.txt
19:00:51     INFO -    http://mochi.test:8888/tests/dom/workers/test/serviceworkers/fetch/fetch_worker_script.js
19:00:51     INFO -    http://mochi.test:8888/tests/dom/workers/test/serviceworkers/fetch/synthesized-redirect-twice-real-file.txt
Flags: needinfo?(ryanvm)
I managed to reproduce this on linux in an rr chaos session.  I guess rr isn't terrible.
I'm having difficulty getting rr to replay on my machine, but at least it reproduces reliably under chaos mode.  Should make it easier to track down even without replay.
Finally have a config which lets me replay a capture of this.  Will debug tomorrow.
Whiteboard: [MemShrink]
Whiteboard: [MemShrink] → [MemShrink:P2]
It seems the leaked URLs correspond to channels going through this could with shouldIntercept=true:

  https://dxr.mozilla.org/mozilla-central/source/netwerk/protocol/http/HttpChannelParent.cpp?from=HttpChannelParent.cpp#516

So e10s parent-side redirection with interception.
Try confirms that we can run leak-free on Windows with just test_fetch_event.html skipped, so I'll be pushing a patch shortly that does just that. Other than that, test_notificationclick_focus.html is permafail on Android debug due to bug 1259338 but we're otherwise skip-free now!
Flags: needinfo?(ryanvm)
Keywords: leave-open
FWIW, I've seen chaos mode leak URLs from other tests in the dir that perform redirects.
I'm not currently working on this.  I think we should wait for Josh's network interception changes.
Assignee: bkelly → nobody
Status: ASSIGNED → NEW
Depends on: 1231222
Intermittent e10s test leak
Priority: -- → P3
See Also: → 1311290
On Feb 3rd this started up on win8-debug in the dom/workers/test/serviceworkers/ directory.

I did a series of retriggers here:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=win%20x64%20debug%20mochitest-e10s-4&tochange=eefc0b77e8cd3634483af8f2ab1c5c8338d85866&fromchange=0a1c94d2176286fe2316988f7641165c3090da8b&selectedJob=74150072

ni myself to follow up here on the retriggers.

It would be nice to bisect this down, that will take some time as there are many tests in this manifest:
https://dxr.mozilla.org/mozilla-central/source/dom/workers/test/serviceworkers/mochitest.ini
Flags: needinfo?(jmaher)
:bkelly- can you help figure out which test is leaking or follow along and help come to a resolution of this bug?
Flags: needinfo?(jmaher) → needinfo?(bkelly)
Andrew, can you find someone to investigate this leak?
Flags: needinfo?(bkelly) → needinfo?(overholt)
Catalin, please take a look.
Flags: needinfo?(overholt) → needinfo?(catalin.badea392)
oh retriggering helped pinpoint the exact failure (bug 1336043):
https://hg.mozilla.org/integration/mozilla-inbound/rev/eefc0b77e8cd3634483af8f2ab1c5c8338d85866

asking :baku for info here as he authored bug 1336043.
Depends on: 1336043
Flags: needinfo?(amarchesini)
Running test_fetch_event.html --repeat=10 triggers the leak consistently on my machine.
Assignee: nobody → catalin.badea392
Flags: needinfo?(catalin.badea392)
This might be the same leak as bug 1326136.
See Also: → 1326136
now that bug 1326136 is fixed, I still see errors daily related to this test, it has dropped in frequency about a week ago though, not sure why.
Whiteboard: [MemShrink:P2] → [MemShrink:P2][stockwell unknown]
there is a leak here:
11:02:30     INFO - GECKO(1812) | nsStringStats
11:02:30     INFO - GECKO(1812) |  => mAllocCount:         276144
11:02:30     INFO - GECKO(1812) |  => mReallocCount:        57096
11:02:30     INFO - GECKO(1812) |  => mFreeCount:          276144
11:02:30     INFO - GECKO(1812) |  => mShareCount:         495693
11:02:30     INFO - GECKO(1812) |  => mAdoptCount:          28101
11:02:30     INFO - GECKO(1812) |  => mAdoptFreeCount:      28101
11:02:30     INFO - GECKO(1812) |  => Process ID: 1813, Thread ID: 140735240848128
11:02:30     INFO - GECKO(1812) | [Parent 1812] WARNING: NS_ENSURE_TRUE(mDB) failed: file /home/worker/workspace/build/src/netwerk/cache/nsDiskCacheDeviceSQL.cpp, line 1426
11:02:30     INFO - GECKO(1812) | [Parent 1812] WARNING: nsAppShell::Exit() called redundantly: file /home/worker/workspace/build/src/widget/cocoa/nsAppShell.mm, line 684
11:02:31     INFO - GECKO(1812) | [Parent 1812] WARNING: NS_ENSURE_TRUE(maybeContext) failed: file /home/worker/workspace/build/src/xpcom/threads/nsThread.cpp, line 1009
11:02:31     INFO - GECKO(1812) | [Parent 1812] WARNING: 'NS_FAILED(RemovePermissionChangeObserver())', file /home/worker/workspace/build/src/dom/notification/Notification.cpp, line 669
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: NS_ENSURE_TRUE(aObserver) failed: file /home/worker/workspace/build/src/modules/libpref/nsPrefBranch.cpp, line 715
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80070057: file /home/worker/workspace/build/src/modules/libpref/Preferences.cpp, line 1709
11:02:33     INFO - GECKO(1812) | --DOMWINDOW == 9 (0x11e192800) [pid = 1812] [serial = 4] [outer = 0x0] [url = about:blank]
11:02:33     INFO - GECKO(1812) | --DOMWINDOW == 8 (0x125c26800) [pid = 1812] [serial = 1] [outer = 0x0] [url = chrome://browser/content/hiddenWindow.xul]
11:02:33     INFO - GECKO(1812) | --DOMWINDOW == 7 (0x12a43b800) [pid = 1812] [serial = 5] [outer = 0x0] [url = about:blank]
11:02:33     INFO - GECKO(1812) | --DOMWINDOW == 6 (0x11b1e8000) [pid = 1812] [serial = 9] [outer = 0x0] [url = about:blank]
11:02:33     INFO - GECKO(1812) | --DOMWINDOW == 5 (0x12a43d800) [pid = 1812] [serial = 6] [outer = 0x0] [url = about:blank]
11:02:33     INFO - GECKO(1812) | --DOMWINDOW == 4 (0x12c6dc800) [pid = 1812] [serial = 10] [outer = 0x0] [url = about:blank]
11:02:33     INFO - GECKO(1812) | --DOMWINDOW == 3 (0x12857bc00) [pid = 1812] [serial = 11] [outer = 0x0] [url = about:newtab]
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!mMainThread', file /home/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 314
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!mMainThread', file /home/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 314
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!mMainThread', file /home/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 314
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!mMainThread', file /home/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 314
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!mMainThread', file /home/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 314
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!mMainThread', file /home/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 314
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!mMainThread', file /home/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 314
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!mMainThread', file /home/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 314
11:02:33     INFO - GECKO(1812) | --DOMWINDOW == 2 (0x129f9ac00) [pid = 1812] [serial = 13] [outer = 0x0] [url = about:newtab]
11:02:33     INFO - GECKO(1812) | --DOMWINDOW == 1 (0x125c27800) [pid = 1812] [serial = 2] [outer = 0x0] [url = about:blank]
11:02:33     INFO - GECKO(1812) | --DOMWINDOW == 0 (0x11e191800) [pid = 1812] [serial = 3] [outer = 0x0] [url = chrome://browser/content/browser.xul]
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!compMgr', file /home/worker/workspace/build/src/xpcom/components/nsComponentManagerUtils.cpp, line 63
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!mMainThread', file /home/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 314
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!mMainThread', file /home/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 314
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!mMainThread', file /home/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 314
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!mMainThread', file /home/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 314
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!mMainThread', file /home/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 314
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!mMainThread', file /home/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 314
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!mMainThread', file /home/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 314
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!mMainThread', file /home/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 314
11:02:33     INFO - GECKO(1812) | [Parent 1812] ###!!! ASSERTION: 2 dynamic atom(s) with non-zero refcount: http://mochi.test:8888,https://example.com: 'nonZeroRefcountAtomsCount == 0', file /home/worker/workspace/build/src/xpcom/ds/nsAtomTable.cpp, line 416
11:02:33     INFO - GECKO(1812) | #01: mozilla::VolatileBuffer::NonHeapSizeOfExcludingThis() const[/builds/slave/test/build/application/NightlyDebug.app/Contents/MacOS/XUL +0xaf800]
11:02:33     INFO - GECKO(1812) | #02: nsXPTCStubBase::Stub249()[/builds/slave/test/build/application/NightlyDebug.app/Contents/MacOS/XUL +0x14d60a]
11:02:33     INFO - GECKO(1812) | #03: workerlz4_maxCompressedSize[/builds/slave/test/build/application/NightlyDebug.app/Contents/MacOS/XUL +0x405ff17]
11:02:33     INFO - GECKO(1812) | #04: workerlz4_maxCompressedSize[/builds/slave/test/build/application/NightlyDebug.app/Contents/MacOS/XUL +0x40691fc]
11:02:33     INFO - GECKO(1812) | #05: workerlz4_maxCompressedSize[/builds/slave/test/build/application/NightlyDebug.app/Contents/MacOS/XUL +0x4069693]
11:02:33     INFO - GECKO(1812) | #06: start[/builds/slave/test/build/application/NightlyDebug.app/Contents/MacOS/firefox +0xeea]
11:02:33     INFO - GECKO(1812) | Leaked URLs:
11:02:33     INFO - GECKO(1812) |   https://example.com/tests/dom/workers/test/serviceworkers/fetch/origin/https/index-https.sjs
11:02:33     INFO - GECKO(1812) |   https://example.com/tests/dom/workers/test/serviceworkers/fetch/origin/https/index-https.sjs
11:02:33     INFO - GECKO(1812) |   http://mochi.test:8888/tests/dom/workers/test/serviceworkers/test_https_origin_after_redirect.html
11:02:33     INFO - GECKO(1812) |   http://mochi.test:8888/tests/dom/workers/test/serviceworkers/test_https_origin_after_redirect.html
11:02:33     INFO - GECKO(1812) |   http://mochi.test:8888/tests/dom/workers/test/serviceworkers/test_https_origin_after_redirect.html
11:02:33     INFO - GECKO(1812) |   https://example.org/tests/dom/workers/test/serviceworkers/fetch/origin/https/realindex.html
11:02:33     INFO - GECKO(1812) |   https://example.com/tests/dom/workers/test/serviceworkers/fetch/origin/https/index-https.sjs
11:02:33     INFO - GECKO(1812) |   http://mochi.test:8888/tests/dom/workers/test/serviceworkers/test_https_origin_after_redirect.html
11:02:33     INFO - GECKO(1812) | nsStringStats
11:02:33     INFO - GECKO(1812) |  => mAllocCount:         612616
11:02:33     INFO - GECKO(1812) |  => mReallocCount:        82475
11:02:33     INFO - GECKO(1812) |  => mFreeCount:          612404  --  LEAKED 212 !!!
11:02:33     INFO - GECKO(1812) |  => mShareCount:         501836
11:02:33     INFO - GECKO(1812) |  => mAdoptCount:          24288
11:02:33     INFO - GECKO(1812) |  => mAdoptFreeCount:      24286  --  LEAKED 2 !!!
11:02:33     INFO - GECKO(1812) |  => Process ID: 1812, Thread ID: 140735240848128
11:02:33     INFO - TEST-INFO | Main app process: exit 0
11:02:33     INFO - runtests.py | Application ran for: 0:01:56.865963
11:02:33     INFO - zombiecheck | Reading PID log: /var/folders/cf/5zhxcwh11mq5cjjb5nhm70lm00000w/T/tmp3X109Npidlog
11:02:33     INFO - ==> process 1812 launched child process 1813
11:02:33     INFO - zombiecheck | Checking for orphan process with PID: 1813
11:02:33     INFO - Stopping web server
11:02:33     INFO - Stopping web socket server
11:02:33     INFO - Stopping ssltunnel
11:02:33     INFO - TEST-INFO | leakcheck | default process: leak threshold set at 0 bytes
11:02:33     INFO - TEST-INFO | leakcheck | plugin process: leak threshold set at 0 bytes
11:02:33     INFO - TEST-INFO | leakcheck | tab process: leak threshold set at 10000 bytes
11:02:33     INFO - TEST-INFO | leakcheck | geckomediaplugin process: leak threshold set at 20000 bytes
11:02:33     INFO - TEST-INFO | leakcheck | gpu process: leak threshold set at 0 bytes
11:02:33     INFO - 
11:02:33     INFO - == BloatView: ALL (cumulative) LEAK AND BLOAT STATISTICS, default process 1812
11:02:33     INFO - 
11:02:33     INFO -      |<----------------Class--------------->|<-----Bytes------>|<----Objects---->|
11:02:33     INFO -      |                                      | Per-Inst   Leaked|   Total      Rem|
11:02:33     INFO -    0 |TOTAL                                 |       24    24114|17142113      375|
11:02:33     INFO -   65 |CacheEntry                            |      256      256|     529        1|
11:02:33     INFO -   68 |CacheEntryHandle                      |       32       32|    7738        1|
11:02:33     INFO -   69 |CacheFile                             |      344      344|     529        1|
11:02:33     INFO -   77 |CacheFileMetadata                     |      200      200|     529        1|
11:02:33     INFO -   78 |CacheFileOutputStream                 |      104      104|     473        1|
11:02:33     INFO -   87 |CacheStorageService                   |      304      304|       1        1|
11:02:33     INFO -   93 |CancelableRunnable                    |       48       48|   48436        1|
11:02:33     INFO -   98 |ChannelEventQueue                     |       72       72|     977        1|
11:02:33     INFO -  132 |CondVar                               |       40      400|    1132       10|
11:02:33     INFO -  149 |ConsoleReportCollector                |       64      128|    1264        2|
11:02:33     INFO -  152 |ContentParent                         |     2200     2200|       1        1|
11:02:33     INFO -  189 |DataStorage                           |      336     1008|       3        3|
11:02:33     INFO -  318 |HttpBaseChannel                       |     1328     1328|    1042        1|
11:02:33     INFO -  319 |HttpChannelParent                     |      248      248|     977        1|
11:02:33     INFO -  320 |HttpChannelParentListener             |       88       88|     942        1|
11:02:33     INFO -  339 |IdlePeriod                            |       24       72|     240        3|
11:02:33     INFO -  362 |InterceptedChannelBase                |       80       80|      17        1|
11:02:33     INFO -  379 |LoadContext                           |      104      208|    1002        2|
11:02:33     INFO -  381 |LoadInfo                              |      192      192|    4133        1|
11:02:33     INFO -  418 |Mutex                                 |       32      768|   14454       24|
11:02:33     INFO -  461 |PBrowserParent                        |      424      424|      20        1|
11:02:33     INFO -  470 |PContentParent                        |     1832     1832|       1        1|
11:02:33     INFO -  475 |PHttpChannelParent                    |       40       40|     977        1|
11:02:33     INFO -  518 |PollableEvent                         |       24       24|       1        1|
11:02:33     INFO -  563 |ReentrantMonitor                      |       40       80|    5984        2|
11:02:33     INFO -  564 |RefCountedMonitor                     |       80       80|     154        1|
11:02:33     INFO -  570 |RequestContextService                 |       88       88|       1        1|
11:02:33     INFO -  580 |Runnable                              |       40       80|  142649        2|
11:02:33     INFO -  606 |Service                               |      144      144|       1        1|
11:02:33     INFO -  623 |SharedMemory                          |       32       32|      61        1|
11:02:33     INFO -  655 |StringAdopt                           |        1        2|   24288        2|
11:02:33     INFO -  666 |TabParent                             |     1248     1248|      20        1|
11:02:33     INFO -  684 |ThirdPartyUtil                        |       32       32|       1        1|
11:02:33     INFO -  843 |ipc::MessageChannel                   |      328      328|     176        1|
11:02:33     INFO -  888 |nsAuthURLParser                       |       24       24|       2        1|
11:02:33     INFO -  944 |nsCategoryObserver                    |       96       96|       8        1|
11:02:33     INFO -  976 |nsCookiePermission                    |       64       64|       1        1|
11:02:33     INFO -  977 |nsCookieService                       |      144      144|       1        1|
11:02:33     INFO -  983 |nsDNSService                          |      160      160|       1        1|
11:02:33     INFO - 1039 |nsEffectiveTLDService                 |       40       40|       1        1|
11:02:33     INFO - 1105 |nsHttpAuthCache::OriginClearObserver  |       32       64|       2        2|
11:02:33     INFO - 1106 |nsHttpChannel                         |     1872     1872|    1042        1|
11:02:33     INFO - 1107 |nsHttpChannelAuthProvider             |      248      248|     995        1|
11:02:33     INFO - 1109 |nsHttpConnectionInfo                  |      184      184|    1004        1|
11:02:33     INFO - 1110 |nsHttpConnectionMgr                   |      296      296|       1        1|
11:02:33     INFO - 1112 |nsHttpHandler                         |      696      696|       1        1|
11:02:33     INFO - 1113 |nsHttpRequestHead                     |      136      136|    1091        1|
11:02:33     INFO - 1116 |nsIDNService                          |       96       96|       1        1|
11:02:33     INFO - 1119 |nsIOService                           |      272      272|       1        1|
11:02:33     INFO - 1147 |nsJSPrincipals                        |       24       72|    9760        3|
11:02:33     INFO - 1158 |nsLocalFile                           |      192      576|   17563        3|
11:02:33     INFO - 1167 |nsMainThreadPtrHolder<T>              |       24       96|    5329        4|
11:02:33     INFO - 1198 |nsNetworkLinkService                  |       88       88|       1        1|
11:02:33     INFO - 1203 |nsNodeWeakReference                   |       32       32|     739        1|
11:02:33     INFO - 1225 |nsPermissionManager                   |      160      160|       1        1|
11:02:33     INFO - 1232 |nsPrefBranch                          |      128      128|      57        1|
11:02:33     INFO - 1241 |nsProxyInfo                           |      104      104|     996        1|
11:02:33     INFO - 1295 |nsSiteSecurityService                 |       80       80|       1        1|
11:02:33     INFO - 1299 |nsSocketTransportService              |      256      256|       1        1|
11:02:33     INFO - 1302 |nsStandardURL                         |      264     2112|   59471        8|
11:02:33     INFO - 1308 |nsStreamConverterService              |       72       72|       1        1|
11:02:33     INFO - 1313 |nsStringBuffer                        |        8     1696|  695090      212|
11:02:33     INFO - 1359 |nsTArray_base                         |        8      264| 3750599       33|
11:02:33     INFO - 1368 |nsThread                              |      352     1056|     239        3|
11:02:33     INFO - 1387 |nsUUIDGenerator                       |       56       56|       1        1|
11:02:33     INFO - 1406 |nsWeakReference                       |       40      360|     883        9|
11:02:33     INFO - 
11:02:33     INFO - nsTraceRefcnt::DumpStatistics: 1478 entries
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 CacheEntry
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 CacheEntryHandle
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 CacheFile
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 CacheFileMetadata
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 CacheFileOutputStream
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 CacheStorageService
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 CancelableRunnable
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 ChannelEventQueue
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 10 CondVar
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 2 ConsoleReportCollector
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 ContentParent
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 3 DataStorage
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 HttpBaseChannel
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 HttpChannelParent
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 HttpChannelParentListener
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 3 IdlePeriod
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 InterceptedChannelBase
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 2 LoadContext
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 LoadInfo
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 24 Mutex
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 PBrowserParent
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 PContentParent
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 PHttpChannelParent
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 PollableEvent
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 2 ReentrantMonitor
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 RefCountedMonitor
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 RequestContextService
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 2 Runnable
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 Service
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 SharedMemory
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 2 StringAdopt
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 TabParent
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 ThirdPartyUtil
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 ipc::MessageChannel
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsAuthURLParser
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsCategoryObserver
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsCookiePermission
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsCookieService
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsDNSService
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsEffectiveTLDService
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 2 nsHttpAuthCache::OriginClearObserver
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsHttpChannel
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsHttpChannelAuthProvider
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsHttpConnectionInfo
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsHttpConnectionMgr
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsHttpHandler
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsHttpRequestHead
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsIDNService
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsIOService
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 3 nsJSPrincipals
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 3 nsLocalFile
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 4 nsMainThreadPtrHolder<T>
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsNetworkLinkService
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsNodeWeakReference
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsPermissionManager
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsPrefBranch
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsProxyInfo
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsSiteSecurityService
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsSocketTransportService
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 8 nsStandardURL
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsStreamConverterService
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 212 nsStringBuffer
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 33 nsTArray_base
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 3 nsThread
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsUUIDGenerator
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 9 nsWeakReference
11:02:33    ERROR - TEST-UNEXPECTED-FAIL | leakcheck | default process: 24114 bytes leaked (CacheEntry, CacheEntryHandle, CacheFile, CacheFileMetadata, CacheFileOutputStream, ...)
11:02:33     INFO - 
11:02:33     INFO - == BloatView: ALL (cumulative) LEAK AND BLOAT STATISTICS, tab process 1813
11:02:33     INFO - 
11:02:33     INFO -      |<----------------Class--------------->|<-----Bytes------>|<----Objects---->|
11:02:33     INFO -      |                                      | Per-Inst   Leaked|   Total      Rem|
11:02:33     INFO -    0 |TOTAL                                 |       29        0| 6061782        0|
11:02:33     INFO - 

from this log file:
https://archive.mozilla.org/pub/firefox/tinderbox-builds/autoland-macosx64-debug/autoland_yosemite_r7-debug_test-mochitest-e10s-4-bm134-tests1-macosx-build510.txt.gz

I see an assertion, maybe that is the root cause as we might not clean up properly after hitting that?


:baku, you are ni? here as the patch author of who originally caused this to spike up a month ago.  Can you look at this bug in the next week?
Whiteboard: [MemShrink:P2][stockwell unknown] → [MemShrink:P2][stockwell needswork]
:overholt, can you help us find someone to look into this assuming :baku is not available.  This is a test we will probably narrow down and disable next week, it is one of the tests in:
dom/workers/test/serviceworkers (which is 109 tests, I assume it is one or two tests causing the leak)
Flags: needinfo?(overholt)
:Mavericks, can you bisect this down to help determine the one or two tests which are causing the leak?
Flags: needinfo?(kshriram18)
Catalin is still assigned to this, and looked at it a month ago, so he's probably the one you should needinfo about this.
Flags: needinfo?(catalin.badea392)
(I asked Catalin on IRC to take another look)
Flags: needinfo?(overholt)
dom/workers/test/serviceworkers/test_fetch_event.html is one of the tests where the leak happens. I'm working on finding the issue.
Flags: needinfo?(catalin.badea392)
in some try pushes, I determined that dom/workers/test/serviceworkers/test_fetch_event.html is the only leaking test in the directory, that narrows this down- :catalinb, let me know if you need help narrowing it down more or hacking on try pushes, etc.
Flags: needinfo?(kshriram18)
Whiteboard: [MemShrink:P2][stockwell needswork] → [MemShrink:P2][stockwell unknown]
:catalinb, a few weeks ago you were looking into why dom/workers/test/serviceworkers/test_fetch_event.html is causing a leak- have you more data?  This is still leaking quite frequently.  Please let me know what if any help you need!
Flags: needinfo?(catalin.badea392)
(In reply to Joel Maher ( :jmaher) from comment #78)
> :catalinb, a few weeks ago you were looking into why
> dom/workers/test/serviceworkers/test_fetch_event.html is causing a leak-
> have you more data?  This is still leaking quite frequently.  Please let me
> know what if any help you need!

Sorry for the late reply.

I spent a couple of days couple investigating this but couldn't find anything more conclusive than: there's a HtppChannelParent that's left hanging which causes everything else to leak. I could spend more time on this, but at this point I think it's better to wait for bug 1231222 to land, which will significantly change the fetch interception code.
Flags: needinfo?(catalin.badea392)
this has been pretty high on the failure list for the last 4 weeks, the last week really high.

:overholt, I see you as the triage owner for this component, can you help find someone from the service workers team to look into this leak issue?
Flags: needinfo?(amarchesini) → needinfo?(overholt)
We're still waiting for bug 1231222 to land. Given previous investigations haven't gone anywhere, can we disable this until bug 1231222 lands?
Flags: needinfo?(overholt) → needinfo?(jmaher)
Flags: needinfo?(jmaher)
Whiteboard: [MemShrink:P2][stockwell unknown] → [MemShrink:P2][stockwell needswork]
yeah, we can disable- I haven't seen much traction on bug 1231222, so lets figure out what to disable and move on.

I am trying to determine which test to disable:
https://treeherder.mozilla.org/#/jobs?repo=try&author=jmaher@mozilla.com&filter-searchStr=Windows%208%20x64%20debug%20Mochitest%20e10s%20Mochitest%20e10s%20M-e10s(4)&fromchange=2447e233a28af394e2d716f83f114834b7b6c416&tochange=e2a02d90c98786f02b98a4fb161740e5919bc9dc

hopefully a bit more work we can narrow this down and resolve this in the short term.
Summary: Intermittent TEST-UNEXPECTED-FAIL | leakcheck | default process: 15870 bytes leaked (CacheEntry, CacheEntryHandle, CacheFile, CacheFileMetadata, CacheFileOutputStream, ...) → Intermittent leakcheck | default process: 15870 bytes leaked (CacheEntry, CacheEntryHandle, CacheFile, CacheFileMetadata, CacheFileOutputStream, ...)
Attachment #8879964 - Flags: review?(gbrown) → review+
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/9b8d77e85563
disable test_cross_origin_url_after_redirect.html as it leaks often. r=gbrown
Keywords: leave-open
Whiteboard: [MemShrink:P2][stockwell needswork] → [MemShrink:P2][stockwell disabled]
Priority: P3 → P5
Whiteboard: [MemShrink:P2][stockwell disabled] → [MemShrink:P2][stockwell needswork:DOM]
Whiteboard: [MemShrink:P2][stockwell needswork:DOM] → [MemShrink:P2][stockwell disabled]
Hey Tom, this is something needs investigation. Would you have bandwidth to take a look?
Flags: needinfo?(ttung)
Based on the previous investigation in comment 80, it's better to wait for bug 1231222 to land. I'd like to work on telemetry first.
Flags: needinfo?(ttung)
Whiteboard: [MemShrink:P2][stockwell needswork:DOM] → [MemShrink:P2][stockwell needswork:owner]
:tt, I don't see any action in bug 1231222 for the last 4 months, this is a frequent intermittent- can you help move bug 1231222 along to get it done faster?
Flags: needinfo?(ttung)
(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #112)
> :tt, I don't see any action in bug 1231222 for the last 4 months, this is a
> frequent intermittent- can you help move bug 1231222 along to get it done
> faster?

I'm afraid that I cannot help too much for bug 1231222 since it's not easy to split the work inside. Moreover, I believe :asuth is still working on it.
Flags: needinfo?(ttung)
:tt, can you recommend someone else to work on this if you don't have time?  bug 1231222 doesn't seem to be in active development
Flags: needinfo?(ttung)
Sorry for not responding more promptly on that bug; we're aggressively trying to resolve all of these things, but they're a big mess, :bkelly came up with a revised plan based around bug 1391693 that simplifies the intercept situation that we're going to try and aggressively pursue.  Re: the parts I've been working on, I've been putting our 55 release regression fires (I miss Aurorora!) but should be back to this area of stuff by Monday.
Flags: needinfo?(ttung)
Priority: -- → P3
You need to log in before you can comment on or make changes to this bug.