Intermittent LeakSanitizer | leak at unknown stack
Categories
(Core :: General, defect, P5)
Tracking
()
| Tracking | Status | |
|---|---|---|
| firefox76 | --- | fixed |
People
(Reporter: intermittent-bug-filer, Assigned: jdescottes)
Details
(Keywords: intermittent-failure, Whiteboard: [stockwell unknown])
Attachments
(3 files)
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
Comment 8•8 years ago
|
||
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
Comment 11•8 years ago
|
||
Comment 12•8 years ago
|
||
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
Comment 15•8 years ago
|
||
Comment 16•8 years ago
|
||
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
Comment 23•7 years ago
|
||
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
Comment 27•7 years ago
|
||
| Comment hidden (Intermittent Failures Robot) |
Comment 29•7 years ago
|
||
Comment 30•7 years ago
|
||
Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=223710198&repo=mozilla-inbound&lineNumber=276860
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
Comment 34•7 years ago
|
||
This bug has 55 failures in the last 7 days. Occurs on Linux asan build type.
Recent log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=226793417&repo=mozilla-inbound&lineNumber=354487
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
Comment 38•7 years ago
|
||
This bug failed 32 times in the last 7 days. Occurs on linux64 debug build types.
Recent log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=227378045&repo=mozilla-inbound&lineNumber=347067
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
Comment 42•7 years ago
|
||
Comment 43•7 years ago
|
||
Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=257517998&repo=mozilla-central&lineNumber=12224
[task 2019-07-20T11:45:11.489Z] 11:45:11 INFO - TEST-START | /websockets/referrer.any.html
[task 2019-07-20T11:45:11.492Z] 11:45:11 INFO - Closing window 358629769217
[task 2019-07-20T11:45:11.581Z] 11:45:11 INFO - PID 26530 | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2019-07-20T11:45:12.365Z] 11:45:12 INFO - .
[task 2019-07-20T11:45:12.365Z] 11:45:12 INFO - TEST-OK | /websockets/referrer.any.html | took 874ms
[task 2019-07-20T11:45:16.434Z] 11:45:16 INFO - PID 26530 | SUMMARY: AddressSanitizer: 10924 byte(s) leaked in 71 allocation(s).
[task 2019-07-20T11:45:17.079Z] 11:45:17 INFO - PID 26530 | -----------------------------------------------------
[task 2019-07-20T11:45:17.079Z] 11:45:17 INFO - PID 26530 | Suppressions used:
[task 2019-07-20T11:45:17.080Z] 11:45:17 INFO - PID 26530 | count bytes template
[task 2019-07-20T11:45:17.080Z] 11:45:17 INFO - PID 26530 | 24 728 nsComponentManagerImpl
[task 2019-07-20T11:45:17.080Z] 11:45:17 INFO - PID 26530 | 4 832 mozJSComponentLoader::LoadModule
[task 2019-07-20T11:45:17.080Z] 11:45:17 INFO - PID 26530 | 611 17509 libfontconfig.so
[task 2019-07-20T11:45:17.081Z] 11:45:17 INFO - PID 26530 | 336 14792 _PR_Getfd
[task 2019-07-20T11:45:17.081Z] 11:45:17 INFO - PID 26530 | 1 29 libglib-2.0.so
[task 2019-07-20T11:45:17.081Z] 11:45:17 INFO - PID 26530 | -----------------------------------------------------
[task 2019-07-20T11:45:17.238Z] 11:45:17 INFO - Browser exited with return code 0
[task 2019-07-20T11:45:17.238Z] 11:45:17 INFO - PROCESS LEAKS None
[task 2019-07-20T11:45:17.238Z] 11:45:17 INFO - ERROR | LeakSanitizer | SUMMARY: AddressSanitizer: 13492 byte(s) leaked in 85 allocation(s).
[task 2019-07-20T11:45:17.239Z] 11:45:17 INFO - LeakSanitizer | To show the addresses of leaked objects add report_objects=1 to LSAN_OPTIONS
[task 2019-07-20T11:45:17.239Z] 11:45:17 INFO - This can be done in testing/mozbase/mozrunner/mozrunner/utils.py
[task 2019-07-20T11:45:17.239Z] 11:45:17 INFO - Allowed depth was 4
[task 2019-07-20T11:45:17.239Z] 11:45:17 INFO - TEST-FAIL | LeakSanitizer | leak at Alloc, nsTSubstring, Gecko_StartBulkWriteString, nsstring::nsAString::start_bulk_write_impl
[task 2019-07-20T11:45:17.239Z] 11:45:17 INFO - INFO | LeakSanitizer | Frame Alloc matched a expected leak
[task 2019-07-20T11:45:17.239Z] 11:45:17 INFO - TEST-FAIL | LeakSanitizer | leak at mozilla::net::CookieSettings::Create, mozilla::net::LoadInfo::LoadInfo, nsDocShell::DoURILoad, nsDocShell::InternalLoad
[task 2019-07-20T11:45:17.239Z] 11:45:17 INFO - INFO | LeakSanitizer | Frame mozilla::net::CookieSettings::Create matched a expected leak
[task 2019-07-20T11:45:17.240Z] 11:45:17 INFO - TEST-FAIL | LeakSanitizer | leak at mozilla::BasePrincipal::CreateContentPrincipal, mozilla::BasePrincipal::CreateContentPrincipal, nsScriptSecurityManager::GetChannelURIPrincipal, nsScriptSecurityManager::GetChannelResultPrincipal
[task 2019-07-20T11:45:17.240Z] 11:45:17 INFO - INFO | LeakSanitizer | Frame mozilla::BasePrincipal::CreateContentPrincipal matched a expected leak
[task 2019-07-20T11:45:17.240Z] 11:45:17 INFO - TEST-FAIL | LeakSanitizer | leak at mozilla::detail::UniqueSelector, mozilla::dom::ClientInfo::operator=, emplace, GetClientInfo
[task 2019-07-20T11:45:17.240Z] 11:45:17 INFO - INFO | LeakSanitizer | Frame GetClientInfo matched a expected leak
[task 2019-07-20T11:45:17.240Z] 11:45:17 INFO - TEST-FAIL | LeakSanitizer | leak at mozilla::detail::UniqueSelector, mozilla::dom::ClientInfo::operator=, nsSupportsWeakReference::GetWeakReference, emplace
[task 2019-07-20T11:45:17.240Z] 11:45:17 INFO - INFO | LeakSanitizer | Frame nsSupportsWeakReference::GetWeakReference matched a expected leak
[task 2019-07-20T11:45:17.240Z] 11:45:17 INFO - TEST-FAIL | LeakSanitizer | leak at js::jit::InvokeFunction, js::jit::InvokeFromInterpreterStub, Alloc, nsTSubstring
[task 2019-07-20T11:45:17.240Z] 11:45:17 INFO - INFO | LeakSanitizer | Frame Alloc matched a expected leak
[task 2019-07-20T11:45:17.240Z] 11:45:17 INFO - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at unknown stack
[task 2019-07-20T11:45:17.241Z] 11:45:17 INFO - TEST-FAIL | LeakSanitizer | leak at mozilla::SchedulerGroup::CreateEventTargetFor, mozilla::SchedulerGroup::CreateEventTargets, mozilla::dom::TabGroup::TabGroup, mozilla::dom::ContentChild::GetSpecificMessageEventTarget
[task 2019-07-20T11:45:17.241Z] 11:45:17 INFO - INFO | LeakSanitizer | Frame mozilla::SchedulerGroup::CreateEventTargetFor matched a expected leak
[task 2019-07-20T11:45:17.241Z] 11:45:17 INFO - TEST-FAIL | LeakSanitizer | leak at mozilla::net::WebSocketEventService::GetOrCreate, mozilla::dom::WebSocketImpl::Init, mozilla::dom::WebSocket::ConstructorCommon, mozilla::dom::WebSocket::Constructor
[task 2019-07-20T11:45:17.241Z] 11:45:17 INFO - INFO | LeakSanitizer | Frame mozilla::net::WebSocketEventService::GetOrCreate matched a expected leak
[task 2019-07-20T11:45:17.241Z] 11:45:17 INFO - TEST-FAIL | LeakSanitizer | leak at mozilla::net::nsStandardURL::TemplatedMutator, Init, mozilla::net::nsStandardURL::TemplatedMutator, operator
[task 2019-07-20T11:45:17.241Z] 11:45:17 INFO - INFO | LeakSanitizer | Frame mozilla::net::nsStandardURL::TemplatedMutator matched a expected leak
[task 2019-07-20T11:45:17.241Z] 11:45:17 INFO - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at setInternal, js::Proxy::set, SetProperty, SetPropertyOperation
[task 2019-07-20T11:45:17.242Z] 11:45:17 INFO - TEST-FAIL | LeakSanitizer | leak at Alloc, nsTSubstring, Gecko_StartBulkWriteCString, nsstring::nsACString::start_bulk_write_impl
[task 2019-07-20T11:45:17.242Z] 11:45:17 INFO - INFO | LeakSanitizer | Frame Alloc matched a expected leak
[task 2019-07-20T11:45:17.246Z] 11:45:17 INFO - TEST-FAIL | LeakSanitizer | leak at Create, nsAtomTable::Atomize, mozilla::BasePrincipal::FinishInit, mozilla::ContentPrincipal::Init
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
Comment 46•7 years ago
|
||
| Comment hidden (Intermittent Failures Robot) |
Comment 48•6 years ago
|
||
Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=290763476&repo=autoland&lineNumber=17847
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
Comment 52•6 years ago
|
||
Julian, this has spiked after Bug 1620194 landed here: https://treeherder.mozilla.org/?#/jobs?repo=autoland&group_state=expanded&searchStr=linux%2C18.04%2Cx64%2Casan%2Copt%2Cmochitests%2Ctest-linux1804-64-asan%2Fopt-mochitest-devtools-chrome-e10s-4%2Cm%28dt4%29&tochange=1f037156fba1beddb204c329515ff27e5b9efcb0&test_paths=devtools%2Fclient%2Finspector%2Frules%2Ftest%2F&fromchange=c620d986a9eb16f3332d767177cdcb5a03735852&selectedJob=294272242
The shutdown leaks happen after https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=293657988&repo=autoland&lineNumber=331665
It has 36 failures since the 18th of March: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2020-03-15&endday=2020-03-22&tree=trunk&bug=1395843
Recent failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=294217423&repo=mozilla-central
[task 2020-03-21T22:53:25.462Z] 22:53:25 INFO - TEST-START | Shutdown
[task 2020-03-21T22:53:25.462Z] 22:53:25 INFO - Browser Chrome Test Summary
[task 2020-03-21T22:53:25.463Z] 22:53:25 INFO - Passed: 4530
[task 2020-03-21T22:53:25.464Z] 22:53:25 INFO - Failed: 0
[task 2020-03-21T22:53:25.464Z] 22:53:25 INFO - Todo: 0
[task 2020-03-21T22:53:25.464Z] 22:53:25 INFO - Mode: e10s
[task 2020-03-21T22:53:25.464Z] 22:53:25 INFO - *** End BrowserChrome Test Results ***
[task 2020-03-21T22:53:26.158Z] 22:53:26 INFO - GECKO(2252) | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2020-03-21T22:53:28.150Z] 22:53:28 INFO - GECKO(2252) | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2020-03-21T22:53:28.156Z] 22:53:28 INFO - GECKO(2252) | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2020-03-21T22:53:29.107Z] 22:53:29 INFO - GECKO(2252) | -----------------------------------------------------
[task 2020-03-21T22:53:29.109Z] 22:53:29 INFO - GECKO(2252) | Suppressions used:
[task 2020-03-21T22:53:29.109Z] 22:53:29 INFO - GECKO(2252) | count bytes template
[task 2020-03-21T22:53:29.109Z] 22:53:29 INFO - GECKO(2252) | 27 832 nsComponentManagerImpl
[task 2020-03-21T22:53:29.109Z] 22:53:29 INFO - GECKO(2252) | 633 18287 libfontconfig.so
[task 2020-03-21T22:53:29.109Z] 22:53:29 INFO - GECKO(2252) | 1 11 libglib-2.0.so
[task 2020-03-21T22:53:29.109Z] 22:53:29 INFO - GECKO(2252) | -----------------------------------------------------
[task 2020-03-21T22:53:29.819Z] 22:53:29 INFO - GECKO(2252) | ==2617==WARNING: failed to fork (errno 12)
[task 2020-03-21T22:53:29.827Z] 22:53:29 INFO - GECKO(2252) | ==2617==WARNING: failed to fork (errno 12)
[task 2020-03-21T22:53:29.827Z] 22:53:29 INFO - GECKO(2252) | ==2617==WARNING: failed to fork (errno 12)
[task 2020-03-21T22:53:29.827Z] 22:53:29 INFO - GECKO(2252) | ==2617==WARNING: failed to fork (errno 12)
[task 2020-03-21T22:53:29.827Z] 22:53:29 INFO - GECKO(2252) | ==2617==WARNING: failed to fork (errno 12)
[task 2020-03-21T22:53:29.827Z] 22:53:29 INFO - GECKO(2252) | ==2617==WARNING: Failed to use and restart external symbolizer!
[task 2020-03-21T22:53:31.445Z] 22:53:31 INFO - GECKO(2252) | 1584831211410 Marionette TRACE Received observer notification xpcom-will-shutdown
[task 2020-03-21T22:53:31.447Z] 22:53:31 INFO - GECKO(2252) | 1584831211414 Marionette INFO Stopped listening on port 2828
[task 2020-03-21T22:53:31.448Z] 22:53:31 INFO - GECKO(2252) | 1584831211415 Marionette DEBUG Marionette stopped listening
[task 2020-03-21T22:53:31.448Z] 22:53:31 INFO - GECKO(2252) | -----------------------------------------------------
[task 2020-03-21T22:53:31.448Z] 22:53:31 INFO - GECKO(2252) | Suppressions used:
[task 2020-03-21T22:53:31.448Z] 22:53:31 INFO - GECKO(2252) | count bytes template
[task 2020-03-21T22:53:31.448Z] 22:53:31 INFO - GECKO(2252) | 27 832 nsComponentManagerImpl
[task 2020-03-21T22:53:31.448Z] 22:53:31 INFO - GECKO(2252) | 633 18287 libfontconfig.so
[task 2020-03-21T22:53:31.448Z] 22:53:31 INFO - GECKO(2252) | 1 11 libglib-2.0.so
[task 2020-03-21T22:53:31.448Z] 22:53:31 INFO - GECKO(2252) | -----------------------------------------------------
[task 2020-03-21T22:53:31.448Z] 22:53:31 INFO - GECKO(2252) | -----------------------------------------------------
[task 2020-03-21T22:53:31.448Z] 22:53:31 INFO - GECKO(2252) | Suppressions used:
[task 2020-03-21T22:53:31.448Z] 22:53:31 INFO - GECKO(2252) | count bytes template
[task 2020-03-21T22:53:31.448Z] 22:53:31 INFO - GECKO(2252) | 45 1360 nsComponentManagerImpl
[task 2020-03-21T22:53:31.448Z] 22:53:31 INFO - GECKO(2252) | 633 18287 libfontconfig.so
[task 2020-03-21T22:53:31.448Z] 22:53:31 INFO - GECKO(2252) | 1 11 libglib-2.0.so
[task 2020-03-21T22:53:31.448Z] 22:53:31 INFO - GECKO(2252) | -----------------------------------------------------
[task 2020-03-21T22:53:31.466Z] 22:53:31 INFO - GECKO(2252) | -----------------------------------------------------
[task 2020-03-21T22:53:31.466Z] 22:53:31 INFO - GECKO(2252) | Suppressions used:
[task 2020-03-21T22:53:31.466Z] 22:53:31 INFO - GECKO(2252) | count bytes template
[task 2020-03-21T22:53:31.466Z] 22:53:31 INFO - GECKO(2252) | 60 1800 nsComponentManagerImpl
[task 2020-03-21T22:53:31.466Z] 22:53:31 INFO - GECKO(2252) | 633 18287 libfontconfig.so
[task 2020-03-21T22:53:31.466Z] 22:53:31 INFO - GECKO(2252) | 1 11 libglib-2.0.so
[task 2020-03-21T22:53:31.467Z] 22:53:31 INFO - GECKO(2252) | -----------------------------------------------------
[task 2020-03-21T22:53:59.715Z] 22:53:59 INFO - GECKO(2252) | =================================================================
[task 2020-03-21T22:53:59.717Z] 22:53:59 ERROR - GECKO(2252) | ==2617==ERROR: LeakSanitizer: detected memory leaks
Could you please have over what's going on here? Thank you.
| Comment hidden (Intermittent Failures Robot) |
| Assignee | ||
Comment 54•6 years ago
|
||
Thanks! Looking, for now the only thing I can see which seems suspicious is that the logs are completely flooded with a huge object being logged:
if (accessible.actorID) {
console.warn(`Error fetching accessible details: `, accessible, error);
}
at https://searchfox.org/mozilla-central/rev/3fd53c47864fedb916f0ed8f002f15456324f729/devtools/client/accessibility/reducers/details.js#42
This log forces a dump of 160,000 lines of text, maybe this stresses the test harness? And maybe my change made the warning more frequent.
Try pushes in progress to see if changing this has any impact on the failure rate. Keeping the ni? for now.
| Assignee | ||
Updated•6 years ago
|
| Assignee | ||
Comment 55•6 years ago
|
||
Updated•6 years ago
|
| Assignee | ||
Comment 56•6 years ago
|
||
(didn't mean to clean the ni)
Updated•6 years ago
|
| Assignee | ||
Comment 57•6 years ago
|
||
Comment 58•6 years ago
|
||
Comment 59•6 years ago
|
||
| Assignee | ||
Comment 60•6 years ago
|
||
Landing a few improvements, although my try pushes don't indicate they fix the leak (the second one might reduce the failure rate?) .
I found failures dating back to before I landed my patches. Some of them are DevTools, some are not. What is interesting is that for the DevTools one they also occurred in the same chunk, running the same test suites (dt4 linux asan), typically at the end of devtools/client/inspector/rules/test/
The logs are very similar to the ones we have now. So this is really not a new failure, but it became much more frequent. Let's see if the patches which landed so far help reduce the failure rate.
| Assignee | ||
Comment 61•6 years ago
|
||
Depends on D67845
Comment 62•6 years ago
|
||
| bugherder | ||
Comment 63•6 years ago
|
||
| Assignee | ||
Updated•6 years ago
|
| Assignee | ||
Comment 64•6 years ago
|
||
Clearing leave-open, try pushes seem to indicate the last patch fixes the issue.
Comment 65•6 years ago
|
||
| bugherder | ||
| Comment hidden (Intermittent Failures Robot) |
Description
•