Open Bug 1634641 Opened 6 months ago Updated 3 days ago

Intermittent LeakSanitizer | leak at mozilla::dom::Console::CreateInstance, mozilla::dom::console_Binding::createInstance, CallJSNative, js::InternalCallOrConstruct

Categories

(Firefox :: Downloads Panel, defect, P5)

defect

Tracking

()

People

(Reporter: intermittent-bug-filer, Unassigned, NeedInfo)

References

(Regression)

Details

(Keywords: intermittent-failure, leave-open, memory-leak, Whiteboard: [stockwell disabled][stockwell needswork:owner])

Attachments

(1 file)

Filed by: nbeleuzu [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=300342402&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/JlG4IiXKQN65Rb36JQvBOg/runs/0/artifacts/public/logs/live_backing.log


[task 2020-05-01T06:48:15.320Z] 06:48:15 INFO - GECKO(2865) | -----------------------------------------------------
[task 2020-05-01T06:48:15.362Z] 06:48:15 INFO - GECKO(2865) | -----------------------------------------------------
[task 2020-05-01T06:48:15.362Z] 06:48:15 INFO - GECKO(2865) | Suppressions used:
[task 2020-05-01T06:48:15.362Z] 06:48:15 INFO - GECKO(2865) | count bytes template
[task 2020-05-01T06:48:15.362Z] 06:48:15 INFO - GECKO(2865) | 24 744 nsComponentManagerImpl
[task 2020-05-01T06:48:15.362Z] 06:48:15 INFO - GECKO(2865) | 633 18287 libfontconfig.so
[task 2020-05-01T06:48:15.362Z] 06:48:15 INFO - GECKO(2865) | 1 11 libglib-2.0.so
[task 2020-05-01T06:48:15.362Z] 06:48:15 INFO - GECKO(2865) | -----------------------------------------------------
[task 2020-05-01T06:48:15.423Z] 06:48:15 INFO - GECKO(2865) | -----------------------------------------------------
[task 2020-05-01T06:48:15.423Z] 06:48:15 INFO - GECKO(2865) | Suppressions used:
[task 2020-05-01T06:48:15.423Z] 06:48:15 INFO - GECKO(2865) | count bytes template
[task 2020-05-01T06:48:15.423Z] 06:48:15 INFO - GECKO(2865) | 24 744 nsComponentManagerImpl
[task 2020-05-01T06:48:15.423Z] 06:48:15 INFO - GECKO(2865) | 633 18287 libfontconfig.so
[task 2020-05-01T06:48:15.423Z] 06:48:15 INFO - GECKO(2865) | 1 11 libglib-2.0.so
[task 2020-05-01T06:48:15.423Z] 06:48:15 INFO - GECKO(2865) | -----------------------------------------------------
[task 2020-05-01T06:48:15.541Z] 06:48:15 INFO - GECKO(2865) | -----------------------------------------------------
[task 2020-05-01T06:48:15.542Z] 06:48:15 INFO - GECKO(2865) | Suppressions used:
[task 2020-05-01T06:48:15.543Z] 06:48:15 INFO - GECKO(2865) | count bytes template
[task 2020-05-01T06:48:15.544Z] 06:48:15 INFO - GECKO(2865) | 24 744 nsComponentManagerImpl
[task 2020-05-01T06:48:15.544Z] 06:48:15 INFO - GECKO(2865) | 633 18287 libfontconfig.so
[task 2020-05-01T06:48:15.545Z] 06:48:15 INFO - GECKO(2865) | 1 11 libglib-2.0.so
[task 2020-05-01T06:48:15.546Z] 06:48:15 INFO - GECKO(2865) | -----------------------------------------------------
[task 2020-05-01T06:48:15.624Z] 06:48:15 INFO - GECKO(2865) | -----------------------------------------------------
[task 2020-05-01T06:48:15.624Z] 06:48:15 INFO - GECKO(2865) | Suppressions used:
[task 2020-05-01T06:48:15.625Z] 06:48:15 INFO - GECKO(2865) | count bytes template
[task 2020-05-01T06:48:15.625Z] 06:48:15 INFO - GECKO(2865) | 24 744 nsComponentManagerImpl
[task 2020-05-01T06:48:15.625Z] 06:48:15 INFO - GECKO(2865) | 633 18287 libfontconfig.so
[task 2020-05-01T06:48:15.626Z] 06:48:15 INFO - GECKO(2865) | 1 11 libglib-2.0.so
[task 2020-05-01T06:48:15.626Z] 06:48:15 INFO - GECKO(2865) | -----------------------------------------------------
[task 2020-05-01T06:48:17.072Z] 06:48:17 INFO - GECKO(2865) | -----------------------------------------------------
[task 2020-05-01T06:48:17.074Z] 06:48:17 INFO - GECKO(2865) | Suppressions used:
[task 2020-05-01T06:48:17.075Z] 06:48:17 INFO - GECKO(2865) | count bytes template
[task 2020-05-01T06:48:17.076Z] 06:48:17 INFO - GECKO(2865) | 33 992 nsComponentManagerImpl
[task 2020-05-01T06:48:17.076Z] 06:48:17 INFO - GECKO(2865) | 633 18083 libfontconfig.so
[task 2020-05-01T06:48:17.077Z] 06:48:17 INFO - GECKO(2865) | 19 119 libglib-2.0.so
[task 2020-05-01T06:48:17.077Z] 06:48:17 INFO - GECKO(2865) | 4 832 mozJSComponentLoader
[task 2020-05-01T06:48:17.079Z] 06:48:17 INFO - GECKO(2865) | -----------------------------------------------------
[task 2020-05-01T06:48:17.180Z] 06:48:17 INFO - TEST-INFO | Main app process: exit 0
[task 2020-05-01T06:48:17.181Z] 06:48:17 INFO - TEST-INFO | LeakSanitizer | To show the addresses of leaked objects add report_objects=1 to LSAN_OPTIONS
[task 2020-05-01T06:48:17.182Z] 06:48:17 INFO - TEST-INFO | LeakSanitizer | This can be done in testing/mozbase/mozrunner/mozrunner/utils.py
[task 2020-05-01T06:48:17.182Z] 06:48:17 ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at mozilla::dom::Console::CreateInstance, mozilla::dom::console_Binding::createInstance, CallJSNative, js::InternalCallOrConstruct

hello, I am new to Mozilla contribution can I work on this with your help.

There were 36 failures in the past week on linux1804-64-asan opt.

Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=305251658&repo=autoland&lineNumber=4962

[task 2020-06-05T18:30:56.037Z] 18:30:56 INFO - TEST-START | dom/security/test/general/test_xfo_error_page.html
[task 2020-06-05T18:30:59.327Z] 18:30:59 INFO - GECKO(2267) | MEMORY STAT | vsize 20974583MB | residentFast 871MB
[task 2020-06-05T18:31:01.351Z] 18:31:01 INFO - TEST-OK | dom/security/test/general/test_xfo_error_page.html | took 5311ms
[task 2020-06-05T18:31:04.359Z] 18:31:04 INFO - Error: Unable to restore focus, expect failures and timeouts.
[task 2020-06-05T18:31:04.397Z] 18:31:04 INFO - TEST-START | Shutdown
[task 2020-06-05T18:31:04.397Z] 18:31:04 INFO - Passed: 186
[task 2020-06-05T18:31:04.398Z] 18:31:04 INFO - Failed: 0
[task 2020-06-05T18:31:04.398Z] 18:31:04 INFO - Todo: 5
[task 2020-06-05T18:31:04.398Z] 18:31:04 INFO - Mode: e10s
[task 2020-06-05T18:31:04.398Z] 18:31:04 INFO - Slowest: 12352ms - /tests/dom/security/test/general/test_cache_split.html
[task 2020-06-05T18:31:04.398Z] 18:31:04 INFO - SimpleTest FINISHED
[task 2020-06-05T18:31:04.398Z] 18:31:04 INFO - TEST-INFO | Ran 1 Loops
[task 2020-06-05T18:31:04.399Z] 18:31:04 INFO - SimpleTest FINISHED
[task 2020-06-05T18:31:06.800Z] 18:31:06 INFO - GECKO(2267) | 1591381866795 Marionette TRACE Received observer notification xpcom-will-shutdown
[task 2020-06-05T18:31:06.802Z] 18:31:06 INFO - GECKO(2267) | 1591381866795 Marionette INFO Stopped listening on port 2828
[task 2020-06-05T18:31:06.802Z] 18:31:06 INFO - GECKO(2267) | 1591381866796 Marionette DEBUG Marionette stopped listening
[task 2020-06-05T18:31:07.058Z] 18:31:07 INFO - GECKO(2267) | -----------------------------------------------------
[task 2020-06-05T18:31:07.058Z] 18:31:07 INFO - GECKO(2267) | Suppressions used:
[task 2020-06-05T18:31:07.059Z] 18:31:07 INFO - GECKO(2267) | count bytes template
[task 2020-06-05T18:31:07.059Z] 18:31:07 INFO - GECKO(2267) | 24 744 nsComponentManagerImpl
[task 2020-06-05T18:31:07.060Z] 18:31:07 INFO - GECKO(2267) | 633 18287 libfontconfig.so
[task 2020-06-05T18:31:07.061Z] 18:31:07 INFO - GECKO(2267) | -----------------------------------------------------
[task 2020-06-05T18:31:07.170Z] 18:31:07 INFO - GECKO(2267) | -----------------------------------------------------
[task 2020-06-05T18:31:07.170Z] 18:31:07 INFO - GECKO(2267) | Suppressions used:
[task 2020-06-05T18:31:07.171Z] 18:31:07 INFO - GECKO(2267) | count bytes template
[task 2020-06-05T18:31:07.171Z] 18:31:07 INFO - GECKO(2267) | 24 744 nsComponentManagerImpl
[task 2020-06-05T18:31:07.172Z] 18:31:07 INFO - GECKO(2267) | 2 288 libfontconfig.so
[task 2020-06-05T18:31:07.172Z] 18:31:07 INFO - GECKO(2267) | -----------------------------------------------------
[task 2020-06-05T18:31:07.571Z] 18:31:07 INFO - GECKO(2267) | [2020-06-05T18:31:07Z ERROR xulstore::persist] removeDocument error: unavailable
[task 2020-06-05T18:31:11.901Z] 18:31:11 INFO - GECKO(2267) | ==2325==WARNING: Symbolizer buffer too small
[task 2020-06-05T18:31:21.389Z] 18:31:21 INFO - GECKO(2267) | =================================================================
[task 2020-06-05T18:31:21.389Z] 18:31:21 ERROR - GECKO(2267) | ==2325==ERROR: LeakSanitizer: detected memory leaks
[task 2020-06-05T18:31:21.390Z] 18:31:21 INFO - GECKO(2267) | Direct leak of 4096 byte(s) in 1 object(s) allocated from:
[task 2020-06-05T18:31:21.391Z] 18:31:21 INFO - GECKO(2267) | #0 0x55f704aca689 in realloc /builds/worker/fetches/llvm-project/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:164:3
[task 2020-06-05T18:31:21.392Z] 18:31:21 INFO - GECKO(2267) | #1 0x7f59f738a71a in js_arena_realloc /builds/worker/workspace/obj-build/dist/include/js/Utility.h:420:10
[task 2020-06-05T18:31:21.392Z] 18:31:21 INFO - GECKO(2267) | #2 0x7f59f738a71a in js_pod_arena_realloc<unsigned char> /builds/worker/workspace/obj-build/dist/include/js/Utility.h:624:26
[task 2020-06-05T18:31:21.392Z] 18:31:21 INFO - GECKO(2267) | #3 0x7f59f738a71a in maybe_pod_arena_realloc<unsigned char> /builds/worker/checkouts/gecko/js/src/vm/MallocProvider.h:72:12
[task 2020-06-05T18:31:21.392Z] 18:31:21 INFO - GECKO(2267) | #4 0x7f59f738a71a in unsigned char* js::MallocProvider<JS::Zone>::pod_arena_realloc<unsigned char>(unsigned long, unsigned char*, unsigned long, unsigned long) /builds/worker/checkouts/gecko/js/src/vm/MallocProvider.h:208:12
[task 2020-06-05T18:31:21.392Z] 18:31:21 INFO - GECKO(2267) | #5 0x7f59f736bd12 in pod_realloc<unsigned char> /builds/worker/checkouts/gecko/js/src/vm/MallocProvider.h:227:12
[task 2020-06-05T18:31:21.392Z] 18:31:21 INFO - GECKO(2267) | #6 0x7f59f736bd12 in js::Nursery::reallocateBuffer(JS::Zone*, js::gc::Cell*, void*, unsigned long, unsigned long) /builds/worker/checkouts/gecko/js/src/gc/Nursery.cpp:635:18
[task 2020-06-05T18:31:21.392Z] 18:31:21 INFO - GECKO(2267) | #7 0x7f59f6b4066c in js::HeapSlot* js::ReallocateObjectBuffer<js::HeapSlot>(JSContext*, JSObject*, js::HeapSlot*, unsigned int, unsigned int) /builds/worker/checkouts/gecko/js/src/gc/Nursery-inl.h:133:45
[task 2020-06-05T18:31:21.392Z] 18:31:21 INFO - GECKO(2267) | #8 0x7f59f6b4033e in js::NativeObject::growSlots(JSContext*, unsigned int, unsigned int) /builds/worker/checkouts/gecko/js/src/vm/NativeObject.cpp:383:7
[task 2020-06-05T18:31:21.392Z] 18:31:21 INFO - GECKO(2267) | #9 0x7f59f6c69af9 in updateSlotsForSpan /builds/worker/checkouts/gecko/js/src/vm/NativeObject-inl.h:549:33
[task 2020-06-05T18:31:21.392Z] 18:31:21 INFO - GECKO(2267) | #10 0x7f59f6c69af9 in setLastProperty /builds/worker/checkouts/gecko/js/src/vm/NativeObject-inl.h:587:7
[task 2020-06-05T18:31:21.392Z] 18:31:21 INFO - GECKO(2267) | #11 0x7f59f6c69af9 in getChildDataProperty /builds/worker/checkouts/gecko/js/src/vm/Shape.cpp:447:13
[task 2020-06-05T18:31:21.392Z] 18:31:21 INFO - GECKO(2267) | #12 0x7f59f6c69af9 in js::NativeObject::addDataPropertyInternal(JSContext*, JS::Handle<js::NativeObject*>, JS::Handle<JS::PropertyKey>, unsigned int, unsigned int, js::ShapeTable*, js::ShapeTable::Entry*, js::AutoKeepShapeCaches const&) /builds/worker/checkouts/gecko/js/src/vm/Shape.cpp:731:13
[task 2020-06-05T18:31:21.395Z] 18:31:21 INFO - GECKO(2267) | #13 0x7f59f6b4ba40 in addDataProperty /builds/worker/checkouts/gecko/js/src/vm/Shape-inl.h:436:10
[task 2020-06-05T18:31:21.396Z] 18:31:21 INFO - GECKO(2267) | #14 0x7f59f6b4ba40 in AddOrChangeProperty<IsAddOrChange::Add> /builds/worker/checkouts/gecko/js/src/vm/NativeObject.cpp:1450:15
[task 2020-06-05T18:31:21.396Z] 18:31:21 INFO - GECKO(2267) | #15 0x7f59f6b4ba40 in js::NativeDefineProperty(JSContext*, JS::Handle<js::NativeObject*>, JS::Handle<JS::PropertyKey>, JS::Handle<JS::PropertyDescriptor>, JS::ObjectOpResult&) /builds/worker/checkouts/gecko/js/src/vm/NativeObject.cpp:1758:10
[task 2020-06-05T18:31:21.396Z] 18:31:21 INFO - GECKO(2267) | #16 0x7f59f6aa65ee in js::DefineDataProperty(JSContext*, JS::Handle<JSObject*>, JS::Handle<JS::PropertyKey>, JS::Handle<JS::Value>, unsigned int, JS::ObjectOpResult&) /builds/worker/checkouts/gecko/js/src/vm/JSObject.cpp:2742:10
[task 2020-06-05T18:31:21.396Z] 18:31:21 INFO - GECKO(2267) | #17 0x7f59f6aa6936 in js::DefineDataProperty(JSContext*, JS::Handle<JSObject*>, JS::Handle<JS::PropertyKey>, JS::Handle<JS::Value>, unsigned int) /builds/worker/checkouts/gecko/js/src/vm/JSObject.cpp:2763:8
[task 2020-06-05T18:31:21.396Z] 18:31:21 INFO - GECKO(2267) | #18 0x7f59f67376a2 in DefineDataPropertyById /builds/worker/checkouts/gecko/js/src/jsapi.cpp:2167:10
[task 2020-06-05T18:31:21.396Z] 18:31:21 INFO - GECKO(2267) | #19 0x7f59f67376a2 in DefineDataProperty(JSContext*, JS::Handle<JSObject*>, char const*, JS::Handle<JS::Value>, unsigned int) /builds/worker/checkouts/gecko/js/src/jsapi.cpp:2247:10
[task 2020-06-05T18:31:21.396Z] 18:31:21 INFO - GECKO(2267) | #20 0x7f59f67382e1 in JS_DefineProperty(JSContext*, JS::Handle<JSObject*>, char const*, JS::Handle<JSObject*>, unsigned int) /builds/worker/checkouts/gecko/js/src/jsapi.cpp:2284:10
[task 2020-06-05T18:31:21.405Z] 18:31:21 INFO - GECKO(2267) | #21 0x7f59f07b4b33 in mozilla::dom::DefineConstructor(JSContext*, JS::Handle<JSObject*>, char const*, JS::Handle<JSObject*>) /builds/worker/checkouts/gecko/dom/bindings/BindingUtils.cpp:792:10
[task 2020-06-05T18:31:21.405Z] 18:31:21 INFO - GECKO(2267) | #22 0x7f59f07835cb in CreateInterfaceObject /builds/worker/checkouts/gecko/dom/bindings/BindingUtils.cpp:888:26
[task 2020-06-05T18:31:21.406Z] 18:31:21 INFO - GECKO(2267) | #23 0x7f59f07835cb in mozilla::dom::CreateInterfaceObjects(JSContext*, JS::Handle<JSObject*>, JS::Handle<JSObject*>, JSClass const*, JS::Heap<JSObject*>, JS::Handle<JSObject>, JSClass const*, unsigned int, mozilla::dom::NamedConstructor const*, JS::Heap<JSObject*>, mozilla::dom::NativePropertiesN<7> const, mozilla::dom::NativePropertiesN<7> const*, char const*, bool, char const* const*, bool, char const* const*) /builds/worker/checkouts/gecko/dom/bindings/BindingUtils.cpp:1061:17
[task 2020-06-05T18:31:21.406Z] 18:31:21 INFO - GECKO(2267) | #24 0x7f59efda5197 in mozilla::dom::XMLDocument_Binding::CreateInterfaceObjects(JSContext*, JS::Handle<JSObject*>, mozilla::dom::ProtoAndIfaceCache&, bool) /builds/worker/workspace/obj-build/dom/bindings/XMLDocumentBinding.cpp:451:3
[task 2020-06-05T18:31:21.406Z] 18:31:21 INFO - GECKO(2267) | #25 0x7f59f079759f in mozilla::dom::GetPerInterfaceObjectHandle(JSContext*, unsigned long, void ()(JSContext, JS::Handle<JSObject*>, mozilla::dom::ProtoAndIfaceCache&, bool), bool) /builds/worker/checkouts/gecko/dom/bindings/BindingUtils.cpp:4182:5
[task 2020-06-05T18:31:21.407Z] 18:31:21 INFO - GECKO(2267) | #26 0x7f59efda3d88 in GetProtoObjectHandle /builds/worker/workspace/obj-build/dist/include/mozilla/dom/XMLDocumentBinding.h:61:12
[task 2020-06-05T18:31:21.407Z] 18:31:21 INFO - GECKO(2267) | #27 0x7f59efda3d88 in mozilla::dom::XMLDocument_Binding::Wrap(JSContext*, mozilla::dom::XMLDocument*, nsWrapperCache*, JS::Handle<JSObject*>, JS::MutableHandle<JSObject*>) /builds/worker/workspace/obj-build/dom/bindings/XMLDocumentBinding.cpp:355:42
[task 2020-06-05T18:31:21.407Z] 18:31:21 INFO - GECKO(2267) | #28 0x7f59f25ecb2e in Wrap<mozilla::dom::XMLDocument> /builds/worker/workspace/obj-build/dist/include/mozilla/dom/XMLDocumentBinding.h:42:12
[task 2020-06-05T18:31:21.408Z] 18:31:21 INFO - GECKO(2267) | #29 0x7f59f25ecb2e in mozilla::dom::XMLDocument::WrapNode(JSContext*, JS::Handle<JSObject*>) /builds/worker/checkouts/gecko/dom/xml/XMLDocument.cpp:343:10
[task 2020-06-05T18:31:21.408Z] 18:31:21 INFO - GECKO(2267) | #30 0x7f59eed8423b in nsINode::WrapObject(JSContext*, JS::Handle<JSObject*>) /builds/worker/checkouts/gecko/dom/base/nsINode.cpp:2974:34
[task 2020-06-05T18:31:21.409Z] 18:31:21 INFO - GECKO(2267) | #31 0x7f59eeb68a22 in DoGetOrCreateDOMReflector<mozilla::dom::Document, mozilla::dom::binding_detail::eWrapIntoContextCompartment> /builds/worker/workspace/obj-build/dist/include/mozilla/dom/BindingUtils.h:1053:18

Flags: needinfo?(odvarko)
Whiteboard: [stockwell needswork:owner]

There are 189 total failures for the last 30 days on linux1804-64-asan platform.

Andrew, can you please take a look?

Flags: needinfo?(continuation)
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]

This is a really generic signature, that basically indicates we leaked a bunch of stuff. It looks like it really spiked up in the last day.

Maybe somebody could narrow down when it started happening frequently a bit more? Specifically the linux1804-64-asan mochitest-browser-chrome-e10s-10 and mochitest-browser-chrome-e10s-11 failures.

Looking at about a dozen recent failures, they are happening in the directory: browser/components/downloads/test/browser/

I would expect that there's a similar very frequent leakcheck failure in that directory as well.

I don't see any recent test changes in that directory. Gijs, any guesses about what code changes in the last 24 hours might have caused a lot of failures here?

Component: JSON Viewer → Downloads Panel
Flags: needinfo?(odvarko) → needinfo?(gijskruitbosch+bugs)
Product: DevTools → Firefox

Contrary to what the graph looks like, the first failure I can see in this downloads test directory is from the 9th: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=305648028&repo=autoland&lineNumber=2627 directory

I don't see any high frequency leakcheck failures on Orange Factor. I'm not sure what that means.

I've started some retriggers in the dozen or so pushes before the failure in comment 14.

Seems this goes back a while, cf. https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedTaskRun=E2vhT00LQJi8sisFfDa29w.0&searchStr=linux%2C18.04%2Cx64%2Casan%2Copt%2Cmochitests%2Ctest-linux1804-64-asan%2Fopt-mochitest-plain-e10s-4%2Cm%284%29&revision=8a4360db7cbe3808d6466a5ad7807cb95ca47eb3

So we might need more retriggers on earlier pushes?

The leak stacks are a bit confusing; they point to JS things calling into a console API, but the JS is called from nsDocShell::InternalLoad, which appears to be from a call to shouldLoadURI(), which hasn't changed for a while... so I'm not sure what's going on. :-(

(In reply to :Gijs (he/him) from comment #18)

Seems this goes back a while, cf. https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedTaskRun=E2vhT00LQJi8sisFfDa29w.0&searchStr=linux%2C18.04%2Cx64%2Casan%2Copt%2Cmochitests%2Ctest-linux1804-64-asan%2Fopt-mochitest-plain-e10s-4%2Cm%284%29&revision=8a4360db7cbe3808d6466a5ad7807cb95ca47eb3

So we might need more retriggers on earlier pushes?

There are at least two fairly frequent leaks. One of them is the Mochitest plain 4, which maybe is happening in dom/security/test/general/, and then there's also the one happening in downloads directory, which is in Mochitest BC. I think it is the BC failure that has started happening fairly recently, and extremely often, so I would to look at trying to figure out what is causing that one, not the other one.

The leak stacks are a bit confusing; they point to JS things calling into a console API, but the JS is called from nsDocShell::InternalLoad, which appears to be from a call to shouldLoadURI(), which hasn't changed for a while... so I'm not sure what's going on. :-(

The individual leak stacks are basically irrelevant. If you look at a full log, something like 80% of the entire log is just leak stacks. This is a symptom of a leak where we are leaking like an entire DOM window, along with all of its elements and tons of JS stuff.

(In reply to Razvan Maries from comment #19)

Continued retriggers: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception%2Csuccess%2Crunning%2Cpending%2Crunnable&searchStr=linux%2C18.04%2Cx64%2Casan%2Copt%2Cmochitests%2Ctest-linux1804-64-asan%2Fopt-mochitest-plain-e10s-4%2Cm%284%29&fromchange=b05abc0736ee1ca8b97c17dbc78c76b595193220&tochange=1a8d79d0832fed96b4df55d28afe3925f6d8414f

I think it would be good to focus on the Mochitest plain BC 11 failures, not the older Mochitest plain 4 failures, because it seems to be a more recent regression, and because it appears to comprise around 90% of recent failures in this bug.

Although looking at the retriggers, the m4 failure rate appears to be quite high as well, so maybe we need to figure that out, too.

Bug 1265637 might be the debug version of at least some of these leaks. Bug 1265637 comment 101 says "The recent occurrences after the dom/security/ mochitests ran start with bug 1629866 according to retriggers." dom/security suggests it could be the same thing as the M4 leak here.

See Also: → 1265637

Specifically, so far I've found failures on this push: https://hg.mozilla.org/integration/autoland/rev/cbd35ab772537ead53ec64cc3783220a3e66b1be
but not this one: https://hg.mozilla.org/integration/autoland/rev/bf91e9e8455eda80299902401c930f085e85b002

That gets us down to a range of about a dozen pushes, and a good chunk of those can't possibly have caused a Linux BC failure.

I think it is likely that bug 1629866 is the regressor for the plain M4 failure. I'll mark the regressor for the BC failure when I find it.

Regressed by: 1629866

Based on this, I think that the regressor for the BC11 failure is bug 1606652: https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=linux%2Casan%2Cbc11&tochange=fc5808ee37a18acef7ff69eba861ccc8f24c3ae2&fromchange=a34e77d6c3bf775acf9ad987e215038fb2edab72

Denis, could you take a look? Thanks.

Note that there was a single failure before that landed, but I did a half dozen retriggers and it didn't happen again, so it may have been a pre-existing very rare intermittent failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedTaskRun=XDKoY3MdST-Q6VLt5eVDWA.0&revision=5cff19bd3c534c55d3ef0070773a91f822135c60&searchStr=linux%2Cbc11

Flags: needinfo?(continuation) → needinfo?(dpalmeiro)
Keywords: memory-leak
Regressed by: 1606652

Peter, could you please look at the intermittent Mochitest 4 leak? This feels like it is probably the same issue as described in Bug 1265637 comment 101.

Flags: needinfo?(peterv)

Thanks Andrew, taking a look asap.

Flags: needinfo?(dpalmeiro)

Clearing my ni given we have regressors now...

Flags: needinfo?(gijskruitbosch+bugs)

Changing the ~LinkedList() assert to a MOZ_RELEASE_ASSERT, I can reproduce the same issue as bug 1265637 here and also gave me a useful traceback. There are some unused tasks remaining in the parseFinishedList_ for the JS helper thread when we call JS_Shutdown. This is to be expected since we are speculatively parsing scripts now, but any remaining preload scripts should be canceled during ~ScriptLoader() to clear this list out. Running this fix through some tests at the moment.

(In reply to Andrew McCreight [:mccr8] from comment #29)

Peter, could you please look at the intermittent Mochitest 4 leak? This feels like it is probably the same issue as described in Bug 1265637 comment 101.

That patch only added a strong reference between BrowsingContext and ChildSHistory, declared to the CC. I'm trying to reproduce so I can get some logs.

(In reply to Peter Van der Beken [:peterv] from comment #37)

That patch only added a strong reference between BrowsingContext and ChildSHistory, declared to the CC. I'm trying to reproduce so I can get some logs.

I've had some success with getting CC logs off of try for leaks I couldn't reproduce locally.

Depends on: 1646793
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]

bug 1646793 has landed but there appear to still be bc failures. Denis, any ideas as to what may be causing that?

Flags: needinfo?(dpalmeiro)

Hmm, I did confirm the bc11 failures were fixed with the changes in bug 1646793: https://treeherder.mozilla.org/#/jobs?repo=try&revision=34e69ad19905812597aee2e7075ebd5c379afcb3

But I'll take another look today to see if I maybe missed something.

Flags: needinfo?(dpalmeiro)

(In reply to Denis Palmeiro [:denispal] from comment #44)

Hmm, I did confirm the bc11 failures were fixed with the changes in bug 1646793: https://treeherder.mozilla.org/#/jobs?repo=try&revision=34e69ad19905812597aee2e7075ebd5c379afcb3

But I'll take another look today to see if I maybe missed something.

Chunks (and which tests are in which chunk number) are not constant - they get re-done when tests are added/removed. For instance, the recent failures shown in orangefactor are for jobs in chunk 13, not 11.

The issue seems to be triggered in runs that include the browser mochitests in browser/components/downloads/test/browser/, and your trypush did not run those tests.

You can use a path param with mach try fuzzy to only run the relevant tests (ie ./mach try fuzzy browser/components/downloads/test/browser/).

Denis, do you have any updates on this? The failure rate here is still high with 177 total failures in the last 7 days: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2020-06-21&endday=2020-06-28&tree=trunk&bug=1634641
Thank you.

Flags: needinfo?(dpalmeiro)
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]

(In reply to Cosmin Sabou [:CosminS] from comment #51)

Denis, do you have any updates on this? The failure rate here is still high with 177 total failures in the last 7 days: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2020-06-21&endday=2020-06-28&tree=trunk&bug=1634641
Thank you.

I thought I found the issue, but it appears to only address the isEmpty asserts from bug 1265637 which appears to be a separate issue. I am still investigating this leak as my main focus, however. It is only reproducible on try for me so debugging has been a bit lengthy. I should also mention that the initial bug that regressed this is under a nightly only pref so this leak should go away in a release build.

Flags: needinfo?(dpalmeiro)

(In reply to Andrew McCreight [:mccr8] from comment #29)

Peter, could you please look at the intermittent Mochitest 4 leak? This feels like it is probably the same issue as described in Bug 1265637 comment 101.

I posted this in bug 1265637, but I'll post it here too since the bugs have become muddled. This was an existing leak, where we were leaking a ton of BrowsingContexts. Bug 1629866 made the BrowsingContext hold an additional object (ChildSHistory), and that made the leak worse, but it doesn't cause it. Somebody needs to figure out the existing leak.

Flags: needinfo?(peterv)

The browser/components/downloads/test/browser/ failure seems to be almost (but not entirely) permanent at this point. I'm not sure if that's a change, but I've looked back to at least Tuesday and it has been that way.

Ok, that's not right. Going back further, it was still orange on m-c earlier than that.

Still not entirely sure where the leak is coming from, but it seems strongly tied to parsing "resource://pdf.js/build/pdf.js" off the main thread for whatever reason. It seems there is a race somewhere since subtle harmless changes can make the leak go away entirely. There are other scripts parsed omt with similar properties but it doesn't seem like parsing any of them introduces any leaks at all which is curious.

Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]

Over in bug 1609920, I think rpl found that browser_pdfjs_preview.js was the cause of the bc leak. We should disable that test with ASan, because this is failing 100% of the time.

Assignee: nobody → nerli
Status: NEW → ASSIGNED
Pushed by csabou@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/f06c9a5fbd67
Skip browser_pdfjs_preview.js in asan builds due to frequent failures. r=jmaher
Assignee: nerli → nobody
Status: ASSIGNED → NEW
Keywords: leave-open
Whiteboard: [stockwell disable-recommended] → [stockwell disabled]
Assignee: nobody → nerli
Status: NEW → ASSIGNED
Depends on: 1652126

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=309460731&repo=autoland&lineNumber=20452 seems to be the same leak in m-bc running toolkit/components/certviewer/tests/browser/browser.ini , which doesn't appear to have anything to do with pdfjs ( https://searchfox.org/mozilla-central/search?q=pdf&path=toolkit%2Fcomponents%2Fcertviewer%2Ftests%2Fbrowser&case=false&regexp=false shows no hits).

There are also a large number of devtools test leaks. Are we any closer to figure out what is going on here, and/or is there asan-only debugging code we could add to get closer to figuring that out?

Flags: needinfo?(dpalmeiro)
Flags: needinfo?(continuation)

Bah, I noticed bug 1652126 too late - I'm guessing that is supposed to fix all of these?

Flags: needinfo?(continuation)

(In reply to :Gijs (he/him) from comment #71)

Bah, I noticed bug 1652126 too late - I'm guessing that is supposed to fix all of these?

Yes, I'm hopeful it should fix the leaks here and the asserts in bug 1265637. The leaks in m-bc look similar as well, so hopefully it should also fix those.

Flags: needinfo?(dpalmeiro)
Assignee: nerli → nobody
Status: ASSIGNED → NEW

In the last 7 days there have been 25 occurrences, all on linux1804-64-asan opt.

Recent failure: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=312497017&repo=mozilla-central&lineNumber=38981

Denis, are you planning more work to fix these, besides bug 1652126?

Flags: needinfo?(dpalmeiro)
Whiteboard: [stockwell disabled] → [stockwell disabled][stockwell needswork]
Whiteboard: [stockwell disabled][stockwell needswork] → [stockwell disabled][stockwell needswork:owner]

(In reply to Bogdan Tara[:bogdan_tara | bogdant] from comment #76)

In the last 7 days there have been 25 occurrences, all on linux1804-64-asan opt.

Recent failure: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=312497017&repo=mozilla-central&lineNumber=38981

Denis, are you planning more work to fix these, besides bug 1652126?

bug 1652126 should address this problem, but I'm still working on it.

Flags: needinfo?(dpalmeiro)

In the last 7 days there were failures of this bug. These happened on linux1804-64-asan opt.

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=313766963&repo=autoland&lineNumber=6186

Denis, it would seem this is still happening despite bug 1652126 being fixed - can you take a look?

Flags: needinfo?(dpalmeiro)

It may not be related. This intermittent failure signature is just what it looks like whenever we leak a lot of JS stuff. It looks like these failures are happening in mochitest-devtools-chrome-e10s-4, in the devtools/client/jsonview/test/ directory.

You need to log in before you can comment on or make changes to this bug.