Closed Bug 1589207 Opened 4 years ago Closed 4 years ago

Perma [tier 2] TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at EntrySlotOrCreate, EntrySlotOrCreate, mozilla::dom::BrowsingContext_Binding::CreateInterfaceObjects, mozilla::dom::GetPerInterfaceObjectHandle and other leaks

Categories

(Core :: DOM: Events, defect)

Product:
Core
Component:
DOM: Events
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla71
Project Flags:
Fission Milestone M4
Tracking Flags:
Tracking Status
firefox-esr60 --- unaffected
firefox-esr68 --- unaffected
firefox67 --- unaffected
firefox68 --- unaffected
firefox69 --- unaffected
firefox70 --- unaffected
firefox71 --- fixed

People

(Reporter: malexandru, Assigned: decoder)

References

(Regression)

Details

(Keywords: intermittent-failure, regression)

Attachments

(1 file)

Bug 1589207 - Suppress two leaks in WebPlatform tests. r?jgraham
47 bytes, text/x-phabricator-request
Details | Review

Parsed log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=271539367&repo=autoland&lineNumber=39327

Raw log: https://queue.taskcluster.net/v1/task/BkPYKzHpQn6ssmXlbb4ZVw/runs/0/artifacts/public/logs/live_backing.log

[task 2019-10-16T18:15:25.551Z] 18:15:25 INFO - PID 8509 | -----------------------------------------------------
[task 2019-10-16T18:15:25.551Z] 18:15:25 INFO - PID 8509 | Suppressions used:
[task 2019-10-16T18:15:25.551Z] 18:15:25 INFO - PID 8509 | count bytes template
[task 2019-10-16T18:15:25.552Z] 18:15:25 INFO - PID 8509 | 497 51429 libc.so
[task 2019-10-16T18:15:25.553Z] 18:15:25 INFO - PID 8509 | 156 24779 nsComponentManagerImpl
[task 2019-10-16T18:15:25.553Z] 18:15:25 INFO - PID 8509 | 2584 101965 libfontconfig.so
[task 2019-10-16T18:15:25.554Z] 18:15:25 INFO - PID 8509 | 1 29 libglib-2.0.so
[task 2019-10-16T18:15:25.554Z] 18:15:25 INFO - PID 8509 | 4513 193780 js::frontend::BytecodeEmitter:
[task 2019-10-16T18:15:25.554Z] 18:15:25 INFO - PID 8509 | 596 21588 js::frontend::GeneralParser
[task 2019-10-16T18:15:25.555Z] 18:15:25 INFO - PID 8509 | 1129 55976 js::frontend::Parse
[task 2019-10-16T18:15:25.555Z] 18:15:25 INFO - PID 8509 | 36 1012 xpc::CIGSHelper
[task 2019-10-16T18:15:25.556Z] 18:15:25 INFO - PID 8509 | 641 60470 mozJSComponentLoader
[task 2019-10-16T18:15:25.557Z] 18:15:25 INFO - PID 8509 | 182 11556 XPCWrappedNativeJSOps
[task 2019-10-16T18:15:25.557Z] 18:15:25 INFO - PID 8509 | -----------------------------------------------------
[task 2019-10-16T18:15:26.107Z] 18:15:26 INFO - PID 8509 | -----------------------------------------------------
[task 2019-10-16T18:15:26.108Z] 18:15:26 INFO - PID 8509 | Suppressions used:
[task 2019-10-16T18:15:26.108Z] 18:15:26 INFO - PID 8509 | count bytes template
[task 2019-10-16T18:15:26.108Z] 18:15:26 INFO - PID 8509 | 24 728 nsComponentManagerImpl
[task 2019-10-16T18:15:26.108Z] 18:15:26 INFO - PID 8509 | 4 832 mozJSComponentLoader::LoadModule
[task 2019-10-16T18:15:26.108Z] 18:15:26 INFO - PID 8509 | 611 17509 libfontconfig.so
[task 2019-10-16T18:15:26.109Z] 18:15:26 INFO - PID 8509 | 1 29 libglib-2.0.so
[task 2019-10-16T18:15:26.109Z] 18:15:26 INFO - PID 8509 | -----------------------------------------------------
[task 2019-10-16T18:15:26.205Z] 18:15:26 INFO - Browser exited with return code 0
[task 2019-10-16T18:15:26.205Z] 18:15:26 INFO - PROCESS LEAKS None
[task 2019-10-16T18:15:26.205Z] 18:15:26 INFO - ERROR | LeakSanitizer | SUMMARY: AddressSanitizer: 124936 byte(s) leaked in 1206 allocation(s).
[task 2019-10-16T18:15:26.205Z] 18:15:26 INFO - LeakSanitizer | To show the addresses of leaked objects add report_objects=1 to LSAN_OPTIONS
[task 2019-10-16T18:15:26.206Z] 18:15:26 INFO - This can be done in testing/mozbase/mozrunner/mozrunner/utils.py
[task 2019-10-16T18:15:26.206Z] 18:15:26 INFO - Allowed depth was 4
[task 2019-10-16T18:15:26.206Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at Alloc, nsTSubstring, Gecko_StartBulkWriteString, nsstring::nsAString::start_bulk_write_impl
[task 2019-10-16T18:15:26.206Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame Alloc matched a expected leak
[task 2019-10-16T18:15:26.206Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at NS_NewDOMPointerEvent, mozilla::dom::PointerEvent::GetCoalescedEvents, mozilla::dom::PointerEvent_Binding::getCoalescedEvents, mozilla::dom::binding_detail::GenericMethod
[task 2019-10-16T18:15:26.207Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame NS_NewDOMPointerEvent matched a expected leak
[task 2019-10-16T18:15:26.207Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at mozilla::SchedulerGroup::Runnable::Run, js_arena_calloc, js_pod_arena_calloc, maybe_pod_calloc
[task 2019-10-16T18:15:26.207Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame js_arena_calloc matched a expected leak
[task 2019-10-16T18:15:26.207Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at js_arena_malloc, js_pod_arena_malloc, maybe_pod_malloc, js::detail::OrderedHashTable
[task 2019-10-16T18:15:26.207Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame js_arena_malloc matched a expected leak
[task 2019-10-16T18:15:26.207Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at js_arena_malloc, js_pod_arena_malloc, maybe_pod_malloc, pod_malloc
[task 2019-10-16T18:15:26.207Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame js_arena_malloc matched a expected leak
[task 2019-10-16T18:15:26.208Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at createTable, mozilla::detail::HashTable, reserve, reserve
[task 2019-10-16T18:15:26.208Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame createTable matched a expected leak
[task 2019-10-16T18:15:26.208Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at js_arena_malloc, js_pod_arena_malloc, maybe_pod_arena_malloc, pod_arena_malloc
[task 2019-10-16T18:15:26.208Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame js_arena_malloc matched a expected leak
[task 2019-10-16T18:15:26.208Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at Alloc, nsTSubstring, nsTSubstring, NonVoidStringToJsval
[task 2019-10-16T18:15:26.209Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame Alloc matched a expected leak
[task 2019-10-16T18:15:26.209Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at mozilla::WidgetPointerEvent::Duplicate, mozilla::dom::Event::DuplicatePrivateData, mozilla::dom::UIEvent::DuplicatePrivateData, mozilla::EventDispatcher::Dispatch
[task 2019-10-16T18:15:26.209Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame mozilla::WidgetPointerEvent::Duplicate matched a expected leak
[task 2019-10-16T18:15:26.209Z] 18:15:26 INFO - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at Malloc, nsTArray_base, AppendElement, mozilla::dom::PointerEvent::GetCoalescedEvents
[task 2019-10-16T18:15:26.210Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at createTable, mozilla::detail::HashTable, rehashIfOverloaded, putNew
[task 2019-10-16T18:15:26.210Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame createTable matched a expected leak
[task 2019-10-16T18:15:26.210Z] 18:15:26 INFO - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at mozilla::ipc::MessagePump::Run, init, js::Shape::cachify, maybeCreateCacheForLookup
[task 2019-10-16T18:15:26.210Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at js_arena_realloc, js_pod_arena_realloc, maybe_pod_realloc, pod_realloc
[task 2019-10-16T18:15:26.210Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame js_arena_realloc matched a expected leak
[task 2019-10-16T18:15:26.210Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at std::sys::unix::alloc::_$LT$impl$u20$core..alloc..GlobalAlloc$u20$for$u20$std..alloc..System$GT$::alloc, __rdl_alloc, alloc::alloc::alloc, _$LT$alloc..alloc..Global$u20$as$u20$core..alloc..Alloc$GT$::alloc
[task 2019-10-16T18:15:26.211Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame alloc::alloc::alloc::h09060bea804b74c6 matched a expected leak
[task 2019-10-16T18:15:26.211Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at AllocateProtoAndIfaceCache, mozilla::dom::CreateGlobal, mozilla::dom::Window_Binding::Wrap, CreateNativeGlobalForInner
[task 2019-10-16T18:15:26.211Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame AllocateProtoAndIfaceCache matched a expected leak
[task 2019-10-16T18:15:26.211Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at NS_ProcessNextEvent, js_arena_malloc, js_pod_arena_malloc, maybe_pod_malloc
[task 2019-10-16T18:15:26.211Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame js_arena_malloc matched a expected leak
[task 2019-10-16T18:15:26.211Z] 18:15:26 INFO - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at mozilla::net::nsStandardURL::TemplatedMutator, Init, mozilla::net::nsStandardURL::TemplatedMutator, operator
[task 2019-10-16T18:15:26.212Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at mozilla::WidgetPointerEvent::Duplicate, mozilla::dom::Event::DuplicatePrivateData, mozilla::dom::UIEvent::DuplicatePrivateData, mozilla::dom::PointerEvent::GetCoalescedEvents
[task 2019-10-16T18:15:26.212Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame mozilla::WidgetPointerEvent::Duplicate matched a expected leak
[task 2019-10-16T18:15:26.212Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at nsThread::ProcessNextEvent, js_arena_malloc, js_pod_arena_malloc, maybe_pod_arena_malloc
[task 2019-10-16T18:15:26.212Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame nsThread::ProcessNextEvent matched a expected leak
[task 2019-10-16T18:15:26.212Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at NS_NewDOMPointerEvent, mozilla::EventDispatcher::CreateEvent, mozilla::EventListenerManager::HandleEventInternal, HandleEvent
[task 2019-10-16T18:15:26.213Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame NS_NewDOMPointerEvent matched a expected leak
[task 2019-10-16T18:15:26.213Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at js_arena_calloc, js_pod_arena_calloc, maybe_pod_calloc, js::ShapeTable::init
[task 2019-10-16T18:15:26.213Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame js_arena_calloc matched a expected leak
[task 2019-10-16T18:15:26.214Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at js_arena_calloc, js_pod_arena_calloc, maybe_pod_calloc, js::ShapeTable::change
[task 2019-10-16T18:15:26.214Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame js_arena_calloc matched a expected leak
[task 2019-10-16T18:15:26.215Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at Alloc, nsTSubstring, nsTSubstring, nsTSubstring
[task 2019-10-16T18:15:26.215Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame Alloc matched a expected leak
[task 2019-10-16T18:15:26.215Z] 18:15:26 INFO - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at EntrySlotOrCreate, EntrySlotOrCreate, mozilla::dom::BrowsingContext_Binding::CreateInterfaceObjects, mozilla::dom::GetPerInterfaceObjectHandle
[task 2019-10-16T18:15:26.215Z] 18:15:26 INFO - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at EntrySlotOrCreate, EntrySlotOrCreate, mozilla::dom::WindowGlobalChild_Binding::CreateInterfaceObjects, mozilla::dom::GetPerInterfaceObjectHandle
[task 2019-10-16T18:15:26.215Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at Alloc, nsTSubstring, nsTSubstring, mozilla::net::nsStandardURL::BuildNormalizedSpec
[task 2019-10-16T18:15:26.216Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame Alloc matched a expected leak

This is in web platform tests. They have their own per-directory whitelisting set up, so perhaps the removal interfered with that somehow.

James, I would like to understand more about how the WPT per-directory LSan suppressions actually work.

In bug 1587463 we went away from having external textual suppression lists (which would still work though) and as a result, I also simplified our CI and removed LSan suppression file support from some things including WPT. We can undo some of this, but I would like to better understand how WPT works before deciding on what the best way to move forward is.

Also, after a discussion with :mccr8 we also agreed that in the long run, it is not ideal for us to auto-whitelist/suppress leaks or other sanitizer results (TSan will be coming soon) because they end up remaining suppressed with the issues potentially never being addressed. We should have bugs on file for every suppression that we use to ensure we don't miss an important bug.

Thanks in advance!

Flags: needinfo?(james)

The important functional requirement for wpt is that we are able to do regular imports without blocking on developers fixing code issues. Having some mechanism to permit behaviour that would usually cause tests to fail is essential to avoid such blocking. That said, I obviously agree that we shouldn't be leaving these things in the tree long-term and having some mechanism to ensure that the problems actually get fixed, albeit asynchronously, is neccesary.

In general we use metadata files under testing/web-platform/meta to store the expected behaviour of tests. LSAN failures are represented by a lsan-allowed key that contains a list of stack frames e.g. [1]. Typically for LSAN failuers we put these at the directory level at most 3 levels deep under the wpt root because tests in that subtree all run in the same browser process, and we can only have one LSAN suppression list per process, so anything finer grained will cause us to restart the browser each time the suppression list changes. At testing time, if we are running with LSAN enabled, the list of stack frames is taken from the metadata and passed into mozleak [2].

Inside mozleak the list of stack frames from the metadata file is compiled into a regexp [3] which we run against the LSAN output to decide if the failure was allowed or not [4]. The complete list of stack frames and whether any matched the allow list is passed into mozlog, which records it in some way according to how it's configured [5].

When we import wpt, we do a try push. The "wptreport' logs for this contain an entry for LSAN failures [6] which contains pretty much all the data we logged. In the case where we get a LSAN failure in the try push which doesn't have "allowed_match": true, we update the corresponding metadata file to add the topmost stack frame of the leak to the allow list [7].

In addition to all of this we also previously used the external suppressions list, since our setup also calls test_environment [8].

In terms of ensuring the failures get fixed, there's currently work in progress to create bugs for certain kinds of failure we see on import e.g. crashes, or gecko-only failures. Using that infrastruture to create bugs for new LSAN failures seems quite plausible.

[1] https://searchfox.org/mozilla-central/source/testing/web-platform/meta/2dcontext/imagebitmap/__dir__.ini#1
[2] https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wptrunner/wptrunner/browsers/firefox.py#256,277
[3] https://searchfox.org/mozilla-central/source/testing/mozbase/mozleak/mozleak/lsan.py#60-61
[4] https://searchfox.org/mozilla-central/source/testing/mozbase/mozleak/mozleak/lsan.py#169
[5] https://searchfox.org/mozilla-central/source/testing/mozbase/mozleak/mozleak/lsan.py#143
[6] https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wptrunner/wptrunner/formatters/wptreport.py#138-144
[7] https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wptrunner/wptrunner/manifestupdate.py#784
[8] https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wptrunner/wptrunner/browsers/firefox.py#280

Flags: needinfo?(james)

Thank you for the detailed explanation. From this explanation it seems like no additional suppressions are used by WPT (the code for ignoring certain leaks here is part of the test framework instead).

I do have another theory about what is going on: It might be that we are now enabling LSan for more processes than before, because with my patch, we pass useLSan=True unconditionally. I am making a try run now to verify this assumption. If that is the case, then we need to convert the lsan_dir parameter into a bool and propagate it from wptcommandline.py appropriately to not break this logic.

Component: DOM: Core & HTML → Document Navigation

Turns out my theory in comment 5 was not correct, still seeing the leaks. However, it is two leaks in WPT14, all other WPTs are green. I don't think it is worth spending more time to figure out why these two show up right now. We should rather suppress them and leave this bug open for actually fixing the leaks in question.

That said, with those suppressions WPT14 is still orange for me due to various intermittents. It looks like that test chunk is particularly unstable and we should maybe try to find the root cause for that.

These leaks are in the /pointerevents/ directory, so presumably there's something leaky with pointer events. (As discussed above, this is an existing leak that we've just started to fail to ignore.)

That said, these are the existing suppressions:
lsan-allowed: [Alloc, mozilla::dom::ChromeUtils::GenerateQI]
leak-threshold: [default:153600, tab:307200]

Note that there is already a large leak-threshold (this is for the debug leak checking), but not many things in the lsan-allowed list. I'm not sure why this showed up, but I think the easiest fix here is just to add a few new things to the white list for this directory.

Component: Document Navigation → DOM: Events

(In reply to Christian Holler (:decoder) from comment #7)

Turns out my theory in comment 5 was not correct, still seeing the leaks. However, it is two leaks in WPT14, all other WPTs are green. I don't think it is worth spending more time to figure out why these two show up right now. We should rather suppress them and leave this bug open for actually fixing the leaks in question.

Yes, I agree that suppressing the leaks makes sense. I think we're suppressing most of the leaks. It is just that a bit of it is falling through the cracks.

Pushed by choller@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/6a3fa6243593
Suppress two leaks in WebPlatform tests. r=mccr8

:decoder , there is a leak that continues after you push.
Can you please take a look?

Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=271904043&repo=autoland&lineNumber=39916

Flags: needinfo?(choller)
Keywords: leave-open
Regressions: 1589920

Filed bug 1589920 for the leak Narcis mentioned above.

Flags: needinfo?(choller)
Keywords: leave-open
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]

No failures here since the 19th. Leaks have moved to 1589920.

Assignee: nobody → choller
Status: NEW → RESOLVED
Closed: 4 years ago
status-firefox71: --- → fixed
Resolution: --- → FIXED
Whiteboard: [stockwell disable-recommended]
Target Milestone: --- → mozilla71

Retroactively moving fixed bugs whose summaries mention "Fission" (or other Fission-related keywords) but are not assigned to a Fission Milestone to an appropriate Fission Milestone.

This will generate a lot of bugmail, so you can filter your bugmail for the following UUID and delete them en masse:

0ee3c76a-bc79-4eb2-8d12-05dc0b68e732

Fission Milestone: --- → M4
Has Regression Range: --- → yes
You need to log in before you can comment on or make changes to this bug.