1589207 - Perma [tier 2] TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at EntrySlotOrCreate, EntrySlotOrCreate, mozilla::dom::BrowsingContext_Binding::CreateInterfaceObjects, mozilla::dom::GetPerInterfaceObjectHandle and other leaks

Status: RESOLVED FIXED

(Core :: DOM: Events, defect)

Description

[Alexandru Michis [:malexandru]]

Parsed log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=271539367&repo=autoland&lineNumber=39327

Raw log: https://queue.taskcluster.net/v1/task/BkPYKzHpQn6ssmXlbb4ZVw/runs/0/artifacts/public/logs/live_backing.log

[task 2019-10-16T18:15:25.551Z] 18:15:25 INFO - PID 8509 | -----------------------------------------------------
[task 2019-10-16T18:15:25.551Z] 18:15:25 INFO - PID 8509 | Suppressions used:
[task 2019-10-16T18:15:25.551Z] 18:15:25 INFO - PID 8509 | count bytes template
[task 2019-10-16T18:15:25.552Z] 18:15:25 INFO - PID 8509 | 497 51429 libc.so
[task 2019-10-16T18:15:25.553Z] 18:15:25 INFO - PID 8509 | 156 24779 nsComponentManagerImpl
[task 2019-10-16T18:15:25.553Z] 18:15:25 INFO - PID 8509 | 2584 101965 libfontconfig.so
[task 2019-10-16T18:15:25.554Z] 18:15:25 INFO - PID 8509 | 1 29 libglib-2.0.so
[task 2019-10-16T18:15:25.554Z] 18:15:25 INFO - PID 8509 | 4513 193780 js::frontend::BytecodeEmitter:
[task 2019-10-16T18:15:25.554Z] 18:15:25 INFO - PID 8509 | 596 21588 js::frontend::GeneralParser
[task 2019-10-16T18:15:25.555Z] 18:15:25 INFO - PID 8509 | 1129 55976 js::frontend::Parse
[task 2019-10-16T18:15:25.555Z] 18:15:25 INFO - PID 8509 | 36 1012 xpc::CIGSHelper
[task 2019-10-16T18:15:25.556Z] 18:15:25 INFO - PID 8509 | 641 60470 mozJSComponentLoader
[task 2019-10-16T18:15:25.557Z] 18:15:25 INFO - PID 8509 | 182 11556 XPCWrappedNativeJSOps
[task 2019-10-16T18:15:25.557Z] 18:15:25 INFO - PID 8509 | -----------------------------------------------------
[task 2019-10-16T18:15:26.107Z] 18:15:26 INFO - PID 8509 | -----------------------------------------------------
[task 2019-10-16T18:15:26.108Z] 18:15:26 INFO - PID 8509 | Suppressions used:
[task 2019-10-16T18:15:26.108Z] 18:15:26 INFO - PID 8509 | count bytes template
[task 2019-10-16T18:15:26.108Z] 18:15:26 INFO - PID 8509 | 24 728 nsComponentManagerImpl
[task 2019-10-16T18:15:26.108Z] 18:15:26 INFO - PID 8509 | 4 832 mozJSComponentLoader::LoadModule
[task 2019-10-16T18:15:26.108Z] 18:15:26 INFO - PID 8509 | 611 17509 libfontconfig.so
[task 2019-10-16T18:15:26.109Z] 18:15:26 INFO - PID 8509 | 1 29 libglib-2.0.so
[task 2019-10-16T18:15:26.109Z] 18:15:26 INFO - PID 8509 | -----------------------------------------------------
[task 2019-10-16T18:15:26.205Z] 18:15:26 INFO - Browser exited with return code 0
[task 2019-10-16T18:15:26.205Z] 18:15:26 INFO - PROCESS LEAKS None
[task 2019-10-16T18:15:26.205Z] 18:15:26 INFO - ERROR | LeakSanitizer | SUMMARY: AddressSanitizer: 124936 byte(s) leaked in 1206 allocation(s).
[task 2019-10-16T18:15:26.205Z] 18:15:26 INFO - LeakSanitizer | To show the addresses of leaked objects add report_objects=1 to LSAN_OPTIONS
[task 2019-10-16T18:15:26.206Z] 18:15:26 INFO - This can be done in testing/mozbase/mozrunner/mozrunner/utils.py
[task 2019-10-16T18:15:26.206Z] 18:15:26 INFO - Allowed depth was 4
[task 2019-10-16T18:15:26.206Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at Alloc, nsTSubstring, Gecko_StartBulkWriteString, nsstring::nsAString::start_bulk_write_impl
[task 2019-10-16T18:15:26.206Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame Alloc matched a expected leak
[task 2019-10-16T18:15:26.206Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at NS_NewDOMPointerEvent, mozilla::dom::PointerEvent::GetCoalescedEvents, mozilla::dom::PointerEvent_Binding::getCoalescedEvents, mozilla::dom::binding_detail::GenericMethod
[task 2019-10-16T18:15:26.207Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame NS_NewDOMPointerEvent matched a expected leak
[task 2019-10-16T18:15:26.207Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at mozilla::SchedulerGroup::Runnable::Run, js_arena_calloc, js_pod_arena_calloc, maybe_pod_calloc
[task 2019-10-16T18:15:26.207Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame js_arena_calloc matched a expected leak
[task 2019-10-16T18:15:26.207Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at js_arena_malloc, js_pod_arena_malloc, maybe_pod_malloc, js::detail::OrderedHashTable
[task 2019-10-16T18:15:26.207Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame js_arena_malloc matched a expected leak
[task 2019-10-16T18:15:26.207Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at js_arena_malloc, js_pod_arena_malloc, maybe_pod_malloc, pod_malloc
[task 2019-10-16T18:15:26.207Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame js_arena_malloc matched a expected leak
[task 2019-10-16T18:15:26.208Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at createTable, mozilla::detail::HashTable, reserve, reserve
[task 2019-10-16T18:15:26.208Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame createTable matched a expected leak
[task 2019-10-16T18:15:26.208Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at js_arena_malloc, js_pod_arena_malloc, maybe_pod_arena_malloc, pod_arena_malloc
[task 2019-10-16T18:15:26.208Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame js_arena_malloc matched a expected leak
[task 2019-10-16T18:15:26.208Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at Alloc, nsTSubstring, nsTSubstring, NonVoidStringToJsval
[task 2019-10-16T18:15:26.209Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame Alloc matched a expected leak
[task 2019-10-16T18:15:26.209Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at mozilla::WidgetPointerEvent::Duplicate, mozilla::dom::Event::DuplicatePrivateData, mozilla::dom::UIEvent::DuplicatePrivateData, mozilla::EventDispatcher::Dispatch
[task 2019-10-16T18:15:26.209Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame mozilla::WidgetPointerEvent::Duplicate matched a expected leak
[task 2019-10-16T18:15:26.209Z] 18:15:26 INFO - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at Malloc, nsTArray_base, AppendElement, mozilla::dom::PointerEvent::GetCoalescedEvents
[task 2019-10-16T18:15:26.210Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at createTable, mozilla::detail::HashTable, rehashIfOverloaded, putNew
[task 2019-10-16T18:15:26.210Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame createTable matched a expected leak
[task 2019-10-16T18:15:26.210Z] 18:15:26 INFO - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at mozilla::ipc::MessagePump::Run, init, js::Shape::cachify, maybeCreateCacheForLookup
[task 2019-10-16T18:15:26.210Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at js_arena_realloc, js_pod_arena_realloc, maybe_pod_realloc, pod_realloc
[task 2019-10-16T18:15:26.210Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame js_arena_realloc matched a expected leak
[task 2019-10-16T18:15:26.210Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at std::sys::unix::alloc::_$LT$impl$u20$core..alloc..GlobalAlloc$u20$for$u20$std..alloc..System$GT$::alloc, __rdl_alloc, alloc::alloc::alloc, _$LT$alloc..alloc..Global$u20$as$u20$core..alloc..Alloc$GT$::alloc
[task 2019-10-16T18:15:26.211Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame alloc::alloc::alloc::h09060bea804b74c6 matched a expected leak
[task 2019-10-16T18:15:26.211Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at AllocateProtoAndIfaceCache, mozilla::dom::CreateGlobal, mozilla::dom::Window_Binding::Wrap, CreateNativeGlobalForInner
[task 2019-10-16T18:15:26.211Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame AllocateProtoAndIfaceCache matched a expected leak
[task 2019-10-16T18:15:26.211Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at NS_ProcessNextEvent, js_arena_malloc, js_pod_arena_malloc, maybe_pod_malloc
[task 2019-10-16T18:15:26.211Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame js_arena_malloc matched a expected leak
[task 2019-10-16T18:15:26.211Z] 18:15:26 INFO - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at mozilla::net::nsStandardURL::TemplatedMutator, Init, mozilla::net::nsStandardURL::TemplatedMutator, operator
[task 2019-10-16T18:15:26.212Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at mozilla::WidgetPointerEvent::Duplicate, mozilla::dom::Event::DuplicatePrivateData, mozilla::dom::UIEvent::DuplicatePrivateData, mozilla::dom::PointerEvent::GetCoalescedEvents
[task 2019-10-16T18:15:26.212Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame mozilla::WidgetPointerEvent::Duplicate matched a expected leak
[task 2019-10-16T18:15:26.212Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at nsThread::ProcessNextEvent, js_arena_malloc, js_pod_arena_malloc, maybe_pod_arena_malloc
[task 2019-10-16T18:15:26.212Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame nsThread::ProcessNextEvent matched a expected leak
[task 2019-10-16T18:15:26.212Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at NS_NewDOMPointerEvent, mozilla::EventDispatcher::CreateEvent, mozilla::EventListenerManager::HandleEventInternal, HandleEvent
[task 2019-10-16T18:15:26.213Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame NS_NewDOMPointerEvent matched a expected leak
[task 2019-10-16T18:15:26.213Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at js_arena_calloc, js_pod_arena_calloc, maybe_pod_calloc, js::ShapeTable::init
[task 2019-10-16T18:15:26.213Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame js_arena_calloc matched a expected leak
[task 2019-10-16T18:15:26.214Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at js_arena_calloc, js_pod_arena_calloc, maybe_pod_calloc, js::ShapeTable::change
[task 2019-10-16T18:15:26.214Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame js_arena_calloc matched a expected leak
[task 2019-10-16T18:15:26.215Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at Alloc, nsTSubstring, nsTSubstring, nsTSubstring
[task 2019-10-16T18:15:26.215Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame Alloc matched a expected leak
[task 2019-10-16T18:15:26.215Z] 18:15:26 INFO - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at EntrySlotOrCreate, EntrySlotOrCreate, mozilla::dom::BrowsingContext_Binding::CreateInterfaceObjects, mozilla::dom::GetPerInterfaceObjectHandle
[task 2019-10-16T18:15:26.215Z] 18:15:26 INFO - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at EntrySlotOrCreate, EntrySlotOrCreate, mozilla::dom::WindowGlobalChild_Binding::CreateInterfaceObjects, mozilla::dom::GetPerInterfaceObjectHandle
[task 2019-10-16T18:15:26.215Z] 18:15:26 INFO - TEST-FAIL | LeakSanitizer | leak at Alloc, nsTSubstring, nsTSubstring, mozilla::net::nsStandardURL::BuildNormalizedSpec
[task 2019-10-16T18:15:26.216Z] 18:15:26 INFO - INFO | LeakSanitizer | Frame Alloc matched a expected leak

Comment 1

[Andrew McCreight [:mccr8]]

This is in web platform tests. They have their own per-directory whitelisting set up, so perhaps the removal interfered with that somehow.

Comment 2

[Christian Holler (:decoder)]

James, I would like to understand more about how the WPT per-directory LSan suppressions actually work.

In bug 1587463 we went away from having external textual suppression lists (which would still work though) and as a result, I also simplified our CI and removed LSan suppression file support from some things including WPT. We can undo some of this, but I would like to better understand how WPT works before deciding on what the best way to move forward is.

Also, after a discussion with :mccr8 we also agreed that in the long run, it is not ideal for us to auto-whitelist/suppress leaks or other sanitizer results (TSan will be coming soon) because they end up remaining suppressed with the issues potentially never being addressed. We should have bugs on file for every suppression that we use to ensure we don't miss an important bug.

Thanks in advance!

Comment 3

[Intermittent Failures Robot]

43 failures in 800 pushes (0.054 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 43

Platform breakdown:

• linux64: 43

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1589207&startday=2019-10-16&endday=2019-10-16&tree=all

Comment 4

[James Graham [:jgraham]]

The important functional requirement for wpt is that we are able to do regular imports without blocking on developers fixing code issues. Having some mechanism to permit behaviour that would usually cause tests to fail is essential to avoid such blocking. That said, I obviously agree that we shouldn't be leaving these things in the tree long-term and having some mechanism to ensure that the problems actually get fixed, albeit asynchronously, is neccesary.

In general we use metadata files under testing/web-platform/meta to store the expected behaviour of tests. LSAN failures are represented by a lsan-allowed key that contains a list of stack frames e.g. [1]. Typically for LSAN failuers we put these at the directory level at most 3 levels deep under the wpt root because tests in that subtree all run in the same browser process, and we can only have one LSAN suppression list per process, so anything finer grained will cause us to restart the browser each time the suppression list changes. At testing time, if we are running with LSAN enabled, the list of stack frames is taken from the metadata and passed into mozleak [2].

Inside mozleak the list of stack frames from the metadata file is compiled into a regexp [3] which we run against the LSAN output to decide if the failure was allowed or not [4]. The complete list of stack frames and whether any matched the allow list is passed into mozlog, which records it in some way according to how it's configured [5].

When we import wpt, we do a try push. The "wptreport' logs for this contain an entry for LSAN failures [6] which contains pretty much all the data we logged. In the case where we get a LSAN failure in the try push which doesn't have "allowed_match": true, we update the corresponding metadata file to add the topmost stack frame of the leak to the allow list [7].

In addition to all of this we also previously used the external suppressions list, since our setup also calls test_environment [8].

In terms of ensuring the failures get fixed, there's currently work in progress to create bugs for certain kinds of failure we see on import e.g. crashes, or gecko-only failures. Using that infrastruture to create bugs for new LSAN failures seems quite plausible.

[1] https://searchfox.org/mozilla-central/source/testing/web-platform/meta/2dcontext/imagebitmap/__dir__.ini#1
[2] https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wptrunner/wptrunner/browsers/firefox.py#256,277
[3] https://searchfox.org/mozilla-central/source/testing/mozbase/mozleak/mozleak/lsan.py#60-61
[4] https://searchfox.org/mozilla-central/source/testing/mozbase/mozleak/mozleak/lsan.py#169
[5] https://searchfox.org/mozilla-central/source/testing/mozbase/mozleak/mozleak/lsan.py#143
[6] https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wptrunner/wptrunner/formatters/wptreport.py#138-144
[7] https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wptrunner/wptrunner/manifestupdate.py#784
[8] https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wptrunner/wptrunner/browsers/firefox.py#280

Comment 5

[Christian Holler (:decoder)]

Thank you for the detailed explanation. From this explanation it seems like no additional suppressions are used by WPT (the code for ignoring certain leaks here is part of the test framework instead).

I do have another theory about what is going on: It might be that we are now enabling LSan for more processes than before, because with my patch, we pass useLSan=True unconditionally. I am making a try run now to verify this assumption. If that is the case, then we need to convert the lsan_dir parameter into a bool and propagate it from wptcommandline.py appropriately to not break this logic.

Comment 6

[Christian Holler (:decoder)]

Attachment: Bug 1589207 - Suppress two leaks in WebPlatform tests. r?jgraham

Comment 7

[Christian Holler (:decoder)]

Turns out my theory in comment 5 was not correct, still seeing the leaks. However, it is two leaks in WPT14, all other WPTs are green. I don't think it is worth spending more time to figure out why these two show up right now. We should rather suppress them and leave this bug open for actually fixing the leaks in question.

That said, with those suppressions WPT14 is still orange for me due to various intermittents. It looks like that test chunk is particularly unstable and we should maybe try to find the root cause for that.

Comment 8

[Andrew McCreight [:mccr8]]

These leaks are in the /pointerevents/ directory, so presumably there's something leaky with pointer events. (As discussed above, this is an existing leak that we've just started to fail to ignore.)

That said, these are the existing suppressions:
lsan-allowed: [Alloc, mozilla::dom::ChromeUtils::GenerateQI]
leak-threshold: [default:153600, tab:307200]

Note that there is already a large leak-threshold (this is for the debug leak checking), but not many things in the lsan-allowed list. I'm not sure why this showed up, but I think the easiest fix here is just to add a few new things to the white list for this directory.

Comment 9

[Andrew McCreight [:mccr8]]

(In reply to Christian Holler (:decoder) from comment #7)

Turns out my theory in comment 5 was not correct, still seeing the leaks. However, it is two leaks in WPT14, all other WPTs are green. I don't think it is worth spending more time to figure out why these two show up right now. We should rather suppress them and leave this bug open for actually fixing the leaks in question.

Yes, I agree that suppressing the leaks makes sense. I think we're suppressing most of the leaks. It is just that a bit of it is falling through the cracks.

Comment 10

[Intermittent Failures Robot]

85 failures in 834 pushes (0.102 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 79
• mozilla-central: 2
• try: 2
• mozilla-inbound: 2

Platform breakdown:

• linux64: 85

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1589207&startday=2019-10-17&endday=2019-10-17&tree=all

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6a3fa6243593
Suppress two leaks in WebPlatform tests. r=mccr8

Comment 12

[Narcis Beleuzu [:NarcisB]]

:decoder , there is a leak that continues after you push.
Can you please take a look?

Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=271904043&repo=autoland&lineNumber=39916

Comment 13

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/6a3fa6243593

Comment 14

[Intermittent Failures Robot]

27 failures in 747 pushes (0.036 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 24
• mozilla-inbound: 1
• mozilla-central: 2

Platform breakdown:

• linux64: 27

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1589207&startday=2019-10-18&endday=2019-10-18&tree=all

Comment 15

[Andreea Pavel [:apavel]]

Filed bug 1589920 for the leak Narcis mentioned above.

Comment 16

[Intermittent Failures Robot]

156 failures in 3964 pushes (0.039 failures/push) were associated with this bug in the last 7 days.

This is the #3 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 147
• mozilla-central: 4
• try: 2
• mozilla-inbound: 3

Platform breakdown:

• linux64: 156

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1589207&startday=2019-10-14&endday=2019-10-20&tree=all

Comment 17

[Cosmin Sabou [:CosminS]]

No failures here since the 19th. Leaks have moved to 1589920.

Comment 18

[Chris Peterson [:cpeterson]]

Retroactively moving fixed bugs whose summaries mention "Fission" (or other Fission-related keywords) but are not assigned to a Fission Milestone to an appropriate Fission Milestone.

This will generate a lot of bugmail, so you can filter your bugmail for the following UUID and delete them en masse:

0ee3c76a-bc79-4eb2-8d12-05dc0b68e732