1363976 - stylo: ASan detected leaks

Status: RESOLVED FIXED

(Core :: CSS Parsing and Computation, defect, P1)

Description

[Shing Lyu [:shinglyu]]

Attachment: List of "leaks at ..." log entries

Linux64-asan (with stylo enabled) found a lot of leaks in mochitests and reftests.

Step to reproduce:
* Add "--enable-stylo" to asan mozconfigs: [1]
* Push try: "try: -b do -p linux64-asan -u all -t none"

An example push: [2].

I also attached the list of leaking locations, extracted from the logs.

[1] https://hg.mozilla.org/try/rev/47bef5c8de54ee1edac4f6d924550123fe84daba
[2] https://treeherder.mozilla.org/#/jobs?repo=try&revision=d31cf95b1ac021ba56d010754062a7f3134c7e42

Comment 1

[Bobby Holley (:bholley)]

Looks to me like we're leaking servo stylesheets, but not the corresponding gecko data structures.

Comment 2

[Cameron McCormack (:heycam)]

I wonder if we still get these leaks.  Shing, could you do another try run?  If we've still got leaks we should look at them soon.

Comment 3

[Bobby Holley (:bholley)]

jryans, would you mind doing a try run to see if this is still happening?

Comment 4

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Here's a fresh try run, I'll update this bug when results come in.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=7ad4a48b9fee15c6ed85fa3288870df57aab0245

Comment 5

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

It appears things have improved since the previous run, and some chunks do complete without any leaks, but most are still orange.

The most common report by far looks like the following:

Direct leak of 64 byte(s) in 4 object(s) allocated from:
    #0 0x4bba1c in malloc /builds/slave/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:64:3
    #1 0x7fdfbabf1759 in alloc_system::imp::allocate /checkout/src/liballoc_system/lib.rs:90
    #2 0x7fdfbabf1759 in alloc_system::__rust_allocate /checkout/src/liballoc_system/lib.rs:44
    #3 0x7fdfbabf1759 in alloc::heap::allocate /checkout/src/liballoc/heap.rs:59
    #4 0x7fdfbabf1759 in alloc::heap::exchange_malloc /checkout/src/liballoc/heap.rs:154
    #5 0x7fdfbabf1759 in std::sys::imp::thread::Thread::new::h67bdde99cb81e0df /checkout/src/libstd/sys/unix/thread.rs:49
    #6 0x7fdfbab8bdd7 in std::thread::{{impl}}::spawn<closure,()> /checkout/src/libstd/thread/mod.rs:374
    #7 0x7fdfbab8bdd7 in rayon_core::registry::Registry::new::hfeb2d53c3f4b12e9 /home/worker/workspace/build/src/third_party/rust/rayon-core/src/registry.rs:145
    #8 0x7fdfbab42f2d in rayon_core::thread_pool::{{impl}}::new /home/worker/workspace/build/src/third_party/rust/rayon-core/src/thread_pool/mod.rs:56
    #9 0x7fdfbab42f2d in style::gecko::global_style_data::{{impl}}::deref::__static_ref_initialize /home/worker/workspace/build/src/servo/components/style/gecko/global_style_data.rs:95
    #10 0x7fdfbab42f2d in core::ops::FnOnce::call_once<fn() -> style::gecko::global_style_data::StyleThreadPool,()> /checkout/src/libcore/ops.rs:2683
    #11 0x7fdfbab42f2d in lazy_static::lazy::{{impl}}::get::{{closure}}<style::gecko::global_style_data::StyleThreadPool,fn() -> style::gecko::global_style_data::StyleThreadPool> /home/worker/workspace/build/src/third_party/rust/lazy_static/src/lazy.rs:16
    #12 0x7fdfbab42f2d in std::sync::once::Once::call_once::_$u7b$$u7b$closure$u7d$$u7d$::h2649cc33cd477db2 /checkout/src/libstd/sync/once.rs:227
    #13 0x7fdfbabe5d22 in std::sync::once::Once::call_inner::h15de809733866c18 /checkout/src/libstd/sync/once.rs:307

I believe this suggests the thread object inside Stylo's Rayon thread pool is being leaked.

Comment 6

[Bobby Holley (:bholley)]

(In reply to J. Ryan Stinnett [:jryans] (use ni?) from comment #5)
> I believe this suggests the thread object inside Stylo's Rayon thread pool
> is being leaked.

Leaking singleton static data like that seems fine. Can we whitelist it somehow?

Comment 7

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

(In reply to Bobby Holley (:bholley) (busy with Stylo) from comment #6)
> (In reply to J. Ryan Stinnett [:jryans] (use ni?) from comment #5)
> > I believe this suggests the thread object inside Stylo's Rayon thread pool
> > is being leaked.
> 
> Leaking singleton static data like that seems fine. Can we whitelist it
> somehow?

For C++ data, there's a MOZ_LSAN_INTENTIONALLY_LEAK_OBJECT macro[1] that can be used to ignore objects.

There's also a suppressions[2] file you can add entries to.

I'll try the suppressions file first.

[1]: http://searchfox.org/mozilla-central/rev/bd39b6170f04afeefc751a23bb04e18bbd10352b/mfbt/MemoryChecking.h#123
[2]: http://searchfox.org/mozilla-central/source/build/sanitizers/lsan_suppressions.txt

Comment 8

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c946a0ed32d8f784d70ba5e3b3b470ac72a8ae87

Comment 9

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Looks like suppressing the global style data is enough here.  The remaining failures are known test issues (not leaks).

(On a side note, this ASAN run did log a UAF related to bug 1388165, so it's great to have this tooling working!)

Comment 10

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Attachment: Bug 1363976 - Suppress global_style_data for LSAN.

Suppress leaks for allocations where any stack frame matches the string
style::gecko::global_style_data.  This could be dangerous if someone were to
allocate non-global things with this on the stack, but hopefully that can be
avoided, given the name of the module.

Review commit: https://reviewboard.mozilla.org/r/165918/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/165918/

Comment 11

[Bobby Holley (:bholley)]

Comment on attachment 8894746 [details]
Bug 1363976 - Suppress global_style_data for LSAN.

https://reviewboard.mozilla.org/r/165918/#review171026

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/85a69f9c265a
Suppress global_style_data for LSAN. r=bholley

Comment 13

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/85a69f9c265a