1116230 - Shutdown leaks in toolkit/components/aboutmemory/tests when run as a standalone directory

Status: NEW

(Toolkit :: about:memory, defect)

Description

[Vaibhav (:vaibhav1994)]

In bug 1110982, we are looking to enable tests where we run a fresh browser instance per directory.  Usually what happens is that a few tests fail because they accidentally depend on the state of the browser from an earlier test.

When the tests are run in toolkit/components/aboutmemory/tests/chrome.ini
cause leaks in linux asan. Example try run: https://treeherder.mozilla.org/#/jobs?repo=try&revision=1fb77b605b06

The leaks:

SUMMARY: AddressSanitizer: 1792 byte(s) leaked in 14 allocation(s).

SUMMARY: AddressSanitizer: 87094 byte(s) leaked in 2 allocation(s).

TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at /usr/lib/x86_64-linux-gnu/libX11.so.6 

can be seen on linux asan job.

We have bisected down the probable cause and the green try push is:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=506a7fcead24

in which we have disabled these tests: https://hg.mozilla.org/try/diff/0070a30de170/toolkit/components/aboutmemory/tests/chrome.ini

Comment 1

[Andrew McCreight [:mccr8]]

Interesting.

It looks like this is leaking across all debug builds in addition to ASan.

Maybe ClientLayerManager::MemoryPressureObserver could also observe xpcom-shutdown and destroy itself in that case?

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

I could verify this locally by downloading the asan build/tests, then running the command:
python runtests.py --chrome --app ../firefox/firefox --xre-path ../firefox --utility-path ../bin --certificate-path ../certs --autorun --close-when-done --extra-profile-file=../bin/plugins --console-level=INFO --setpref=webgl.force-enabled=true --test-path toolkit/components/aboutmemory/tests


I found that test_aboutmemory5.xul was the root cause for me locally.

I assume if you had an ASAN build locally, you could do:
./mach mochitest-chrome toolkit/components/aboutmemory/tests

Comment 3

[Andrew McCreight [:mccr8]]

FWIW I wasn't able to reproduce this locally with the steps in comment 2 in a regular debug build.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

I suspect you need an ASAN build vs a debug build to reproduce this.  So many flavors of builds/tests to take care of!

Comment 5

[Andrew McCreight [:mccr8]]

(In reply to Joel Maher (:jmaher) from comment #4)
> I suspect you need an ASAN build vs a debug build to reproduce this.  So
> many flavors of builds/tests to take care of!

Well, the first try run in comment 0 shows it failing in debug builds, too.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

Andrew, got it.

I believe the debug leaks are related to docshell/test/chrome and filed in bug 1116195.  There is a good chance that both the linux debug and linux asan leaks are leaking the same bits and could be the same cause.

Comment 7

[Andrew McCreight [:mccr8]]

Sorry, I wasn't looking at this too closely.  I got distracted by all those debug leaks.  You are right, all of the other LSan and debug leaks look like some docshell/test/chrome things.

Investigating the LSan leak will be a bit of a pain.  Somebody will have to locally install some kind of debug version of libX11 so that the stackwalker can actually get useful stacks.  If this particular failure becomes a blocker, we can whitelist the leak.  If it is only showing up in this one particular test it hopefully is not a widespread problem.

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

the single test case that is causing this leak appears to be:
test_aboutmemory5.xul

Is there anything else we can do here?  I am nearing the end of the cycle of tests to fix, and would like to either disable this for asan, or work on getting a fix.

Comment 9

[Andrew McCreight [:mccr8]]

I'll look at this today.

Comment 10

[Andrew McCreight [:mccr8]]

Joel, is there a recent-ish run of this similar to that in comment 0?  I want to make sure I'm seeing the same thing locally.

Comment 11

[Joel Maher ( :jmaher ) (UTC -8)]

here is a run I did this weekend:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=bee45701f366

Comment 12

[Andrew McCreight [:mccr8]]

Thanks.

The log shows a leak that looks like this:
Direct leak of 46392 byte(s) in 1 object(s) allocated from:
    #0 0x471d71 in malloc /builds/slave/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:74
    #1 0x7f4544862b83 (/usr/lib/x86_64-linux-gnu/libX11.so.6+0x53b83)
Direct leak of 46392 byte(s) in 1 object(s) allocated from:
    #0 0x471d71 in malloc /builds/slave/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:74
    #1 0x7f4544862b83 (/usr/lib/x86_64-linux-gnu/libX11.so.6+0x53b83)

Locally, I tried:
  ./mach mochitest --run-by-dir toolkit/components/aboutmemory/tests/
I see a different leak, oddly enough caused by the same test, but we're instead leaking an IonScript.  I can't imagine how that would be related.

So, without being able to reproduce locally, I'm not sure what I could do about this.  I'm not sure if it would be better to just disable this test for ASan, or to add a suppression for libX11.so that would apply to all tests.  Nick, do you have an opinion?

Comment 13

[Nicholas Nethercote [inactive]]

That test creates some content processes and then does some memory reporting. The creation of the content processes distinguishes it from all the other tests in that same directory, so the leak must somehow be an e10s thing.

My initial inclination was to suppress all leaks from libX11.so but I guess disabling the test is a more conservative action. I could live with either.

Comment 14

[Joel Maher ( :jmaher ) (UTC -8)]

I am planning on disabling this test come Monday, do speak up if this is not a good course of action.  As a reminder this is only disabled for Linux64 ASAN builds.

Comment 15

[Andrew McCreight [:mccr8]]

Yeah, that's probably the best thing to do.  Thanks.