1353882 - Intermittent ArenaAllocator.AllocationsPerChunk | Expected: (uintptr_t(x) + kAlignment) != (uintptr_t(y)), actual: 37466688 vs 37466688 @ xpcom/tests/gtest/TestArenaAllocator.cpp:148

Status: RESOLVED FIXED

(Core :: XPCOM, defect)

Description

[Treeherder Bug Filer]

Filed by: wkocher [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=88947574&repo=mozilla-central

https://queue.taskcluster.net/v1/task/MZ28UNVaQpquTQ2GevgBhA/runs/0/artifacts/public/logs/live_backing.log

Comment 1

[Andrew McCreight [:mccr8]]

Eric just added this test, I think.

Comment 2

[Eric Rahm [:erahm]]

(In reply to Andrew McCreight [:mccr8] from comment #1)
> Eric just added this test, I think.

Whoa that's sketchy. I'll take a look.

Comment 3

[Eric Rahm [:erahm]]

Wes, is there a way to see other failures with this signature? I'd like to see if it's only on a certain platform (ccov in this case).

Comment 4

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

I only see this one failure.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

erahm, I believe this is the main failure we see in code coverage (almost perma fail) and mentioned in bug 1358163.  should ArenaAllocator be running in coverage?  I don't mind skipping a few tests to get reliable coverage results.

We currently run coverage only on mozilla-central (search inside of treeherder for 'ccov'):
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=ccov&selectedJob=94927018

Comment 6

[Eric Rahm [:erahm]]

(In reply to Joel Maher ( :jmaher) from comment #5)
> erahm, I believe this is the main failure we see in code coverage (almost
> perma fail) and mentioned in bug 1358163.  should ArenaAllocator be running
> in coverage?  I don't mind skipping a few tests to get reliable coverage
> results.
> 
> We currently run coverage only on mozilla-central (search inside of
> treeherder for 'ccov'):
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-
> searchStr=ccov&selectedJob=94927018

The test runs fine under asan and v, I'm inclined to say there's something wrong with ccov. I don't particularly care whether it's run under ccov or not, but this does imply ccov is doing some rather alarming things.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

coverage will change the timing and it will increase the overall I/O and memory usage on the box due to the collection of coverage data.  These are standard gcov tools, nothing tricky.  It is easy to skip this- although our coverage would go down slightly, but not of big concern.

Comment 8

[Marco Castelluccio [:marco]]

erahm, could it be related to the fact that jemalloc is disabled in ccov builds?

Comment 9

[Eric Rahm [:erahm]]

(In reply to Marco Castelluccio [:marco] from comment #8)
> erahm, could it be related to the fact that jemalloc is disabled in ccov
> builds?

I believe it's disabled under ASAN and Valgrind as well.

Comment 10

[Marco Castelluccio [:marco]]

(In reply to Eric Rahm [:erahm] from comment #9)
> (In reply to Marco Castelluccio [:marco] from comment #8)
> > erahm, could it be related to the fact that jemalloc is disabled in ccov
> > builds?
> 
> I believe it's disabled under ASAN and Valgrind as well.

Looks like it is not disabled under ASAN (https://hg.mozilla.org/mozilla-central/file/2cca333f546f38860f84940d4c72d7470a3410f4/browser/config/mozconfigs/linux64/code-coverage), but it is disabled under Valgrind (https://hg.mozilla.org/mozilla-central/file/2cca333f546f38860f84940d4c72d7470a3410f4/browser/config/mozconfigs/linux64/valgrind).

The Valgrind build also has the "--disable-gtest-in-build" option, which, I suppose, disables GTest in automation?

Comment 11

[Marco Castelluccio [:marco]]

(In reply to Marco Castelluccio [:marco] from comment #10)
> Looks like it is not disabled under ASAN
> (https://hg.mozilla.org/mozilla-central/file/
> 2cca333f546f38860f84940d4c72d7470a3410f4/browser/config/mozconfigs/linux64/
> code-coverage)

Correct URL: https://hg.mozilla.org/mozilla-central/file/2cca333f546f38860f84940d4c72d7470a3410f4/browser/config/mozconfigs/linux64/debug-asan.

Comment 12

[Eric Rahm [:erahm]]

(In reply to Marco Castelluccio [:marco] from comment #11)
> (In reply to Marco Castelluccio [:marco] from comment #10)
> > Looks like it is not disabled under ASAN
> > (https://hg.mozilla.org/mozilla-central/file/
> > 2cca333f546f38860f84940d4c72d7470a3410f4/browser/config/mozconfigs/linux64/
> > code-coverage)
> 
> Correct URL:
> https://hg.mozilla.org/mozilla-central/file/
> 2cca333f546f38860f84940d4c72d7470a3410f4/browser/config/mozconfigs/linux64/
> debug-asan.

It includes a base file [1] that disables jemalloc.

[1] http://searchfox.org/mozilla-central/source/build/unix/mozconfig.asan

Comment 13

[Marco Castelluccio [:marco]]

Right, I thought it was importing the debug base file, which doesn't allow to disable jemalloc.

Anyway, in the past few days this test has not been failing anymore (since April 29). Let's see if it fails again before disabling.

Comment 14

[Marco Castelluccio [:marco]]

Attachment: Patch

It's failing again, let's disable it in code coverage builds.

Comment 15

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8867150 [details] [diff] [review]
Patch

Review of attachment 8867150 [details] [diff] [review]:
-----------------------------------------------------------------

do we have #define MOZ_CODE_COVERAGE?

Comment 16

[Marco Castelluccio [:marco]]

(In reply to Joel Maher ( :jmaher) from comment #15)
> Comment on attachment 8867150 [details] [diff] [review]
> Patch
> 
> Review of attachment 8867150 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> do we have #define MOZ_CODE_COVERAGE?

We have this line (https://dxr.mozilla.org/mozilla-central/rev/8a7d0b15595f9916123848ca906f29c62d4914c9/browser/config/mozconfigs/linux64/code-coverage#12) which defines it, correct?

Comment 17

[Joel Maher ( :jmaher ) (UTC -8)]

I am not sure, maybe a push to try |mach try -b o -p linux64-ccov -u gtest -t none| would help validate that?

Comment 18

[Marco Castelluccio [:marco]]

(In reply to Joel Maher ( :jmaher) from comment #17)
> I am not sure, maybe a push to try |mach try -b o -p linux64-ccov -u gtest
> -t none| would help validate that?

Good idea: https://treeherder.mozilla.org/#/jobs?repo=try&revision=095832ac96a83107ddc9b919f3b3d445c6cbf0f9

Comment 19

[Marco Castelluccio [:marco]]

Attachment: Define MOZ_CODE_COVERAGE in xpcom/tests/gtest/*

Looking at the logs, the test was running anyway, so MOZ_CODE_COVERAGE was not defined.
This patch defines it for files under xpcom/tests/gtest/*, but I'm not sure this is the proper way to do it.

I've tested the patch locally and it appears to work (AllocationsPerChunk is not run with the patch, it is run without the patch).

Comment 20

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8867922 [details] [diff] [review]
Define MOZ_CODE_COVERAGE in xpcom/tests/gtest/*

Review of attachment 8867922 [details] [diff] [review]:
-----------------------------------------------------------------

good find, it is sort of sad that we have to hack it in the individual moz.build file, but I view this as an exception.

Comment 21

[Marco Castelluccio [:marco]]

Attachment: disable_failing_gtests

Sorry for the noise, I've looked at a couple failing GTests and looks like both AllocationsPerChunk and MemoryIsValid are intermittently failing.
So, this patch disables them both.

Green try run: https://treeherder.mozilla.org/#/jobs?repo=try&revision=bf3ae93ca8b6d4e977b1e66b737e3207e3aefdbe&selectedJob=99393926.

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/393ed85d8f8b
Disable TestArenaAllocator.AllocationsPerChunk and TestArenaAllocator.MemoryIsValid tests in code coverage builds. r=jmaher

Comment 23

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/393ed85d8f8b