Closed Bug 1353882 Opened 3 years ago Closed 2 years ago

Intermittent ArenaAllocator.AllocationsPerChunk | Expected: (uintptr_t(x) + kAlignment) != (uintptr_t(y)), actual: 37466688 vs 37466688 @ xpcom/tests/gtest/TestArenaAllocator.cpp:148

Categories

(Core :: XPCOM, defect)

defect
Not set

Tracking

()

RESOLVED FIXED
mozilla55
Tracking Status
firefox-esr52 --- unaffected
firefox53 --- unaffected
firefox54 --- disabled
firefox55 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: marco)

References

(Depends on 1 open bug)

Details

(Keywords: intermittent-failure)

Attachments

(1 file, 2 obsolete files)

Eric just added this test, I think.
Component: Memory Allocator → XPCOM
Flags: needinfo?(erahm)
(In reply to Andrew McCreight [:mccr8] from comment #1)
> Eric just added this test, I think.

Whoa that's sketchy. I'll take a look.
Wes, is there a way to see other failures with this signature? I'd like to see if it's only on a certain platform (ccov in this case).
Flags: needinfo?(erahm) → needinfo?(wkocher)
I only see this one failure.
Flags: needinfo?(wkocher)
erahm, I believe this is the main failure we see in code coverage (almost perma fail) and mentioned in bug 1358163.  should ArenaAllocator be running in coverage?  I don't mind skipping a few tests to get reliable coverage results.

We currently run coverage only on mozilla-central (search inside of treeherder for 'ccov'):
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=ccov&selectedJob=94927018
(In reply to Joel Maher ( :jmaher) from comment #5)
> erahm, I believe this is the main failure we see in code coverage (almost
> perma fail) and mentioned in bug 1358163.  should ArenaAllocator be running
> in coverage?  I don't mind skipping a few tests to get reliable coverage
> results.
> 
> We currently run coverage only on mozilla-central (search inside of
> treeherder for 'ccov'):
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-
> searchStr=ccov&selectedJob=94927018

The test runs fine under asan and v, I'm inclined to say there's something wrong with ccov. I don't particularly care whether it's run under ccov or not, but this does imply ccov is doing some rather alarming things.
coverage will change the timing and it will increase the overall I/O and memory usage on the box due to the collection of coverage data.  These are standard gcov tools, nothing tricky.  It is easy to skip this- although our coverage would go down slightly, but not of big concern.
erahm, could it be related to the fact that jemalloc is disabled in ccov builds?
(In reply to Marco Castelluccio [:marco] from comment #8)
> erahm, could it be related to the fact that jemalloc is disabled in ccov
> builds?

I believe it's disabled under ASAN and Valgrind as well.
(In reply to Eric Rahm [:erahm] from comment #9)
> (In reply to Marco Castelluccio [:marco] from comment #8)
> > erahm, could it be related to the fact that jemalloc is disabled in ccov
> > builds?
> 
> I believe it's disabled under ASAN and Valgrind as well.

Looks like it is not disabled under ASAN (https://hg.mozilla.org/mozilla-central/file/2cca333f546f38860f84940d4c72d7470a3410f4/browser/config/mozconfigs/linux64/code-coverage), but it is disabled under Valgrind (https://hg.mozilla.org/mozilla-central/file/2cca333f546f38860f84940d4c72d7470a3410f4/browser/config/mozconfigs/linux64/valgrind).

The Valgrind build also has the "--disable-gtest-in-build" option, which, I suppose, disables GTest in automation?
(In reply to Marco Castelluccio [:marco] from comment #10)
> Looks like it is not disabled under ASAN
> (https://hg.mozilla.org/mozilla-central/file/
> 2cca333f546f38860f84940d4c72d7470a3410f4/browser/config/mozconfigs/linux64/
> code-coverage)

Correct URL: https://hg.mozilla.org/mozilla-central/file/2cca333f546f38860f84940d4c72d7470a3410f4/browser/config/mozconfigs/linux64/debug-asan.
(In reply to Marco Castelluccio [:marco] from comment #11)
> (In reply to Marco Castelluccio [:marco] from comment #10)
> > Looks like it is not disabled under ASAN
> > (https://hg.mozilla.org/mozilla-central/file/
> > 2cca333f546f38860f84940d4c72d7470a3410f4/browser/config/mozconfigs/linux64/
> > code-coverage)
> 
> Correct URL:
> https://hg.mozilla.org/mozilla-central/file/
> 2cca333f546f38860f84940d4c72d7470a3410f4/browser/config/mozconfigs/linux64/
> debug-asan.

It includes a base file [1] that disables jemalloc.

[1] http://searchfox.org/mozilla-central/source/build/unix/mozconfig.asan
Right, I thought it was importing the debug base file, which doesn't allow to disable jemalloc.

Anyway, in the past few days this test has not been failing anymore (since April 29). Let's see if it fails again before disabling.
Attached patch Patch (obsolete) — Splinter Review
It's failing again, let's disable it in code coverage builds.
Attachment #8867150 - Flags: review?(jmaher)
Comment on attachment 8867150 [details] [diff] [review]
Patch

Review of attachment 8867150 [details] [diff] [review]:
-----------------------------------------------------------------

do we have #define MOZ_CODE_COVERAGE?
Attachment #8867150 - Flags: review?(jmaher) → review+
(In reply to Joel Maher ( :jmaher) from comment #15)
> Comment on attachment 8867150 [details] [diff] [review]
> Patch
> 
> Review of attachment 8867150 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> do we have #define MOZ_CODE_COVERAGE?

We have this line (https://dxr.mozilla.org/mozilla-central/rev/8a7d0b15595f9916123848ca906f29c62d4914c9/browser/config/mozconfigs/linux64/code-coverage#12) which defines it, correct?
I am not sure, maybe a push to try |mach try -b o -p linux64-ccov -u gtest -t none| would help validate that?
(In reply to Joel Maher ( :jmaher) from comment #17)
> I am not sure, maybe a push to try |mach try -b o -p linux64-ccov -u gtest
> -t none| would help validate that?

Good idea: https://treeherder.mozilla.org/#/jobs?repo=try&revision=095832ac96a83107ddc9b919f3b3d445c6cbf0f9
Looking at the logs, the test was running anyway, so MOZ_CODE_COVERAGE was not defined.
This patch defines it for files under xpcom/tests/gtest/*, but I'm not sure this is the proper way to do it.

I've tested the patch locally and it appears to work (AllocationsPerChunk is not run with the patch, it is run without the patch).
Attachment #8867922 - Flags: review?(jmaher)
Comment on attachment 8867922 [details] [diff] [review]
Define MOZ_CODE_COVERAGE in xpcom/tests/gtest/*

Review of attachment 8867922 [details] [diff] [review]:
-----------------------------------------------------------------

good find, it is sort of sad that we have to hack it in the individual moz.build file, but I view this as an exception.
Attachment #8867922 - Flags: review?(jmaher) → review+
Sorry for the noise, I've looked at a couple failing GTests and looks like both AllocationsPerChunk and MemoryIsValid are intermittently failing.
So, this patch disables them both.

Green try run: https://treeherder.mozilla.org/#/jobs?repo=try&revision=bf3ae93ca8b6d4e977b1e66b737e3207e3aefdbe&selectedJob=99393926.
Assignee: nobody → mcastelluccio
Attachment #8867150 - Attachment is obsolete: true
Attachment #8867922 - Attachment is obsolete: true
Status: NEW → ASSIGNED
Attachment #8868092 - Flags: review?(jmaher)
Pushed by mcastelluccio@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/393ed85d8f8b
Disable TestArenaAllocator.AllocationsPerChunk and TestArenaAllocator.MemoryIsValid tests in code coverage builds. r=jmaher
https://hg.mozilla.org/mozilla-central/rev/393ed85d8f8b
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla55
Depends on: 1371009
You need to log in before you can comment on or make changes to this bug.