1267227 - gtest.GrowToMaxCapacity fails often when running in windows7 VM in AWS

Status: RESOLVED FIXED

(Core :: XPCOM, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

we are trying to get our windows 7 tests running in AWS for better scalability.  One problem I see is that GTEST fails about 20% of the time:
https://treeherder.mozilla.org/#/jobs?repo=try&author=[e-mail]&selectedJob=19903076&filter-searchStr=gtest

if you look at a successful log, you see something like this:
07:54:48     INFO -  TEST-START | PLDHashTableTest.GrowToMaxCapacity
07:55:00     INFO -  TEST-PASS | PLDHashTableTest.GrowToMaxCapacity | test completed (time: 12805ms)


and the failed log, I see:
07:49:28     INFO -  TEST-START | PLDHashTableTest.GrowToMaxCapacity
07:49:58  WARNING -  TEST-UNEXPECTED-FAIL | PLDHashTableTest.GrowToMaxCapacity | Value of: false
07:49:58     INFO -    Actual: false
07:49:58     INFO -  Expected: true @ c:/builds/moz2_slave/try-w32-0000000000000000000000/build/src/xpcom/tests/gtest/TestPLDHash.cpp:353
07:49:58  WARNING -  TEST-UNEXPECTED-FAIL | PLDHashTableTest.GrowToMaxCapacity | test completed (time: 30573ms)

What I notice is the time difference of 12 seconds vs 30 seconds- I am not sure how to get other information here?

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

:njn, I see you wrote this test over a few different patches, can you help me figure out why this is causing problems in the cloud?

Comment 2

[Nicholas Nethercote [inactive]]

The test keeps adding elements to a PLDHashTable until we hit the maximum size, which is 1<<26. This requires repeatedly doubling the table size up to a max of 512 MiB. If any of the allocations fail the test will fail.

For this reason the test is already disabled on Android. Are the Windows machines in question 32-bit? If so, failing to allocate 512 MiB is probably not unreasonable. We could perhaps disable this test on all 32-bit platforms.

Comment 3

[Q]

They are 32bit systems. Win 7 has 3.5 gb available to the OS we have some additional overhead on these boxes that are not on hardware with the xenagent and ec2config services.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

would a 32 bit machine intermittently fail?  If that sounds plausible, then I agree on disabling this for 32 bit systems.  Is there a way to detect this at runtime vs compile time?

Comment 5

[Nicholas Nethercote [inactive]]

(In reply to Joel Maher (:jmaher) from comment #4)
> would a 32 bit machine intermittently fail?

It's conceivable.

> If that sounds plausible, then
> I agree on disabling this for 32 bit systems.  Is there a way to detect this
> at runtime vs compile time?

Yes. I'll write a patch tomorrow.

Comment 6

[Nicholas Nethercote [inactive]]

Attachment: Don't run the PLDHashTableTest.GrowToMaxCapacity gtest on 32-bit machines, because it sometimes OOMs

Comment 7

[Eric Rahm [:erahm]]

Comment on attachment 8745877 [details] [diff] [review]
Don't run the PLDHashTableTest.GrowToMaxCapacity gtest on 32-bit machines, because it sometimes OOMs

Review of attachment 8745877 [details] [diff] [review]:
-----------------------------------------------------------------

Seems reasonable, I guess we're not concerned about 64-bit android?

Comment 8

[Nicholas Nethercote [inactive]]

> Seems reasonable, I guess we're not concerned about 64-bit android?

Is that a thing? Huh. I figure we can worry about it if we start getting failures.

Comment 9

[Nicholas Nethercote [inactive]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/984828b14c451b8570a01e7ebbfac369e39dc762
Bug 1267227 - Don't run the PLDHashTableTest.GrowToMaxCapacity gtest on 32-bit machines, because it sometimes OOMs. r=erahm.

Comment 10

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/984828b14c45