gtest.GrowToMaxCapacity fails often when running in windows7 VM in AWS

RESOLVED FIXED in Firefox 49

Status

()

Core
XPCOM
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: jmaher, Assigned: njn)

Tracking

Trunk
mozilla49
Points:
---

Firefox Tracking Flags

(firefox48 affected, firefox49 fixed)

Details

Attachments

(1 attachment)

(Reporter)

Description

2 years ago
we are trying to get our windows 7 tests running in AWS for better scalability.  One problem I see is that GTEST fails about 20% of the time:
https://treeherder.mozilla.org/#/jobs?repo=try&author=jmaher@mozilla.com&selectedJob=19903076&filter-searchStr=gtest

if you look at a successful log, you see something like this:
07:54:48     INFO -  TEST-START | PLDHashTableTest.GrowToMaxCapacity
07:55:00     INFO -  TEST-PASS | PLDHashTableTest.GrowToMaxCapacity | test completed (time: 12805ms)


and the failed log, I see:
07:49:28     INFO -  TEST-START | PLDHashTableTest.GrowToMaxCapacity
07:49:58  WARNING -  TEST-UNEXPECTED-FAIL | PLDHashTableTest.GrowToMaxCapacity | Value of: false
07:49:58     INFO -    Actual: false
07:49:58     INFO -  Expected: true @ c:/builds/moz2_slave/try-w32-0000000000000000000000/build/src/xpcom/tests/gtest/TestPLDHash.cpp:353
07:49:58  WARNING -  TEST-UNEXPECTED-FAIL | PLDHashTableTest.GrowToMaxCapacity | test completed (time: 30573ms)

What I notice is the time difference of 12 seconds vs 30 seconds- I am not sure how to get other information here?
(Reporter)

Comment 1

2 years ago
:njn, I see you wrote this test over a few different patches, can you help me figure out why this is causing problems in the cloud?
Flags: needinfo?(n.nethercote)
(Assignee)

Comment 2

2 years ago
The test keeps adding elements to a PLDHashTable until we hit the maximum size, which is 1<<26. This requires repeatedly doubling the table size up to a max of 512 MiB. If any of the allocations fail the test will fail.

For this reason the test is already disabled on Android. Are the Windows machines in question 32-bit? If so, failing to allocate 512 MiB is probably not unreasonable. We could perhaps disable this test on all 32-bit platforms.
Flags: needinfo?(n.nethercote)

Comment 3

2 years ago
They are 32bit systems. Win 7 has 3.5 gb available to the OS we have some additional overhead on these boxes that are not on hardware with the xenagent and ec2config services.
(Reporter)

Comment 4

2 years ago
would a 32 bit machine intermittently fail?  If that sounds plausible, then I agree on disabling this for 32 bit systems.  Is there a way to detect this at runtime vs compile time?
Flags: needinfo?(n.nethercote)
(Assignee)

Comment 5

2 years ago
(In reply to Joel Maher (:jmaher) from comment #4)
> would a 32 bit machine intermittently fail?

It's conceivable.

> If that sounds plausible, then
> I agree on disabling this for 32 bit systems.  Is there a way to detect this
> at runtime vs compile time?

Yes. I'll write a patch tomorrow.
Flags: needinfo?(n.nethercote)
(Assignee)

Comment 6

2 years ago
Created attachment 8745877 [details] [diff] [review]
Don't run the PLDHashTableTest.GrowToMaxCapacity gtest on 32-bit machines, because it sometimes OOMs
Attachment #8745877 - Flags: review?(erahm)
(Assignee)

Updated

2 years ago
Assignee: nobody → n.nethercote
Status: NEW → ASSIGNED
Comment on attachment 8745877 [details] [diff] [review]
Don't run the PLDHashTableTest.GrowToMaxCapacity gtest on 32-bit machines, because it sometimes OOMs

Review of attachment 8745877 [details] [diff] [review]:
-----------------------------------------------------------------

Seems reasonable, I guess we're not concerned about 64-bit android?
Attachment #8745877 - Flags: review?(erahm) → review+
(Assignee)

Comment 8

2 years ago
> Seems reasonable, I guess we're not concerned about 64-bit android?

Is that a thing? Huh. I figure we can worry about it if we start getting failures.
(Assignee)

Comment 9

2 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/984828b14c451b8570a01e7ebbfac369e39dc762
Bug 1267227 - Don't run the PLDHashTableTest.GrowToMaxCapacity gtest on 32-bit machines, because it sometimes OOMs. r=erahm.

Comment 10

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/984828b14c45
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
status-firefox49: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla49
You need to log in before you can comment on or make changes to this bug.