Closed Bug 1196464 Opened 9 years ago Closed 9 years ago

Split Gtests into their own automation job and turn them on to ride the trains

Categories

(Testing :: General, defect)

Product:
Testing
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: chmanchester, Assigned: chmanchester)

References

Details

Attachments

(3 files)

Split the gtests out of cpp-gtest, try only
2.61 KB, patch
rail
: review+
Details | Diff | Splinter Review
PR with updated symbol
46 bytes, text/x-github-pull-request
emorley
: review+
Details | Review
MozReview Request: Bug 1196464 - Schedule gtests on integration branches and to ride the trains. r=rail
40 bytes, text/x-review-board-request
rail
: review+
Details
In bug 1179955 I got them running next to Cpp unittests because their runtime is so short. As we're getting closer to being able to turn them on to ride the trains, I'm thinking these will be better off in their own job. This will simplify the initial roll out because we wont have to worry about replacing an existing job, will increase visibility for these, and will save us the trouble of splitting these out later as new GTests are added. This bug will cover the buildbot-config and treeherder work to make the split.
Builders added: + Rev4 MacOSX Snow Leopard 10.6 try debug test gtest + Rev4 MacOSX Snow Leopard 10.6 try opt test gtest + Rev5 MacOSX Yosemite 10.10 try debug test gtest + Rev5 MacOSX Yosemite 10.10 try opt test gtest + Ubuntu ASAN VM 12.04 x64 try opt test gtest + Ubuntu Code Coverage VM 12.04 x64 try opt test gtest + Ubuntu TSAN VM 12.04 x64 try opt test gtest + Ubuntu VM 12.04 try debug test gtest + Ubuntu VM 12.04 try opt test gtest + Ubuntu VM 12.04 x64 try debug test gtest + Ubuntu VM 12.04 x64 try opt test gtest + Windows 10 64-bit try debug test gtest + Windows 10 64-bit try opt test gtest + Windows 7 32-bit try debug test gtest + Windows 7 32-bit try opt test gtest + Windows 8 64-bit try debug test gtest + Windows 8 64-bit try opt test gtest + Windows XP 32-bit try debug test gtest + Windows XP 32-bit try opt test gtest Builders removed - Rev4 MacOSX Snow Leopard 10.6 try debug test cpp_gtest - Rev4 MacOSX Snow Leopard 10.6 try opt test cpp_gtest - Rev5 MacOSX Yosemite 10.10 try debug test cpp_gtest - Rev5 MacOSX Yosemite 10.10 try opt test cpp_gtest - Ubuntu ASAN VM 12.04 x64 try opt test cpp_gtest - Ubuntu Code Coverage VM 12.04 x64 try opt test cpp_gtest - Ubuntu TSAN VM 12.04 x64 try opt test cpp_gtest - Ubuntu VM 12.04 try debug test cpp_gtest - Ubuntu VM 12.04 try opt test cpp_gtest - Ubuntu VM 12.04 x64 try debug test cpp_gtest - Ubuntu VM 12.04 x64 try opt test cpp_gtest - Windows 10 64-bit try debug test cpp_gtest - Windows 10 64-bit try opt test cpp_gtest - Windows 7 32-bit try debug test cpp_gtest - Windows 7 32-bit try opt test cpp_gtest - Windows 8 64-bit try debug test cpp_gtest - Windows 8 64-bit try opt test cpp_gtest - Windows XP 32-bit try debug test cpp_gtest - Windows XP 32-bit try opt test cpp_gtest
Attachment #8650195 - Flags: review?(rail) → review+
Attachment #8650196 - Flags: review?(cdawson) → review+
Bug 1196464 - Follow up to fix keys specifying config files for gtests. r=rail
Attachment #8655105 - Flags: review?(rail)
Comment on attachment 8655105 [details] MozReview Request: Bug 1196464 - Schedule gtests on integration branches and to ride the trains. r=rail https://reviewboard.mozilla.org/r/17823/#review15911
Attachment #8655105 - Flags: review?(rail) → review+
Attachment #8655105 - Attachment description: MozReview Request: Bug 1196464 - Follow up to fix keys specifying config files for gtests. r=rail → MozReview Request: Bug 1196464 - Schedule gtests on integration branches and to ride the trains. r=rail
Comment on attachment 8655105 [details] MozReview Request: Bug 1196464 - Schedule gtests on integration branches and to ride the trains. r=rail Bug 1196464 - Schedule gtests on integration branches and to ride the trains. r=rail
Builders added: + Rev4 MacOSX Snow Leopard 10.6 alder debug test gtest + Rev4 MacOSX Snow Leopard 10.6 alder opt test gtest + Rev4 MacOSX Snow Leopard 10.6 ash debug test gtest + Rev4 MacOSX Snow Leopard 10.6 ash opt test gtest + Rev4 MacOSX Snow Leopard 10.6 b2g-inbound debug test gtest + Rev4 MacOSX Snow Leopard 10.6 b2g-inbound opt test gtest + Rev4 MacOSX Snow Leopard 10.6 fx-team debug test gtest + Rev4 MacOSX Snow Leopard 10.6 fx-team opt test gtest + Rev4 MacOSX Snow Leopard 10.6 gum debug test gtest + Rev4 MacOSX Snow Leopard 10.6 gum opt test gtest + Rev4 MacOSX Snow Leopard 10.6 jamun debug test gtest + Rev4 MacOSX Snow Leopard 10.6 jamun opt test gtest + Rev4 MacOSX Snow Leopard 10.6 maple debug test gtest + Rev4 MacOSX Snow Leopard 10.6 maple opt test gtest + Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test gtest + Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test gtest + Rev4 MacOSX Snow Leopard 10.6 mozilla-inbound debug test gtest + Rev4 MacOSX Snow Leopard 10.6 mozilla-inbound opt test gtest + Rev4 MacOSX Snow Leopard 10.6 oak debug test gtest + Rev4 MacOSX Snow Leopard 10.6 oak opt test gtest + Rev5 MacOSX Yosemite 10.10 alder debug test gtest + Rev5 MacOSX Yosemite 10.10 alder opt test gtest + Rev5 MacOSX Yosemite 10.10 ash debug test gtest + Rev5 MacOSX Yosemite 10.10 ash opt test gtest + Rev5 MacOSX Yosemite 10.10 fx-team debug test gtest + Rev5 MacOSX Yosemite 10.10 fx-team opt test gtest + Rev5 MacOSX Yosemite 10.10 gum debug test gtest + Rev5 MacOSX Yosemite 10.10 gum opt test gtest + Rev5 MacOSX Yosemite 10.10 jamun debug test gtest + Rev5 MacOSX Yosemite 10.10 jamun opt test gtest + Rev5 MacOSX Yosemite 10.10 maple debug test gtest + Rev5 MacOSX Yosemite 10.10 maple opt test gtest + Rev5 MacOSX Yosemite 10.10 mozilla-central debug test gtest + Rev5 MacOSX Yosemite 10.10 mozilla-central opt test gtest + Rev5 MacOSX Yosemite 10.10 mozilla-inbound debug test gtest + Rev5 MacOSX Yosemite 10.10 mozilla-inbound opt test gtest + Rev5 MacOSX Yosemite 10.10 oak debug test gtest + Rev5 MacOSX Yosemite 10.10 oak opt test gtest + Ubuntu ASAN VM 12.04 x64 alder opt test gtest + Ubuntu ASAN VM 12.04 x64 ash opt test gtest + Ubuntu ASAN VM 12.04 x64 b2g-inbound opt test gtest + Ubuntu ASAN VM 12.04 x64 fx-team opt test gtest + Ubuntu ASAN VM 12.04 x64 gum opt test gtest + Ubuntu ASAN VM 12.04 x64 jamun opt test gtest + Ubuntu ASAN VM 12.04 x64 maple opt test gtest + Ubuntu ASAN VM 12.04 x64 mozilla-central opt test gtest + Ubuntu ASAN VM 12.04 x64 mozilla-inbound opt test gtest + Ubuntu ASAN VM 12.04 x64 oak opt test gtest + Ubuntu VM 12.04 alder debug test gtest + Ubuntu VM 12.04 alder opt test gtest + Ubuntu VM 12.04 ash debug test gtest + Ubuntu VM 12.04 ash opt test gtest + Ubuntu VM 12.04 b2g-inbound debug test gtest + Ubuntu VM 12.04 b2g-inbound opt test gtest + Ubuntu VM 12.04 b2g-inbound pgo test gtest + Ubuntu VM 12.04 elm debug test gtest + Ubuntu VM 12.04 elm opt test gtest + Ubuntu VM 12.04 fx-team debug test gtest + Ubuntu VM 12.04 fx-team opt test gtest + Ubuntu VM 12.04 fx-team pgo test gtest + Ubuntu VM 12.04 gum debug test gtest + Ubuntu VM 12.04 gum opt test gtest + Ubuntu VM 12.04 jamun debug test gtest + Ubuntu VM 12.04 jamun opt test gtest + Ubuntu VM 12.04 maple debug test gtest + Ubuntu VM 12.04 maple opt test gtest + Ubuntu VM 12.04 mozilla-central debug test gtest + Ubuntu VM 12.04 mozilla-central opt test gtest + Ubuntu VM 12.04 mozilla-central pgo test gtest + Ubuntu VM 12.04 mozilla-inbound debug test gtest + Ubuntu VM 12.04 mozilla-inbound opt test gtest + Ubuntu VM 12.04 mozilla-inbound pgo test gtest + Ubuntu VM 12.04 oak debug test gtest + Ubuntu VM 12.04 oak opt test gtest + Ubuntu VM 12.04 x64 alder debug test gtest + Ubuntu VM 12.04 x64 alder opt test gtest + Ubuntu VM 12.04 x64 ash debug test gtest + Ubuntu VM 12.04 x64 ash opt test gtest + Ubuntu VM 12.04 x64 b2g-inbound debug test gtest + Ubuntu VM 12.04 x64 b2g-inbound opt test gtest + Ubuntu VM 12.04 x64 b2g-inbound pgo test gtest + Ubuntu VM 12.04 x64 elm debug test gtest + Ubuntu VM 12.04 x64 elm opt test gtest + Ubuntu VM 12.04 x64 fx-team debug test gtest + Ubuntu VM 12.04 x64 fx-team opt test gtest + Ubuntu VM 12.04 x64 fx-team pgo test gtest + Ubuntu VM 12.04 x64 gum debug test gtest + Ubuntu VM 12.04 x64 gum opt test gtest + Ubuntu VM 12.04 x64 jamun debug test gtest + Ubuntu VM 12.04 x64 jamun opt test gtest + Ubuntu VM 12.04 x64 maple debug test gtest + Ubuntu VM 12.04 x64 maple opt test gtest + Ubuntu VM 12.04 x64 mozilla-central debug test gtest + Ubuntu VM 12.04 x64 mozilla-central opt test gtest + Ubuntu VM 12.04 x64 mozilla-central pgo test gtest + Ubuntu VM 12.04 x64 mozilla-inbound debug test gtest + Ubuntu VM 12.04 x64 mozilla-inbound opt test gtest + Ubuntu VM 12.04 x64 mozilla-inbound pgo test gtest + Ubuntu VM 12.04 x64 oak debug test gtest + Ubuntu VM 12.04 x64 oak opt test gtest + Windows 7 32-bit alder debug test gtest + Windows 7 32-bit alder opt test gtest + Windows 7 32-bit ash debug test gtest + Windows 7 32-bit ash opt test gtest + Windows 7 32-bit b2g-inbound debug test gtest + Windows 7 32-bit b2g-inbound opt test gtest + Windows 7 32-bit b2g-inbound pgo test gtest + Windows 7 32-bit fx-team debug test gtest + Windows 7 32-bit fx-team opt test gtest + Windows 7 32-bit fx-team pgo test gtest + Windows 7 32-bit gum debug test gtest + Windows 7 32-bit gum opt test gtest + Windows 7 32-bit jamun debug test gtest + Windows 7 32-bit jamun opt test gtest + Windows 7 32-bit maple debug test gtest + Windows 7 32-bit maple opt test gtest + Windows 7 32-bit mozilla-central debug test gtest + Windows 7 32-bit mozilla-central opt test gtest + Windows 7 32-bit mozilla-central pgo test gtest + Windows 7 32-bit mozilla-inbound debug test gtest + Windows 7 32-bit mozilla-inbound opt test gtest + Windows 7 32-bit mozilla-inbound pgo test gtest + Windows 7 32-bit oak debug test gtest + Windows 7 32-bit oak opt test gtest + Windows 8 64-bit alder debug test gtest + Windows 8 64-bit alder opt test gtest + Windows 8 64-bit ash debug test gtest + Windows 8 64-bit ash opt test gtest + Windows 8 64-bit b2g-inbound debug test gtest + Windows 8 64-bit b2g-inbound opt test gtest + Windows 8 64-bit b2g-inbound pgo test gtest + Windows 8 64-bit fx-team debug test gtest + Windows 8 64-bit fx-team opt test gtest + Windows 8 64-bit fx-team pgo test gtest + Windows 8 64-bit gum debug test gtest + Windows 8 64-bit gum opt test gtest + Windows 8 64-bit jamun debug test gtest + Windows 8 64-bit jamun opt test gtest + Windows 8 64-bit maple debug test gtest + Windows 8 64-bit maple opt test gtest + Windows 8 64-bit mozilla-central debug test gtest + Windows 8 64-bit mozilla-central opt test gtest + Windows 8 64-bit mozilla-central pgo test gtest + Windows 8 64-bit mozilla-inbound debug test gtest + Windows 8 64-bit mozilla-inbound opt test gtest + Windows 8 64-bit mozilla-inbound pgo test gtest + Windows 8 64-bit oak debug test gtest + Windows 8 64-bit oak opt test gtest Builders removed - Windows XP 32-bit try debug test gtest - Windows XP 32-bit try opt test gtest
Comment on attachment 8655105 [details] MozReview Request: Bug 1196464 - Schedule gtests on integration branches and to ride the trains. r=rail I have a question about this patch: we can't run the GTests on Windows PGO builds, but I can't find a way to exclude that in the buildbot config. There's already code to prevent us building them on PGO builds, so the tests will fail immediately. Treeherder has the granularity to hide these, is our best option to go with this patch and add leave the jobs hidden in Treeherder?
Attachment #8655105 - Flags: review+ → review?
Attachment #8655105 - Flags: review? → review?(rail)
Comment on attachment 8655105 [details] MozReview Request: Bug 1196464 - Schedule gtests on integration branches and to ride the trains. r=rail https://reviewboard.mozilla.org/r/17823/#review16305
Attachment #8655105 - Flags: review?(rail) → review+
Thanks for the review. Do you have any input for my question in comment 11?
Flags: needinfo?(rail)
To be honest I don't know how to disable those on PGO builds...
Flags: needinfo?(rail)
jlund, do you have any ideas for comment 11? The trouble is we can't build these on windows whenever PGO is enabled (which will be all our nightlies as well if I understand correctly, so this might get complicated), so the resources to run them wont exist for those builds.
Flags: needinfo?(jlund)
afaik we do not have logic in place to disable specific pgo test jobs. it's either disable all of them or none. To add such logic, we would need to do something like: for every slave_platform add a 'disabled_pgo_unittests' list that's empty. e.g. below this line[1] add: 'disabled_pgo_unittests': [], then, in your patch under say here[2] (or under your incoming 43 trunk enable patch) you would add: BRANCHES['try']['platforms'][platform][slave_platform]['disabled_pgo_unittests'] += GTEST[0] # 'gtest' finally, we would need to patch the part where we go from configuration to actual generating builders. you would need to change this[3] line and probably all of these[4] lines to something like: disabled_pgo_unittests = branch_config['platforms'][platform][slave_platform]['disabled_pgo_unittests'] if create_pgo_builders and test_type == 'opt' and suite_name not in disabled_pgo_unittests: [1] http://hg.mozilla.org/build/buildbot-configs/file/40e7ca1f953d/mozilla-tests/config.py#l794 [2] http://hg.mozilla.org/build/buildbot-configs/file/40e7ca1f953d/mozilla-tests/config.py#l2284 [3] https://dxr.mozilla.org/build-central/source/buildbotcustom/misc.py#2568 [4] https://dxr.mozilla.org/build-central/search?q=create_pgo_builders%20and%20test_type%20%3D%3D%20%27opt%27&case=true&=build-central&redirect=true
Flags: needinfo?(jlund)
saying all that, if these tests fail immediately, it is probably worth the small hit in slave pool efficiency and just hide these in th assuming sheriffs don't care. does this help?
Thanks jlund, that does help. I think any further work here would be better spent getting these to run on PGO builds (this would be good, given the nightlies are PGO), so I'm going to lean towards hiding them.
This is working as I expect.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
(In reply to Chris Manchester [:chmanchester] from comment #19) > Thanks jlund, that does help. I think any further work here would be better > spent getting these to run on PGO builds (this would be good, given the > nightlies are PGO), so I'm going to lean towards hiding them. The problem with this is that building gtests means linking a second copy of libxul. Actually now that I say that I realize that there's very little value in running gtests on PGO--because they're in a second copy of libxul, that library isn't going to be used during the profiling run, so the second link isn't going to do the same optimizations as the normal copy of libxul, so we wouldn't wind up testing the PGOed code anyway.
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #23) > (In reply to Chris Manchester [:chmanchester] from comment #19) > > Thanks jlund, that does help. I think any further work here would be better > > spent getting these to run on PGO builds (this would be good, given the > > nightlies are PGO), so I'm going to lean towards hiding them. > > The problem with this is that building gtests means linking a second copy of > libxul. > > Actually now that I say that I realize that there's very little value in > running gtests on PGO--because they're in a second copy of libxul, that > library isn't going to be used during the profiling run, so the second link > isn't going to do the same optimizations as the normal copy of libxul, so we > wouldn't wind up testing the PGOed code anyway. That makes sense. Nightlies don't work the way I thought they did when I made that comment.
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: