1403222 - Add a test suite family to SCHEDULES.exclusive

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Andrew Halberstadt [:ahal]]

Another group of mutually exclusive jobs are test suites. This isn't as impactful as os_family as it only applies to test harness code and tests/manifests. But it's still worth adding as it'll reduce test only changes by *a lot*.

To start I'd just like to apply the tags to the test harnesses, though we could go a lot further and apply the tags to individual tests and manifests as well.

Comment 1

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1403222 - Add test suite families to SCHEDULES.exclusive

Review commit: https://reviewboard.mozilla.org/r/183638/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/183638/

Comment 2

[Andrew Halberstadt [:ahal]]

I *think* this patch will do what we want, though I'm not sure how to test it on try since pushing these changes means that I'm modifying files outside of any given test harness. Guess I could add a blanket ** in the root moz.build just for try purposes.

Comment 3

[Andrew Halberstadt [:ahal]]

Comment on attachment 8912284 [details]
Bug 1403222 - Add test suite families to SCHEDULES.exclusive

Is this doing what it looks like it's doing? Can you think of any reasons we shouldn't do this?

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8912284 [details]
Bug 1403222 - Add test suite families to SCHEDULES.exclusive

https://reviewboard.mozilla.org/r/183638/#review188910

It makes sense to me!  This is pretty much exactly how I envisioned this.

Comment 5

[Andrew Halberstadt [:ahal]]

Comment on attachment 8912284 [details]
Bug 1403222 - Add test suite families to SCHEDULES.exclusive

Fyi this is unchanged since you f+'ed.

I'm still not sure how to test this on try. Since my push contains these changes it won't optimize based on SCHEDULES.exclusive. Should we just land on central and then verify things work as expected afterwards?

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8912284 [details]
Bug 1403222 - Add test suite families to SCHEDULES.exclusive

https://reviewboard.mozilla.org/r/183638/#review190230

YOLO?  I don't have any good ideas either..

Comment 7

[Andrew Halberstadt [:ahal]]

I just pushed to try and the decision task didn't even run? I'm pretty sure it ran in an earlier try push. Could be an unrelated issue, but won't have time to investigate until next week.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ddfc127526b2
Add test suite families to SCHEDULES.exclusive r=dustin

Comment 9

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Backed out for failing python/mozbuild/mozbuild/test/frontend/test_reader.py::TestBuildReader::test_schedules:

https://hg.mozilla.org/integration/autoland/rev/b1588798f186f15d41b891346899cdc3eae9344d

Push with bustage: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=ddfc127526b292037d2e3298c65afd15c37ddc91&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=usercancel&filter-resultStatus=runnable
Build log: https://treeherder.mozilla.org/logviewer.html#?job_id=134399326&repo=autoland

[task 2017-10-02T14:00:00.247Z] 14:00:00  WARNING - ../python/mozbuild/mozbuild/test/frontend/test_reader.py::TestBuildReader::test_schedules TEST-UNEXPECTED-FAIL
[task 2017-10-02T14:00:00.247Z] 14:00:00     INFO - =================================== FAILURES ===================================
[task 2017-10-02T14:00:00.247Z] 14:00:00     INFO - ________________________ TestBuildReader.test_schedules ________________________
[task 2017-10-02T14:00:00.247Z] 14:00:00     INFO - self = <mozbuild.test.frontend.test_reader.TestBuildReader testMethod=test_schedules>
[task 2017-10-02T14:00:00.247Z] 14:00:00     INFO -     def test_schedules(self):
[task 2017-10-02T14:00:00.247Z] 14:00:00     INFO -         reader = self.reader('schedules')
[task 2017-10-02T14:00:00.247Z] 14:00:00     INFO -         info = reader.files_info(['somefile', 'foo.win', 'foo.osx', 'subd/aa.py', 'subd/yaml.py'])
[task 2017-10-02T14:00:00.247Z] 14:00:00     INFO -         # default: all exclusive, no inclusive
[task 2017-10-02T14:00:00.247Z] 14:00:00     INFO -         self.assertEqual(info['somefile']['SCHEDULES'].inclusive, [])
[task 2017-10-02T14:00:00.247Z] 14:00:00     INFO - >       self.assertEqual(info['somefile']['SCHEDULES'].exclusive, ['android', 'linux', 'macosx', 'windows'])
[task 2017-10-02T14:00:00.248Z] 14:00:00     INFO - E       AssertionError: [u'android', u'awsy', u'cppunittest', u'firefox-ui', u'geckoview', u'gtest', u'jittest', u'linux', u'macosx', u'marionette', u'mochitest', u'reftest', u'robocop', u'talos', u'telemetry-tests-client', u'web-platform-tests', u'web-platform-tests-reftests', u'web-platform-tests-wdspec', u'windows', u'xpcshell', u'xpcshell-coverage'] != [u'android', u'linux', u'macosx', u'windows']
[task 2017-10-02T14:00:00.248Z] 14:00:00     INFO - ../python/mozbuild/mozbuild/test/frontend/test_reader.py:488: AssertionError
[task 2017-10-02T14:00:00.248Z] 14:00:00     INFO - ===================== 1 failed, 32 passed in 0.94 seconds ======================

Comment 10

[Andrew Halberstadt [:ahal]]

My mistake, used artifact builds as I didn't realize there was a mozbuild test for this.

Comment 11

[Andrew Halberstadt [:ahal]]

Comment on attachment 8912284 [details]
Bug 1403222 - Add test suite families to SCHEDULES.exclusive

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/183638/diff/1-2/

Comment 12

[Andrew Halberstadt [:ahal]]

Test fixed:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=1062833968754633a388d6c49f21f516fbe173d7

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e34125164eb5
Add test suite families to SCHEDULES.exclusive r=dustin

Comment 14

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/e34125164eb5