Closed Bug 1403222 Opened 2 years ago Closed 2 years ago

Add a test suite family to SCHEDULES.exclusive

Categories

(Firefox Build System :: Task Configuration, task)

task
Not set

Tracking

(Not tracked)

RESOLVED FIXED
mozilla58

People

(Reporter: ahal, Assigned: ahal)

References

(Blocks 2 open bugs)

Details

Attachments

(1 file)

Another group of mutually exclusive jobs are test suites. This isn't as impactful as os_family as it only applies to test harness code and tests/manifests. But it's still worth adding as it'll reduce test only changes by *a lot*.

To start I'd just like to apply the tags to the test harnesses, though we could go a lot further and apply the tags to individual tests and manifests as well.
I *think* this patch will do what we want, though I'm not sure how to test it on try since pushing these changes means that I'm modifying files outside of any given test harness. Guess I could add a blanket ** in the root moz.build just for try purposes.
Comment on attachment 8912284 [details]
Bug 1403222 - Add test suite families to SCHEDULES.exclusive

Is this doing what it looks like it's doing? Can you think of any reasons we shouldn't do this?
Attachment #8912284 - Flags: feedback?(dustin)
Comment on attachment 8912284 [details]
Bug 1403222 - Add test suite families to SCHEDULES.exclusive

https://reviewboard.mozilla.org/r/183638/#review188910

It makes sense to me!  This is pretty much exactly how I envisioned this.
Attachment #8912284 - Flags: feedback?(dustin) → feedback+
See Also: → 1403703
Comment on attachment 8912284 [details]
Bug 1403222 - Add test suite families to SCHEDULES.exclusive

Fyi this is unchanged since you f+'ed.

I'm still not sure how to test this on try. Since my push contains these changes it won't optimize based on SCHEDULES.exclusive. Should we just land on central and then verify things work as expected afterwards?
Attachment #8912284 - Flags: review?(dustin)
Assignee: nobody → ahalberstadt
Status: NEW → ASSIGNED
Comment on attachment 8912284 [details]
Bug 1403222 - Add test suite families to SCHEDULES.exclusive

https://reviewboard.mozilla.org/r/183638/#review190230

YOLO?  I don't have any good ideas either..
Attachment #8912284 - Flags: review?(dustin) → review+
I just pushed to try and the decision task didn't even run? I'm pretty sure it ran in an earlier try push. Could be an unrelated issue, but won't have time to investigate until next week.
Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/ddfc127526b2
Add test suite families to SCHEDULES.exclusive r=dustin
Backed out for failing python/mozbuild/mozbuild/test/frontend/test_reader.py::TestBuildReader::test_schedules:

https://hg.mozilla.org/integration/autoland/rev/b1588798f186f15d41b891346899cdc3eae9344d

Push with bustage: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=ddfc127526b292037d2e3298c65afd15c37ddc91&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=usercancel&filter-resultStatus=runnable
Build log: https://treeherder.mozilla.org/logviewer.html#?job_id=134399326&repo=autoland

[task 2017-10-02T14:00:00.247Z] 14:00:00  WARNING - ../python/mozbuild/mozbuild/test/frontend/test_reader.py::TestBuildReader::test_schedules TEST-UNEXPECTED-FAIL
[task 2017-10-02T14:00:00.247Z] 14:00:00     INFO - =================================== FAILURES ===================================
[task 2017-10-02T14:00:00.247Z] 14:00:00     INFO - ________________________ TestBuildReader.test_schedules ________________________
[task 2017-10-02T14:00:00.247Z] 14:00:00     INFO - self = <mozbuild.test.frontend.test_reader.TestBuildReader testMethod=test_schedules>
[task 2017-10-02T14:00:00.247Z] 14:00:00     INFO -     def test_schedules(self):
[task 2017-10-02T14:00:00.247Z] 14:00:00     INFO -         reader = self.reader('schedules')
[task 2017-10-02T14:00:00.247Z] 14:00:00     INFO -         info = reader.files_info(['somefile', 'foo.win', 'foo.osx', 'subd/aa.py', 'subd/yaml.py'])
[task 2017-10-02T14:00:00.247Z] 14:00:00     INFO -         # default: all exclusive, no inclusive
[task 2017-10-02T14:00:00.247Z] 14:00:00     INFO -         self.assertEqual(info['somefile']['SCHEDULES'].inclusive, [])
[task 2017-10-02T14:00:00.247Z] 14:00:00     INFO - >       self.assertEqual(info['somefile']['SCHEDULES'].exclusive, ['android', 'linux', 'macosx', 'windows'])
[task 2017-10-02T14:00:00.248Z] 14:00:00     INFO - E       AssertionError: [u'android', u'awsy', u'cppunittest', u'firefox-ui', u'geckoview', u'gtest', u'jittest', u'linux', u'macosx', u'marionette', u'mochitest', u'reftest', u'robocop', u'talos', u'telemetry-tests-client', u'web-platform-tests', u'web-platform-tests-reftests', u'web-platform-tests-wdspec', u'windows', u'xpcshell', u'xpcshell-coverage'] != [u'android', u'linux', u'macosx', u'windows']
[task 2017-10-02T14:00:00.248Z] 14:00:00     INFO - ../python/mozbuild/mozbuild/test/frontend/test_reader.py:488: AssertionError
[task 2017-10-02T14:00:00.248Z] 14:00:00     INFO - ===================== 1 failed, 32 passed in 0.94 seconds ======================
Flags: needinfo?(ahalberstadt)
My mistake, used artifact builds as I didn't realize there was a mozbuild test for this.
Flags: needinfo?(ahalberstadt)
Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/e34125164eb5
Add test suite families to SCHEDULES.exclusive r=dustin
https://hg.mozilla.org/mozilla-central/rev/e34125164eb5
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla58
Blocks: fastci
Product: TaskCluster → Firefox Build System
You need to log in before you can comment on or make changes to this bug.