1624902 - Limit the number of platforms bugbug manifests run on

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Andrew Halberstadt [:ahal]]

The bugbug service only returns manfiests to run. As of now, we simply run every task that contains this manifest. This means we are often running the same set of tests on 20 different configurations (or whatever it is).

Instead we should limit what gets run. Maybe we only run debug platforms (as they are more likely to fail). Maybe we try to round robin manifests into different configurations such that we get coverage across all configurations but don't run all tests on all of them. Maybe we do some combination of both.

Comment 1

[Marco Castelluccio [:marco]]

We should probably also limit the platforms for tasks returned by bugbug. Since it returns all possible tasks that might fail, many of them might be redundant with each other. I'm not sure how to best do this though.

Comment 2

[Andrew Halberstadt [:ahal]]

Yeah, because the scheduling algorithm doesn't know why those tasks were chosen (i.e, which manifests are important).. then it's kind of tricky to do. I think a better use of time would be getting reftest/wpt working with taskgraph chunking.

Comment 3

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1624902 - [taskgraph] Move experimental strategies into a common namespace, r?marco

Small cleanup to help compatmentalize these strategies.

Comment 4

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1624902 - [tryselect] Validate '--strategy' argument to ./mach try auto, r?marco

This ensures we fail with invalid module paths early. Otherwise users wouldn't
find out until the decision task fails.

Depends on D68463

Comment 5

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1624902 - [taskgraph] Implement substrategies in the bugbug scheduler for dealing with platforms, r?marco

The bugbug scheduler currently chooses which manifests are important, and we
then run every task that contains those manifests. This is likely overkill
and we can reduce the number of configurations we run these manifests on.

Depends on D68464

Comment 6

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1624902 - [ci] Add a shadow-scheduler for the 'bugbug-debug' scheduling algorithm, r?marco

Depends on D68466

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/08356d478d16
[taskgraph] Move experimental strategies into a common namespace, r=marco
https://hg.mozilla.org/integration/autoland/rev/c7a62f2d9034
[tryselect] Validate '--strategy' argument to ./mach try auto, r=marco
https://hg.mozilla.org/integration/autoland/rev/bf817363344e
[taskgraph] Implement substrategies in the bugbug scheduler for dealing with platforms, r=marco
https://hg.mozilla.org/integration/autoland/rev/b500403daa0c
[ci] Add a shadow-scheduler for the 'bugbug-debug' scheduling algorithm, r=marco

Comment 8

[Natalia Csoregi [:nataliaCs]]

Backed out for failures on test_optimize_strategies.py

backout: https://hg.mozilla.org/integration/autoland/rev/7ec9113ff5260558001e4fca1f8c9a73c7ffd06d

push: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&revision=b500403daa0cb31be4b9bb349af9911f7784bc6a&searchStr=Linux%2C18.04%2Cx64%2Copt&selectedJob=295650289

failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=295650289&repo=autoland&lineNumber=533

[task 2020-03-31T20:55:40.098Z] 0:34.73 taskcluster/taskgraph/test/test_optimize_strategies.py::test_bugbug_push_schedules[all-data3-expected3] PASSED
[task 2020-03-31T20:55:40.098Z] 0:34.73 taskcluster/taskgraph/test/test_optimize_strategies.py::test_bugbug_push_schedules[all-data4-expected4] PASSED
[task 2020-03-31T20:55:40.099Z] 0:34.73 taskcluster/taskgraph/test/test_optimize_strategies.py::test_bugbug_push_schedules[debug-data5-expected5] PASSED
[task 2020-03-31T20:55:40.099Z] 0:34.73 taskcluster/taskgraph/test/test_optimize_strategies.py::test_bugbug_timeout TEST-UNEXPECTED-FAIL
[task 2020-03-31T20:55:40.099Z] 0:34.73 taskcluster/taskgraph/test/test_optimize_strategies.py::test_bugbug_timeout ERROR
[task 2020-03-31T20:55:40.100Z] 0:34.74
[task 2020-03-31T20:55:40.100Z] 0:34.74 ==================================== ERRORS ====================================
[task 2020-03-31T20:55:40.101Z] 0:34.74 ___________________ ERROR at teardown of test_bugbug_timeout ___________________
[task 2020-03-31T20:55:40.101Z] 0:34.74
[task 2020-03-31T20:55:40.101Z] 0:34.74 @pytest.fixture
[task 2020-03-31T20:55:40.102Z] 0:34.74 def responses():
[task 2020-03-31T20:55:40.102Z] 0:34.74 with RequestsMock() as rsps:
[task 2020-03-31T20:55:40.102Z] 0:34.74 > yield rsps
[task 2020-03-31T20:55:40.103Z] 0:34.74
[task 2020-03-31T20:55:40.103Z] 0:34.74 taskcluster/taskgraph/test/conftest.py:13:

Comment 9

[Andrew Halberstadt [:ahal]]

I did a bad merge and forgot to update one of the new tests I had added in another bug to handle the changes in this one.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3ab0ab42da13
[taskgraph] Move experimental strategies into a common namespace, r=marco
https://hg.mozilla.org/integration/autoland/rev/89778a10b27e
[tryselect] Validate '--strategy' argument to ./mach try auto, r=marco
https://hg.mozilla.org/integration/autoland/rev/6d15fe012042
[taskgraph] Implement substrategies in the bugbug scheduler for dealing with platforms, r=marco
https://hg.mozilla.org/integration/autoland/rev/f6072df56b3f
[ci] Add a shadow-scheduler for the 'bugbug-debug' scheduling algorithm, r=marco

Comment 11

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/3ab0ab42da13
https://hg.mozilla.org/mozilla-central/rev/89778a10b27e
https://hg.mozilla.org/mozilla-central/rev/6d15fe012042
https://hg.mozilla.org/mozilla-central/rev/f6072df56b3f