1629642 - Implement a smarter platform filter for the bugbug scheduling algorithm

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, enhancement)

Description

[Andrew Halberstadt [:ahal]]

In bug 1624902 I implemented a filter that reduces bugbug's chosen platforms to debug only. However we can do something smarter. My current idea is something like this (though may morph over time):

1. Break each manifest into tiers based on their confidence intervals, e.g:
   a. Tier 1: >0.9
   b. Tier 2: >0.75
   c. Tier 3: >0.5
   d. Tier 4: >0

2. Assign constraints to each tier, e.g:

   a. A manifest in tier 1 is guaranteed to run in at least one task from every major configuration.
   b. A manifest in tier 2 is guaranteed to run in at least one task from every OS family.
   c. A manifest in tier 3 is guaranteed to run in at least one mobile and one desktop platform.
   d. A manifest in tier 4 is guaranteed to run in at least one task

3. Overall, try to cover as many different configurations as possible while satisfying the above constraints. This way we try to spread out our coverage across as many configurations as possible while minimizing duplication.

This is just an example, we may end up with more or fewer tiers, different constraints, different thresholds etc.

Comment 1

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1629642 - [taskgraph] Pull bugbug's 'platform filter' into a standalone strategy, r?marco

This change is beneficial for two reasons:

1. Improvement on the single responsibility principle
2. Platform filter isn't 'bugbug' specific. E.g, the 'relevant tests' optimizer
   could also theoretically use this. Having it as a standalone optimizer allows
   us to compose it with other strategies.

Comment 2

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1629642 - [taskgraph] Add an option to sum confidence thresholds of manifests in bugbug scheduler, r?marco

Comment 3

[Andrew Halberstadt [:ahal]]

Hm, implementing the above as stated won't be possible. We won't be able to make those guarantees as the optimizers process tasks one at a time, so have no idea what is coming. I'm implementing something a lot simpler for now, and we can iterate on it over time.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f94d27cca400
[taskgraph] Pull bugbug's 'platform filter' into a standalone strategy, r=marco

Comment 5

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/f94d27cca400

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/25119864bcc1
[taskgraph] Add an option to sum confidence thresholds of manifests in bugbug scheduler, r=marco

Comment 7

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1629642 - [taskgraph] Split 'test' optimization strategy arg into two

Depends on D71314

Comment 8

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1629642 - [taskgraph] Implement a 'disperse' platform optimization strategy

Depends on D72464

Comment 9

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1629642 - [ci] Add a shadow-scheduler for the 'bugbug.disperse' scheduling algorithm, r?marco

Depends on D72465

Comment 10

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/25119864bcc1

Comment 11

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1629642 - [ci] Fix wrong strategy name in bugbug_combined_high shadow-scheduler, DONTBUILD, r?marco

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d7d94860086b
[ci] Fix wrong strategy name in bugbug_combined_high shadow-scheduler, DONTBUILD, r=marco

Comment 13

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/d7d94860086b

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0414f57713e8
[taskgraph] Split 'test' optimization strategy arg into two r=marco
https://hg.mozilla.org/integration/autoland/rev/c5df25796d9b
[taskgraph] Implement a 'disperse' platform optimization strategy r=marco
https://hg.mozilla.org/integration/autoland/rev/a34695d9b99d
[ci] Add a shadow-scheduler for the 'bugbug.disperse' scheduling algorithm, r=marco

Comment 15

[Oana Pop-Rus]

https://hg.mozilla.org/mozilla-central/rev/0414f57713e8
https://hg.mozilla.org/mozilla-central/rev/c5df25796d9b
https://hg.mozilla.org/mozilla-central/rev/a34695d9b99d