1648723 - Introduce a "half backstop" where we select a subset of tests to run using the smart scheduler

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, enhancement)

Description

[Marco Castelluccio [:marco]]

On normal pushes, we use the smart scheduler with a high confidence threshold.
On backstop pushes, we run everything.

We could introduce a third kind, a "half" backstop, where we use the smart scheduler with a low confidence threshold, scheduling all tests that would have been selected with low confidence threshold between the previous backstop and now.
With a low confidence threshold, we catch almost all regressions, but we'd run at most half the tests, so it'd be a big win compared to a full backstop.

Comment 1

[Marco Castelluccio [:marco]]

Attachment: Bug 1648723 - Run full backstops every 20 pushes, and backstops optimized by bugbug every 10 pushes. r=ahal!,jmaher

Comment 2

[Marco Castelluccio [:marco]]

Attachment: Bug 1648723 - Make it possible to run the bugbug optimization strategy on a set of pushes. r=ahal

Comment 3

[Marco Castelluccio [:marco]]

Attachment: Bug 1648723 - Use the util/hg.py module in the backstop module to avoid code duplication. r=ahal

Depends on D85275

Comment 4

[Marco Castelluccio [:marco]]

Attachment: Bug 1648723 - Define a new push-interval-20 strategy that optimizes everything except every 20th push. r=ahal

Depends on D85276

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b99b7e7967bb
Make it possible to run the bugbug optimization strategy on a set of pushes. r=ahal
https://hg.mozilla.org/integration/autoland/rev/55bcd608e834
Use the util/hg.py module in the backstop module to avoid code duplication. r=ahal
https://hg.mozilla.org/integration/autoland/rev/0026f4c69875
Define a new push-interval-20 strategy that optimizes everything except every 20th push. r=ahal

Comment 6

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/b99b7e7967bb
https://hg.mozilla.org/mozilla-central/rev/55bcd608e834
https://hg.mozilla.org/mozilla-central/rev/0026f4c69875

Comment 7

[Marco Castelluccio [:marco]]

Forgot to add leave-open, there is one last patch to land.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/17a6cf8777b1
Run full backstops every 20 pushes, and backstops optimized by bugbug every 10 pushes. r=ahal,jmaher

Comment 9

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/17a6cf8777b1