Closed Bug 1648723 Opened 7 months ago Closed 6 months ago

Introduce a "half backstop" where we select a subset of tests to run using the smart scheduler

Categories

(Firefox Build System :: Task Configuration, enhancement)

enhancement

Tracking

(firefox81 fixed)

RESOLVED FIXED
81 Branch
Tracking Status
firefox81 --- fixed

People

(Reporter: marco, Assigned: marco)

References

(Blocks 2 open bugs)

Details

Attachments

(4 files)

On normal pushes, we use the smart scheduler with a high confidence threshold.
On backstop pushes, we run everything.

We could introduce a third kind, a "half" backstop, where we use the smart scheduler with a low confidence threshold, scheduling all tests that would have been selected with low confidence threshold between the previous backstop and now.
With a low confidence threshold, we catch almost all regressions, but we'd run at most half the tests, so it'd be a big win compared to a full backstop.

Assignee: nobody → mcastelluccio
Attachment #9160687 - Attachment description: Bug 1648723 - Introduce a new backstop strategy which optimizes tasks away using bugbug on the N/2th push → Bug 1648723 - Make it possible to run the bugbug optimization strategy on a set of pushes and use it to optimize some backstops. r=ahal,jmaher
Status: NEW → ASSIGNED
Attachment #9160687 - Attachment description: Bug 1648723 - Make it possible to run the bugbug optimization strategy on a set of pushes and use it to optimize some backstops. r=ahal,jmaher → Bug 1648723 - Run full backstops every 20 pushes, and a backstop optimized by bugbug every 10 pushes. r=ahal!,jmaher
Attachment #9166808 - Attachment description: Bug 1648723 - Use new pushdata module in the backstop module to avoid code duplication. r=ahal → Bug 1648723 - Use the util/hg.py module in the backstop module to avoid code duplication. r=ahal
Attachment #9160687 - Attachment description: Bug 1648723 - Run full backstops every 20 pushes, and a backstop optimized by bugbug every 10 pushes. r=ahal!,jmaher → Bug 1648723 - Run full backstops every 20 pushes, and backstops optimized by bugbug every 10 pushes. r=ahal!,jmaher
Pushed by mcastelluccio@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/b99b7e7967bb
Make it possible to run the bugbug optimization strategy on a set of pushes. r=ahal
https://hg.mozilla.org/integration/autoland/rev/55bcd608e834
Use the util/hg.py module in the backstop module to avoid code duplication. r=ahal
https://hg.mozilla.org/integration/autoland/rev/0026f4c69875
Define a new push-interval-20 strategy that optimizes everything except every 20th push. r=ahal
Regressions: 1656315

Forgot to add leave-open, there is one last patch to land.

Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Pushed by mcastelluccio@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/17a6cf8777b1
Run full backstops every 20 pushes, and backstops optimized by bugbug every 10 pushes. r=ahal,jmaher
Status: REOPENED → RESOLVED
Closed: 6 months ago6 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.