1630708 - Increase confidence thresholds for the bugbug shadow schedulers

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, enhancement)

Description

[Marco Castelluccio [:marco]]

The bugbug-based shadow schedulers have very high regression detection rate, but are scheduling many tasks (except the last added one, bugbug_reduced).

For try, we want a high regression detection rate even at the cost of scheduling more tasks, but for autoland, we're OK with a smaller regression detection rate.

We should try increasing the confidence thresholds (considering that currently the one with 0.7 threshold, which is the max, detects a large percentage of regressions anyway).

Comment 1

[Marco Castelluccio [:marco]]

Attachment: Bug 1630708 - Increase the confidence thresholds used by the bugbug-based strategies. r=ahal

Keeping the same for the currently chosen strategy for try auto, since we
don't want to decrease its regression detection rate.

We also add a new shadow scheduler which uses the reduced set with a higher
confidence threshold.

Comment 2

[Marco Castelluccio [:marco]]

Attachment: Bug 1630708 - Rename bugbug_push_schedules shadow scheduler to bugbug_all for consistency. r=ahal

Depends on D71205

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6844825a2ada
Increase the confidence thresholds used by the bugbug-based strategies. r=ahal
https://hg.mozilla.org/integration/autoland/rev/b2cb3fea55e6
Rename bugbug_push_schedules shadow scheduler to bugbug_all for consistency. r=ahal

Comment 4

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/6844825a2ada
https://hg.mozilla.org/mozilla-central/rev/b2cb3fea55e6

Comment 5

[Alexandru Ionescu (needinfo me) [:alexandrui]]

[alert posted in the wrong bug]

Comment 6

[Marco Castelluccio [:marco]]

(In reply to Alexandru Ionescu (needinfo me) :alexandrui from comment #5)

== Change summary for alert #25648 (as of Sat, 18 Apr 2020 05:43:36 GMT) ==

Improvements:

22% tscrollx windows10-64-shippable-qr opt e10s stylo 1.06 -> 0.83
10% tart windows10-64-shippable-qr opt e10s stylo 2.82 -> 2.54

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=25648

It's definitely not from this bug.

Comment 7

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Fixed, thanks Maco.