1979870 - Consider using "Windows2012-64" for alerting on build-metrics (non-shippable)

Status: NEW

(Testing :: Performance, enhancement, P3)

Description

[Mayank Bansal]

https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&highlightCommonAlerts=0&replicates=0&series=autoland,1921052,1,2&series=autoland,1457010,1,2&timerange=7776000

• This graphs shows the comparison between "installer size opt nightly" and "installer size opt nightly shippable". The first does not have alerts, but is far less noisy, is always run at each commit, and is so sensitive that changes of 5kb are also clearly visible. It also has very clear correlation with "installer size opt nightly shippable" - so any change in the former also means a change in the latter.
• IMHO, it should also be used as the metric for alerts. And because it is run on each commit, any change will be very identified very early adn easily.
• will save sheriff time and effort (noise-free), will save CI capacity (because no bisection needed), and will be easy to identify smaller changes (very sensitive).
• Continue to monitor the "shippable" metric, but the primary metric for alerts should be the "non-shippable" metric. In most cases, the "non-shippable" will err on the side of caution - large changes on "non-shippable" will usually lead to smaller changes in "shippable" builds. So very low chance of missing-out on big changes
• PGO builds reduce the build size - at the same time PGO builds can also increase build size due to inlining. But since we continue to monitor both, chances of regression slipping are very low.
• On at-least two occasions, I have used the "non-shippable" builds to post comment on bugs causing regressions. The dev then fixed the regression. And all of this before even an alert was created for the shippable builds.
• This needs agreement with Product and/or Build team of course

cc davehunt who has initial interest in exploring this further.