(In reply to Simon Giesecke [:sg] [he/him] from comment #2)
Hm. It might depend on what kind of changes you are doing, but I ran quite frequently into the issue that either the
build-linux64-base-toolchains failed. Removing them completely from autoland doesn't feel like a good idea to me, if we still care about their results. If we don't, we should increase the minimum requirement for building Firefox (but this would only resolve the need to the build-linux64-base-toolchains-clang builds). Fixing issues with the builds with a time distance makes that more complicated. and since this will require separate bugs to be filed, there might be a significant time distance. The alternative of backing out changes that already landed on mozilla-central would be even worse.
I agree with you, but we have to weigh the cost of running these builds with the probability they find unique regressions (e.g. if it's 50k$ per year, 40 failures out of 23144 pushes might not be enough to justify the expense).
There's no real correlation in what the older gcc and clang compilers complain about, which are typically compiler bugs.
Restricting to only the debug builds might be acceptable. This would bear the risk of masking issues with non-debug-only code, but this is rather rare.
There were 8 cases where a base-toolchain opt build failed and the equivalent debug didn't. So, one order of magnitude more rare.
I am not sure what "backstop pushes" are, and if restricting these builds to such pushes might be more acceptable.
Once every 10 pushes on autoland, we have a "backstop" push which runs everything. So moving to backstop, compared to moving to mozilla-central only, would decrease the delay between landing the patch and noticing the failure (at a higher cost as we'd be running the builds more often than if we moved to mozilla-central).
I think we should:
- at least move these builds to "backstop" pushes only, given their low failure frequency (and they have no tests associated to them, so no need to backfill);
- if the cost is high, move the opt builds to mozilla-central only;
- if the cost is very high, move both the debug builds and the opt builds to mozilla-central only.