1637542 - Don't run opt builds on autoland when we have a corresponding instrumented build

Status: RESOLVED FIXED

(Testing :: General, enhancement)

Description

[Marco Castelluccio [:marco]]

E.g. instrumented-build-linux64-shippable/opt and build-linux64/opt likely find the same regressions.

We could run instrumented-build-linux64-shippable/opt only, and stop running build-linux64/opt (either running it only on backstop pushes, or even better run it only on mozilla-central).

Comment 1

[Marco Castelluccio [:marco]]

We have to:

• check how often build-linux64/opt fails when instrumented-build-linux64-shippable/opt or build-linux64-shippable/opt don't;
• check what's the cost of build-linux64/opt.

Same for all other platforms where we have the same builds.

Comment 2

[Marco Castelluccio [:marco]]

For Linux, the opt build failed exclusively 22 times (although most of them seem to be false positives, e.g. wrong classifications or some builds were canceled after the first failed).
For win32, same.
For win64, a few less, 16.

I think we can definitely run these at least on backstop pushes, but probably even just on mozilla-central.

The savings should be similar to bug 1636902 (precisely, around 3/4), given that we now cross-compile the Windows build on Linux. So around 18k if we move to mozilla-central, a bit less if we run on backstop pushes.

Comment 3

[Marco Castelluccio [:marco]]

We should stop running webgpu on opt (or somehow run it off of the instrumented build if for some reason we can't run it off of the pgoed one).

Comment 4

[[e-mail] (:bhearsum)]

So just to be clear, these are the potentially relevant tasks I currently see on autoland:
artifact-build-linux64-artifact/opt
build-android-api-16-gcp/opt
build-android-api-16/opt
build-android-api-16/pgo
build-linux64/opt
build-linux64-shippable/opt
build-macosx64-shippable/opt
build-win32/opt
build-win32-shippable/opt
build-win64/opt
build-win64-shippable/opt
instrumented-build-android-api-16/pgo
instrumented-build-linux64-shippable/opt
instrumented-build-macosx64-shippable/opt
instrumented-build-win32-shippable/opt
instrumented-build-win64-shippable/opt

I'm pretty sure the artifact & gcp ones can be ignored for our purposes here, so let's ignore them.

Aside from that, it sounds like you're saying that we should explicitly ensure that build-{linux64,macosx64,win32,win64}{,-shippable} should not be runnable on autoland -- is that right? Should they be allowed to run as part of backfilling, or not even that?

Comment 5

[Marco Castelluccio [:marco]]

I was thinking we should disable the following (still allowed to run as part of backfilling):
build-linux64/opt
build-win32/opt
build-win64/opt
As they are redundant with their instrumented or shippable counterparts.
It seems like the Android opt build is still used for running tests, so we can't disable it.

After today's discussion though, maybe we should consider making autoland run tests on opt normally and run shippable builds only on backstop pushes and for perf tests. This could be a separate bug though, it might take longer and might require buy-in from more people.

Comment 6

[[e-mail] (:bhearsum)]

(In reply to Marco Castelluccio [:marco] from comment #5)

I was thinking we should disable the following (still allowed to run as part of backfilling):
build-linux64/opt
build-win32/opt
build-win64/opt
As they are redundant with their instrumented or shippable counterparts.
It seems like the Android opt build is still used for running tests, so we can't disable it.

Thanks for the clarifications! This should be relatively straightforward to do.

After today's discussion though, maybe we should consider making autoland run tests on opt normally and run shippable builds only on backstop pushes and for perf tests. This could be a separate bug though, it might take longer and might require buy-in from more people.

Yeah, let's move this out to something else.

Comment 7

[[e-mail] (:bhearsum)]

Attachment: diff --git a/taskclu

OK, I have a minimal patch that does this:
```

Comment 8

[[e-mail] (:bhearsum)]

I have no idea what happened with the previous comment, but in any case, the attachment is a patch that removes some of the builds we mentioned (along with some other things, because I need to tweak it more).

However, investigating this uncovered some things we need to think about before we can proceed, including:

• What to do with artifact builds on autoland. They exist for linux64, and depend on the opt builds. Whatever we do with opt, needs to happen for artifact builds as well. My understanding is that artifact builds merely exist to speed up certain types of local builds (and possibly on try? that part isn't clear to me), so my gut says we can keep them off on autoland just like opt.
• Can we disable or move to shippable all of: browsertime-ares6, browsertime-speedometer, and mochitest-webgpu for the opt builds (including the quantum render variants)? The two browsertime suites are tier3 and currently only run on opt. webgpu seems to run everywhere except on shippable.

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

browsertime* is perf, so those need shippable, mochitest-webgpu can be whatever we need.

:sparky, what is path for browsertime tests on desktop?

Comment 10

[Greg Mierzwinski [:sparky]]

IIUC, the problematic browsertime ares6/speedometer tests will be disabled in this bug/patch and only run on linux64-shippable in mozilla-central: https://bugzilla.mozilla.org/show_bug.cgi?id=1639374

Comment 11

[[e-mail] (:bhearsum)]

Attachment: Bug 1637542: Disable {linux64,win32,win64} opt and linux64 artifact builds on autoland.

The opt builds were originally disabled in bug 1522111, but were re-enabled when webgpu was added.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e4b856d5476a
Disable {linux64,win32,win64} opt and linux64 artifact builds on autoland. r=jmaher

Comment 13

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/e4b856d5476a

Comment 14

[[e-mail] (:bhearsum)]

I've been using https://datastudio.google.com/reporting/1YOak7Cu3jvFXcupE1UvHnVC5JFKA-elY/page/GX6QB/edit and https://console.cloud.google.com/bigquery?project=moz-fx-data-taskclu-prod-8fbf&supportedpurview=project&j=bq:US:bquxjob_bea212a_1725b5bc4e4&page=queryresults to help calculate savings here.