1619966 - mach try fuzzy + artifact build + macOS mochitest-browser job generates 3-part PGO build dependency instead of artifact one

Status: NEW

(Firefox Build System :: Task Configuration, defect, P3)

Description

[:Gijs (he/him)]

STR:

1. update to central
2. use artifact mozconfig
3. ./mach try fuzzy browser/components/downloads
4. filter for 'macos 'opt
5. tick that one job
6. submit to try

ER:
artifact build + the one test with that directory

AR:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=8b699292ad3c0a12352dda53085d88c1926fd4ce

Which runs a 3-part pgo build instead of the artifact one, which means it takes me 4 hours instead of 20 minutes to get feedback - and in this case, the build didn't even run successfully. :-\

macOS only has 1 type of opt build, which I assume is related...

Comment 1

[Andrew Halberstadt [:ahal]]

It looks like the selected task is shippable which will build with PGO (and where artifact builds aren't supported). Looking at the list of tasks, non-shippable OSX opt builds don't seem to exist. I believe that was a conscious decision to reduce costs.

Artifact builds in mach try are best effort, i.e it'll use artifact builds where it can but doesn't preclude builds that don't support them. Imo precluding these builds (and associated tests) would cause more frustration than not, especially given the default behaviour of running artifact builds based on your buildconfig.

Maybe one thing we could do here is make the fact that non-artifact builds could still be scheduled more clear. Otherwise I'm not sure what else we can do from mach try. Looks like on OSX debug are the only builds that support artifacts.

Comment 2

[:Gijs (he/him)]

(In reply to Andrew Halberstadt [:ahal] from comment #1)

Looks like on OSX debug are the only builds that support artifacts.

Is there a bug open on making macos opt (shippable) builds support artifacts? I thought they used to... Certainly locally, we support that.

Comment 3

[Andrew Halberstadt [:ahal]]

I'm not sure, here's where the taskgraph determines what's eligible for an artifact build:
https://searchfox.org/mozilla-central/rev/d5b34cc8872177d3ee071e06f787c2a14268595b/taskcluster/taskgraph/transforms/build.py#188

Looks like it uses the presence of a 'nightly' attribute and inspects that ARTIFACT_JOBS object. It's very possible that this should be getting selected and isn't. Better question for the build team.

Mike, should OSX shippable builds be supporting artifact builds? Maybe someone forgot to update the list in artifact_builds.py?

Comment 4

[:Gijs (he/him)]

In fact, it seems that in this configuration the third part of the PGO process, the (B) build job, does in fact do an artifact build, completely ignoring the results of (instr) and (run) jobs, see e.g. this log: https://firefoxci.taskcluster-artifacts.net/G07LNzNIS3-trJmIsXMVXA/0/public/logs/live_backing.log (from https://treeherder.mozilla.org/#/jobs?repo=try&revision=7bb0ade60dabb4a2b2426b2a0b5942bd86cf7c26&selectedJob=291686422 )

Comment 5

[Michael Shal [:mshal]]

(In reply to Andrew Halberstadt [:ahal] from comment #3)

Looks like it uses the presence of a 'nightly' attribute and inspects that ARTIFACT_JOBS object. It's very possible that this should be getting selected and isn't. Better question for the build team.

Mike, should OSX shippable builds be supporting artifact builds? Maybe someone forgot to update the list in artifact_builds.py?

I'm afraid I don't really know. Nick, do you have any insight here?

Comment 6

[Nick Alexander :nalexander [he/him]]

(In reply to Michael Shal [:mshal] from comment #5)

(In reply to Andrew Halberstadt [:ahal] from comment #3)

Looks like it uses the presence of a 'nightly' attribute and inspects that ARTIFACT_JOBS object. It's very possible that this should be getting selected and isn't. Better question for the build team.

Mike, should OSX shippable builds be supporting artifact builds? Maybe someone forgot to update the list in artifact_builds.py?

I'm afraid I don't really know. Nick, do you have any insight here?

Sadly, not really -- chmanchester was the most knowledgeable. I think this is just drift over time, as we moved PGO to more platforms and never caught artifact build configurations up.

I think it's clear that we want to support the flow Gijs expects to work: we want to be able to use an artifact build to iterate on tests in automation. I looked at the index for macosx64-pgo and it looks like that is stale (3 months old already). So my insight, such as it is, is to figure out what the shippable index title is and make the amendment to JOB_CHOICES that :ahal suggests.

Comment 7

[Tom Prince [:tomprince]]

I think there are a number of things going on here:

• shippable builds are not eligible for being turned into artifact builds in taskgraph

  This requires them being added to JOB_CHOICES. The name there corresponds to the index name of the job. Looking at that though, the value there should correspond to the job-name from the index definition of the task; however it looks like the shippable jobs use a different index type and overlapping job-names (e.g here). There'll need to be some work here; maybe allowing multiple types of indexes, or maybe including the default index type in the shippable index type. (I suspect this difference might be part of why nightly attribute builds were excluded from artifact builds)

• There is code in mozbuild to guess the type of build to search for. I think in automation, it would make more sense to pass at least the job-type from taskgraph, rather than guessing based on configure values.[1]

• The shippable builds tend to have three-tier PGO enabled, and the first two tiers will be run even if the build was an artifact build. This can be fixed by switching the artifact code and the PGO

I suspect there may be more things that I am missing as well.

[1] We could also figure out the task to use in taskgraph. This would probably require:

• using some of the code from mozbuild.artifacts to find the right push-head
• use information in taskgraph to construct the appropriate index for the task and construct a dummy task with optimization: {index-search: ...} to represent it
• pass that task via MOZ_ARTIFACT_TASK like is done here