1751059 - GeckoView shippable artifact builds fail in CI with "Artifact builds do not support localization"

Status: RESOLVED DUPLICATE of bug 1489128

(Firefox Build System :: General, defect, P3)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 10/03 - 10/13)]

I'm using artifact builds locally and used mach try fuzzy to trigger wdspec jobs on try. The shippable artifact build job is failing:

https://treeherder.mozilla.org/jobs?repo=try&revision=c32ad6ccfcaacca55528a78b783b3cfd44df54cb&selectedTaskRun=erP2gppLT2qQpwZ9p5TT_Q.0

https://treeherder.mozilla.org/logviewer?job_id=364687494&repo=try&lineNumber=9184-9192

[task 2022-01-19T20:07:25.054Z] 20:07:25     INFO -  20:07:25     INFO -  Artifact builds do not support localization. If you know what you are doing, you can use:
[task 2022-01-19T20:07:25.054Z] 20:07:25     INFO -  20:07:25     INFO -  ac_add_options --disable-compile-environment
[task 2022-01-19T20:07:25.054Z] 20:07:25     INFO -  20:07:25     INFO -  export BUILD_BACKENDS=FasterMake,RecursiveMake
[task 2022-01-19T20:07:25.054Z] 20:07:25     INFO -  20:07:25     INFO -  in your mozconfig.
[task 2022-01-19T20:07:25.110Z] 20:07:25     INFO -  20:07:25    ERROR - Return code: 1
[task 2022-01-19T20:07:25.110Z] 20:07:25     INFO -  20:07:25    FATAL - 'mach package-multi-locale --locales an ar ast az be bg bn br bs ca cak cs cy da de dsb el en-CA en-GB eo es-AR es-CL es-ES es-MX et eu fa ff fi fr fy-NL ga-IE gd gl gn gu-IN he hi-IN hr hsb hu hy-AM id is it ja ka kab kk kn ko lij lo lt lv ml mr ms my nb-NO ne-NP nl nn-NO oc pa-IN pl pt-BR pt-PT rm ro ru sk sl son sq sr sv-SE ta te th tr trs uk ur uz vi wo xh zam zh-CN zh-TW' did not run successfully. Please check log for errors.
[task 2022-01-19T20:07:25.110Z] 20:07:25     INFO -  20:07:25    FATAL - Running post_fatal callback...
[task 2022-01-19T20:07:25.110Z] 20:07:25     INFO -  20:07:25    FATAL - Exiting -1
[task 2022-01-19T20:07:25.110Z] 20:07:25     INFO -  20:07:25     INFO - [mozharness: 2022-01-19 20:07:25.110455Z] Finished package-multi step (failed)

Comment 1

[Intermittent Failures Robot]

1 failures in 3444 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform and build breakdown:

• android-5-0-x86_64-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1751059&startday=2022-01-17&endday=2022-01-23&tree=all

Comment 2

[Mitchell Hentges [:mhentges] 🦀]

Hmm, a couple questions here:

• Which wdspec job are you running? ./mach try fuzzy --full is showing no matches if I punch in android-wdspec. I looked at your existing task graph and found this task, but if I punch in its name as-is (test-android-em-7.0-x86_64-shippable-qr/opt-geckoview-web-platform-tests-wdspec-headless-e10s-2) I still don't get any matches.
• Where is the "shippable artifact mode Android build" defined? I'm only seeing this one, which is associated with this (non-artifact-mode) job.
• Do you know what's up with the localization error? It sounds correct to me, perhaps your wdspec jobs should be based off of a non-artifact mode build if they need localization? I'm grasping at straws here, I'm not sure what's going on.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 10/03 - 10/13)]

(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #2)

Hmm, a couple questions here:

• Which wdspec job are you running? ./mach try fuzzy --full is showing no matches if I punch in android-wdspec. I looked at your existing task graph and found this task, but if I punch in its name as-is (test-android-em-7.0-x86_64-shippable-qr/opt-geckoview-web-platform-tests-wdspec-headless-e10s-2) I still don't get any matches.

Ah, yes. You would need the following changeset which actually enabled Wdspec on Android:
https://hg.mozilla.org/try/rev/705fef0526d7f68255afae96145f1d986a816cca

• Where is the "shippable artifact mode Android build" defined? I'm only seeing this one, which is associated with this (non-artifact-mode) job.

Maybe the web-platform TaskCluster config for wdspec jobs should not trigger shippable builds then and I would have to update my patch?

• Do you know what's up with the localization error? It sounds correct to me, perhaps your wdspec jobs should be based off of a non-artifact mode build if they need localization? I'm grasping at straws here, I'm not sure what's going on.

We don't need localization just the default en-US. But as written above I assume a problem with the taskcluster config?

Comment 4

[Mitchell Hentges [:mhentges] 🦀]

Hey :whimboo, sorry the delay, I'm juggling some other build-related issues this week :)
I'm still struggling to reproduce this, though:

• Here's my try push on Treeherder.
• It's based on 705fef0526d7f68255afae96145f1d986a816cca, as you've recommended
• Here's my build-android-x86_64-shippable/opt task, which built successfully. Note that it doesn't have 'USE_ARTIFACT': '1' in the logs.
  • Meanwhile, your build-android-x86_64-shippable/opt task, which failed, does have USE_ARTIFACT': '1' in its logs

I'll try again with your exact try commit (here's mine) to see if that's the difference, and report back by the end of the day.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 10/03 - 10/13)]

Does it matter if you have locally set artifact builds? I mean I have and run mach try fuzzy that way. Or was there a different issue that maybe got fixed in the meantime? I could try myself again if you cannot reproduce.

Comment 6

[Mitchell Hentges [:mhentges] 🦀]

Hmm, maybe it does, because it looks like my copy of your commit has use-artifact-builds: true, so it depends on that's set.
(EDIT: Ah, there it is, we do set use-artifact-builds based on your local config).
Fortunately, looks like I have a repro now :)
(EDIT 2: I'll bring this up in my Monday meeting with :glandium - I'm guessing the right move here is for artifact android builds shouldn't be localized, but I'd like to confirm this assumption).

Comment 7

[Mitchell Hentges [:mhentges] 🦀]

After discussion with :glandium, it sounds like these artifact builds should support localization.
However, in the meantime, I'm adding supports-artifact-builds: false to each shippable Android CI job.

Comment 8

[Mitchell Hentges [:mhentges] 🦀]

Attachment: WIP: Bug 1751059: Mark Android shippable CI builds as non-artifact-buildable

The shippable Android builds do localization, and we currently don't
support Android localization with artifact builds.

Comment 9

[Mitchell Hentges [:mhentges] 🦀]

Hey :ahal, no rush, but when you're back: I have a patch here that adds support-artifact-builds: false to android-x86_64-shippable/opt (among others), yet the job still ran with USE_ARTIFACT=1.
The try run is here.

I assumed that filter_unsupported_artifact_builds() was what I needed to get the behaviour I wanted (still schedule the job, but don't set USE_ARTIFACT=1), but that didn't appear to do it. (I'm guessing that filter_unsupported_artifact_builds() simply prunes jobs, but since a different job I was scheduling still depended on my "android build job", it got scheduled anyways as-is?)

What do you think is the right step here?

Comment 10

[Andrew Halberstadt [:ahal]]

If you look, the _try_config method doesn't apply that filter, only the try syntax one and the mach try auto ones:
https://searchfox.org/mozilla-central/source/taskcluster/gecko_taskgraph/target_tasks.py#286

I think the reasoning behind this was that people would be surprised if they explicitly selected a task, and then it just magically didn't show up. Better to have it show up and fail so they realize the mistake.

I think the ideal solution would be for the mach try invocation to check this locally and either error out before pushing, or print a warning like "The following tasks will be skipped:". But no ones gotten around to implementing this.. I think there might even be a bug on file.

Comment 11

[Mitchell Hentges [:mhentges] 🦀]

Hmm, thanks, that makes sense. Sounds like this is less of of a "low hanging fruit" than I anticipated.

I think the ideal solution would be for the mach try invocation to check this locally and either error out before pushing, or print a warning like "The following tasks will be skipped:". But no ones gotten around to implementing this.. I think there might even be a bug on file.

That's fair, but it's still a bummer - ideally, all jobs that can be artifact-mode'd would use artifacts, and those that need to opt out (such as Android Shippable) would opt out.

I don't think I can prioritize the significant chunk of work that would enable that, and I don't think that the alternative solutions' benefits are sufficiently worth the work to implement them today. I'm going to plonk this into the backlog.

Workaround

Edit your mozconfig file and disable the ac_add_options --enable-artifact-builds line before pushing to try.

Comment 12

[Andrew Halberstadt [:ahal]]

Note there's also --artifact/--no-artifact flags on mach try that override whatever is in your mozconfig. This bug could likely be marked as a dupe of one of the ones I added to the See Also.

Comment 13

[Mitchell Hentges [:mhentges] 🦀]

Sure, I pick 1489128 :)