Closed Bug 1751059 Opened 3 years ago Closed 3 years ago

GeckoView shippable artifact builds fail in CI with "Artifact builds do not support localization"

Categories

(Firefox Build System :: General, defect, P3)

Unspecified
All
defect

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 1489128

People

(Reporter: whimboo, Unassigned)

References

Details

Attachments

(1 obsolete file)

I'm using artifact builds locally and used mach try fuzzy to trigger wdspec jobs on try. The shippable artifact build job is failing:

https://treeherder.mozilla.org/jobs?repo=try&revision=c32ad6ccfcaacca55528a78b783b3cfd44df54cb&selectedTaskRun=erP2gppLT2qQpwZ9p5TT_Q.0

https://treeherder.mozilla.org/logviewer?job_id=364687494&repo=try&lineNumber=9184-9192

[task 2022-01-19T20:07:25.054Z] 20:07:25     INFO -  20:07:25     INFO -  Artifact builds do not support localization. If you know what you are doing, you can use:
[task 2022-01-19T20:07:25.054Z] 20:07:25     INFO -  20:07:25     INFO -  ac_add_options --disable-compile-environment
[task 2022-01-19T20:07:25.054Z] 20:07:25     INFO -  20:07:25     INFO -  export BUILD_BACKENDS=FasterMake,RecursiveMake
[task 2022-01-19T20:07:25.054Z] 20:07:25     INFO -  20:07:25     INFO -  in your mozconfig.
[task 2022-01-19T20:07:25.110Z] 20:07:25     INFO -  20:07:25    ERROR - Return code: 1
[task 2022-01-19T20:07:25.110Z] 20:07:25     INFO -  20:07:25    FATAL - 'mach package-multi-locale --locales an ar ast az be bg bn br bs ca cak cs cy da de dsb el en-CA en-GB eo es-AR es-CL es-ES es-MX et eu fa ff fi fr fy-NL ga-IE gd gl gn gu-IN he hi-IN hr hsb hu hy-AM id is it ja ka kab kk kn ko lij lo lt lv ml mr ms my nb-NO ne-NP nl nn-NO oc pa-IN pl pt-BR pt-PT rm ro ru sk sl son sq sr sv-SE ta te th tr trs uk ur uz vi wo xh zam zh-CN zh-TW' did not run successfully. Please check log for errors.
[task 2022-01-19T20:07:25.110Z] 20:07:25     INFO -  20:07:25    FATAL - Running post_fatal callback...
[task 2022-01-19T20:07:25.110Z] 20:07:25     INFO -  20:07:25    FATAL - Exiting -1
[task 2022-01-19T20:07:25.110Z] 20:07:25     INFO -  20:07:25     INFO - [mozharness: 2022-01-19 20:07:25.110455Z] Finished package-multi step (failed)
Product: GeckoView → Firefox Build System
Assignee: nobody → mhentges
Status: NEW → ASSIGNED

Hmm, a couple questions here:

  • Which wdspec job are you running? ./mach try fuzzy --full is showing no matches if I punch in android-wdspec. I looked at your existing task graph and found this task, but if I punch in its name as-is (test-android-em-7.0-x86_64-shippable-qr/opt-geckoview-web-platform-tests-wdspec-headless-e10s-2) I still don't get any matches.
  • Where is the "shippable artifact mode Android build" defined? I'm only seeing this one, which is associated with this (non-artifact-mode) job.
  • Do you know what's up with the localization error? It sounds correct to me, perhaps your wdspec jobs should be based off of a non-artifact mode build if they need localization? I'm grasping at straws here, I'm not sure what's going on.
Flags: needinfo?(hskupin)

(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #2)

Hmm, a couple questions here:

  • Which wdspec job are you running? ./mach try fuzzy --full is showing no matches if I punch in android-wdspec. I looked at your existing task graph and found this task, but if I punch in its name as-is (test-android-em-7.0-x86_64-shippable-qr/opt-geckoview-web-platform-tests-wdspec-headless-e10s-2) I still don't get any matches.

Ah, yes. You would need the following changeset which actually enabled Wdspec on Android:
https://hg.mozilla.org/try/rev/705fef0526d7f68255afae96145f1d986a816cca

Maybe the web-platform TaskCluster config for wdspec jobs should not trigger shippable builds then and I would have to update my patch?

  • Do you know what's up with the localization error? It sounds correct to me, perhaps your wdspec jobs should be based off of a non-artifact mode build if they need localization? I'm grasping at straws here, I'm not sure what's going on.

We don't need localization just the default en-US. But as written above I assume a problem with the taskcluster config?

Flags: needinfo?(hskupin)

Hey :whimboo, sorry the delay, I'm juggling some other build-related issues this week :)
I'm still struggling to reproduce this, though:

I'll try again with your exact try commit (here's mine) to see if that's the difference, and report back by the end of the day.

Does it matter if you have locally set artifact builds? I mean I have and run mach try fuzzy that way. Or was there a different issue that maybe got fixed in the meantime? I could try myself again if you cannot reproduce.

Hmm, maybe it does, because it looks like my copy of your commit has use-artifact-builds: true, so it depends on that's set.
(EDIT: Ah, there it is, we do set use-artifact-builds based on your local config).
Fortunately, looks like I have a repro now :)
(EDIT 2: I'll bring this up in my Monday meeting with :glandium - I'm guessing the right move here is for artifact android builds shouldn't be localized, but I'd like to confirm this assumption).

After discussion with :glandium, it sounds like these artifact builds should support localization.
However, in the meantime, I'm adding supports-artifact-builds: false to each shippable Android CI job.

See Also: → 1753053

The shippable Android builds do localization, and we currently don't
support Android localization with artifact builds.

Attachment #9261716 - Attachment description: Bug 1751059: Mark Android shippable CI builds as non-artifact-buildable → WIP: Bug 1751059: Mark Android shippable CI builds as non-artifact-buildable

Hey :ahal, no rush, but when you're back: I have a patch here that adds support-artifact-builds: false to android-x86_64-shippable/opt (among others), yet the job still ran with USE_ARTIFACT=1.
The try run is here.

I assumed that filter_unsupported_artifact_builds() was what I needed to get the behaviour I wanted (still schedule the job, but don't set USE_ARTIFACT=1), but that didn't appear to do it. (I'm guessing that filter_unsupported_artifact_builds() simply prunes jobs, but since a different job I was scheduling still depended on my "android build job", it got scheduled anyways as-is?)

What do you think is the right step here?

Flags: needinfo?(ahal)

If you look, the _try_config method doesn't apply that filter, only the try syntax one and the mach try auto ones:
https://searchfox.org/mozilla-central/source/taskcluster/gecko_taskgraph/target_tasks.py#286

I think the reasoning behind this was that people would be surprised if they explicitly selected a task, and then it just magically didn't show up. Better to have it show up and fail so they realize the mistake.

I think the ideal solution would be for the mach try invocation to check this locally and either error out before pushing, or print a warning like "The following tasks will be skipped:". But no ones gotten around to implementing this.. I think there might even be a bug on file.

Flags: needinfo?(ahal)

Hmm, thanks, that makes sense. Sounds like this is less of of a "low hanging fruit" than I anticipated.

I think the ideal solution would be for the mach try invocation to check this locally and either error out before pushing, or print a warning like "The following tasks will be skipped:". But no ones gotten around to implementing this.. I think there might even be a bug on file.

That's fair, but it's still a bummer - ideally, all jobs that can be artifact-mode'd would use artifacts, and those that need to opt out (such as Android Shippable) would opt out.

I don't think I can prioritize the significant chunk of work that would enable that, and I don't think that the alternative solutions' benefits are sufficiently worth the work to implement them today. I'm going to plonk this into the backlog.

Workaround

Edit your mozconfig file and disable the ac_add_options --enable-artifact-builds line before pushing to try.

Assignee: mhentges → nobody
Status: ASSIGNED → NEW
Priority: -- → P3

Note there's also --artifact/--no-artifact flags on mach try that override whatever is in your mozconfig. This bug could likely be marked as a dupe of one of the ones I added to the See Also.

Sure, I pick 1489128 :)

Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → DUPLICATE
Attachment #9261716 - Attachment is obsolete: true
See Also: → 1890553
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: