GeckoView shippable artifact builds fail in CI with "Artifact builds do not support localization"
Categories
(Firefox Build System :: General, defect, P3)
Tracking
(Not tracked)
People
(Reporter: whimboo, Unassigned)
References
Details
Attachments
(1 obsolete file)
I'm using artifact builds locally and used mach try fuzzy
to trigger wdspec jobs on try. The shippable artifact build job is failing:
https://treeherder.mozilla.org/logviewer?job_id=364687494&repo=try&lineNumber=9184-9192
[task 2022-01-19T20:07:25.054Z] 20:07:25 INFO - 20:07:25 INFO - Artifact builds do not support localization. If you know what you are doing, you can use:
[task 2022-01-19T20:07:25.054Z] 20:07:25 INFO - 20:07:25 INFO - ac_add_options --disable-compile-environment
[task 2022-01-19T20:07:25.054Z] 20:07:25 INFO - 20:07:25 INFO - export BUILD_BACKENDS=FasterMake,RecursiveMake
[task 2022-01-19T20:07:25.054Z] 20:07:25 INFO - 20:07:25 INFO - in your mozconfig.
[task 2022-01-19T20:07:25.110Z] 20:07:25 INFO - 20:07:25 ERROR - Return code: 1
[task 2022-01-19T20:07:25.110Z] 20:07:25 INFO - 20:07:25 FATAL - 'mach package-multi-locale --locales an ar ast az be bg bn br bs ca cak cs cy da de dsb el en-CA en-GB eo es-AR es-CL es-ES es-MX et eu fa ff fi fr fy-NL ga-IE gd gl gn gu-IN he hi-IN hr hsb hu hy-AM id is it ja ka kab kk kn ko lij lo lt lv ml mr ms my nb-NO ne-NP nl nn-NO oc pa-IN pl pt-BR pt-PT rm ro ru sk sl son sq sr sv-SE ta te th tr trs uk ur uz vi wo xh zam zh-CN zh-TW' did not run successfully. Please check log for errors.
[task 2022-01-19T20:07:25.110Z] 20:07:25 INFO - 20:07:25 FATAL - Running post_fatal callback...
[task 2022-01-19T20:07:25.110Z] 20:07:25 INFO - 20:07:25 FATAL - Exiting -1
[task 2022-01-19T20:07:25.110Z] 20:07:25 INFO - 20:07:25 INFO - [mozharness: 2022-01-19 20:07:25.110455Z] Finished package-multi step (failed)
Updated•3 years ago
|
Comment hidden (Intermittent Failures Robot) |
Updated•3 years ago
|
Comment 2•3 years ago
|
||
Hmm, a couple questions here:
- Which
wdspec
job are you running?./mach try fuzzy --full
is showing no matches if I punch inandroid-wdspec
. I looked at your existing task graph and found this task, but if I punch in its name as-is (test-android-em-7.0-x86_64-shippable-qr/opt-geckoview-web-platform-tests-wdspec-headless-e10s-2
) I still don't get any matches. - Where is the "shippable artifact mode Android build" defined? I'm only seeing this one, which is associated with this (non-artifact-mode) job.
- Do you know what's up with the localization error? It sounds correct to me, perhaps your
wdspec
jobs should be based off of a non-artifact mode build if they need localization? I'm grasping at straws here, I'm not sure what's going on.
Reporter | ||
Comment 3•3 years ago
|
||
(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #2)
Hmm, a couple questions here:
- Which
wdspec
job are you running?./mach try fuzzy --full
is showing no matches if I punch inandroid-wdspec
. I looked at your existing task graph and found this task, but if I punch in its name as-is (test-android-em-7.0-x86_64-shippable-qr/opt-geckoview-web-platform-tests-wdspec-headless-e10s-2
) I still don't get any matches.
Ah, yes. You would need the following changeset which actually enabled Wdspec on Android:
https://hg.mozilla.org/try/rev/705fef0526d7f68255afae96145f1d986a816cca
- Where is the "shippable artifact mode Android build" defined? I'm only seeing this one, which is associated with this (non-artifact-mode) job.
Maybe the web-platform TaskCluster config for wdspec jobs should not trigger shippable builds then and I would have to update my patch?
- Do you know what's up with the localization error? It sounds correct to me, perhaps your
wdspec
jobs should be based off of a non-artifact mode build if they need localization? I'm grasping at straws here, I'm not sure what's going on.
We don't need localization just the default en-US. But as written above I assume a problem with the taskcluster config?
Reporter | ||
Updated•3 years ago
|
Comment 4•3 years ago
|
||
Hey :whimboo, sorry the delay, I'm juggling some other build-related issues this week :)
I'm still struggling to reproduce this, though:
- Here's my
try
push on Treeherder. - It's based on
705fef0526d7f68255afae96145f1d986a816cca
, as you've recommended - Here's my
build-android-x86_64-shippable/opt
task, which built successfully. Note that it doesn't have'USE_ARTIFACT': '1'
in the logs.- Meanwhile, your
build-android-x86_64-shippable/opt
task, which failed, does haveUSE_ARTIFACT': '1'
in its logs
- Meanwhile, your
I'll try again with your exact try
commit (here's mine) to see if that's the difference, and report back by the end of the day.
Reporter | ||
Comment 5•3 years ago
|
||
Does it matter if you have locally set artifact builds? I mean I have and run mach try fuzzy
that way. Or was there a different issue that maybe got fixed in the meantime? I could try myself again if you cannot reproduce.
Comment 6•3 years ago
•
|
||
Hmm, maybe it does, because it looks like my copy of your commit has use-artifact-builds: true
, so it depends on that's set.
(EDIT: Ah, there it is, we do set use-artifact-builds
based on your local config).
Fortunately, looks like I have a repro now :)
(EDIT 2: I'll bring this up in my Monday meeting with :glandium - I'm guessing the right move here is for artifact android builds shouldn't be localized, but I'd like to confirm this assumption).
Comment 7•3 years ago
|
||
After discussion with :glandium, it sounds like these artifact builds should support localization.
However, in the meantime, I'm adding supports-artifact-builds: false
to each shippable Android CI job.
Comment 8•3 years ago
|
||
The shippable Android builds do localization, and we currently don't
support Android localization with artifact builds.
Updated•3 years ago
|
Comment 9•3 years ago
|
||
Hey :ahal, no rush, but when you're back: I have a patch here that adds support-artifact-builds: false
to android-x86_64-shippable/opt
(among others), yet the job still ran with USE_ARTIFACT=1
.
The try run is here.
I assumed that filter_unsupported_artifact_builds()
was what I needed to get the behaviour I wanted (still schedule the job, but don't set USE_ARTIFACT=1
), but that didn't appear to do it. (I'm guessing that filter_unsupported_artifact_builds()
simply prunes jobs, but since a different job I was scheduling still depended on my "android build job", it got scheduled anyways as-is?)
What do you think is the right step here?
Updated•3 years ago
|
Comment 10•3 years ago
|
||
If you look, the _try_config
method doesn't apply that filter, only the try syntax one and the mach try auto
ones:
https://searchfox.org/mozilla-central/source/taskcluster/gecko_taskgraph/target_tasks.py#286
I think the reasoning behind this was that people would be surprised if they explicitly selected a task, and then it just magically didn't show up. Better to have it show up and fail so they realize the mistake.
I think the ideal solution would be for the mach try
invocation to check this locally and either error out before pushing, or print a warning like "The following tasks will be skipped:". But no ones gotten around to implementing this.. I think there might even be a bug on file.
Updated•3 years ago
|
Comment 11•3 years ago
•
|
||
Hmm, thanks, that makes sense. Sounds like this is less of of a "low hanging fruit" than I anticipated.
I think the ideal solution would be for the mach try invocation to check this locally and either error out before pushing, or print a warning like "The following tasks will be skipped:". But no ones gotten around to implementing this.. I think there might even be a bug on file.
That's fair, but it's still a bummer - ideally, all jobs that can be artifact-mode'd would use artifacts, and those that need to opt out (such as Android Shippable) would opt out.
I don't think I can prioritize the significant chunk of work that would enable that, and I don't think that the alternative solutions' benefits are sufficiently worth the work to implement them today. I'm going to plonk this into the backlog.
Workaround
Edit your mozconfig
file and disable the ac_add_options --enable-artifact-builds
line before pushing to try
.
Comment 12•3 years ago
|
||
Note there's also --artifact/--no-artifact
flags on mach try
that override whatever is in your mozconfig. This bug could likely be marked as a dupe of one of the ones I added to the See Also.
Comment 13•3 years ago
|
||
Sure, I pick 1489128 :)
Updated•3 years ago
|
Description
•