buildbot mac l10n repacks fail to run mar

RESOLVED FIXED

Status

P2
major
RESOLVED FIXED
a year ago
a year ago

People

(Reporter: rail, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

a year ago
14:45:06     INFO -  /builds/slave/rel-jamun_fx_m64_l10n_rpk-0000/build/jamun/tools/update-packaging/make_full_update.sh: line 116: ../../dist/host/bin/mar: cannot execute binary file

https://archive.mozilla.org/pub/firefox/tinderbox-builds/jamun-l10n/release-jamun_firefox_macosx64_l10n_repack-bm84-build1-build242.txt.gz

Could be related to 2 things:
1) path issues
2) the mar binary is built on linux, so we cannot execute it on mac slaves

Possible solutions:
1) cross compile mar/mbsdiff for darwin. I think this is the safest option.
2) use mar/mbsdiff binaries from tooltool. As a bandaid this can work, but it may bite use when we switch to LZMA compression.
3) build mar/mbsdiff as a part of l10n repack
4) modify release promotion to use l10n repacks from the nightly graph. This will require a lot of work and testing, but it'd be closer to the desired final state where we use in-tree scheduling for release promotion.
(Reporter)

Comment 1

a year ago
Mike, do you need that we can cross-build mar/mbsdiff on linux so it's runnable on macs?
Flags: needinfo?(mh+mozilla)
https://archive.mozilla.org/pub/firefox/tinderbox-builds/jamun-l10n/release-jamun_firefox_macosx64_l10n_repack-bm84-build1-build242.txt.gz

scripts/scripts/desktop_l10n.py --branch-config single_locale/dev-mozilla-beta.py --platform-config single_locale/macosx64.py --environment-config single_locale/production.py --balrog-config balrog/production.py

in the "buildbot" props:
14:16:06     INFO -     "mar_tools_url": "https://queue.taskcluster.net/v1/task/JEbZxejaTRaYV7IBvvGG4A/artifacts/public/build/host/bin", 

14:16:06     INFO - Overriding mar_tools_url with https://queue.taskcluster.net/v1/task/JEbZxejaTRaYV7IBvvGG4A/artifacts/public/build/host/bin
We hardcoded mar_tools_url to prefer the buildbot_configs value here:
https://hg.mozilla.org/projects/jamun/file/tip/testing/mozharness/scripts/desktop_l10n.py#l278

We could make this conditional on something in self.config, and then override in jamun/m-b config files. This would ride the trains. We would have to reapply this change on m-b merge day if it takes us >6 weeks to get us on TC release l10n.
(Reporter)

Comment 5

a year ago
Correct, the initial idea was to download the mar/bsdiff binaries from the corresponding en-US builds in case something changes in those binaries. release-runner resolves the tasks and passes that property to releasetasks in https://github.com/mozilla-releng/releasetasks/blob/master/releasetasks/templates/desktop/l10n.yml.tmpl#L33
10:44 <aki> where does releasetasks l10n_config come from?
10:44 jmaher → jmaher|afk
10:45 <rail> aki: https://hg.mozilla.org/build/tools/file/tip/lib/python/kickoff/__init__.py#l153
10:46 <rail> aki: my last patch for jamun: https://gist.github.com/rail/e9829640d592587ecf139d90e29e24b8
10:48 <rail> other patches are in https://bugzilla.mozilla.org/show_bug.cgi?id=1379261
10:48 <aki> we could do a platform check here: https://gist.github.com/rail/e9829640d592587ecf139d90e29e24b8#file-gistfile-diff-L45 =\  if we're worried about lzma riding the trains, in-tree patches make more sense

10:51 <rail> aki: not sure what you want to do after that...
10:51 mtabara → mtabara|bbl
10:51 <aki> rail: after your patches, or after my patches?
10:52 <rail> aki: in general :) download the tools from somewhere else or something else?

10:53 <aki> rail: currently thinking in-tree mh changes > releaserunner changes, since they can ride the trains. 1) in-tree mh changes to download a different set of mar tools for mac, 2) prepare for merge day 3) hopefully g.landium can help us x-compile mar tools so we can throw away my patches sooner :) 4) otherwise, prioritize moving to tc l10n releasetasks sooner to
10:53 <aki> ditch this hack

10:55 <rail> aki: download from tooltool or some stable URL?

10:55 <aki> rail: i was thinking a specific bb osx nightly url off archive.m.o to make it easier, but could do tooltool

10:56 <rail> aki: nightly url should work, not sure what to do when we land lzma
10:57 <aki> rail: we'd change urls for that train. when is lzma? is it going to ride the trains?
10:57 <rail> 56.0 or 56.0.1, not sure...
10:58 <aki> so the main trick is finding a good osx mar without bb nightlies
10:58 <rail> they want to have a special dot release for this, but I'm not sure when it's landing
10:58 <rail> yeah
10:58 <rail> we can probably build ourselves :)
10:58 <aki> yeah
10:58 <rail> "on my laptop" :)
10:59 <rail> err, "your"
10:59 <aki> then push to a known url and change the in-tree url
10:59 <aki> haha
10:59 <aki> so ugly
New messages
10:59 <rail> yeah
10:59 <rail> very ugly
10:59 <rail> cross-compiling would be much better
10:59 <aki> i'm just taking this approach because it's a known way to unblock us
10:59 <rail> yup
11:04 <catlee> that's kind of like the tooltool appraoch
11:04 <catlee> but even more hacky :)
11:04 <aki> we can do tooltool
11:05 <Callek> at least tooltool would guarantee us a known sha
11:05 <aki> rather than just making sure mar_tools_url is set to a known url, we'll have to pass in tooltool info and run tooltool instead of download_mar_tools
11:05 <aki> larger diff, but sha verification
11:08 <aki> we may need 2x tooltool runs as well
11:08 <aki> i think i can do that
We switched trunk to x-compiled on June 21.

The mar and mbsdiff files in
https://archive.mozilla.org/pub/firefox/nightly/2017/06/2017-06-22-00-03-07-mozilla-central/mar-tools/macosx64/
are linux.

The mar and mbsdiff files in
https://archive.mozilla.org/pub/firefox/nightly/2017/06/2017-06-21-03-02-08-mozilla-central/
are osx. We don't have linux in here, so we aren't overwriting!

Let's use the mar and mbsdiff files from 2017-06-21-03-02-08-mozilla-central/ until lzma complicates everything.
(Reporter)

Comment 9

a year ago
Thanks for the investigation. I added a hack to use https://archive.mozilla.org/pub/firefox/nightly/2017/06/2017-06-21-03-02-08-mozilla-central/ for mac. So far it looks good. \o/
If we don't use mar during the build, an easy way out can be to switch it from HostProgram to Program, which will cross compile it.
Flags: needinfo?(mh+mozilla)
(that would change its path from dist/host/bin to dist/bin, though)
(Reporter)

Comment 12

a year ago
I think we need to generate both HostProgram and Program here. :/
(In reply to Rail Aliiev [:rail] ⌚️ET from comment #12)
> I think we need to generate both HostProgram and Program here. :/

well, that's possible too.
(Reporter)

Updated

a year ago
Severity: critical → normal
(Reporter)

Comment 14

a year ago
https://hg.mozilla.org/build/tools/rev/077a4adc87dc6c02665e21c68ac9974eb866990e#l2.26 is the hack I've just landed. We need to figure out our next steps here.
(Reporter)

Comment 15

a year ago
I'm not sure if I will be around to address this for reals. Back to the pool.
Assignee: rail → nobody
Severity: normal → major
Priority: P1 → P2
Current plan:

* get native osx lzma martools from the Oak team
* upload them somewhere we can grab them from (archive.m.o?)
* update releasetools hack to point to the lzma martools for the appropriate trains
* prioritize in-tree tc release l10n so we stop needing native osx martools
13:06 <rstrong> catlee: with that in mind why is a new mar needed for l10n?
13:06 <•catlee> it's not now
13:06 <•catlee> done!
13:06 <rstrong> catlee: I'll skip trying to build the mac one then :)
13:07 <aki> oh, so we can create bz2 mars and we're good?
13:07 <rail> ship it
13:07 <•catlee> aki: no, but the compression happens in the wrapper scripts
13:07 <•catlee> using cmdline xz
13:07 <rstrong> aki: that is controlled by the shell scripts
13:07 <aki> ok
13:07 <•catlee> that wasn't the case last week :)
13:07 <aki> then we don't have to worry about this anymore. yay!

We don't need new native mar tools. This hack can live until we move to tc l10n releases, or something else changes.
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.