Closed Bug 1381495 Opened 8 years ago Closed 8 years ago

buildbot mac l10n repacks fail to run mar

Categories

(Release Engineering :: Release Automation, enhancement, P2)

Product:
Release Engineering
Component:
Release Automation
Type:
enhancement

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: rail, Unassigned)

References

Details

14:45:06 INFO - /builds/slave/rel-jamun_fx_m64_l10n_rpk-0000/build/jamun/tools/update-packaging/make_full_update.sh: line 116: ../../dist/host/bin/mar: cannot execute binary file https://archive.mozilla.org/pub/firefox/tinderbox-builds/jamun-l10n/release-jamun_firefox_macosx64_l10n_repack-bm84-build1-build242.txt.gz Could be related to 2 things: 1) path issues 2) the mar binary is built on linux, so we cannot execute it on mac slaves Possible solutions: 1) cross compile mar/mbsdiff for darwin. I think this is the safest option. 2) use mar/mbsdiff binaries from tooltool. As a bandaid this can work, but it may bite use when we switch to LZMA compression. 3) build mar/mbsdiff as a part of l10n repack 4) modify release promotion to use l10n repacks from the nightly graph. This will require a lot of work and testing, but it'd be closer to the desired final state where we use in-tree scheduling for release promotion.
Mike, do you need that we can cross-build mar/mbsdiff on linux so it's runnable on macs?
Flags: needinfo?(mh+mozilla)
https://archive.mozilla.org/pub/firefox/tinderbox-builds/jamun-l10n/release-jamun_firefox_macosx64_l10n_repack-bm84-build1-build242.txt.gz scripts/scripts/desktop_l10n.py --branch-config single_locale/dev-mozilla-beta.py --platform-config single_locale/macosx64.py --environment-config single_locale/production.py --balrog-config balrog/production.py in the "buildbot" props: 14:16:06 INFO - "mar_tools_url": "https://queue.taskcluster.net/v1/task/JEbZxejaTRaYV7IBvvGG4A/artifacts/public/build/host/bin", 14:16:06 INFO - Overriding mar_tools_url with https://queue.taskcluster.net/v1/task/JEbZxejaTRaYV7IBvvGG4A/artifacts/public/build/host/bin
We hardcoded mar_tools_url to prefer the buildbot_configs value here: https://hg.mozilla.org/projects/jamun/file/tip/testing/mozharness/scripts/desktop_l10n.py#l278 We could make this conditional on something in self.config, and then override in jamun/m-b config files. This would ride the trains. We would have to reapply this change on m-b merge day if it takes us >6 weeks to get us on TC release l10n.
Correct, the initial idea was to download the mar/bsdiff binaries from the corresponding en-US builds in case something changes in those binaries. release-runner resolves the tasks and passes that property to releasetasks in https://github.com/mozilla-releng/releasetasks/blob/master/releasetasks/templates/desktop/l10n.yml.tmpl#L33
10:44 <aki> where does releasetasks l10n_config come from? 10:44 jmaher → jmaher|afk 10:45 <rail> aki: https://hg.mozilla.org/build/tools/file/tip/lib/python/kickoff/__init__.py#l153 10:46 <rail> aki: my last patch for jamun: https://gist.github.com/rail/e9829640d592587ecf139d90e29e24b8 10:48 <rail> other patches are in https://bugzilla.mozilla.org/show_bug.cgi?id=1379261 10:48 <aki> we could do a platform check here: https://gist.github.com/rail/e9829640d592587ecf139d90e29e24b8#file-gistfile-diff-L45 =\ if we're worried about lzma riding the trains, in-tree patches make more sense 10:51 <rail> aki: not sure what you want to do after that... 10:51 mtabara → mtabara|bbl 10:51 <aki> rail: after your patches, or after my patches? 10:52 <rail> aki: in general :) download the tools from somewhere else or something else? 10:53 <aki> rail: currently thinking in-tree mh changes > releaserunner changes, since they can ride the trains. 1) in-tree mh changes to download a different set of mar tools for mac, 2) prepare for merge day 3) hopefully g.landium can help us x-compile mar tools so we can throw away my patches sooner :) 4) otherwise, prioritize moving to tc l10n releasetasks sooner to 10:53 <aki> ditch this hack 10:55 <rail> aki: download from tooltool or some stable URL? 10:55 <aki> rail: i was thinking a specific bb osx nightly url off archive.m.o to make it easier, but could do tooltool 10:56 <rail> aki: nightly url should work, not sure what to do when we land lzma 10:57 <aki> rail: we'd change urls for that train. when is lzma? is it going to ride the trains? 10:57 <rail> 56.0 or 56.0.1, not sure... 10:58 <aki> so the main trick is finding a good osx mar without bb nightlies 10:58 <rail> they want to have a special dot release for this, but I'm not sure when it's landing 10:58 <rail> yeah 10:58 <rail> we can probably build ourselves :) 10:58 <aki> yeah 10:58 <rail> "on my laptop" :) 10:59 <rail> err, "your" 10:59 <aki> then push to a known url and change the in-tree url 10:59 <aki> haha 10:59 <aki> so ugly New messages 10:59 <rail> yeah 10:59 <rail> very ugly 10:59 <rail> cross-compiling would be much better 10:59 <aki> i'm just taking this approach because it's a known way to unblock us 10:59 <rail> yup
11:04 <catlee> that's kind of like the tooltool appraoch 11:04 <catlee> but even more hacky :) 11:04 <aki> we can do tooltool 11:05 <Callek> at least tooltool would guarantee us a known sha 11:05 <aki> rather than just making sure mar_tools_url is set to a known url, we'll have to pass in tooltool info and run tooltool instead of download_mar_tools 11:05 <aki> larger diff, but sha verification 11:08 <aki> we may need 2x tooltool runs as well 11:08 <aki> i think i can do that
We switched trunk to x-compiled on June 21. The mar and mbsdiff files in https://archive.mozilla.org/pub/firefox/nightly/2017/06/2017-06-22-00-03-07-mozilla-central/mar-tools/macosx64/ are linux. The mar and mbsdiff files in https://archive.mozilla.org/pub/firefox/nightly/2017/06/2017-06-21-03-02-08-mozilla-central/ are osx. We don't have linux in here, so we aren't overwriting! Let's use the mar and mbsdiff files from 2017-06-21-03-02-08-mozilla-central/ until lzma complicates everything.
Thanks for the investigation. I added a hack to use https://archive.mozilla.org/pub/firefox/nightly/2017/06/2017-06-21-03-02-08-mozilla-central/ for mac. So far it looks good. \o/
If we don't use mar during the build, an easy way out can be to switch it from HostProgram to Program, which will cross compile it.
Flags: needinfo?(mh+mozilla)
(that would change its path from dist/host/bin to dist/bin, though)
I think we need to generate both HostProgram and Program here. :/
(In reply to Rail Aliiev [:rail] ⌚️ET from comment #12) > I think we need to generate both HostProgram and Program here. :/ well, that's possible too.
Severity: critical → normal
https://hg.mozilla.org/build/tools/rev/077a4adc87dc6c02665e21c68ac9974eb866990e#l2.26 is the hack I've just landed. We need to figure out our next steps here.
I'm not sure if I will be around to address this for reals. Back to the pool.
Assignee: rail → nobody
Severity: normal → major
Priority: P1 → P2
Current plan: * get native osx lzma martools from the Oak team * upload them somewhere we can grab them from (archive.m.o?) * update releasetools hack to point to the lzma martools for the appropriate trains * prioritize in-tree tc release l10n so we stop needing native osx martools
13:06 <rstrong> catlee: with that in mind why is a new mar needed for l10n? 13:06 <•catlee> it's not now 13:06 <•catlee> done! 13:06 <rstrong> catlee: I'll skip trying to build the mac one then :) 13:07 <aki> oh, so we can create bz2 mars and we're good? 13:07 <rail> ship it 13:07 <•catlee> aki: no, but the compression happens in the wrapper scripts 13:07 <•catlee> using cmdline xz 13:07 <rstrong> aki: that is controlled by the shell scripts 13:07 <aki> ok 13:07 <•catlee> that wasn't the case last week :) 13:07 <aki> then we don't have to worry about this anymore. yay! We don't need new native mar tools. This hack can live until we move to tc l10n releases, or something else changes.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.