Support cross compiling to Linux using docker from any platform

NEW
Unassigned

Status

4 years ago
7 months ago

People

(Reporter: BenWa, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

4 years ago
From a discussion I had with gps that I wanted to log:

It would be nice if our build system supported something like:
ac_add_options --enable-docker
ac_add_options --target=<something linux>

Config would see enable-docker, make sure that you have it installed, create your objdir, pull a blessed docker image for the target from a docker register (hub.docker.io) and mount your srcdir and (empty) objdir and build.

Supporting mach transparently would be a bit more work.
(Reporter)

Updated

2 years ago
Depends on: 1278036
(Reporter)

Comment 1

2 years ago
Created attachment 8759995 [details]
Bug 1123827 - Support cross compiling to Linux using docker from any platform.

Review commit: https://reviewboard.mozilla.org/r/57746/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/57746/
Attachment #8759995 - Flags: review?(gps)
(Reporter)

Comment 2

2 years ago
Usage (PWD set to the m-c tree):
docker build --no-cache=true -t mozilla/platform-dev -f testing/docker/platform-dev/Dockerfile testing/docker/platform-dev
docker run -it -v $PWD:/home/worker/tree mozilla/platform-dev ./mach build

This is less ambitious than the original suggestion but it works well in practice.

Right now docker has various performance issues with mounting and generally very slow disk performance so this is only handy if you're willing to wait 30 minutes for a build:
https://github.com/boot2docker/boot2docker/issues/593#issuecomment-62309867
https://forums.docker.com/t/file-access-in-mounted-volumes-extremely-slow-cpu-bound/8076

Running this script will setup /Users (assuming your tree is there) via NFS and will substantially increase performance, but still slow overall:
https://gist.github.com/olalonde/3f7512c0bd2bc8abb46d

Seperatly we can consider how to integrate this with mach. I have a few ideas but first I'd rather focus in landing this.
Why not just use the taskcluster docker images? Take a build on treeherder, click "inspect task", then on the "task" tab, and you'll find instructions how to get the image and how to reproduce the build with it.
(Reporter)

Comment 4

2 years ago
I spoke with Dustin about using the taskcluster image as that was my first approach:

(Spinet highlights from our discussion)
> dustin: no, compilers come from tooltool
> dustin: mach bootstrap might be a better choice - the desktop-build image is fairly specific to automation (things like not including a compiler, as you've noticed)
> BenWa: Should I create a docker image that extends from your image?
> dustin: it really depends how important it is to hew closely to the automation build environment (centos 6, etc.)
> dustin: vs. having access to a modern environment
> BenWa: Yes, a modern environment is probably better
> dustin: in that case, I think mach bootstrap is probably the better choice

From what I gather the images aren't meant to be ran outside of taskcluster and grab artifact from tooltool however I'm not familiar with it.

And mach bootstrap right now requires interaction. I'd rather use that for my setup script rather than have a list of packages (mostly imported from mach bootstrap anyways). Hopefully we can have mach bootstrap detect there's no tty and do something sane.
(Reporter)

Comment 5

2 years ago
Maybe in a follow-up we might want to change this to use Docker data volume to overcome the disk performance. This way we would use hg/git to sync into docker, and a persistent data volume to host the obj files.

Upside is much better performance and once your changes have been pushed to docker you can continue to work, downside is mac no longer sees the object files
Downside is more complex implementation and the object files are no longer visible on mac which is handy because the mental model was simple (run docker, have object files show up).

Comment 6

2 years ago
(In reply to Benoit Girard (:BenWa) from comment #4)
> And mach bootstrap right now requires interaction. I'd rather use that for
> my setup script rather than have a list of packages (mostly imported from
> mach bootstrap anyways). Hopefully we can have mach bootstrap detect there's
> no tty and do something sane.

bootstrap.py supports a --no-interactive option. However, `mach bootstrap` doesn't expose it. It isn't very well tested though, so you may find bugs from things that don't take reasonable actions when in non-interactive mode.

Comment 7

2 years ago
Comment on attachment 8759995 [details]
Bug 1123827 - Support cross compiling to Linux using docker from any platform.

https://reviewboard.mozilla.org/r/57746/#review59250

Per the last comment on this bug, change the bootstrap script to run python/mozboot/bin/bootstrap.py to minimize duplication of logic. Because you can only ADD files to Dockerfile in the directory structure underneath, you may have to curl/wget bootstraph.py from hg.mozilla.org and execute it :/ I'm not a fan of that approach (you already have the repo, you should execute locally). But Docker doesn't give you many options unless you want to write code to call Docker's APIs (you can provide the image build API a tarball containing files available to ADD). See the cool trick we do at https://hg.mozilla.org/hgcustom/version-control-tools/file/530a34bf549a/testing/vcttesting/docker.py#l340 :)
Attachment #8759995 - Flags: review?(gps)

Updated

7 months ago
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.