1133877 - Combine MozBoot with docker build-environments (Docker images based on Mozilla's in house builders)

Status: NEW

(Firefox Build System :: General, defect)

Description

[Morgan Phillips [:mrrrgn]]

At least the CentOS bootstrapper is out of date, http://mxr.mozilla.org/mozilla-central/source/python/mozboot/mozboot/centos.py compared to the mozharness configs and mock configs which are being used to populate our in house builders.

Given that we now have a project which produces docker containers, based on our most recent builder configs it makes sense to encourage developers to do builds inside of these containers instead of bootstrapping: https://github.com/mrrrgn/build-mozilla-build-environments

Comment 1

[(not currently active) Ted Mielczarek]

These have always been best-effort, which sucks, but they've been better than the situation before they existed which was "try to figure out what set of packages you need to build". I'm sure we'd all love to have a standardized way to specify our build dependencies.

I think it'd be great to make it simple (a one-liner even) to replicate our official Docker images locally. I don't think this is going to fit everyone's needs, especially since it's almost certainly going to come with another large download (and we're already rough on people with crappy bandwidth).

I'd love to see what we can do, though!

Comment 2

[Gregory Szorc [:gps]]

We should absolutely leverage the official build environments in some manner.

Some people will complain about the loss of flexibility that comes with Docker (harder to view state, etc).

Some will complain that we should continue to support building natively on all operating systems and not require a specialized build environment.

Yes, downloading pre-built Docker images or even building the images locally will incur a lot of bandwidth. That is unfortunate.

But we should not let these things stand in the way. Developers want and need a mechanism to reproduce the official build environments so they can test and debug against the official build environment with little hassle.

I think we should start by having mozboot prompt for whether to use "local" or "Docker" development mode. "local" is what we do today. "Docker" would install Docker and pull or build the images that Morgan just developed.

A tricky part here is figuring out how the source directory interacts with the Docker image. Should we put the source directory on the host and mount it inside the Docker container? If so, does that bloat scope to mean build system / mach support for "tunneling" into a Docker container for execution?

Comment 3

[Joshua Cranmer [:jcranmer]]

My experience with using Docker is that it makes the edit-build-run cycle much more painful (handling X is a particular pain). For a local developer, its main value is in being able to represent the official build environment, which is relatively unimportant for most developers (particularly casual developers, for whom mach bootstrap would be most useful).

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

OK, so what I'm getting from this is that mozbootstrap is a relatively loose "here's the stuff you need to develop" that isn't guaranteed to generate something very close to the official build environment, but will generally build a browser of some sort.  The docker images are specifically intended to replicate the official build environment.

In which case, we in releng can continue on our merry way without worrying about mozbootstrap.

Comment 5

[Morgan Phillips [:mrrrgn]]

So, I think getting x to work, and mounting the the source directory can be solved fairly painlessly. Though, interacting with the container would certainly require a slightly modified developer workflow - and perhaps a few new mach commands.

I understand that for casual developers this may seem lke overkill, but there are some very nice advantages to shipping an entire environment to devs (as opposed to just installing dependencies).

Given that, while I agree that the scope of this ticket is wrong [replacing mozbootstrap], I think supporting docker based development in a way like Greg mentioned is worth discussing (i.e. local mode versus container mode).

Comment 6

[Morgan Phillips [:mrrrgn]]

All that said: "In which case, we in releng can continue on our merry way without worrying about mozbootstrap." If this turns out to be our choice, that's quite understandable. The conversation is worth having nonetheless.

Comment 7

[(not currently active) Ted Mielczarek]

(In reply to Dustin J. Mitchell [:dustin] from comment #4)
> OK, so what I'm getting from this is that mozbootstrap is a relatively loose
> "here's the stuff you need to develop" that isn't guaranteed to generate
> something very close to the official build environment, but will generally
> build a browser of some sort.  The docker images are specifically intended
> to replicate the official build environment.

This is true, but!


(In reply to Morgan Phillips [:mrrrgn] from comment #5)
> I understand that for casual developers this may seem lke overkill, but
> there are some very nice advantages to shipping an entire environment to
> devs (as opposed to just installing dependencies).
> 
> Given that, while I agree that the scope of this ticket is wrong [replacing
> mozbootstrap], I think supporting docker based development in a way like
> Greg mentioned is worth discussing (i.e. local mode versus container mode).

Yes, we are in violent agreement here. As with most things it's hard to find a one-size-fits-all approach given the broad range of our contributors, but having an easy way to replicate the official build environment in a docker image would be fantastic and a lot of people would use it. I don't think it will replace everyone's local development workflow, but it's a great thing to do.

If we can figure out a way to share the list of dependencies between the bootstrap scripts and the docker images that'd be even better. We had this same discussion in bug 1115107.

Comment 8

[Morgan Phillips [:mrrrgn]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #7)
> (In reply to Dustin J. Mitchell [:dustin] from comment #4)
> > OK, so what I'm getting from this is that mozbootstrap is a relatively loose
> > "here's the stuff you need to develop" that isn't guaranteed to generate
> > something very close to the official build environment, but will generally
> > build a browser of some sort.  The docker images are specifically intended
> > to replicate the official build environment.
> 
> This is true, but!
> 
> 
> (In reply to Morgan Phillips [:mrrrgn] from comment #5)
> > I understand that for casual developers this may seem lke overkill, but
> > there are some very nice advantages to shipping an entire environment to
> > devs (as opposed to just installing dependencies).
> > 
> > Given that, while I agree that the scope of this ticket is wrong [replacing
> > mozbootstrap], I think supporting docker based development in a way like
> > Greg mentioned is worth discussing (i.e. local mode versus container mode).
> 
> Yes, we are in violent agreement here. As with most things it's hard to find
> a one-size-fits-all approach given the broad range of our contributors, but
> having an easy way to replicate the official build environment in a docker
> image would be fantastic and a lot of people would use it. I don't think it
> will replace everyone's local development workflow, but it's a great thing
> to do.
> 
> If we can figure out a way to share the list of dependencies between the
> bootstrap scripts and the docker images that'd be even better. We had this
> same discussion in bug 1115107.

I left a comment on 1115107 as well, but, in short, I use the mozharness configs to generate our dockerfiles so I think that makes a reasonable canonical place to find dependencies.

Comment 9

[Mike Hommey [:glandium]]

Note that we have too many mock configs, and we should bring that down so that people who would like to build, say desktop and android don't need two different docker images just because our configs are not consolidated (and they really could be).

Comment 10

[Morgan Phillips [:mrrrgn]]

Okay, so I'm pretty excited about this, here's my current plan of attack:

1.) Get mozbootstrap up to date, and make sure there's a method in place for _keeping_ it up to date.
2.) Generate a docker image based on the mozbootstrap dependencies. This should be created by running bootstrap from inside of a container and saving it.
3.) Add new ./mack commands which are something like:

  ./mach docker-bootstrap # downloads our docker image or raises a prompt for the user to install docker if they don't have it already
  ./mach docker-build # attempts to build from within the docker container (we'll always chdir to wherever the mach script is in order to know which directory to mount) 
  ./mach docker-run # run ./mach run from within the container (we can share x sessions between the container and host)

Comment 11

[Morgan Phillips [:mrrrgn]]

(In reply to Mike Hommey [:glandium] from comment #9)
> Note that we have too many mock configs, and we should bring that down so
> that people who would like to build, say desktop and android don't need two
> different docker images just because our configs are not consolidated (and
> they really could be).

I agree, my plan as of now is to create one set of dependencies to rule them all, then, generate a docker image which is based on simply running bootstrap inside of a centos container. That way there is only one container, and users can generate an identical environment on a vm/bare metal machine if they so wish.

Comment 12

[(not currently active) Ted Mielczarek]

(In reply to Morgan Phillips [:mrrrgn] from comment #10)
> Okay, so I'm pretty excited about this, here's my current plan of attack:
> 
> 1.) Get mozbootstrap up to date, and make sure there's a method in place for
> _keeping_ it up to date.
> 2.) Generate a docker image based on the mozbootstrap dependencies. This
> should be created by running bootstrap from inside of a container and saving
> it.

This sounds pretty fantastic.

> 3.) Add new ./mack commands which are something like:

I'm not sure if I'm 100% sold on having a parallel set of commands for this, but we can have that bikeshed argument later. :)

Comment 14

[Pete Moore [:pmoore][:pete]]

I think also it makes sense that when we generate a task graph (from mach) ideally we should be able to run any of the tasks locally, i.e. so locally you can run a given test suite, and mach will take care of grabbing the docker image, and running the job (not sure if it would require docker-worker for this) but not publish any artifacts etc...

Ideally we can build/pull any docker image that is referenced in the task graph, and potentially run the task too, locally.

Would be nice if we have a uniform process across desktop/fennec/b2g.

Comment 15

[Pete Moore [:pmoore][:pete]]

We should also make sure to update MDN documents about how to build/update your system with info about how to do this using in-tree definitions when we are done... This would then be the canonical way, and would be good to remove references to the old world.

Comment 16

[Morgan Phillips [:mrrrgn]]

> I think also it makes sense that when we generate a task graph (from mach)

That's a very interesting idea. From what I can see, the current job definitions live in 'testing/taskcluster/tasks' my hope would be that there's enough information here to kick off a job without needing a taskcluster worker.