1129789 - Write a script which publishes a [Linux] build's bare environment

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Morgan Phillips [:mrrrgn]]

This could just be a stripped version of our current mozharness scripts used in builds. Ultimately, this script should spit out a tar.gz'd chroot environment by default, or, when a flag is set create and/or publish a docker container.

The idea here, is that the nicest way to publish build chroot environments will be via docker containers, due to their build-in versioning etc... This may also make moving the builds into taskcluster less painful.

Comment 1

[Gregory Szorc [:gps]]

Unless it's 5 minutes of work, I think Docker image generation should be deferred. This isn't because it isn't useful (it is) but because it is trivial to create a Docker image from a chroot tarball using `docker import`.

This bug excites me, BTW \o/

Comment 2

[Morgan Phillips [:mrrrgn]]

(In reply to Gregory Szorc [:gps] from comment #1)
> Unless it's 5 minutes of work, I think Docker image generation should be
> deferred. This isn't because it isn't useful (it is) but because it is
> trivial to create a Docker image from a chroot tarball using `docker import`.
> 
> This bug excites me, BTW \o/

Yay, thank you. :) Yes, I agree with you. Though, my thinking was that publishing a docker image would provide a more familiar interface for developers to pull down the environments. It seems more elegant than hitting an s3 bucket.

Comment 3

[Morgan Phillips [:mrrrgn]]

I've published several docker containers formed via this utility: https://github.com/mrrrgn/build-mozilla-build-environments/blob/master/utils/compile_mozharness_config/compile_mozharness_config.py

It compiles jinja2 templates against variables in mozharness configs. I was able to create a Dockerfile jinja2 template which gets me most of what I need. The publishing of new containers could easily be accomplished with a makefile.

Comment 4

[Morgan Phillips [:mrrrgn]]

Attachment: GitHub Repo

So, I have a makefile and some bash scripts here which generate dockerfiles and optionally publish them. platforms/ contains our 3 base mock configs and builders contains our 3 base builders, and 6 additional Linux build sub-types.

Do these environments seem acceptable so far?

Comment 5

[Gregory Szorc [:gps]]

This work looks great and I think they are worthy of being published!

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

I'm a little leery of this diverging from the other ways that build environments are built.

Does this replace MozBootstrap?  If so, should we rm -rf that?

Also, it looks like the platforms (an overloaded word, btw) are generated from the upstream yum repositories, which will result in different package versions than what we run in production (which is based on the mockbuild-repos alone).

I think something is definitely better than nothing here, but without some strong guarantees that the environments match, I think we may leave developers battling unknown unknowns, which is never fun.

Comment 7

[Morgan Phillips [:mrrrgn]]

(In reply to Dustin J. Mitchell [:dustin] from comment #6)
> I'm a little leery of this diverging from the other ways that build
> environments are built.
> 
> Does this replace MozBootstrap?  If so, should we rm -rf that?
> 
> Also, it looks like the platforms (an overloaded word, btw) are generated
> from the upstream yum repositories, which will result in different package
> versions than what we run in production (which is based on the
> mockbuild-repos alone).
> 
> I think something is definitely better than nothing here, but without some
> strong guarantees that the environments match, I think we may leave
> developers battling unknown unknowns, which is never fun.

Google/Wiki searches are teaching me that Mozbootstrap is a loaded term as well. >.< I don't think this should replace that, if it's what the name implies (a way for developers to get started contributing, no?)

As for the platforms, I can regenerate those with a more true form (in the process of that now). Because the builders are all built on top of those three bases, I can propagate it asap.

Like you said, something may be better than nothing. If we can get these in good enough shape, I think they'll decrease the number of unknowns developers are battling.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Yeah, but it's also trying to install packages, e.g., http://mxr.mozilla.org/mozilla-central/source/python/mozboot/mozboot/centos.py

Comment 9

[Morgan Phillips [:mrrrgn]]

I'm considering this finished; but we can/should continue the conversation on the parent task.