1247646 - Chain docker-image generation tasks

Status: RESOLVED INCOMPLETE

(Taskcluster Graveyard :: Docker Images, defect)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

Both of these images are currently set up with three separate docker images, each building on the last.  The idea was to minimize the amount devs would need to download to reproduce.  However, with the in-tree building system, the whole thing gets flattened and devs are required to download it all.

The downside of the three images is that modifying either of the first two requires permission to push to the docker hub taskcluster repo, which not everyone has.

To work around that shortcoming, let's collapse the three images into one, so that any change will be automatically rebuilt in-tree.  If we can figure out how to make the downloads smaller, and it requires breaking them out again, we can consider that.

Greg, am I totally insane?

Comment 1

[Greg Arndt [:garndt]]

I don't think it's insane at all.  I was going to do this for the b2gbuild and tester images before the announcement about b2g and I shifted focus.  

I think it's a lot easier to iterate on the images and reason about what's in the images by looking at a single image context.

Comment 2

[Pete Moore [:pmoore][:pete]]

This might be completely off-base, but could funsize help us to generate binary diffs on demand and serve those up? Probably a very naive comment...

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

I think funsize takes advantage of some knowledge of the file structure, and anyway for devs to take advantage of any caching, it needs to be in the shape of multiple docker layers, which are likely not in mar format.  It's a cool idea though!

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Yesterday, :nalexander wanted to make a new docker iamge based on desktop-test, with a bunch of extra stuff (gradle) installed.  There's really no decent way to do that -- to have two in-tree images, one of which depends on the other.

I think that, rather than flatten these images, we need to come up with a way to chain image dependencies into multiple "I" jobs.  Part of that is figuring out how to do so in in-tree configurations, but a lot of the challenge is figuring out how to convince Docker to do so, since it will likely involve editing the FROM line of the Dockerfiles.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

The images are no longer chained, based on :gps's work, so nothing to do here.