1316519 - squash docker image layers in image_builder

Status: RESOLVED WONTFIX

(Taskcluster Graveyard :: Docker Images, defect)

Description

[Jonas Finnemann Jensen (:jonasfj)]

There is a few decent tools to do this:
  https://github.com/goldmann/docker-squash
  https://github.com/jwilder/docker-squash

It might be that the tooling isn't stable enough yet, in which case we should just drop the idea.

Future versions of docker will likely make it easier to squash images.
So porting a newer version to dind-service is another approach to this.

Comment 1

[Gregory Szorc [:gps]]

Presumably we only produce a single image layer during our image build process because we build with the equivalent of `docker build --rm`. That means other layers must come from base images we pull from Docker Registry.

One solution might be to squash the base images we use into a single tarball/layer and use those as the base. That will get us down to a single base layer and a layer for our stuff. 2 layers feels pretty manageable.

As an added bonus, if we host our own base layers, we'll have better determinism and reproducibility (pulling images without specifying the hash isn't deterministic).

Comment 2

[Gregory Szorc [:gps]]

FWIW, one of the benefits of the "layer tarballs" is they contain metadata such as which command to run as startup. Since we're providing our own metadata and essentially using the images as filesystem snapshots, we can likely throw away the metadata+layers bits.

Comment 3

[Greg Arndt [:garndt]]

This is set for the 1.13 milestone fwiw https://github.com/docker/docker/pull/22641

Comment 4

[Jonas Finnemann Jensen (:jonasfj)]

@gps,
No, each RUN command in the Dockerfile produces a file-system layer.
Docker build does this so it can cache every build step and quickly rebuild.
Given that docker builds are non-deterministic this doesn't always work too well in practice.

Comment 5

[Jonas Finnemann Jensen (:jonasfj)]

Attachment: Bug 1316519 - Squash docker image layers.

Review commit: https://reviewboard.mozilla.org/r/92458/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/92458/

Comment 6

[Jonas Finnemann Jensen (:jonasfj)]

@dustin,
If you r+ I'll build and push the docker image.

Comment 7

[Jonas Finnemann Jensen (:jonasfj)]

note: on the 500mb lint image (which is 200mb after zstd) this saved about 12mb (after zstd).

But ideally we get some runtime wins by having fewer layers.

Comment 8

[Gregory Szorc [:gps]]

If 2 images share the same base layer, presumably when we import that image the import of an existing layer is a no-op. While we'd incur time to download and decompress the image, Docker wouldn't process it. Is this true? And if so, does squashing layers undo that optimization?

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8809979 [details]
Bug 1316519 - Squash docker image layers.

https://reviewboard.mozilla.org/r/92458/#review92580

The patch looks good.  I'm not totally convinced of the wisdom of this -- I think we achieve a very similar effect by using system-setup.sh, meaning that the layers in the final image are largely orthogonal (upstream base, then an image with our system-setup, with a few intervening  "0-byte" layers).  :gps raises a good point, and anyway I'd be interested to know if this shaves more than ~10mb off desktop-test or desktop-build.

Comment 10

[Jonas Finnemann Jensen (:jonasfj)]

> Docker wouldn't process it. Is this true?
Hmm, I don't know...

Anyways, points.

I'll test it with desktop1604-test and see what this gives us.

Comment 11

[Jonas Finnemann Jensen (:jonasfj)]

After testing it some more it seems that docker-squash takes forever on large images (3.5 GB).
I tried a different squash utility, but it provided no worthwhile size difference, load time also seemed to be the same.

Given these results, I suggest we park this for now and reconsider it when docker adds official support for squashing when building images.

Comment 12

[Jonas Finnemann Jensen (:jonasfj)]

Comment on attachment 8809979 [details]
Bug 1316519 - Squash docker image layers.

https://reviewboard.mozilla.org/r/92458/#review93634

After testing with the desktop1604-test image it seems that a 3.5GB image layer causes problems for docker-squash which is written in python.
Furthermore, after trying it with different squash utility the win seemed to be negligible.

Further investigation needed, but this probably isn't worth trying out before docker comes with built-in squash on-build features.