Closed Bug 1419638 Opened 2 years ago Closed 2 years ago

Allow to pass arguments to docker when building docker-images

Categories

(Firefox Build System :: Task Configuration, task)

task
Not set

Tracking

(Not tracked)

RESOLVED FIXED
mozilla59

People

(Reporter: glandium, Assigned: glandium)

References

Details

Attachments

(3 files)

There are two usecases I'm interested in:
- I need to pass down task-ids for package artifacts to install in the docker-image.
- I would like to introduce some sort of variants for a docker image. The concrete example here is that for bug 1399679, I need two different images, and their Dockerfile is almost entirely identical, and the other files attached to the docker image are the exact same. By duplicating those, we risk the footgun where one is updated and not the other, or they're not updated in the exact same way, etc. By having the relevant parts of the Dockerfile influenced by an ARG, it is possible to share the same file for those two images.

Now, while the first usecase is straightforward, I'm not sure how to alter the kind.yml for docker images to allow the second.

Maybe something like:

  foo1:
    symbol: I(foo1)
    definition: foo
    args:
      VARIANT: foo1

  foo2:
    symbol: I(foo2)
    definition: foo2
    args:
      VARIANT: foo2

where definition would default to the name if not set, and correspond to the directory under taskcluster/docker/ where the Dockerfile is.
(modulo bikeshedding)

Dustin, what do you think?
Can the args be environment variables?  That's already well supported by TC kind.yml files (I think), and definitely is standard for parameterizing Dockerfiles.
Flags: needinfo?(mh+mozilla)
(In reply to Nick Alexander :nalexander from comment #1)
> Can the args be environment variables?  That's already well supported by TC
> kind.yml files (I think), and definitely is standard for parameterizing
> Dockerfiles.

Isn't the standard for parameterizing Dockerfiles --build-arg/ARG?
Flags: needinfo?(mh+mozilla)
(In reply to Mike Hommey [:glandium] from comment #2)
> (In reply to Nick Alexander :nalexander from comment #1)
> > Can the args be environment variables?  That's already well supported by TC
> > kind.yml files (I think), and definitely is standard for parameterizing
> > Dockerfiles.
> 
> Isn't the standard for parameterizing Dockerfiles --build-arg/ARG?

Huh, sure looks more standard than env variables.  I stand corrected :)
Flags: needinfo?(dustin)
I like this!!
Flags: needinfo?(dustin)
So, it turns out that past the hurdles of having to change image_builder to make this happen, this is blocked on the fact that the version of docker used to build docker images is too old and doesn't support passing build arguments through the API (it might not even support ARG in Dockerfile either).

Bummer.
Depends on: 1426282
Blocks: 1427326
Comment on attachment 8938918 [details]
Bug 1419638 - Allow to pass arguments to docker when building docker-images.

https://reviewboard.mozilla.org/r/209390/#review215444

::: taskcluster/taskgraph/transforms/docker_image.py:44
(Diff revision 4)
>      # relative path (from config.path) to the file the docker image was defined
>      # in.
>      Optional('job-from'): basestring,
> +
> +    # Arguments to use for the Dockerfile.
> +    Optional('args', default={}): {basestring: basestring},

Per catlee's recent landing, omit `default=` here and apply the default in `pop` below.

::: taskcluster/taskgraph/util/docker.py:199
(Diff revision 4)
>      volumes = set()
>  
>      with open(os.path.join(IMAGE_DIR, image, 'Dockerfile'), 'rb') as fh:
>          for line in fh:
>              line = line.strip()
> +            # We assume VOLUME definitions don't use %ARGS.

Worth noting this in the docstring describing %ARGS
Attachment #8938918 - Flags: review?(dustin) → review+
Comment on attachment 8938986 [details]
Bug 1419638 - Allow to share docker image definitions.

https://reviewboard.mozilla.org/r/209438/#review215446

::: taskcluster/taskgraph/transforms/docker_image.py:47
(Diff revision 4)
>  
>      # Arguments to use for the Dockerfile.
>      Optional('args', default={}): {basestring: basestring},
> +
> +    # Name of the docker image definition under taskcluster/docker, when
> +    # difference from the docker image name.

*different
Attachment #8938986 - Flags: review?(dustin) → review+
Comment on attachment 8938917 [details]
Bug 1419638 - Add schema validation to docker image transform.

https://reviewboard.mozilla.org/r/209388/#review215440

::: taskcluster/taskgraph/transforms/docker_image.py:6
(Diff revision 4)
>  # This Source Code Form is subject to the terms of the Mozilla Public
>  # License, v. 2.0. If a copy of the MPL was not distributed with this
>  # file, You can obtain one at http://mozilla.org/MPL/2.0/.
>  """
> -Transform the upload-symbols task description template,
> -  taskcluster/ci/upload-symbols/job-template.yml
> +Transform the docker-images task description template,
> +  taskcluster/ci/upload-symbols/kind.yml

missed a bit here :)

This whole docstring can probably be dropped.
Attachment #8938917 - Flags: review?(dustin) → review+
Pushed by mh@glandium.org:
https://hg.mozilla.org/integration/autoland/rev/e2d65a3c816b
Add schema validation to docker image transform. r=dustin
https://hg.mozilla.org/integration/autoland/rev/a8a58f122f88
Allow to pass arguments to docker when building docker-images. r=dustin
https://hg.mozilla.org/integration/autoland/rev/2cda01d56646
Allow to share docker image definitions. r=dustin
Pushed by apavel@mozilla.com:
https://hg.mozilla.org/mozilla-central/rev/48faf408be3b
Add schema validation to docker image transform. r=dustin
https://hg.mozilla.org/mozilla-central/rev/7b087c4f2aa7
Allow to pass arguments to docker when building docker-images. r=dustin
https://hg.mozilla.org/mozilla-central/rev/ab480c23769b
Allow to share docker image definitions. r=dustin
https://hg.mozilla.org/mozilla-central/rev/48faf408be3b
https://hg.mozilla.org/mozilla-central/rev/7b087c4f2aa7
https://hg.mozilla.org/mozilla-central/rev/ab480c23769b
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla59
Blocks: 1429670
Product: TaskCluster → Firefox Build System
You need to log in before you can comment on or make changes to this bug.