Closed Bug 1244189 Opened 4 years ago Closed 4 years ago

Add a mach command to download docker images from artifacts

Categories

(Taskcluster Graveyard :: Docker Images, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED
mozilla48

People

(Reporter: dustin, Assigned: dustin)

Details

Attachments

(1 file)

./mach docker-load desktop-test

would do the usual arithmetic to figure out what task artifact corresponds to the desktop-test directory in-tree, docker-load it, and tag it as `desktop-test:latest`.
The command helpfully downloads a docker image created by a taskcluster task,
which can be helpful when trying to replicate subtle bugs that only occur
in-tree.

This also fixes a bug in hashing Dockerfile directories, where the full
pathname was taken into account.  While this pathname is consistent from
decision task to decision task, it is not consistent with developers' home
directories.  This change omits the directory prefix, which will cause a
one-time shift in all directory hashes.

Review commit: https://reviewboard.mozilla.org/r/42505/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/42505/
Attachment #8734951 - Flags: review?(garndt)
Comment on attachment 8734951 [details]
MozReview Request: Bug 1244189: add 'mach taskcluster-load-image'; r?garndt

https://reviewboard.mozilla.org/r/42505/#review39015

::: testing/taskcluster/mach_commands.py:238
(Diff revision 1)
> +    @CommandArgument('--task-id',
> +        help="Load the image at public/image.tar in this task, rather than "
> +             "searching the index")
> +    @CommandArgument('image_name', nargs='?',
> +        help="Load the image of this name based on the current contents of the tree "
> +             "(as built for mozilla-central or mozilla-inbound)")

Hrm, why is this "or mozilla-inbound" but we only seem to be loading from mozilla-central below.

::: testing/taskcluster/taskcluster_graph/image_builder.py:250
(Diff revision 1)
> +    filename = 'temp-docker-image.tar'
> +
> +    print("Downloading {}".format(url))
> +    subprocess.check_call(['curl', '-#', '-L', '-o', filename, url])
> +
> +    print("Loading image into docker")

So the tarball is downloaded, which could be many gigabytes, before trying to load the image.  Would it be helpful to catch CalledProcessError in the case that docker load fails or that docker daemon is unreachable (happens a lot when running docker-machine and the current environment isn't configured for whatever machine you want to use).  This way we could include an error message indicating what might have happened, where the image was stored, and what to do once docker is configured/working correctly.  Maybe the user could then run docker load themselves or something since the file was already downloaded.

::: testing/taskcluster/taskcluster_graph/image_builder.py:251
(Diff revision 1)
> +
> +    print("Downloading {}".format(url))
> +    subprocess.check_call(['curl', '-#', '-L', '-o', filename, url])
> +
> +    print("Loading image into docker")
> +    subprocess.check_call(['docker', 'load', '-i', filename])

I wish 'load' returned back the image name, or at least the image ID that it loaded it with, but after talking with the docker team I guess that would not be nice since an image tarball could actually contain multiple images.
Attachment #8734951 - Flags: review?(garndt)
https://reviewboard.mozilla.org/r/42505/#review39015

> Hrm, why is this "or mozilla-inbound" but we only seem to be loading from mozilla-central below.

Whoops, I changed the message but not the implementation.  The idea is to check both branches so you can use the latest and greatest even if it hasn't merged to central yet.  I'll fix.

> So the tarball is downloaded, which could be many gigabytes, before trying to load the image.  Would it be helpful to catch CalledProcessError in the case that docker load fails or that docker daemon is unreachable (happens a lot when running docker-machine and the current environment isn't configured for whatever machine you want to use).  This way we could include an error message indicating what might have happened, where the image was stored, and what to do once docker is configured/working correctly.  Maybe the user could then run docker load themselves or something since the file was already downloaded.

Good idea - thanks!

> I wish 'load' returned back the image name, or at least the image ID that it loaded it with, but after talking with the docker team I guess that would not be nice since an image tarball could actually contain multiple images.

You would think it could just output one `name:tag` combination per line or something.  It ain't rocket science.  And having to depend on the structure of the tarball feels pretty gross.
Comment on attachment 8734951 [details]
MozReview Request: Bug 1244189: add 'mach taskcluster-load-image'; r?garndt

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/42505/diff/1-2/
Attachment #8734951 - Flags: review?(garndt)
Comment on attachment 8734951 [details]
MozReview Request: Bug 1244189: add 'mach taskcluster-load-image'; r?garndt

https://reviewboard.mozilla.org/r/42505/#review39023
Attachment #8734951 - Flags: review?(garndt) → review+
https://hg.mozilla.org/mozilla-central/rev/4840ae780ac1
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla48
Product: Taskcluster → Taskcluster Graveyard
You need to log in before you can comment on or make changes to this bug.