1247166 - "Run locally" instructions should contain the actual image name/tag

Status: RESOLVED INVALID

(Taskcluster Graveyard :: Docker Images, defect)

Description

[Mike Hommey [:glandium]]

This is what I see on try builds now:

  # Image appears to be a task image
  # Download image tarball from task
  curl -L -o image.tar https://queue.taskcluster.net/v1/task/aqt_YdmkTvugYB5b-OvvJw/artifacts/public/image.tar

  # Extract image name and tag from image tarball
  # Note: jq is required.  Download the package appropriate
  # for your OS at https://stedolan.github.io/jq/
  image=$(tar xf image.tar -O repositories | jq -r 'keys[0]')
  image_tag=$(tar xf image.tar -O repositories | jq -r '.[keys[0]] | keys[0]')
  image_name=$image:$image_tag


And I did that, and it worked. But that was a week ago, and I don't know if the image I got back then is the same. It would be a shame to redownload 2GB to figure it's the one I already have `docker load`ed.

Note, that also means that any benefit from incremental updates from previous images is lost...

Comment 1

[Jonas Finnemann Jensen (:jonasfj)]

Do you see a way to solve this while still storing images as artifacts?

Images as artifacts as some benefits in that artifacts are stored on S3 and we copy them between regions.

Ideas are welcome...

Comment 2

[Mike Hommey [:glandium]]

I don't know, but that was just a side note. The core problem is that I don't know if what I already have is the same as what I'd be downloading...

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

The request is to include the image_name explicitly in the instructions, so a 'docker images' could be used to see if it's already cached.

In this case, the image name is
  mozilla-central:cfef5f18d0c329fc25d179228f409000da00fa6d9c640c6d0c86c76127b6652f
and in fact that comes from
  + docker save mozilla-central:cfef5f18d0c329fc25d179228f409000da00fa6d9c640c6d0c86c76127b6652f
in https://tools.taskcluster.net/task-inspector/#aqt_YdmkTvugYB5b-OvvJw/0

The translation from repo:hash to taskId is performed in the decision task, unfortunately, so the tools page has no good way to discover that information.

One option here may be to encode this information in the task description explicitly for purposes of generating the "Run locally" instructions.

Comment 4

[Aman Mehta]

I am unable to understand what the problem/bug lies? Can anybody elaborate about the bug and the required solution more? 
Thanks in advance.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Hi!

If you look at https://tools.taskcluster.net/task-inspector/#H2YT_OOxSPeBLi6jZJorzg and click "Run Locally", you'll see some instructions appear.  Those tell you to download image.tar.zst, and gives this code to determine the image and tag:

image=$(tar xf image.tar -O repositories | jq -r 'keys[0]')
image_tag=$(tar xf image.tar -O repositories | jq -r '.[keys[0]] | keys[0]')
image_name=$image:$image_tag

Mike is looking for a way to determine that information, so he can tell if he needs to re-download the image (it's really big!)

Comment 3 suggests that's pretty hard, actually: the task inspector has no good way to know that information.  To be honest, this probably should have been marked "mentored" but not "[good-first-bug]"!

If you look at the logs for run 0 of the task linked above, you'll see 

[taskcluster 2017-01-11 16:36:19.929Z] Image 'public/image.tar.zst' from task 'UHPVLHR5QSyf9QZAktnLrQ' loaded.  Using image ID sha256:07bca4a1573a321943c1a1e346558cf5d0542b44fea8610aeabd8d54af5862d5.

So maybe an easier approach is to include some comments telling users how to see the sha256 of an image locally, and how to find the image ID in the logs, so that they can be compared.  That can still be a bit confusing -- in bug 1315415 comment 1 greg points out that there are a few sha's, so you need to make sure you're looking at the right one.

Comment 6

[Aman Mehta]

Ok, I get it now. So I need to store a sha256sum of image.tar and view it in the comments itself and ask the user to match the sha256sum to check if their downloaded version of image.tar is up to date and correctly downloaded. Am I right?

Can you give me the files which has the information to be printed stored after "Run locally" is clicked. I will add sha256sum component for users to let user know if they need to download again or not.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

I don't think the sha256 is of the entire image.tar.  Rather, it's some sort of docker-related signature.  That part is going to take a bit of digging on your part to figure out -- I don't know it myself.

Here's where the run-locally instructions are generated:
  https://github.com/taskcluster/taskcluster-tools/blob/8e4bff2e4dd7f3c87d0cd557a83d57017d27a4bf/src/lib/ui/taskinfo.jsx#L309

Comment 8

[Aman Mehta]

Will sha256 not complete the requirement of user to let them know whether to re-download or not and sha256 is easy to generate. Why not simply add sha256? I would like to know what's wrong with sha256.

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

You're right that anything we could provide for the user, and they could compare to a value they calculate on their laptop, would solve this issue.

The trick is to find something that we already know on the server side, and that the user can calculate.  So if you want to use a hash of the un-compressed tarball, that's fine, but you'll need to find a way to get that data in the task inspector.

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

Run locally is gone now, so nothing more to do here :)