"Run locally" instructions should contain the actual image name/tag



Docker Images
2 years ago
9 months ago


(Reporter: glandium, Unassigned, Mentored)



(Whiteboard: [good-first-bug])



2 years ago
This is what I see on try builds now:

  # Image appears to be a task image
  # Download image tarball from task
  curl -L -o image.tar https://queue.taskcluster.net/v1/task/aqt_YdmkTvugYB5b-OvvJw/artifacts/public/image.tar

  # Extract image name and tag from image tarball
  # Note: jq is required.  Download the package appropriate
  # for your OS at https://stedolan.github.io/jq/
  image=$(tar xf image.tar -O repositories | jq -r 'keys[0]')
  image_tag=$(tar xf image.tar -O repositories | jq -r '.[keys[0]] | keys[0]')

And I did that, and it worked. But that was a week ago, and I don't know if the image I got back then is the same. It would be a shame to redownload 2GB to figure it's the one I already have `docker load`ed.

Note, that also means that any benefit from incremental updates from previous images is lost...
Do you see a way to solve this while still storing images as artifacts?

Images as artifacts as some benefits in that artifacts are stored on S3 and we copy them between regions.

Ideas are welcome...

Comment 2

2 years ago
I don't know, but that was just a side note. The core problem is that I don't know if what I already have is the same as what I'd be downloading...
The request is to include the image_name explicitly in the instructions, so a 'docker images' could be used to see if it's already cached.

In this case, the image name is
and in fact that comes from
  + docker save mozilla-central:cfef5f18d0c329fc25d179228f409000da00fa6d9c640c6d0c86c76127b6652f
in https://tools.taskcluster.net/task-inspector/#aqt_YdmkTvugYB5b-OvvJw/0

The translation from repo:hash to taskId is performed in the decision task, unfortunately, so the tools page has no good way to discover that information.

One option here may be to encode this information in the task description explicitly for purposes of generating the "Run locally" instructions.
Mentor: dustin
Component: General → Docker Images
Whiteboard: [good-first-bug]

Comment 4

a year ago
I am unable to understand what the problem/bug lies? Can anybody elaborate about the bug and the required solution more? 
Thanks in advance.
Flags: needinfo?(dustin)

If you look at https://tools.taskcluster.net/task-inspector/#H2YT_OOxSPeBLi6jZJorzg and click "Run Locally", you'll see some instructions appear.  Those tell you to download image.tar.zst, and gives this code to determine the image and tag:

image=$(tar xf image.tar -O repositories | jq -r 'keys[0]')
image_tag=$(tar xf image.tar -O repositories | jq -r '.[keys[0]] | keys[0]')

Mike is looking for a way to determine that information, so he can tell if he needs to re-download the image (it's really big!)

Comment 3 suggests that's pretty hard, actually: the task inspector has no good way to know that information.  To be honest, this probably should have been marked "mentored" but not "[good-first-bug]"!

If you look at the logs for run 0 of the task linked above, you'll see 

[taskcluster 2017-01-11 16:36:19.929Z] Image 'public/image.tar.zst' from task 'UHPVLHR5QSyf9QZAktnLrQ' loaded.  Using image ID sha256:07bca4a1573a321943c1a1e346558cf5d0542b44fea8610aeabd8d54af5862d5.

So maybe an easier approach is to include some comments telling users how to see the sha256 of an image locally, and how to find the image ID in the logs, so that they can be compared.  That can still be a bit confusing -- in bug 1315415 comment 1 greg points out that there are a few sha's, so you need to make sure you're looking at the right one.
Flags: needinfo?(dustin)

Comment 6

a year ago
Ok, I get it now. So I need to store a sha256sum of image.tar and view it in the comments itself and ask the user to match the sha256sum to check if their downloaded version of image.tar is up to date and correctly downloaded. Am I right?

Can you give me the files which has the information to be printed stored after "Run locally" is clicked. I will add sha256sum component for users to let user know if they need to download again or not.
I don't think the sha256 is of the entire image.tar.  Rather, it's some sort of docker-related signature.  That part is going to take a bit of digging on your part to figure out -- I don't know it myself.

Here's where the run-locally instructions are generated:

Comment 8

a year ago
Will sha256 not complete the requirement of user to let them know whether to re-download or not and sha256 is easy to generate. Why not simply add sha256? I would like to know what's wrong with sha256.
You're right that anything we could provide for the user, and they could compare to a value they calculate on their laptop, would solve this issue.

The trick is to find something that we already know on the server side, and that the user can calculate.  So if you want to use a hash of the un-compressed tarball, that's fine, but you'll need to find a way to get that data in the task inspector.
Run locally is gone now, so nothing more to do here :)
Last Resolved: 9 months ago
Resolution: --- → INVALID
You need to log in before you can comment on or make changes to this bug.