1175561 - docker-worker: pull images by hash

Status: RESOLVED FIXED

(Taskcluster :: Workers, defect)

Description

[Rail Aliiev [:rail]]

I tried to use hashes instead of tags for docker images, but for some reason it fails with the following error:

[taskcluster] taskId: LYA4WRr2SQ2OfgRbPVMPxQ, workerId: i-cd990c06

[taskcluster] Error: Pulling docker image
"rail/funsize-update-generator@sha256:77901eadbdf52da5b36fdbf811b273d9098ede055e99dac6aa2f365c2be09536"
has failed this may indicate an Error with the registry used or an
authentication error in the worker try pulling the image locally. Error: Error:
HTTP code is 500 which indicates error: server error - Invalid repository name
(funsize-update-generator@sha256), only [a-z0-9-_.] are allowed


The task is here: https://tools.taskcluster.net/task-graph-inspector/#4h4baOrCT6-jNtExvxlaOg/LYA4WRr2SQ2OfgRbPVMPxQ/

The payload looks like this:
...
"payload": {
    "image": "rail/funsize-update-generator@sha256:77901eadbdf52da5b36fdbf811b273d9098ede055e99dac6aa2f365c2be09536",
...

Comment 1

[Rail Aliiev [:rail]]

IIRC, we use dockerode in docker-worker. Looks like it tries to parse the string in https://github.com/apocas/dockerode/blob/master/lib/util.js#L28 and fails to detect sha256.

CC'ing James, who wrote that! https://github.com/apocas/dockerode/commit/d868c39bb9df117e54f78881b97fdb2b7ca81fd2 :)

Comment 2

[Rail Aliiev [:rail]]

Also looks like this is a v2 feature of docker hub API...

Comment 3

[Rail Aliiev [:rail]]

Upstream bug: https://github.com/apocas/dockerode/issues/152

Comment 4

[Rail Aliiev [:rail]]

Submitted a PR to upstream: https://github.com/apocas/dockerode/pull/153

Comment 5

[Rail Aliiev [:rail]]

The PR has been merged, waiting on next release.

Comment 6

[Rail Aliiev [:rail]]

Attachment: bump dockerode version

Comment 7

[Rail Aliiev [:rail]]

BTW, TC CI seems failing due to unrelated issue:
https://tools.taskcluster.net/task-graph-inspector/#WbazMkfERLWrjABUsxkqXg/


[taskcluster] taskId: yqOUvftATbCSMfElML90yA, workerId: i-541b24a3

taskcluster/worker-ci:0.0.2 exists in the cache.
[taskcluster] Error: Docker configuration could not be created.  This may
indicate an authentication error when validating scopes necessary for running
the task. 
Error: Cannot run task using docker privileged mode.  Worker must be enabled to
allow running of privileged tasks.
[taskcluster] Unsuccessful task run with exit code: -1 completed in 4.668
seconds

Comment 8

[Greg Arndt [:garndt]]

Error above was from a different issue that's been fixed.  Taskgraph reran and the only failure that's persisting is a known issue with our custom registry proxy that's only used for tests. 

This looks good to me and has been merged https://github.com/taskcluster/docker-worker/commit/5b26c5d227b87ab864e48189f3812b78e8dee1f0

This will roll out into a new ami soon after https://github.com/taskcluster/docker-worker/pull/115 is reviewed and merged.

Comment 9

[Greg Arndt [:garndt]]

Also, I will resolve this bug once the new ami is deployed

Comment 10

[Rail Aliiev [:rail]]

Thanks! \o/

Comment 11

[Rail Aliiev [:rail]]

I just tested the new ami and it looks like it's working! \o/
https://tools.taskcluster.net/task-inspector/#cEc1JVRDTy2q1JQ3oKnGPQ/

Comment 12

[Rail Aliiev [:rail]]

Greg, do we have any blockers there?

Comment 13

[Greg Arndt [:garndt]]

Nope, it was rolled out to a few of the workers already, but then a bug about artifact upload was discovered and needed to be fixed before I continued to roll out the ami changes.  Tomorrow morning the new ami will be rolled out to the other workers and this bug closed once complete.

Comment 14

[Rail Aliiev [:rail]]

Thanks!

Comment 15

[Greg Arndt [:garndt]]

AMIs were rolled out earlier today.  All should be good.