1518759 - taskcluster-client.py with taskcluster-proxy seems to be broken for queue requests

Status: RESOLVED FIXED

(Taskcluster :: Services, defect)

Description

[Rob Thijssen [:grenade (EET/UTC+0300)]]

i'm attempting to write a simple decision task for a github integration:

https://github.com/mozilla-platform-ops/relops-image-builder/blob/master/ci/decision-task.py

the python code works if taskcluster-proxy is not used. eg:

queue = taskcluster.Queue({'rootUrl': 'https://taskcluster.net'})

and doesn't work if taskcluster-proxy is used. eg:

queue = taskcluster.Queue({
  'rootUrl': 'http://localhost:8080',
  'credentials': {
    'clientId': '...',
    'accessToken': '...'
  }
})

the output from tc-proxy during a failing call looks like this:

grenade@quadbrat ~ $ go/bin/taskcluster-proxy --client-id "..." --access-token ... --root-url https://taskcluster.net
2019/01/09 13:01:40 Version: Taskcluster proxy 5.0.0 (unknown git revision)
2019/01/09 13:01:40 Listening on: :8080
2019/01/09 13:01:40 Root URL: 'https://taskcluster.net'
2019/01/09 13:01:40 Client ID: 'mozilla-auth0/ad|Mozilla-LDAP|rthijssen/windows-ami-builder-dev'
2019/01/09 13:01:40 Access Token: <not shown>
2019/01/09 13:01:40 Warning - no taskcluster certificate set - assuming permanent credentials are being used
2019/01/09 13:01:40 Proxy has full scopes of provided credentials - no scope reduction being applied
root handler /api/queue/v1/task/MDabJsYRRkG02G0f0bCaOA
2019/01/09 13:01:53 Proxying /api/queue/v1/task/MDabJsYRRkG02G0f0bCaOA | PUT | https://api.taskcluster.net/queue/v1/task/MDabJsYRRkG02G0f0bCaOA
2019/01/09 13:01:53 Error: Put https://api.taskcluster.net/queue/v1/task/MDabJsYRRkG02G0f0bCaOA: dial tcp: lookup api.taskcluster.net on 192.168.1.1:53: no such host
2019/01/09 13:01:53 Error: Put https://api.taskcluster.net/queue/v1/task/MDabJsYRRkG02G0f0bCaOA: dial tcp: lookup api.taskcluster.net on 192.168.1.1:53: no such host
2019/01/09 13:01:54 Error: Put https://api.taskcluster.net/queue/v1/task/MDabJsYRRkG02G0f0bCaOA: dial tcp: lookup api.taskcluster.net on 192.168.1.1:53: no such host
2019/01/09 13:01:55 Error: Put https://api.taskcluster.net/queue/v1/task/MDabJsYRRkG02G0f0bCaOA: dial tcp: lookup api.taskcluster.net on 192.168.1.1:53: no such host
2019/01/09 13:01:57 Error: Put https://api.taskcluster.net/queue/v1/task/MDabJsYRRkG02G0f0bCaOA: dial tcp: lookup api.taskcluster.net on 192.168.1.1:53: no such host

i think tc-proxy is using the wrong urls (https://api.taskcluster.net/...) for the queue calls. or maybe i'm supposed to configure this somewhere...

Comment 1

[Pete Moore [:pmoore][:pete]]

Thanks Rob!

Indeed, this looks like fallout from https://github.com/taskcluster/taskcluster-proxy/pull/38.

That PR was meant to add support for /api/ urls - looks like it isn't working...

Comment 2

[Pete Moore [:pmoore][:pete]]

Attachment: GitHub Pull Request for taskcluster-proxy

Comment 3

[Pete Moore [:pmoore][:pete]]

Release script is running now to release taskcluster-proxy 5.0.1 to github releases and to dockerhub.

Should be published to both locations in the next 15 mins or so...

Comment 4

[Rob Thijssen [:grenade (EET/UTC+0300)]]

yes, that's fixed it. many thanks!

Comment 5

[Pete Moore [:pmoore][:pete]]

Attachment: GitHub Pull Request for docker-worker

This bumps the version of taskcluster-proxy in docker-worker default config.

I believe we'll also need to roll this out to production worker types after this lands.

@grenade is using github-worker worker type, but we probably want to roll out all of them.

taskcluster-proxy 5.0.1 has been released to github releases, and dockerhub.

Comment 6

[Pete Moore [:pmoore][:pete]]

Hi Wander,

Thanks for merging the docker-worker PR. Would you be able to roll it out to the affected worker types?

Many thanks!

Comment 7

[Rob Thijssen [:grenade (EET/UTC+0300)]]

it seems that the taskcluster-proxy docker-worker is still running taskcluster-proxy:4.0.1:

(from: https://papertrailapp.com/systems/2828441812/events?focus=1020317140764491777)

Jan 11 11:43:05 docker-worker.aws-provisioner.us-east-1c.ami-0026287f.m3-xlarge.i-04177ef4db619aeb6 docker-worker: {"type":"ensure image","source":"top","provisionerId":"aws-provisioner-v1","workerId":"i-04177ef4db619aeb6","workerGroup":"us-east-1","workerType":"github-worker","workerNodeType":"m3.xlarge","image":{"name":"taskcluster/taskcluster-proxy:4.0.1","type":"docker-image"}}

so i'm still unable to reach the queue when running tc-client.py code from the github-worker (it works when run locally using a local tc-proxy 5.0.1).

here's a task that tries to call the queue via tc-proxy but times out after no response from proxy:
https://tools.taskcluster.net/groups/EqsT8NiiSgWPSqACiR7r_w

Comment 8

[Rob Thijssen [:grenade (EET/UTC+0300)]]

• https://papertrailapp.com/events?q=%22taskcluster-proxy%3A4.0.1%22 (many results)
• https://papertrailapp.com/events?q=%22taskcluster-proxy%3A5.0.0%22 (no results)
• https://papertrailapp.com/events?q=%22taskcluster-proxy%3A5.0.1%22 (no results)

Comment 9

[Pete Moore [:pmoore][:pete]]

Sorry about that Rob, Wander is currently testing a release.

FWIW you can also grab the taskcluster-proxy version with a task like this (if you want to avoid reading server logs):

https://tools.taskcluster.net/groups/Hko0w7L_TkqBhBBBpj6ugA/tasks/Hko0w7L_TkqBhBBBpj6ugA/details

Just dumping this here it helps people in future who are struggling to determine version of taskcluster-proxy deployed on a given worker type...

Comment 10

[Pete Moore [:pmoore][:pete]]

Hey Rob,

It looks like the upgrade is completed now[1], can you check if it is working?

Thanks!

--

[1] https://tools.taskcluster.net/groups/Hko0w7L_TkqBhBBBpj6ugA/tasks/HFb096RaQleKJ6ZRIJ4InQ/runs/0/logs/public%2Flogs%2Flive.log#L25

Comment 11

[Rob Thijssen [:grenade (EET/UTC+0300)]]

hmm. i'm still seeing 4.0.1

https://tools.taskcluster.net/groups/cMO9Nfq8RBes2gvFsgCDJw/tasks/cMO9Nfq8RBes2gvFsgCDJw/runs/0/logs/public%2Flogs%2Flive.log

maybe i haven't properly understood the relationship between github-worker, docker-worker & the docker image in use (we use a custom docker image although i assumed the tc-proxy was running on the docker worker host)?

Comment 12

[Pete Moore [:pmoore][:pete]]

Rob, that is indeed strange. It looks like 3 days ago at least one worker was running 5.0.1 so I'm not sure why a worker from an hour ago is running 4.0.1.

Wander, do you know?

Comment 13

[Wander Lairson Costa]

(In reply to Pete Moore [:pmoore][:pete] from comment #12)

Rob, that is indeed strange. It looks like 3 days ago at least one worker
was running 5.0.1 so I'm not sure why a worker from an hour ago is running
4.0.1.

Wander, do you know?

I put newer AMIs to GitHub-worker as a test but pulled it back because I suspected it broke a task. Newer docker-worker should be deployed, but I had some reported issues to verify and have to put tests aside.

Comment 14

[Pete Moore [:pmoore][:pete]]

(In reply to Wander Lairson Costa [:wcosta] from comment #13)

I put newer AMIs to GitHub-worker as a test but pulled it back because I suspected it broke a task. Newer docker-worker should be deployed, but I had some reported issues to verify and have to put tests aside.

No worries - but please update bugs when you make changes (roll-outs/back-outs) so everyone can see what is going on. :)

Do you have details about the test failure you encountered, or an idea how much work is left to do? Thanks.

Comment 15

[Wander Lairson Costa]

(In reply to Pete Moore [:pmoore][:pete] from comment #14)

(In reply to Wander Lairson Costa [:wcosta] from comment #13)

I put newer AMIs to GitHub-worker as a test but pulled it back because I suspected it broke a task. Newer docker-worker should be deployed, but I had some reported issues to verify and have to put tests aside.

No worries - but please update bugs when you make changes
(roll-outs/back-outs) so everyone can see what is going on. :)

Do you have details about the test failure you encountered, or an idea how
much work is left to do? Thanks.

It turned out to be unrelated.

Comment 16

[Wander Lairson Costa]

New docker-worker has been deployed.