1405889 - fetching private artifacts through taskcluster-proxy doesn't work

Status: RESOLVED FIXED

(Taskcluster :: Workers, defect)

Description

[Jonas Finnemann Jensen (:jonasfj)]

A) Auth issue
  $ curl -L http://taskcluster/queue.taskcluster.net/v1/task/.../artifacts/...
causes 403 from S3, because taskcluster-client-go always attaches:
  content-type: application/json
regardless of whether or not there is a body (GET requests never have a body).
And golang (net/http) forwards the content-type header causing the StringToSign
that S3 computes to change, rendering the signature invalid.

B) taskcluster-proxy doesn't stream the response, so downloading a large artifact would run it out of memory

Comment 1

[Jonas Finnemann Jensen (:jonasfj)]

PR for A: https://github.com/taskcluster/taskcluster-client-go/pull/21

Comment 2

[[github robot]]

Commit pushed to master at https://github.com/taskcluster/taskcluster-client-go

https://github.com/taskcluster/taskcluster-client-go/commit/7450d4c392165a03bd7cc982a5b13de3788be405
Merge pull request #21 from taskcluster/bug-1405889

Bug 1405889 - only set content-type of there is a body

Comment 3

[Pete Moore [:pmoore][:pete]]

Released in taskcluster-proxy 4.0.1/latest: https://hub.docker.com/r/taskcluster/taskcluster-proxy/tags/

Comment 4

[Pete Moore [:pmoore][:pete]]

Attachment: Github Pull Request for docker-worker

Comment 5

[[github robot]]

Commit pushed to master at https://github.com/taskcluster/taskcluster-client-go

https://github.com/taskcluster/taskcluster-client-go/commit/0d58436eb4e739ae2fa2b799d1bc336c00839707
Bug 1405889 - add a further test that Content-Type is set if there is a request body

Comment 6

[Pete Moore [:pmoore][:pete]]

Attachment: Github Pull Request for taskcluster-proxy

This is a new integration test in taskcluster-proxy that checks that a private artifact can be retrieved.

Note, the test fails without the taskcluster-client-go patch with the same error that :nalexander had, I believe (see the PR for the output).

When applying the fix in taskcluster-client-go, the test passes.

Comment 7

[[github robot]]

Commit pushed to master at https://github.com/taskcluster/taskcluster-proxy

https://github.com/taskcluster/taskcluster-proxy/commit/3ca831ff3f698fa3cb8df67c9aa51080b2b80533
Bug 1405889 - Add unit test for retrieval of a private artifact (#28)

Comment 8

[[github robot]]

Commits pushed to master at https://github.com/taskcluster/docker-worker

https://github.com/taskcluster/docker-worker/commit/af6037836b76b025bbd83e00b608ee208bfe91bd
Bug 1405889 - Upgrade taskcluster-proxy from 4.0.0 to 4.0.1

https://github.com/taskcluster/docker-worker/commit/e165850227b81e86f46ebaf983b1256ca5ff1e52
Merge pull request #325 from taskcluster/bug1405889

Bug 1405889 - Upgrade taskcluster-proxy from 4.0.0 to 4.0.1

Comment 9

[Greg Arndt [:garndt]]

Docker worker change has been merged, but not deployed.  Deployment can happen Monday morning.

Comment 10

[Greg Arndt [:garndt]]

Docker worker changes were rolled out yesterday to use the new proxy (which fixes problem A in the description, but not problem B).  Any new docker-worker instance will be using this new proxy.

Comment 11

[Chris Cooper [:coop] (he/him)]

Found in triage.

Pete: are you still working on this? What's left to do?

Comment 12

[Pete Moore [:pmoore][:pete]]

(In reply to Chris Cooper [:coop] from comment #11)
> Found in triage.
> 
> Pete: are you still working on this? What's left to do?

All done!