1519532 - ClientId is "undefined" when fetching a secret

Status: RESOLVED FIXED

(Taskcluster :: Workers, defect)

Description

[Bianca Danforth [:bdanforth]]

Attachment: taskcluster_raw_log.txt

Steps to Reproduce[1]:

1. For a given GitHub repo (that triggers a Taskcluster task on PR open/reopen/synchronize), obtain permissions from an admin to: create/edit roles for the repo and read/write secrets from/to the repo.
2. Create a new secret in Taskcluster's Secret Manager.
3. Create a new role with a new scope in the Role Manager that allows a task from this repo to read the secret created in step 2.
4. Make the following changes to the repo's .taskcluster.yml file[2]:
   4.1. Add the scope from Step 3 under task.scopes.
   4.2. Add payload.features.taskclusterProxy key with a value of true.
   4.3. Set policy.pullRequests to collaborators.
5. Open/update a PR that attempts to fetch the secret within the task's execution[3].

Expected Results:

The secret is successfully retrieved.

Actual Results:

The secret is not retrieved, and an "Insufficient Scopes" error is logged[4] noting that ClientId is "undefined".

For posterity, I have also attached the full log for this task to this bug; the Insufficient Scopes error occurs at the very bottom of the log file.

[1]: For exact details on secret, role, scope, etc. used, see https://github.com/biancadanforth/taskcluster-integration-poc/pull/9#issue-243515655
[2]: https://github.com/biancadanforth/taskcluster-integration-poc/blob/22b0a4ac1ca5fb3751e7eb4574fef9c614c180d2/.taskcluster.yml
[3]: https://github.com/biancadanforth/taskcluster-integration-poc/blob/22b0a4ac1ca5fb3751e7eb4574fef9c614c180d2/bin/get_secret.js
[4]: https://tools.taskcluster.net/groups/JPpEaTP5Td27VoJM3_JOgw/tasks/JPpEaTP5Td27VoJM3_JOgw/runs/0/logs/public%2Flogs%2Flive.log#L1974

Comment 1

[Pete Moore [:pmoore][:pete]]

Hi Bianca,

It looks like the taskcluster client is calling https://secrets.taskcluster.net/v1/secret/github.com%2Fbiancadanforth%2Ftaskcluster-integration-poc%2Fssh-key directly, rather than the calling the proxy url.

I see the root url is set here:

• https://github.com/biancadanforth/taskcluster-integration-poc/blob/ca964ae655d8606e99ae88c74fdebcb8d103bc69/bin/get_secret.js

And the version of the client being used is 12.2.0:

• https://github.com/biancadanforth/taskcluster-integration-poc/blob/ca964ae655d8606e99ae88c74fdebcb8d103bc69/package.json#L32

It looks like this version of the client doesn't support TASKCLUSTER_PROXY_URL:

• https://github.com/taskcluster/taskcluster-client/blob/v12.2.0/src/client.js

So my guess is that you need to set TASKCLUSTER_ROOT_URL to http://taskcluster in order that the client makes calls through the taskcluster proxy, rather than directly calling the service.

You should be able to do this by changing this line with this diff:

-const secrets = new taskcluster.Secrets({rootUrl: "https://taskcluster.net"});
+const secrets = new taskcluster.Secrets({rootUrl: "http://taskcluster"});

The proxy is hosted in a sidecar docker container pointed to by the locally resolved hostname taskcluster, and the proxy is preconfigured to talk to the production cluster deployed at https://taskcluster.net.

Comment 2

[Bianca Danforth [:bdanforth]]

Thanks :pmoore for the detailed explanation.

I made the change you suggested, and I am now getting some kind of timeout error for each attempt; would you know why?:

  superagent GET http://taskcluster/api/secrets/v1/secret/github.com%2Fbiancadanforth%2Ftaskcluster-integration-poc%2Fssh-key +30s
  taskcluster-client Request error calling get (retrying), err: Error: Timeout of 30000ms exceeded, JSON: Error: Timeout of 30000ms exceeded +30s
  taskcluster-client Calling: get, retry: 3 +808ms
  superagent GET http://taskcluster/api/secrets/v1/secret/github.com%2Fbiancadanforth%2Ftaskcluster-integration-poc%2Fssh-key +31s
  taskcluster-client Request error calling get (retrying), err: Error: Timeout of 30000ms exceeded, JSON: Error: Timeout of 30000ms exceeded +30s
  taskcluster-client Calling: get, retry: 4 +2s
  superagent GET http://taskcluster/api/secrets/v1/secret/github.com%2Fbiancadanforth%2Ftaskcluster-integration-poc%2Fssh-key +32s
  taskcluster-client Request error calling get (retrying), err: Error: Timeout of 30000ms exceeded, JSON: Error: Timeout of 30000ms exceeded +30s
  taskcluster-client Calling: get, retry: 5 +3s
  superagent GET http://taskcluster/api/secrets/v1/secret/github.com%2Fbiancadanforth%2Ftaskcluster-integration-poc%2Fssh-key +33s
  taskcluster-client Request error calling get NOT retrying!, err: Error: Timeout of 30000ms exceeded, JSON: Error: Timeout of 30000ms exceeded +30s
(node:1) UnhandledPromiseRejectionWarning: Error: Timeout of 30000ms exceeded
    at RequestBase._timeoutError (/repo/node_modules/superagent/lib/request-base.js:670:13)
    at Timeout.<anonymous> (/repo/node_modules/superagent/lib/request-base.js:685:12)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10)

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

I think that's bug 1518759, the fix for which hasn't been deployed yet. Hopefully we can get that deployed soon...

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Bianca, did the fix to bug 1518759 fix this for you?

Comment 5

[Bianca Danforth [:bdanforth]]

Yes it did thanks.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

\o/