"Create Interactive Task" on Treeherder for docker workers fails with: "Docker configuration could not be created. This may indicate an authentication error when validating scopes necessary for running the task"
Categories
(Tree Management :: Treeherder, defect)
Tracking
(Not tracked)
People
(Reporter: whimboo, Assigned: jmaher)
Details
Attachments
(1 file)
I'm trying to create an interactive task for one of the Wd jobs on this try build:
As such I'm using the Create Interactive Task entry in the lower left pane. While the create-inter job turns out to be green the created interactive task actually fails to run due to insufficient scopes. Here an example:
https://firefox-ci-tc.services.mozilla.com/tasks/fGS8iApiSfiKZlIknEvryw
From the live log:
[taskcluster:error] Docker configuration could not be created. This may indicate an authentication error when validating scopes necessary for running the task.
Error: Insufficient scopes to run task in privileged mode. Try adding docker-worker:capability:privileged to the .scopes array
at runAsPrivileged (/home/ubuntu/docker-worker/src/lib/task.js:115:11)
at Task.dockerConfig (/home/ubuntu/docker-worker/src/lib/task.js:327:26)
at Task.run (/home/ubuntu/docker-worker/src/lib/task.js:887:33)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (internal/process/task_queues.js:93:5)
at Task.start (/home/ubuntu/docker-worker/src/lib/task.js:700:17)
at TaskListener.runTaskset (/home/ubuntu/docker-worker/src/lib/task_listener.js:524:9)
at async Promise.all (index 0)
[taskcluster 2022-01-13 13:57:48.802Z] Unsuccessful task run with exit code: -1 completed in 0.68 seconds
When I'm going through the TaskCluster UI myself and trigger an interactive task manually it works all fine. Comparing the full task details I can see that a lot of scopes have been removed including the required docker-worker:capability:privileged one.
Scopes of initial job as triggered via mach try:
"scopes": [
"secrets:get:project/taskcluster/gecko/hgfingerprint",
"secrets:get:project/taskcluster/gecko/hgmointernal",
"project:releng:services/tooltool/api/download/public",
"project:releng:services/tooltool/api/download/internal",
"queue:get-artifact:project/gecko/android-emulator/*",
"queue:get-artifact:project/gecko/android-system-images/*",
"docker-worker:feature:allowPtrace",
"docker-worker:capability:device:loopbackVideo",
"docker-worker:capability:privileged",
"docker-worker:cache:gecko-level-1-checkouts-hg58-v3-35e6d2147228a7dd8319",
"docker-worker:cache:gecko-level-1-tooltool-cache-v3-35e6d2147228a7dd8319"
],
Scopes of the interactive task as created by Treeherder:
"scopes": [
"secrets:get:project/taskcluster/gecko/hgfingerprint",
"secrets:get:project/taskcluster/gecko/hgmointernal",
"project:releng:services/tooltool/api/download/public",
"project:releng:services/tooltool/api/download/internal",
"queue:get-artifact:project/gecko/android-emulator/*",
"queue:get-artifact:project/gecko/android-system-images/*",
"docker-worker:feature:allowPtrace",
"docker-worker:capability:device:loopbackVideo"
],
|
Reporter | |
Comment 1•3 years ago
|
||
Hi Joel, is there something we could easily do for Treeherder to make it easier to run an interactive task? Thanks.
|
Assignee | |
Comment 2•3 years ago
|
||
this is a good find. It appears that we use the in-tree actions when triggering an interactive task. Those can be found here:
https://searchfox.org/mozilla-central/source/taskcluster/gecko_taskgraph/actions/create_interactive.py#83
on line 132 in that file there is code to set the scopes:
# filter scopes with the SCOPE_WHITELIST
task.task["scopes"] = [
s
for s in task.task.get("scopes", [])
if any(p.match(s) for p in SCOPE_WHITELIST)
]
you can find SCOPE_WHITELIST in the same file, here it is (with comments removed for simplicity):
SCOPE_WHITELIST = [
re.compile(r"^secrets:get:project/taskcluster/gecko/(hgfingerprint|hgmointernal)$"),
re.compile(r"^docker-worker:relengapi-proxy:tooltool.download.public$"),
re.compile(r"^project:releng:services/tooltool/api/download/public$"),
re.compile(r"^docker-worker:relengapi-proxy:tooltool.download.internal$"),
re.compile(r"^project:releng:services/tooltool/api/download/internal$"),
re.compile(r"^queue:get-artifact:project/gecko/.*$"),
re.compile(r"^secrets:get:project/releng/gecko/build/level-[0-9]/\*"),
re.compile(r"^docker-worker:feature:allowPtrace$"),
re.compile(r"^docker-worker:capability:device:.*$"),
]
the scopes you are missing in the above example are:
"docker-worker:capability:privileged",
"docker-worker:cache:gecko-level-1-checkouts-hg58-v3-35e6d2147228a7dd8319",
"docker-worker:cache:gecko-level-1-tooltool-cache-v3-35e6d2147228a7dd8319"
I think all that is needed is:
re.compile(r"^docker-worker:capability:privileged$"), # this might not be ideal, but if this is what taskcluster does...
re.compile(r"^docker-worker:cache:gecko-level-1-checkouts.*$"), # is this really needed?
re.compile(r"^docker-worker:cache:gecko-level-1-tooltool-cache.*$"), # probably what is really missing
could you play with this in-tree to see if it works for you?
|
|
Reporter | |
Comment 3•3 years ago
|
||
I played with it directly in TaskCluster because that's easier and safes quite an amount of resources. So in regards of all these three entries all of them are needed. Otherwise the task will fail.
I pushed a new try with the above proposed additional whitelist entries. Lets see if it works...
https://treeherder.mozilla.org/jobs?repo=try&revision=fd8ed954d5ca075fc2c0b89d6b25299497b1b956
|
|
Assignee | |
Comment 4•3 years ago
|
||
this seems to be working, did it work for you?
|
|
Reporter | |
Comment 5•3 years ago
|
||
Oh yes, completely forgot to reply. In indeed works for me when applying the above 3 lines. Could you maybe create the final patch and add the appropriate comments to each of the lines? I would have to speculate a bit what I would have to use. Or tell me please and I can do on Monday. Thanks!
|
|
Assignee | |
Comment 6•3 years ago
|
||
|
||
Updated•3 years ago
|
|
|
Assignee | |
Updated•3 years ago
|
|
||
Comment 8•3 years ago
|
||
| bugherder | ||
Description
•