Insufficient scopes to backfill Talos jobs on mozilla-inbound

RESOLVED FIXED

Status

Taskcluster
General
RESOLVED FIXED
11 months ago
7 months ago

People

(Reporter: igoldan, Unassigned)

Tracking

Details

Attachments

(2 attachments)

Created attachment 8872584 [details]
cannot_backfill.png

When I try to backfill from an existing Talos job, using the "Backfill" button, from the lower console in Treeherder, I get the following popup error:

Taskcluster: You do not have sufficient scopes. This request requires you to have one of the following sets of scopes: [ [ "assume:repo:hg.mozilla.org/integration/mozilla-inbound:*", "queue:route:tc-treeherder.v2.mozilla-inbound.85477a7a8e1a956fd01acbe39ab09a5a5c625e77.90299", "queue:route:tc-treeherder-stage.v2.mozilla-inbound.85477a7a8e1a956fd01acbe39ab09a5a5c625e77.90299" ] ] You only have the scopes: [ "assume:moz-tree:level:1", "assume:mozilla-group:scm_level_1", "assume:mozilla-user:ionut.goldan@softvision.ro", "assume:project:releng:nightly:level-1:stylo-try", "assume:project:releng:nightly:level-1:try", "assume:project:taskcluster:level-1-sccache-buckets", "assume:project:taskcluster:tutorial", "assume:repo:hg.mozilla.org/incubator/stylo-try:*", "assume:repo:hg.mozilla.org/try-comm-central:*", "assume:repo:hg.mozilla.org/try:*", "auth:aws-s3:read-write:taskcluster-level-1-sccache-eu-central-1/*", "auth:aws-s3:read-write:taskcluster-level-1-sccache-us-east-1/*", "auth:aws-s3:read-write:taskcluster-level-1-sccache-us-west-1/*", "auth:aws-s3:read-write:taskcluster-level-1-sccache-us-west-2/*", "auth:create-client:mozilla-ldap/ionut.goldan@softvision.ro/*", "auth:delete-client:mozilla-ldap/ionut.goldan@softvision.ro/*", "auth:reset-access-token:mozilla-ldap/ionut.goldan@softvision.ro/*", "auth:update-client:mozilla-ldap/ionut.goldan@softvision.ro/*", "docker-worker:cache:level-1-*", "docker-worker:cache:tooltool-cache", "docker-worker:capability:device:loopbackAudio", "docker-worker:capability:device:loopbackVideo", "docker-worker:capability:privileged", "docker-worker:feature:allowPtrace", "docker-worker:image:quay.io/mozilla/builder:*", "docker-worker:image:quay.io/mozilla/decision:*", "docker-worker:image:taskcluster/builder:*", "docker-worker:image:taskcluster/tester:*", "docker-worker:image:taskclusterprivate/upload_symbols:*", "docker-worker:relengapi-proxy:tooltool.download.internal", "docker-worker:relengapi-proxy:tooltool.download.public", "generic-worker:cache:level-1-*", "generic-worker:os-group:Administrators", "generic-worker:os-group:GenericWorkerTaskUsers", "index:insert-task:buildbot.branches.stylo-try.*", "index:insert-task:buildbot.branches.try.*", "index:insert-task:buildbot.revisions.*", "index:insert-task:docker.images.v1.stylo-try.*", "index:insert-task:docker.images.v1.try.*", "index:insert-task:docker.images.v2.level-1.*", "index:insert-task:garbage.*", "index:insert-task:gecko.cache.level-1.*", "index:insert-task:gecko.v2.stylo-try.*", "index:insert-task:gecko.v2.try.*", "queue:create-task:aws-provisioner-v1/ami-test*", "queue:create-task:aws-provisioner-v1/android-api-*", "queue:create-task:aws-provisioner-v1/b2gbuild*", "queue:create-task:aws-provisioner-v1/b2gtest*", "queue:create-task:aws-provisioner-v1/balrog", "queue:create-task:aws-provisioner-v1/dbg-*", "queue:create-task:aws-provisioner-v1/desktop-test*", "queue:create-task:aws-provisioner-v1/flame-kk*", "queue:create-task:aws-provisioner-v1/gecko-1-*", "queue:create-task:aws-provisioner-v1/gecko-decision", "queue:create-task:aws-provisioner-v1/gecko-images", "queue:create-task:aws-provisioner-v1/gecko-misc", "queue:create-task:aws-provisioner-v1/gecko-symbol-upload", "queue:create-task:aws-provisioner-v1/gecko-t-*", "queue:create-task:aws-provisioner-v1/loan-1-*", "queue:create-task:aws-provisioner-v1/loan-t-*", "queue:create-task:aws-provisioner-v1/mulet-debug", "queue:create-task:aws-provisioner-v1/mulet-opt", "queue:create-task:aws-provisioner-v1/opt-*", "queue:create-task:aws-provisioner-v1/rustbuild", "queue:create-task:aws-provisioner-v1/spidermonkey", "queue:create-task:aws-provisioner-v1/symbol-upload", "queue:create-task:aws-provisioner-v1/taskcluster-images", "queue:create-task:aws-provisioner-v1/tutorial", "queue:create-task:buildbot-bridge/buildbot-bridge", "queue:create-task:dummy-test-provisioner/dummy-test-type", "queue:create-task:gecko-t-tc-worker/*", "queue:create-task:localprovisioner/*", "queue:create-task:null-provisioner/buildbot", "queue:create-task:null-provisioner/buildbot-try", "queue:create-task:packetnet/*", "queue:create-task:releng-hardware/gecko-t-*", "queue:create-task:scl3-puppet/os-x-10-10-gw", "queue:create-task:scl3-puppet/os-x-build-gw", "queue:create-task:tc-worker-provisioner/*", "queue:create-task:test-dummy-provisioner/*", "queue:create-task:very-low:aws-provisioner-v1/ami-test*", "queue:create-task:very-low:aws-provisioner-v1/android-api-*", "queue:create-task:very-low:aws-provisioner-v1/b2gbuild*", "queue:create-task:very-low:aws-provisioner-v1/b2gtest*", "queue:create-task:very-low:aws-provisioner-v1/balrog", "queue:create-task:very-low:aws-provisioner-v1/dbg-*", "queue:create-task:very-low:aws-provisioner-v1/desktop-test*", "queue:create-task:very-low:aws-provisioner-v1/flame-kk*", "queue:create-task:very-low:aws-provisioner-v1/gecko-1-*", "queue:create-task:very-low:aws-provisioner-v1/gecko-decision", "queue:create-task:very-low:aws-provisioner-v1/gecko-images", "queue:create-task:very-low:aws-provisioner-v1/gecko-misc", "queue:create-task:very-low:aws-provisioner-v1/gecko-symbol-upload", "queue:create-task:very-low:aws-provisioner-v1/gecko-t-*", "queue:create-task:very-low:aws-provisioner-v1/loan-1-*", "queue:create-task:very-low:aws-provisioner-v1/loan-t-*", "queue:create-task:very-low:aws-provisioner-v1/mulet-debug", "queue:create-task:very-low:aws-provisioner-v1/mulet-opt", "queue:create-task:very-low:aws-provisioner-v1/opt-*", "queue:create-task:very-low:aws-provisioner-v1/rustbuild", "queue:create-task:very-low:aws-provisioner-v1/spidermonkey", "queue:create-task:very-low:aws-provisioner-v1/symbol-upload", "queue:create-task:very-low:aws-provisioner-v1/taskcluster-images", "queue:create-task:very-low:buildbot-bridge/buildbot-bridge", "queue:create-task:very-low:dummy-test-provisioner/dummy-test-type", "queue:create-task:very-low:gecko-t-tc-worker/*", "queue:create-task:very-low:localprovisioner/*", "queue:create-task:very-low:null-provisioner/buildbot", "queue:create-task:very-low:null-provisioner/buildbot-try", "queue:create-task:very-low:packetnet/*", "queue:create-task:very-low:releng-hardware/gecko-t-*", "queue:create-task:very-low:scl3-puppet/os-x-10-10-gw", "queue:create-task:very-low:scl3-puppet/os-x-build-gw", "queue:create-task:very-low:tc-worker-provisioner/*", "queue:create-task:very-low:test-dummy-provisioner/*", "queue:define-task:aws-provisioner-v1/build-c4-2xlarge", "queue:define-task:aws-provisioner-v1/taskcluster-images", "queue:define-task:aws-provisioner-v1/test-c4-2xlarge", "queue:define-task:dummy-test-provisioner/dummy-test-type", "queue:route:coalesce.v1.builds.stylo-try.*", "queue:route:coalesce.v1.builds.try.*", "queue:route:index.buildbot.branches.stylo-try.*", "queue:route:index.buildbot.branches.try.*", "queue:route:index.buildbot.revisions.*", "queue:route:index.docker.images.v1.stylo-try.*", "queue:route:index.docker.images.v1.try.*", "queue:route:index.docker.images.v2.level-1.*", "queue:route:index.garbage.*", "queue:route:index.gecko.cache.level-1.*", "queue:route:index.gecko.v2.stylo-try.*", "queue:route:index.gecko.v2.try.*", "queue:route:tc-treeherder-stage.stylo-try.*", "queue:route:tc-treeherder-stage.try.*", "queue:route:tc-treeherder-stage.v2.stylo-try.*", "queue:route:tc-treeherder-stage.v2.try.*", "queue:route:tc-treeherder.stylo-try.*", "queue:route:tc-treeherder.try.*", "queue:route:tc-treeherder.v2.stylo-try.*", "queue:route:tc-treeherder.v2.try.*", "queue:scheduler-id:gecko-level-1", "scheduler:create-task-graph", "scheduler:extend-task-graph:*", "secrets:get:garbage/*", "secrets:get:project/releng/gecko/build/level-1/*", "secrets:get:project/taskcluster/gecko/hgfingerprint", "secrets:set:garbage/*" ] In other words you are missing scopes from one of the options: * Option 0: - "assume:repo:hg.mozilla.org/integration/mozilla-inbound:*", and - "queue:route:tc-treeherder.v2.mozilla-inbound.85477a7a8e1a956fd01acbe39ab09a5a5c625e77.90299", and - "queue:route:tc-treeherder-stage.v2.mozilla-inbound.85477a7a8e1a956fd01acbe39ab09a5a5c625e77.90299"

I've never experienced this before. Seems to me that some security configs were updated.

Comment 1

11 months ago
I think this means you just need additional scopes being set, which is a Tasckluster thing rather than a Treeherder thing.

Brian, could you confirm this is expected (re the comment about this scope not being required before) / suggest what scopes (or LDAP groups) need to be set?
Component: Treeherder: Infrastructure → General
Flags: needinfo?(bstack)
Product: Tree Management → Taskcluster
Summary: Cannot backfill Talos data points → Insufficient scopes to backfill Talos jobs on mozilla-inbound
Version: --- → unspecified
I have also noticed that osx talos doesn't work for backfill or even retriggers.

We need to ensure Ionut can do backfilling, otherwise we need to turn SETA off :(

Comment 3

11 months ago
It appears that the login used for backfilling only has level 1 access, whereas those branches require level 3, which includes the necessary scopes that are reported missing:

In other words you are missing scopes from one of the options: * Option 0: - "assume:repo:hg.mozilla.org/integration/mozilla-inbound:*", and - "queue:route:tc-treeherder.v2.mozilla-inbound.85477a7a8e1a956fd01acbe39ab09a5a5c625e77.90299", and - "queue:route:tc-treeherder-stage.v2.mozilla-inbound.85477a7a8e1a956fd01acbe39ab09a5a5c625e77.90299"


regarding retriggering, I was able to on this push but did notice that it took many minutes for it to show up on TH:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=2a5ce0b6f2cd2d6567e307942990329697661ac1&filter-searchStr=talos%20os%20x%20opt%20g&duplicate_jobs=visible&group_state=expanded&selectedJob=103032260

g1, g2, and g4 were the retriggers here.
ok, maybe it was a temporary issue with osx backfilling and retriggering, glad to see examples of that working.

I need to get Ionut level 3 access then- did this change recently?

Comment 5

11 months ago
Have we decided the issue here is just that ionut does not have level 3 scopes or should I dig deeper?
Flags: needinfo?(bstack)

Comment 6

11 months ago
(In reply to Joel Maher ( :jmaher) from comment #4)
> ok, maybe it was a temporary issue with osx backfilling and retriggering,
> glad to see examples of that working.
> 
> I need to get Ionut level 3 access then- did this change recently?

Requiring level 3 is not new for backfilling a taskcluster task on a level 3 repo.  One must have the correct level of scopes for the branch they are backfilling on.

Aside from getting the scope issue figured out, have we tried backfilling again to ensure that it works for the OS X jobs?
Flags: needinfo?(jmaher)
I tried backfilling osx, and got the purple job again:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=osx%20talos&fromchange=c738c30dc6a248508e00d4d53bd909ef03f65be6&tochange=0b3be4de7ac365ed6e79efb4046d013f7d020711

04:36:15     INFO - Can't download from https://queue.taskcluster.net/v1/task/A5DU4lOSSPOf3ScNa1iUHA/artifacts/public/build/firefox-55.0a1.en-US.mac.test_packages.json to /builds/slave/test/build/firefox-55.0a1.en-US.mac.test_packages.json!
04:36:15     INFO - Caught exception: HTTP Error 404: Not Found
04:36:15     INFO - Caught exception: HTTP Error 404: Not Found
04:36:15     INFO - Caught exception: HTTP Error 404: Not Found
04:36:15     INFO - Caught exception: HTTP Error 404: Not Found
04:36:15     INFO - Caught exception: HTTP Error 404: Not Found
04:36:15    FATAL - Failed to download from all available URLs, aborting

I think fixing this specific error will get us going again.
Flags: needinfo?(jmaher)

Comment 8

9 months ago
Created attachment 8886943 [details]
TaskclusterTriggerError.txt

I'm getting a very similar error (attached) when trying to trigger new jobs on try.
Is this a known problem?
I've been able to do this successfully in the past.
Flags: needinfo?(bstack)

Comment 9

9 months ago
bobowen: Do you have multiple ldap accounts? 95% of the time that we see this sort of issue it is because people are using an account that doesn't have the correct level of permissions.

If that's not the case, perhaps somebody changed the scopes needed to trigger those jobs from before when you've done it? 

```
you are missing scopes from one of the options: * Option 0: - "queue:create-task:highest:aws-provisioner-v1/gecko-decision", and - "queue:scheduler-id:gecko-level-1"
```
Flags: needinfo?(bstack) → needinfo?(bobowencode)

Comment 10

9 months ago
(In reply to Brian Stack [:bstack] from comment #9)
> bobowen: Do you have multiple ldap accounts? 95% of the time that we see
> this sort of issue it is because people are using an account that doesn't
> have the correct level of permissions.

Ah yes, it had picked up my mozilla one (or I'd logged in incorrectly by accident) not my contributor one, thanks.
Flags: needinfo?(bobowencode)
I'm having this issue again, with the following popup error:

Taskcluster: Supplied credentials do not satisfy authorizedScopes; credentials have scopes [assume:hook-id:garbage/*,assume:mozilla-group:IntranetWiki,assume:mozilla-group:StatsDashboard,assume:mozilla-group:all-moco-mofo@mozilla.com,assume:mozilla-group:all-moco@mozilla.com,assume:mozilla-group:corp-contractors@mozilla.com,assume:mozilla-group:corp-vpn,assume:mozilla-group:egencia_de,assume:mozilla-group:eu-corp-contractors@mozilla.com,assume:mozilla-group:eu@mozilla.com,assume:mozilla-group:irccloud,assume:mozilla-group:irccloud-users@mozilla.com,assume:mozilla-group:jmaher-directs@mozilla.com,assume:mozilla-group:moztravel-de@mozilla.com,assume:mozilla-group:okta_mfa,assume:mozilla-group:phonebook_access,assume:mozilla-group:team_moco,assume:mozilla-group:vpn_corp,assume:mozilla-group:vpn_default,assume:mozilla-user:igoldan@mozilla.com,assume:mozillians-user:igoldan,assume:project:taskcluster:tutorial,assume:worker-id:*,auth:create-client:mozilla-ldap/igoldan@mozilla.com/*,auth:create-role:hook-id:garbage/*,auth:delete-client:mozilla-ldap/igoldan@mozilla.com/*,auth:delete-role:hook-id:garbage/*,auth:reset-access-token:mozilla-ldap/igoldan@mozilla.com/*,auth:update-client:mozilla-ldap/igoldan@mozilla.com/*,auth:update-role:hook-id:garbage/*,hooks:modify-hook:garbage/*,hooks:trigger-hook:garbage/*,queue:create-task:aws-provisioner-v1/b2gtest,queue:create-task:aws-provisioner-v1/tutorial,queue:get-artifact:private/*,queue:rerun-task,queue:resolve-task,queue:route:index.garbage.*,queue:route:notify.email.*,scheduler:create-task-graph,scheduler:extend-task-graph,secrets:get:garbage/*,secrets:set:garbage/*]; authorizedScopes are [assume:repo:hg.mozilla.org/integration/autoland:*,queue:route:notify.email.kgupta@mozilla.com.*] 

Brian, could you help me with this?
Flags: needinfo?(bstack)
comment 11 is a false alarm: I logged in with the wrong auth service. Sorry for the ni?
Flags: needinfo?(bstack)
Status: NEW → RESOLVED
Last Resolved: 7 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.