1368657 - Insufficient scopes to backfill Talos jobs on mozilla-inbound

Status: RESOLVED FIXED

(Taskcluster :: General, defect)

Description

[Ionuț Goldan [:igoldan]]

Attachment: cannot_backfill.png

When I try to backfill from an existing Talos job, using the "Backfill" button, from the lower console in Treeherder, I get the following popup error:

Taskcluster: You do not have sufficient scopes. This request requires you to have one of the following sets of scopes: [ [ "assume:repo:hg.mozilla.org/integration/mozilla-inbound:*", "queue:route:tc-treeherder.v2.mozilla-inbound.85477a7a8e1a956fd01acbe39ab09a5a5c625e77.90299", "queue:route:tc-treeherder-stage.v2.mozilla-inbound.85477a7a8e1a956fd01acbe39ab09a5a5c625e77.90299" ] ] You only have the scopes: [ "assume:moz-tree:level:1", "assume:mozilla-group:scm_level_1", "assume:mozilla-user:[e-mail]", "assume:project:releng:nightly:level-1:stylo-try", "assume:project:releng:nightly:level-1:try", "assume:project:taskcluster:level-1-sccache-buckets", "assume:project:taskcluster:tutorial", "assume:repo:hg.mozilla.org/incubator/stylo-try:*", "assume:repo:hg.mozilla.org/try-comm-central:*", "assume:repo:hg.mozilla.org/try:*", "auth:aws-s3:read-write:taskcluster-level-1-sccache-eu-central-1/*", "auth:aws-s3:read-write:taskcluster-level-1-sccache-us-east-1/*", "auth:aws-s3:read-write:taskcluster-level-1-sccache-us-west-1/*", "auth:aws-s3:read-write:taskcluster-level-1-sccache-us-west-2/*", "auth:create-client:mozilla-ldap/[e-mail]/*", "auth:delete-client:mozilla-ldap/[e-mail]/*", "auth:reset-access-token:mozilla-ldap/[e-mail]/*", "auth:update-client:mozilla-ldap/[e-mail]/*", "docker-worker:cache:level-1-*", "docker-worker:cache:tooltool-cache", "docker-worker:capability:device:loopbackAudio", "docker-worker:capability:device:loopbackVideo", "docker-worker:capability:privileged", "docker-worker:feature:allowPtrace", "docker-worker:image:quay.io/mozilla/builder:*", "docker-worker:image:quay.io/mozilla/decision:*", "docker-worker:image:taskcluster/builder:*", "docker-worker:image:taskcluster/tester:*", "docker-worker:image:taskclusterprivate/upload_symbols:*", "docker-worker:relengapi-proxy:tooltool.download.internal", "docker-worker:relengapi-proxy:tooltool.download.public", "generic-worker:cache:level-1-*", "generic-worker:os-group:Administrators", "generic-worker:os-group:GenericWorkerTaskUsers", "index:insert-task:buildbot.branches.stylo-try.*", "index:insert-task:buildbot.branches.try.*", "index:insert-task:buildbot.revisions.*", "index:insert-task:docker.images.v1.stylo-try.*", "index:insert-task:docker.images.v1.try.*", "index:insert-task:docker.images.v2.level-1.*", "index:insert-task:garbage.*", "index:insert-task:gecko.cache.level-1.*", "index:insert-task:gecko.v2.stylo-try.*", "index:insert-task:gecko.v2.try.*", "queue:create-task:aws-provisioner-v1/ami-test*", "queue:create-task:aws-provisioner-v1/android-api-*", "queue:create-task:aws-provisioner-v1/b2gbuild*", "queue:create-task:aws-provisioner-v1/b2gtest*", "queue:create-task:aws-provisioner-v1/balrog", "queue:create-task:aws-provisioner-v1/dbg-*", "queue:create-task:aws-provisioner-v1/desktop-test*", "queue:create-task:aws-provisioner-v1/flame-kk*", "queue:create-task:aws-provisioner-v1/gecko-1-*", "queue:create-task:aws-provisioner-v1/gecko-decision", "queue:create-task:aws-provisioner-v1/gecko-images", "queue:create-task:aws-provisioner-v1/gecko-misc", "queue:create-task:aws-provisioner-v1/gecko-symbol-upload", "queue:create-task:aws-provisioner-v1/gecko-t-*", "queue:create-task:aws-provisioner-v1/loan-1-*", "queue:create-task:aws-provisioner-v1/loan-t-*", "queue:create-task:aws-provisioner-v1/mulet-debug", "queue:create-task:aws-provisioner-v1/mulet-opt", "queue:create-task:aws-provisioner-v1/opt-*", "queue:create-task:aws-provisioner-v1/rustbuild", "queue:create-task:aws-provisioner-v1/spidermonkey", "queue:create-task:aws-provisioner-v1/symbol-upload", "queue:create-task:aws-provisioner-v1/taskcluster-images", "queue:create-task:aws-provisioner-v1/tutorial", "queue:create-task:buildbot-bridge/buildbot-bridge", "queue:create-task:dummy-test-provisioner/dummy-test-type", "queue:create-task:gecko-t-tc-worker/*", "queue:create-task:localprovisioner/*", "queue:create-task:null-provisioner/buildbot", "queue:create-task:null-provisioner/buildbot-try", "queue:create-task:packetnet/*", "queue:create-task:releng-hardware/gecko-t-*", "queue:create-task:scl3-puppet/os-x-10-10-gw", "queue:create-task:scl3-puppet/os-x-build-gw", "queue:create-task:tc-worker-provisioner/*", "queue:create-task:test-dummy-provisioner/*", "queue:create-task:very-low:aws-provisioner-v1/ami-test*", "queue:create-task:very-low:aws-provisioner-v1/android-api-*", "queue:create-task:very-low:aws-provisioner-v1/b2gbuild*", "queue:create-task:very-low:aws-provisioner-v1/b2gtest*", "queue:create-task:very-low:aws-provisioner-v1/balrog", "queue:create-task:very-low:aws-provisioner-v1/dbg-*", "queue:create-task:very-low:aws-provisioner-v1/desktop-test*", "queue:create-task:very-low:aws-provisioner-v1/flame-kk*", "queue:create-task:very-low:aws-provisioner-v1/gecko-1-*", "queue:create-task:very-low:aws-provisioner-v1/gecko-decision", "queue:create-task:very-low:aws-provisioner-v1/gecko-images", "queue:create-task:very-low:aws-provisioner-v1/gecko-misc", "queue:create-task:very-low:aws-provisioner-v1/gecko-symbol-upload", "queue:create-task:very-low:aws-provisioner-v1/gecko-t-*", "queue:create-task:very-low:aws-provisioner-v1/loan-1-*", "queue:create-task:very-low:aws-provisioner-v1/loan-t-*", "queue:create-task:very-low:aws-provisioner-v1/mulet-debug", "queue:create-task:very-low:aws-provisioner-v1/mulet-opt", "queue:create-task:very-low:aws-provisioner-v1/opt-*", "queue:create-task:very-low:aws-provisioner-v1/rustbuild", "queue:create-task:very-low:aws-provisioner-v1/spidermonkey", "queue:create-task:very-low:aws-provisioner-v1/symbol-upload", "queue:create-task:very-low:aws-provisioner-v1/taskcluster-images", "queue:create-task:very-low:buildbot-bridge/buildbot-bridge", "queue:create-task:very-low:dummy-test-provisioner/dummy-test-type", "queue:create-task:very-low:gecko-t-tc-worker/*", "queue:create-task:very-low:localprovisioner/*", "queue:create-task:very-low:null-provisioner/buildbot", "queue:create-task:very-low:null-provisioner/buildbot-try", "queue:create-task:very-low:packetnet/*", "queue:create-task:very-low:releng-hardware/gecko-t-*", "queue:create-task:very-low:scl3-puppet/os-x-10-10-gw", "queue:create-task:very-low:scl3-puppet/os-x-build-gw", "queue:create-task:very-low:tc-worker-provisioner/*", "queue:create-task:very-low:test-dummy-provisioner/*", "queue:define-task:aws-provisioner-v1/build-c4-2xlarge", "queue:define-task:aws-provisioner-v1/taskcluster-images", "queue:define-task:aws-provisioner-v1/test-c4-2xlarge", "queue:define-task:dummy-test-provisioner/dummy-test-type", "queue:route:coalesce.v1.builds.stylo-try.*", "queue:route:coalesce.v1.builds.try.*", "queue:route:index.buildbot.branches.stylo-try.*", "queue:route:index.buildbot.branches.try.*", "queue:route:index.buildbot.revisions.*", "queue:route:index.docker.images.v1.stylo-try.*", "queue:route:index.docker.images.v1.try.*", "queue:route:index.docker.images.v2.level-1.*", "queue:route:index.garbage.*", "queue:route:index.gecko.cache.level-1.*", "queue:route:index.gecko.v2.stylo-try.*", "queue:route:index.gecko.v2.try.*", "queue:route:tc-treeherder-stage.stylo-try.*", "queue:route:tc-treeherder-stage.try.*", "queue:route:tc-treeherder-stage.v2.stylo-try.*", "queue:route:tc-treeherder-stage.v2.try.*", "queue:route:tc-treeherder.stylo-try.*", "queue:route:tc-treeherder.try.*", "queue:route:tc-treeherder.v2.stylo-try.*", "queue:route:tc-treeherder.v2.try.*", "queue:scheduler-id:gecko-level-1", "scheduler:create-task-graph", "scheduler:extend-task-graph:*", "secrets:get:garbage/*", "secrets:get:project/releng/gecko/build/level-1/*", "secrets:get:project/taskcluster/gecko/hgfingerprint", "secrets:set:garbage/*" ] In other words you are missing scopes from one of the options: * Option 0: - "assume:repo:hg.mozilla.org/integration/mozilla-inbound:*", and - "queue:route:tc-treeherder.v2.mozilla-inbound.85477a7a8e1a956fd01acbe39ab09a5a5c625e77.90299", and - "queue:route:tc-treeherder-stage.v2.mozilla-inbound.85477a7a8e1a956fd01acbe39ab09a5a5c625e77.90299"

I've never experienced this before. Seems to me that some security configs were updated.

Comment 1

[Ed Morley [:emorley]]

I think this means you just need additional scopes being set, which is a Tasckluster thing rather than a Treeherder thing.

Brian, could you confirm this is expected (re the comment about this scope not being required before) / suggest what scopes (or LDAP groups) need to be set?

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

I have also noticed that osx talos doesn't work for backfill or even retriggers.

We need to ensure Ionut can do backfilling, otherwise we need to turn SETA off :(

Comment 3

[Greg Arndt [:garndt]]

It appears that the login used for backfilling only has level 1 access, whereas those branches require level 3, which includes the necessary scopes that are reported missing:

In other words you are missing scopes from one of the options: * Option 0: - "assume:repo:hg.mozilla.org/integration/mozilla-inbound:*", and - "queue:route:tc-treeherder.v2.mozilla-inbound.85477a7a8e1a956fd01acbe39ab09a5a5c625e77.90299", and - "queue:route:tc-treeherder-stage.v2.mozilla-inbound.85477a7a8e1a956fd01acbe39ab09a5a5c625e77.90299"


regarding retriggering, I was able to on this push but did notice that it took many minutes for it to show up on TH:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=2a5ce0b6f2cd2d6567e307942990329697661ac1&filter-searchStr=talos%20os%20x%20opt%20g&duplicate_jobs=visible&group_state=expanded&selectedJob=103032260

g1, g2, and g4 were the retriggers here.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

ok, maybe it was a temporary issue with osx backfilling and retriggering, glad to see examples of that working.

I need to get Ionut level 3 access then- did this change recently?

Comment 5

[Brian Stack [:bstack]]

Have we decided the issue here is just that ionut does not have level 3 scopes or should I dig deeper?

Comment 6

[Greg Arndt [:garndt]]

(In reply to Joel Maher ( :jmaher) from comment #4)
> ok, maybe it was a temporary issue with osx backfilling and retriggering,
> glad to see examples of that working.
> 
> I need to get Ionut level 3 access then- did this change recently?

Requiring level 3 is not new for backfilling a taskcluster task on a level 3 repo.  One must have the correct level of scopes for the branch they are backfilling on.

Aside from getting the scope issue figured out, have we tried backfilling again to ensure that it works for the OS X jobs?

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

I tried backfilling osx, and got the purple job again:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=osx%20talos&fromchange=c738c30dc6a248508e00d4d53bd909ef03f65be6&tochange=0b3be4de7ac365ed6e79efb4046d013f7d020711

04:36:15     INFO - Can't download from https://queue.taskcluster.net/v1/task/A5DU4lOSSPOf3ScNa1iUHA/artifacts/public/build/firefox-55.0a1.en-US.mac.test_packages.json to /builds/slave/test/build/firefox-55.0a1.en-US.mac.test_packages.json!
04:36:15     INFO - Caught exception: HTTP Error 404: Not Found
04:36:15     INFO - Caught exception: HTTP Error 404: Not Found
04:36:15     INFO - Caught exception: HTTP Error 404: Not Found
04:36:15     INFO - Caught exception: HTTP Error 404: Not Found
04:36:15     INFO - Caught exception: HTTP Error 404: Not Found
04:36:15    FATAL - Failed to download from all available URLs, aborting

I think fixing this specific error will get us going again.

Comment 8

[Bob Owen (:bobowen)]

Attachment: TaskclusterTriggerError.txt

I'm getting a very similar error (attached) when trying to trigger new jobs on try.
Is this a known problem?
I've been able to do this successfully in the past.

Comment 9

[Brian Stack [:bstack]]

bobowen: Do you have multiple ldap accounts? 95% of the time that we see this sort of issue it is because people are using an account that doesn't have the correct level of permissions.

If that's not the case, perhaps somebody changed the scopes needed to trigger those jobs from before when you've done it? 

```
you are missing scopes from one of the options: * Option 0: - "queue:create-task:highest:aws-provisioner-v1/gecko-decision", and - "queue:scheduler-id:gecko-level-1"
```

Comment 10

[Bob Owen (:bobowen)]

(In reply to Brian Stack [:bstack] from comment #9)
> bobowen: Do you have multiple ldap accounts? 95% of the time that we see
> this sort of issue it is because people are using an account that doesn't
> have the correct level of permissions.

Ah yes, it had picked up my mozilla one (or I'd logged in incorrectly by accident) not my contributor one, thanks.

Comment 11

[Ionuț Goldan [:igoldan]]

I'm having this issue again, with the following popup error:

Taskcluster: Supplied credentials do not satisfy authorizedScopes; credentials have scopes [assume:hook-id:garbage/*,assume:mozilla-group:IntranetWiki,assume:mozilla-group:StatsDashboard,assume:mozilla-group:[e-mail],assume:mozilla-group:[e-mail],assume:mozilla-group:[e-mail],assume:mozilla-group:corp-vpn,assume:mozilla-group:egencia_de,assume:mozilla-group:[e-mail],assume:mozilla-group:[e-mail],assume:mozilla-group:irccloud,assume:mozilla-group:[e-mail],assume:mozilla-group:[e-mail],assume:mozilla-group:[e-mail],assume:mozilla-group:okta_mfa,assume:mozilla-group:phonebook_access,assume:mozilla-group:team_moco,assume:mozilla-group:vpn_corp,assume:mozilla-group:vpn_default,assume:mozilla-user:[e-mail],assume:mozillians-user:igoldan,assume:project:taskcluster:tutorial,assume:worker-id:*,auth:create-client:mozilla-ldap/[e-mail]/*,auth:create-role:hook-id:garbage/*,auth:delete-client:mozilla-ldap/[e-mail]/*,auth:delete-role:hook-id:garbage/*,auth:reset-access-token:mozilla-ldap/[e-mail]/*,auth:update-client:mozilla-ldap/[e-mail]/*,auth:update-role:hook-id:garbage/*,hooks:modify-hook:garbage/*,hooks:trigger-hook:garbage/*,queue:create-task:aws-provisioner-v1/b2gtest,queue:create-task:aws-provisioner-v1/tutorial,queue:get-artifact:private/*,queue:rerun-task,queue:resolve-task,queue:route:index.garbage.*,queue:route:notify.email.*,scheduler:create-task-graph,scheduler:extend-task-graph,secrets:get:garbage/*,secrets:set:garbage/*]; authorizedScopes are [assume:repo:hg.mozilla.org/integration/autoland:*,queue:route:[e-mail].*] 

Brian, could you help me with this?

Comment 12

[Ionuț Goldan [:igoldan]]

comment 11 is a false alarm: I logged in with the wrong auth service. Sorry for the ni?