Closed Bug 1458744 Opened 6 years ago Closed 6 years ago

Missing scopes to start ESR60 releases

Categories

(Release Engineering :: Release Automation: Other, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: nthomas, Unassigned)

References

Details

Attempted to start a prod 60.2.0esr release and hit:

2018-05-02 16:57:17,384 - ERROR - Failed to start release "Firefox-60.0esr-build2". Error(s): You do not have sufficient scopes. You are missing the following scopes:

assume:repo:hg.mozilla.org/releases/mozilla\-esr60:branch:default

You have the scopes:

```
\[
  "assume:hook\-id:project\-releng/candidates\-fennec\-beta",
  "assume:hook\-id:project\-releng/candidates\-fennec\-dev",
  "assume:hook\-id:project\-releng/candidates\-fennec\-release",
  "assume:moz\-tree:level:1:gecko",
  "assume:moz\-tree:level:2:gecko",
  "assume:moz\-tree:level:3",
  "assume:moz\-tree:level:3:gecko",
  "assume:project:releng:branch:gecko:level\-3:birch",
  "assume:project:releng:branch:gecko:level\-3:jamun",
  "assume:project:releng:branch:gecko:level\-3:maple",
  "assume:project:releng:branch:gecko:level\-3:mozilla\-beta",
  "assume:project:releng:branch:gecko:level\-3:mozilla\-release",
  "assume:project:releng:feature:buildbot:gecko:level\-3:birch",
  "assume:project:releng:feature:buildbot:gecko:level\-3:jamun",
  "assume:project:releng:feature:buildbot:gecko:level\-3:maple",
  "assume:project:releng:feature:buildbot:gecko:level\-3:mozilla\-beta",
  "assume:project:releng:feature:buildbot:gecko:level\-3:mozilla\-release",
  "assume:project:releng:feature:taskcluster\-docker\-routes\-v1:gecko:level\-3:birch",
  "assume:project:releng:feature:taskcluster\-docker\-routes\-v1:gecko:level\-3:jamun",
  "assume:project:releng:feature:taskcluster\-docker\-routes\-v1:gecko:level\-3:maple",
  "assume:project:releng:feature:taskcluster\-docker\-routes\-v1:gecko:level\-3:mozilla\-beta",
  "assume:project:releng:feature:taskcluster\-docker\-routes\-v1:gecko:level\-3:mozilla\-release",
  "assume:project:releng:feature:taskcluster\-docker\-routes\-v2:gecko:level\-3:birch",
  "assume:project:releng:feature:taskcluster\-docker\-routes\-v2:gecko:level\-3:jamun",
  "assume:project:releng:feature:taskcluster\-docker\-routes\-v2:gecko:level\-3:maple",
  "assume:project:releng:feature:taskcluster\-docker\-routes\-v2:gecko:level\-3:mozilla\-beta",
  "assume:project:releng:feature:taskcluster\-docker\-routes\-v2:gecko:level\-3:mozilla\-release",
  "assume:project:releng:nightly:level\-3:birch",
  "assume:project:releng:nightly:level\-3:jamun",
  "assume:project:releng:nightly:level\-3:maple",
  "assume:project:releng:nightly:level\-3:mozilla\-beta",
  "assume:project:releng:nightly:level\-3:mozilla\-release",
  "assume:project:releng:push:gecko:level\-3:birch",
  "assume:project:releng:push:gecko:level\-3:jamun",
  "assume:project:releng:push:gecko:level\-3:maple",
  "assume:project:releng:push:gecko:level\-3:mozilla\-beta",
  "assume:project:releng:push:gecko:level\-3:mozilla\-release",
  "assume:project:releng:release:mozilla\-beta",
  "assume:project:releng:release:mozilla\-release",
  "assume:project:taskcluster:gecko:level\-1\-sccache\-buckets",
  "assume:project:taskcluster:gecko:level\-2\-sccache\-buckets",
  "assume:project:taskcluster:gecko:level\-3\-sccache\-buckets",
  "assume:project:taskcluster:level\-1\-sccache\-buckets",
  "assume:project:taskcluster:level\-2\-sccache\-buckets",
  "assume:project:taskcluster:level\-3\-sccache\-buckets",
  "assume:repo:hg.mozilla.org/projects/birch:\*",
  "assume:repo:hg.mozilla.org/projects/jamun:\*",
  "assume:repo:hg.mozilla.org/projects/maple:\*",
  "assume:repo:hg.mozilla.org/releases/mozilla\-beta:\*",
  "assume:repo:hg.mozilla.org/releases/mozilla\-release:\*",
  "auth:aws\-s3:read\-write:public\-qemu\-images/repository/hg.mozilla.org/mozilla\-central/\*",
  "auth:aws\-s3:read\-write:taskcluster\-level\-1\-sccache\-eu\-central\-1/\*",
  "auth:aws\-s3:read\-write:taskcluster\-level\-1\-sccache\-us\-east\-1/\*",
  "auth:aws\-s3:read\-write:taskcluster\-level\-1\-sccache\-us\-west\-1/\*",
  "auth:aws\-s3:read\-write:taskcluster\-level\-1\-sccache\-us\-west\-2/\*",
  "auth:aws\-s3:read\-write:taskcluster\-level\-2\-sccache\-eu\-central\-1/\*",
  "auth:aws\-s3:read\-write:taskcluster\-level\-2\-sccache\-us\-east\-1/\*",
  "auth:aws\-s3:read\-write:taskcluster\-level\-2\-sccache\-us\-west\-1/\*",
  "auth:aws\-s3:read\-write:taskcluster\-level\-2\-sccache\-us\-west\-2/\*",
  "auth:aws\-s3:read\-write:taskcluster\-level\-3\-sccache\-eu\-central\-1/\*",
  "auth:aws\-s3:read\-write:taskcluster\-level\-3\-sccache\-us\-east\-1/\*",
  "auth:aws\-s3:read\-write:taskcluster\-level\-3\-sccache\-us\-west\-1/\*",
  "auth:aws\-s3:read\-write:taskcluster\-level\-3\-sccache\-us\-west\-2/\*",
  "auth:aws\-s3:read\-write:tc\-gp\-private\-1d\-us\-east\-1/releng/mbsdiff\-cache/",
  "docker\-worker:\*",
  "generic\-worker:allow\-rdp:aws\-provisioner\-v1/gecko\-1\-b\-win\*",
  "generic\-worker:allow\-rdp:aws\-provisioner\-v1/gecko\-t\-win\*",
  "generic\-worker:cache:level\-1\-\*",
  "generic\-worker:cache:level\-2\-\*",
  "generic\-worker:cache:level\-3\-\*",
  "generic\-worker:os\-group:\*",
  "in\-tree:hook\-action:project\-gecko/in\-tree\-action\-3\-\*",
  "index:insert\-task:buildbot.branches.birch.\*",
  "index:insert\-task:buildbot.branches.jamun.\*",
  "index:insert\-task:buildbot.branches.maple.\*",
  "index:insert\-task:buildbot.branches.mozilla\-beta.\*",
  "index:insert\-task:buildbot.branches.mozilla\-release.\*",
  "index:insert\-task:buildbot.revisions.\*",
  "index:insert\-task:docker.images.v1.birch.\*",
  "index:insert\-task:docker.images.v1.jamun.\*",
  "index:insert\-task:docker.images.v1.maple.\*",
  "index:insert\-task:docker.images.v1.mozilla\-beta.\*",
  "index:insert\-task:docker.images.v1.mozilla\-release.\*",
  "index:insert\-task:docker.images.v2.level\-1.\*",
  "index:insert\-task:docker.images.v2.level\-2.\*",
  "index:insert\-task:docker.images.v2.level\-3.\*",
  "index:insert\-task:garbage.\*",
  "index:insert\-task:gecko.cache.level\-1.\*",
  "index:insert\-task:gecko.cache.level\-2.\*",
  "index:insert\-task:gecko.cache.level\-3.\*",
  "index:insert\-task:gecko.v2.birch.\*",
  "index:insert\-task:gecko.v2.jamun.\*",
  "index:insert\-task:gecko.v2.maple.\*",
  "index:insert\-task:gecko.v2.mozilla\-beta.\*",
  "index:insert\-task:gecko.v2.mozilla\-release.\*",
  "index:insert\-task:project.releng.funsize.level\-3.\*",
  "index:insert\-task:releases.v1.birch.\*",
  "index:insert\-task:releases.v1.jamun.\*",
  "index:insert\-task:releases.v1.maple.\*",
  "index:insert\-task:releases.v1.mozilla\-beta.\*",
  "index:insert\-task:releases.v1.mozilla\-release.\*",
  "project:releng:addons.mozilla.org:server:production",
  "project:releng:addons.mozilla.org:server:staging",
  "project:releng:balrog:action:\*",
  "project:releng:balrog:channel:\*",
  "project:releng:balrog:server:aurora",
  "project:releng:balrog:server:beta",
  "project:releng:balrog:server:dep",
  "project:releng:balrog:server:nightly",
  "project:releng:balrog:server:release",
  "project:releng:beetmover:action:\*",
  "project:releng:beetmover:bucket:dep",
  "project:releng:beetmover:bucket:dep\-partner",
  "project:releng:beetmover:bucket:nightly",
  "project:releng:beetmover:bucket:partner",
  "project:releng:beetmover:bucket:release",
  "project:releng:beetmover:dep",
  "project:releng:bouncer:action:aliases",
  "project:releng:bouncer:action:submission",
  "project:releng:bouncer:server:production",
  "project:releng:bouncer:server:staging",
  "project:releng:buildbot\-bridge:builder\-name:\*",
  "project:releng:googleplay:aurora",
  "project:releng:googleplay:beta",
  "project:releng:googleplay:dep",
  "project:releng:googleplay:release",
  "project:releng:nightly:level\-3:\*",
  "project:releng:ship\-it:production",
  "project:releng:ship\-it:staging",
  "project:releng:signing:cert:\*",
  "project:releng:signing:format:\*",
  "project:releng:snapcraft:firefox:beta",
  "project:releng:snapcraft:firefox:candidate",
  "project:releng:snapcraft:firefox:mock",
  "project:releng:treescript:action:\*",
  "purge\-cache:aws\-provisioner\-v1/\*",
  "queue:\*",
  "scheduler:\*",
  "secrets:get:garbage/\*",
  "secrets:get:project/releng/gecko/build/level\-1/\*",
  "secrets:get:project/releng/gecko/build/level\-2/\*",
  "secrets:get:project/releng/gecko/build/level\-3/\*",
  "secrets:get:project/releng/snapcraft/firefox/candidate",
  "secrets:get:project/releng/snapcraft/firefox/edge",
  "secrets:get:project/taskcluster/gecko/build/level\-2/\*",
  "secrets:get:project/taskcluster/gecko/build/level\-3/\*",
  "secrets:get:project/taskcluster/gecko/hgfingerprint",
  "secrets:set:garbage/\*",
  "signing:cert:\*",
  "signing:format:\*",
  "worker:privileged:manual\-packet/tc\-worker\-docker\-v0"
\]
```

This request requires you to satisfy this scope expression:

```
{
  "AllOf": \[
    "assume:repo:hg.mozilla.org/releases/mozilla\-esr60:branch:default",
    "queue:route:tc\-treeherder.v2.mozilla\-esr60.fcf3e11f9f8b39f42e791b442dbf0d9c7401915f.16",
    "queue:route:index.gecko.v2.mozilla\-esr60.pushlog\-id.16.actions.RNrWROS9RDmaCane1v\_YfA",
    {
      "AnyOf": \[
        {
          "AllOf": \[
            "queue:scheduler\-id:gecko\-level\-3",
            {
              "AnyOf": \[
                "queue:create\-task:highest:aws\-provisioner\-v1/gecko\-3\-decision",
                "queue:create\-task:very\-high:aws\-provisioner\-v1/gecko\-3\-decision",
                "queue:create\-task:high:aws\-provisioner\-v1/gecko\-3\-decision",
                "queue:create\-task:medium:aws\-provisioner\-v1/gecko\-3\-decision",
                "queue:create\-task:low:aws\-provisioner\-v1/gecko\-3\-decision",
                "queue:create\-task:very\-low:aws\-provisioner\-v1/gecko\-3\-decision",
                "queue:create\-task:lowest:aws\-provisioner\-v1/gecko\-3\-decision"
              \]
            }
          \]
        },
        {
          "AnyOf": \[
            "queue:create\-task:aws\-provisioner\-v1/gecko\-3\-decision",
            {
              "AllOf": \[
                "queue:define\-task:aws\-provisioner\-v1/gecko\-3\-decision",
                "queue:task\-group\-id:gecko\-level\-3/HspcWNJKRjOD\_3rdBqdFSQ",
                "queue:schedule\-task:gecko\-level\-3/HspcWNJKRjOD\_3rdBqdFSQ/RNrWROS9RDmaCane1v\_YfA"
              \]
            }
          \]
        }
      \]
    }
  \]
}
Added the project:releng:release:mozilla-esr60 role, and added assume:project:releng:release:mozilla-esr60 to the release promotion client.
The release started OK with those changes, thanks. The action hit a missing scope error for project:releng:balrog:server:esr, so I've changed the role to have that instead of project:releng:balrog:server:release.
That was not the correct fix and bombed out build3. We need to adjust the roles of the task running the action instead.
I should say ....
That was not the correct fix and build3 bombed out the same as build2, trying to submit a balrog task. I think we need to adjust the roles of the decision task running the action instead, which only has 
  assume:repo:hg.mozilla.org/releases/mozilla-esr60:branch:default
Summary: Missing scope assume:repo:hg.mozilla.org/releases/mozilla\-esr60:branch:default → Missing scopes to start ESR60 releases
I added "project:releng:balrog:server:esr" to "moz-tree:level:3:*"[1]. I reran the same decision task[2] which passed. Then I cancelled that graph and triggered a new build. We needed one anyway.

Thanks for the investigation!

[1] https://tools.taskcluster.net/auth/roles/moz-tree%3Alevel%3A3%3A*
[2] https://tools.taskcluster.net/groups/HspcWNJKRjOD_3rdBqdFSQ/tasks/VtwRzO8vQSmsbpFgJXGpqQ/runs/1
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.