1903776 - support multiple branches in projects.yml

Status: RESOLVED FIXED

(Release Engineering :: Firefox-CI Administration, enhancement)

Description

[[e-mail] (:bhearsum)]

At the moment projects.yml assumes a 1:1 relationship between repositories and branches. This has largely held true, but it makes some things impossible. For example, different branches in the same repository having different levels. We're likely to hit other unforseen issues as we migrate Gecko to Github as well.

We should explicitly support multiple branches here.

Comment 1

[[e-mail] (:bhearsum)]

Attachment: WIP: Bug 1903776: Add support for multiple branches per project r?#releng-reviewers!

Comment 2

[[e-mail] (:bhearsum)]

Attachment: Bug 1903776: refactor projects.yml into new format that supports multiple branches r?#releng-reviewers!

Comment 3

[[e-mail] (:bhearsum)]

Attachment: Bug 1903776: refactor some code in grants.py and matching.py to make it more reusable r?#releng-reviewers!

This is prep work for multibranch support, where this code will be needed in places it's currently not accessible.

It also fixes a minor bug where LEVEL_PRIORITIES wasn't respected for PRs and PR actions (due to relying on project.level instead of the overriden local level).

Comment 4

[[e-mail] (:bhearsum)]

Attachment: Bug 1903776: update tests to reflect multiple branches per project r?#releng-reviewers!

Normally I would do this as part of the code changes, but I think it may be easier to review and understand to see this first, which shows off what projects will look like with the new code and schemas.

Comment 5

[[e-mail] (:bhearsum)]

Attachment: Bug 1903776: add support for multiple branches per project r?#releng-reviewers!

My original aim here was to have absolutely no change shown by ci-admin diff, but after a lot of fussing I don't think this is practical.

Besides the changes to some descriptions, everything else is the addition of scopes to non-L1 roles. As best I can tell (and I will be doing some programmatic verification of this), there is no actual change to the expanded scopes of any branch -- these roles are just being directly granted scopes that have been (and still are) being granted to the * role for an entire repository. I think this is a positive, and I intend to follow up with some changes that drop a bunch of scopes for the latter.

Comment 6

[[e-mail] (:bhearsum)]

Attachment: Bug 1903776: update ciadmin checks to support multiple branches per project r?#releng-reviewers!

Note the first hunk which is not technically perfect, but it suits our needs for the moment. I'd prefer to avoid dealing with that complication until we need to, or at the very least, in a later patch.

Comment 7

[[e-mail] (:bhearsum)]

Attachment: Bug 1903776: add support for multiple branches per project r?#releng-reviewers!

I think this is fairly straightforward in the end: the essence of the change is to add branches to the Project class. This has implications for all code that deals with branches, default branches, and levels - most notably, there is no such thing as a level for a project anymore - it is branch specific. (For now, I am replacing project level with "level of project's default branch" in some places, but even that may be worth reconsidering).

Comment 8

[Julien Cristau [:jcristau]]

I was wondering what we're doing with cron. it seems that we support passing a branch to specific hooks, and configuring that in projects.yml (bug 1652501), but for the project-wide, every-15-minute hook, we use project.default_branch? Should we have a hook+role for other branches there (thinking about a future with central+beta+release on the same repo, e.g.)?

Comment 9

[[e-mail] (:bhearsum)]

(In reply to Julien Cristau [:jcristau] from comment #8)

I was wondering what we're doing with cron. it seems that we support passing a branch to specific hooks, and configuring that in projects.yml (bug 1652501), but for the project-wide, every-15-minute hook, we use project.default_branch? Should we have a hook+role for other branches there (thinking about a future with central+beta+release on the same repo, e.g.)?

I think ideally we would have branch specific cron hooks everywhere, yeah. I can tackle that (preferably) in a follow-up, or here if it needs to strictly block this.

Comment 10

[Julien Cristau [:jcristau]]

I don't think it needs to block this, it looks like it should be fairly easy to add if/when we need it.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/39a7d36ba20e
refactor some code in grants.py and matching.py to make it more reusable r=releng-reviewers,jcristau

Comment 12

[[e-mail] (:bhearsum)]

Attachment: Bug 1903776: backout grants.py and matching.py refactor for bustage r?#releng-reviewers!

Eg:

[task 2024-07-04T13:38:08.591Z] 2024-07-04 13:38:08,591 - ERROR - Client ID task-client/SpzL491RR5qZXDLSa10p5g/0/on/us-west1-a/2664258093284268933/until/1720101455.913 does not have sufficient scopes and is missing the following scopes:
[task 2024-07-04T13:38:08.591Z]
[task 2024-07-04T13:38:08.591Z] ```
[task 2024-07-04T13:38:08.591Z] {
[task 2024-07-04T13:38:08.592Z]   "AnyOf": [
[task 2024-07-04T13:38:08.592Z]     "queue:create-task:highest:built-in/succeed",
[task 2024-07-04T13:38:08.592Z]     "queue:create-task:very-high:built-in/succeed",
[task 2024-07-04T13:38:08.592Z]     "queue:create-task:high:built-in/succeed"
[task 2024-07-04T13:38:08.592Z]   ]
[task 2024-07-04T13:38:08.592Z] }
[task 2024-07-04T13:38:08.593Z] ```
[task 2024-07-04T13:38:08.593Z]
[task 2024-07-04T13:38:08.593Z] This request requires the client to satisfy the following scope expression:
[task 2024-07-04T13:38:08.594Z]
[task 2024-07-04T13:38:08.595Z] ```
[task 2024-07-04T13:38:08.595Z] {
[task 2024-07-04T13:38:08.596Z]   "AllOf": [
[task 2024-07-04T13:38:08.596Z]     "queue:route:checks",
[task 2024-07-04T13:38:08.596Z]     "queue:create-task:project:none",
[task 2024-07-04T13:38:08.597Z]     "queue:scheduler-id:scriptworker-level-1",
[task 2024-07-04T13:38:08.597Z]     {
[task 2024-07-04T13:38:08.597Z]       "AnyOf": [
[task 2024-07-04T13:38:08.598Z]         "queue:create-task:highest:built-in/succeed",
[task 2024-07-04T13:38:08.598Z]         "queue:create-task:very-high:built-in/succeed",
[task 2024-07-04T13:38:08.598Z]         "queue:create-task:high:built-in/succeed"
[task 2024-07-04T13:38:08.598Z]       ]
[task 2024-07-04T13:38:08.598Z]     }
[task 2024-07-04T13:38:08.598Z]   ]
[task 2024-07-04T13:38:08.598Z] }
[task 2024-07-04T13:38:08.598Z] ```

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/0f29d5566148
backout grants.py and matching.py refactor for bustage r=releng-reviewers,jcristau

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/a92e4b4918ec
refactor some code in grants.py and matching.py to make it more reusable r=releng-reviewers,jcristau,ahal

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/a285d8d1f187
update tests to reflect multiple branches per project r=releng-reviewers,ahal
https://hg.mozilla.org/ci/ci-configuration/rev/c4bc9e6f704b
add support for multiple branches per project r=releng-reviewers,ahal
https://hg.mozilla.org/ci/ci-configuration/rev/bd66047d060b
update ciadmin checks to support multiple branches per project r=releng-reviewers,ahal
https://hg.mozilla.org/ci/ci-configuration/rev/7c1d805e2a4a
refactor projects.yml into new format that supports multiple branches r=releng-reviewers,ahal

Comment 16

[[e-mail] (:bhearsum)]

Attachment: Bug 1903776: fix comm-esr128 branch configuration to fix scope errors r?#releng-reviewers!

This is causing issues like:

in-tree:hook-action:project-comm/in-tree-action-3-*

This request requires the client to satisfy the following scope expression:

{
  "AllOf": [
    "assume:repo:hg.mozilla.org/releases/comm-esr128:branch:default",
    "queue:route:[e-mail].*",
    "in-tree:hook-action:project-comm/in-tree-action-3-*",
    "index:insert-task:comm.v2.comm-esr128.*",
    "queue:route:tc-treeherder.v2.comm-esr128.5b8f11ce4188f2d46aa8021a390e543e92a23e0d",
    "queue:route:index.comm.v2.comm-esr128.latest.taskgraph.decision",
    "queue:route:index.comm.v2.comm-esr128.revision.5b8f11ce4188f2d46aa8021a390e543e92a23e0d.taskgraph.decision",
    "queue:route:index.comm.v2.comm-esr128.pushlog-id.9.decision",
    "queue:route:[e-mail]-failed",
    "queue:route:[e-mail]-exception",
    "queue:create-task:project:none",
    "queue:scheduler-id:comm-level-3",
    {
      "AnyOf": [
        "queue:create-task:highest:comm-3/decision-gcp",
        "queue:create-task:very-high:comm-3/decision-gcp",
        "queue:create-task:high:comm-3/decision-gcp",
        "queue:create-task:medium:comm-3/decision-gcp",
        "queue:create-task:low:comm-3/decision-gcp",
        "queue:create-task:very-low:comm-3/decision-gcp",
        "queue:create-task:lowest:comm-3/decision-gcp"
      ]
    }
  ]
}

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/3cb40cb69024
fix comm-esr128 branch configuration to fix scope errors r=releng-reviewers,ahal

Comment 18

[BugBot [:suhaib / :marco/ :calixte]]

A patch has been attached on this bug, which was already closed. Filing a separate bug will ensure better tracking. If this was not by mistake and further action is needed, please alert the appropriate party. (Or: if the patch doesn't change behavior -- e.g. landing a test case, or fixing a typo -- then feel free to disregard this message)