1381613 - Populate a shared index route for "trunk" repos

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Gregory Szorc [:gps]]

Currently, TC Index routes have the prefix "index.gecko.v2.{project}." where {project} is the repository name. e.g. "mozilla-central" or "autoland."

The fact that the repository is in the index path is kind of a kludge. In theory, we should be able to index based on revision and if there are variations between different repositories or branches/heads within a repository, those should be denoted by different task names. e.g. a build with a beta configuration should not only vary by the {project}/repo component of the index: it should vary by the task name.

Unfortunately, we can't universally remove the repository component of the index path just yet because we don't yet vary the job names across repositories. However, for the special case of the "trunk" repositories - {mozilla-central, autoland, mozilla-inbound} - we should be able to store results for these in a central/shared index path.

A big advantage of having a repository-less index path is that when searching for data attached to a revision you only need to make 1 HTTP request instead of N. This matters for things like artifact builds. Artifact builds walk the commit ancestry then look at multiple repositories in the index. It would be simpler and faster to just look at 1 thing: the revision.

Comment 1

[Gregory Szorc [:gps]]

Attachment: Bug 1381613 - Populate a repository-less index route for "trunk" repos;

There are a few places where we walk commit ancestry looking for things
attached to a specific revision. Because the repository name is attached
to the index path and because a revision can exist in multiple
repositories, we often have to perform N index lookups to find a result
for a specific revision. This is inefficient.

To facilitate faster index lookups by revision, we introduce a new route
that doesn't contain the repository name. In theory, we should be able
to do this globally - for all repos. However, the configuration of
tasks can vary significantly by repo. So e.g. a linux64 build on
"central" is sufficiently different from a linux64 build on "beta" or
"release." For that reason, this commit takes the conservative
approach and only defines a shared route for repositories with a similar
configuration: the "trunk" repositories.

Review commit: https://reviewboard.mozilla.org/r/157986/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/157986/

Comment 2

[Gregory Szorc [:gps]]

Comment on attachment 8887208 [details]
Bug 1381613 - Populate a repository-less index route for "trunk" repos;

https://reviewboard.mozilla.org/r/157986/#review163092

::: taskcluster/taskgraph/transforms/task.py:483
(Diff revision 1)
>  ]
>  
> +# {central, inbound, autoland} write to a "trunk" index prefix. This facilitates
> +# walking of tasks with similar configurations.
> +V2_TRUNK_ROUTE_TEMPLATES = [
> +    "index.gecko.v2.trunk.revision.{head_rev}.{product}.{job-name}",

I assume this will require TC scope changes in order to grant permissions to write to the new index prefix. I'm not sure what all is involved here. I figure Dustin or someone can make those changes as part of granting r+.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8887208 [details]
Bug 1381613 - Populate a repository-less index route for "trunk" repos;

https://reviewboard.mozilla.org/r/157986/#review163596

Looks good, but we need to have some external record (in production-branches.json) of what the "trunk" branches are first.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: Bug 1381613: annotate the 'release' branches;

Review commit: https://reviewboard.mozilla.org/r/158384/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/158384/

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

https://github.com/taskcluster/taskcluster-admin/pull/5

Comment 6

[Gregory Szorc [:gps]]

Comment on attachment 8887523 [details]
Bug 1381613: annotate the 'release' branches;

Not sure what is up with MozReview. But I landed this as https://hg.mozilla.org/build/tools/rev/931bdc862a80.

Comment 7

[Gregory Szorc [:gps]]

I landed things to build/tools and I see your PR has merged. Does anything else need to happen before the Firefox change lands?

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

$ ./tcadmin make-gecko-branch-role mozilla-central mozilla-inbound autoland
scope changes required for role repo:hg.mozilla.org/integration/autoland:*:
+ queue:route:index.gecko.v2.trunk.revision.*
updating role
done
no changes required for role repo:hg.mozilla.org/integration/autoland:cron:nightly-*
scope changes required for role repo:hg.mozilla.org/integration/mozilla-inbound:*:
+ queue:route:index.gecko.v2.trunk.revision.*
updating role
done
no changes required for role repo:hg.mozilla.org/integration/mozilla-inbound:cron:nightly-*
scope changes required for role repo:hg.mozilla.org/mozilla-central:*:
+ queue:route:index.gecko.v2.trunk.revision.*
updating role
done
no changes required for role repo:hg.mozilla.org/mozilla-central:cron:nightly-*

Now it's time :)

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/543c503f745c
Populate a repository-less index route for "trunk" repos; r=dustin

Comment 10

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Backed out for failing flake8 at taskcluster/taskgraph/transforms/task.py:836 with "too many blank lines":

https://hg.mozilla.org/integration/autoland/rev/a1ea69a3a0a760036e047f6ff89897aa7ffd4e53

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=543c503f745c89360398fd3207ddaea5c262a807&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable
Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=115449740&repo=autoland
> TEST-UNEXPECTED-ERROR | /home/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py:836:5 | too many blank lines (2) (E303)

Comment 11

[Gregory Szorc [:gps]]

Will autoland a trivial fixed changeset.

Comment 12

[Gregory Szorc [:gps]]

Comment on attachment 8887208 [details]
Bug 1381613 - Populate a repository-less index route for "trunk" repos;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/157986/diff/1-2/

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/374180b112f3
Populate a repository-less index route for "trunk" repos; r=dustin

Comment 14

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/374180b112f3