Closed Bug 1381613 Opened 2 years ago Closed 2 years ago

Populate a shared index route for "trunk" repos

Categories

(Firefox Build System :: Task Configuration, task)

task
Not set

Tracking

(Not tracked)

RESOLVED FIXED
mozilla56

People

(Reporter: gps, Assigned: gps)

References

(Blocks 1 open bug)

Details

Attachments

(2 files)

Currently, TC Index routes have the prefix "index.gecko.v2.{project}." where {project} is the repository name. e.g. "mozilla-central" or "autoland."

The fact that the repository is in the index path is kind of a kludge. In theory, we should be able to index based on revision and if there are variations between different repositories or branches/heads within a repository, those should be denoted by different task names. e.g. a build with a beta configuration should not only vary by the {project}/repo component of the index: it should vary by the task name.

Unfortunately, we can't universally remove the repository component of the index path just yet because we don't yet vary the job names across repositories. However, for the special case of the "trunk" repositories - {mozilla-central, autoland, mozilla-inbound} - we should be able to store results for these in a central/shared index path.

A big advantage of having a repository-less index path is that when searching for data attached to a revision you only need to make 1 HTTP request instead of N. This matters for things like artifact builds. Artifact builds walk the commit ancestry then look at multiple repositories in the index. It would be simpler and faster to just look at 1 thing: the revision.
Comment on attachment 8887208 [details]
Bug 1381613 - Populate a repository-less index route for "trunk" repos;

https://reviewboard.mozilla.org/r/157986/#review163092

::: taskcluster/taskgraph/transforms/task.py:483
(Diff revision 1)
>  ]
>  
> +# {central, inbound, autoland} write to a "trunk" index prefix. This facilitates
> +# walking of tasks with similar configurations.
> +V2_TRUNK_ROUTE_TEMPLATES = [
> +    "index.gecko.v2.trunk.revision.{head_rev}.{product}.{job-name}",

I assume this will require TC scope changes in order to grant permissions to write to the new index prefix. I'm not sure what all is involved here. I figure Dustin or someone can make those changes as part of granting r+.
Comment on attachment 8887208 [details]
Bug 1381613 - Populate a repository-less index route for "trunk" repos;

https://reviewboard.mozilla.org/r/157986/#review163596

Looks good, but we need to have some external record (in production-branches.json) of what the "trunk" branches are first.
Attachment #8887208 - Flags: review?(dustin) → review+
Comment on attachment 8887523 [details]
Bug 1381613: annotate the 'release' branches;

Not sure what is up with MozReview. But I landed this as https://hg.mozilla.org/build/tools/rev/931bdc862a80.
Attachment #8887523 - Flags: review?(gps) → review+
I landed things to build/tools and I see your PR has merged. Does anything else need to happen before the Firefox change lands?
Flags: needinfo?(dustin)
$ ./tcadmin make-gecko-branch-role mozilla-central mozilla-inbound autoland
scope changes required for role repo:hg.mozilla.org/integration/autoland:*:
+ queue:route:index.gecko.v2.trunk.revision.*
updating role
done
no changes required for role repo:hg.mozilla.org/integration/autoland:cron:nightly-*
scope changes required for role repo:hg.mozilla.org/integration/mozilla-inbound:*:
+ queue:route:index.gecko.v2.trunk.revision.*
updating role
done
no changes required for role repo:hg.mozilla.org/integration/mozilla-inbound:cron:nightly-*
scope changes required for role repo:hg.mozilla.org/mozilla-central:*:
+ queue:route:index.gecko.v2.trunk.revision.*
updating role
done
no changes required for role repo:hg.mozilla.org/mozilla-central:cron:nightly-*

Now it's time :)
Flags: needinfo?(dustin)
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/543c503f745c
Populate a repository-less index route for "trunk" repos; r=dustin
Will autoland a trivial fixed changeset.
Flags: needinfo?(gps)
Blocks: 1382507
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/374180b112f3
Populate a repository-less index route for "trunk" repos; r=dustin
https://hg.mozilla.org/mozilla-central/rev/374180b112f3
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla56
Product: TaskCluster → Firefox Build System
You need to log in before you can comment on or make changes to this bug.