1534463 - Patch `run-task` to clone from private hgweb mirrors

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Connor Sheehan [:sheehan]]

Private hgweb mirrors available to Firefox CI are almost ready to roll out to production in us-west-2. The largest source of traffic to hg.mo from CI is the initial checkout of mozilla-central by the run-task driver. We would like to do an incremental rollout for some of that traffic to the private mirrors and monitor performance, increasing the traffic as evidence of it's stability becomes apparent. To do so, we will upload an artifact somewhere (such as a Taskcluster secret, Amazon S3, etc) that indicates the percentage of traffic we want switched over to the private instances and generate a random number that determines if we should clone from the private instances or from the public hg.mo service.

I am writing a patch for this bug that currently changes run-task. robustcheckout is another possible candidate for this change, however run-task already handles logic for finding the base/head repository from environment variables and running the clone. My patch will intercept the function that resolves these variables (collect_vcs_options) and swap the URL to the internal mirror when appropriate.

Comment 1

[Tom Prince [:tomprince]]

I think run-task is a good place for this.

Comment 2

[Mike Hommey [:glandium]]

Isn't this something that could be handled at the DNS level with weighted round robin?

Comment 3

[Connor Sheehan [:sheehan]]

(In reply to Mike Hommey [:glandium] from comment #2)

Isn't this something that could be handled at the DNS level with weighted round robin?

We did consider that, but ruled out anything resembling "split horizon" DNS to avoid debugging issues that aren't reproducible in local environments (ie "my clone fails in CI but works fine from my local machine"), among other things.

Comment 4

[Connor Sheehan [:sheehan]]

Attachment: Bug 1534463: grant read access for hgmointernal config to all entities with read access to hg fingerprint r?tomprince

The private hgweb mirror config we intend to store in
a Taskcluster secret will need to be accessed by run-task,
which also access the hg fingerprint secret. Grant read
access to the hgmointernal config to all groups and projects
which have access to the hg fingerprint secret.

Comment 5

[Connor Sheehan [:sheehan]]

Attachment: Bug 1534463: allow `vpn_hg_admin` group to set Mercurial related secrets r?tomprince

Now that we have given scopes to the correct contexts, we need
to provide the relevant parties with the ability to modify
the secrets. This commit grants write access for these secrets
to the vpn_hg_admin group. This group defines the set of
people with SSH access to the hgweb/hgssh cluster behind hgmo.

Comment 6

[Connor Sheehan [:sheehan]]

Attachment: Bug 1534463: give tasks access to `hgmointernal` Taskcluster secret r?tomprince

Now that we have added the necessary scopes to ci-configuration,
we can add the in-tree scopes to give tasks access to the
hgmointernal config Taskcluster secret.

Comment 7

[Connor Sheehan [:sheehan]]

Attachment: Bug 1534463: patch `run-task` to clone from private hgweb mirrors r?tomprince

With tasks able to access the hgmointernal config from a Taskcluster
secret, we can now add functionality to run-task to support checking
out from the private hg service. Here we add add a resolve_checkout_url
function which takes the base/head repository URLs and determines
whether we should clone from the public or private service, returning
the resolved URL. The function pulls down the secret and checks that
the region the task is executing in is in the set of supported regions.
Then we generate a random number and default to the public service if
the number is lower than our "rate". If all the above conditions are
met, we replace hg.mozilla.org with the resolved domain name for the
given region.

We add a call to this function to collect_vcs_options, and skip
resolving the private URL if we aren't performing a checkout from
within run-task.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a498ba7a0992
give tasks access to `hgmointernal` Taskcluster secret r=tomprince
https://hg.mozilla.org/integration/autoland/rev/3595f2b4875f
patch `run-task` to clone from private hgweb mirrors r=tomprince

Comment 9

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/a498ba7a0992
https://hg.mozilla.org/mozilla-central/rev/3595f2b4875f