1587958 - stand up CI-only hgweb mirrors in GCP

Status: REOPENED

(Developer Services :: Mercurial: hg.mozilla.org, task)

Description

[Connor Sheehan [:sheehan]]

Similar to the work done in bug 1428466, but for GCP.

Fortunately we can re-use most of that work here, so this should be much more turnkey.

Comment 1

[Chris Cooper [:coop] (he/him)]

Even with only a small subset of tier 3 builds running in GCP builds, we are already starting to hit quota limits. To me, this indicates that we need to expand into multiple regions like we do in AWS.

If we need help from Google to help figure out how to properly implement and deploy hg services in multiple regions, we should loop in :jbraddock to help find someone at Google to help us. Google has historically been quite responsive to these requests.

Comment 2

[Chris Cooper [:coop] (he/him)]

Connor: I don't know what else is on your plate. Will this happen before Q1? Just trying to plan accordingly.

Comment 3

[Connor Sheehan [:sheehan]]

This work will likely be completed in Q1 next year. I doubt we'll have everything in place by the end of December.

Comment 4

[Connor Sheehan [:sheehan]]

Attachment: hgmo: filter region specifier by referencing instance data object (Bug 1587958) r?mhentges,smacleod,zeid

Previously we referenced the instance data object to determine which
region the request originated from, and assumed AWS was the cloud
we were operating in. Now that we are moving into GCE as well, we
need to differentiate between the different clouds to correctly
serve bundles from those regions.

This commit adds a cloud_region_specifier function which takes a
parsed instance data dict and returns a byte-string that can be used
to filter clonebundles manifests to serve bundles from the same cloud
region as the originating request.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/60f3b4d51a29
hgmo: filter region specifier by referencing instance data object r=mhentges,smacleod

Comment 6

[:dhouse]

Connor, are hg mirrors set up in GCP? We're planning to start moving CI workers into GCP (again, more than try-ing ;) and I'm wondering if your team will need to do some setup work for us to move a large volume of CI tasks to GCP this year.

Comment 7

[Connor Sheehan [:sheehan]]

(In reply to Dave House [:dhouse] from comment #6)

Connor, are hg mirrors set up in GCP?

No, not at the moment. :)

We're planning to start moving CI workers into GCP (again, more than try-ing ;) and I'm wondering if your team will need to do some setup work for us to move a large volume of CI tasks to GCP this year.

Yes we will have some things to do in coordination with the migration. We have some tentative plans to try and move hgmo to the cloud this year (another "move to GCP" project that has come up once or twice ;)) so how much setup work I actually need to do will depend on the timeline around that project. I'll follow up with glob on this next week when he returns from holiday PTO.

Comment 8

[:dhouse]

Thanks! That's not a problem and could line up well then. I don't think we'll be moving ci workers quickly, and so we should be able to coordinate and move when either the mirrors or cloud! hgmo is ready :)

Comment 9

[Andrew Halberstadt [:ahal]]

Hey Connor, is this still blocking builds given https://bugzilla.mozilla.org/show_bug.cgi?id=1749820 ?

Comment 10

[Connor Sheehan [:sheehan]]

No, this is not blocking builds. At the time this was filed there was a deficiency in how GCP advertised which IP address blocks belonged to which region. Adding hgweb mirrors to GCP would have worked around that issue, but it has been resolved after bug 1749810. Whether we WONTFIX this bug remains to be seen.