Open Bug 1587958 Opened 3 years ago Updated 6 months ago

stand up CI-only hgweb mirrors in GCP

Categories

(Developer Services :: Mercurial: hg.mozilla.org, task)

task
Not set
normal

Tracking

(Not tracked)

REOPENED

People

(Reporter: sheehan, Assigned: sheehan)

References

(Blocks 2 open bugs)

Details

(Keywords: leave-open)

Attachments

(1 file)

Similar to the work done in bug 1428466, but for GCP.

Fortunately we can re-use most of that work here, so this should be much more turnkey.

Blocks: 1596794

Even with only a small subset of tier 3 builds running in GCP builds, we are already starting to hit quota limits. To me, this indicates that we need to expand into multiple regions like we do in AWS.

If we need help from Google to help figure out how to properly implement and deploy hg services in multiple regions, we should loop in :jbraddock to help find someone at Google to help us. Google has historically been quite responsive to these requests.

Blocks: 1547111

Connor: I don't know what else is on your plate. Will this happen before Q1? Just trying to plan accordingly.

Flags: needinfo?(sheehan)

This work will likely be completed in Q1 next year. I doubt we'll have everything in place by the end of December.

Flags: needinfo?(sheehan)

Previously we referenced the instance data object to determine which
region the request originated from, and assumed AWS was the cloud
we were operating in. Now that we are moving into GCE as well, we
need to differentiate between the different clouds to correctly
serve bundles from those regions.

This commit adds a cloud_region_specifier function which takes a
parsed instance data dict and returns a byte-string that can be used
to filter clonebundles manifests to serve bundles from the same cloud
region as the originating request.

Pushed by cosheehan@mozilla.com:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/60f3b4d51a29
hgmo: filter region specifier by referencing instance data object r=mhentges,smacleod

Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Status: RESOLVED → REOPENED
Keywords: leave-open
Resolution: FIXED → ---

Connor, are hg mirrors set up in GCP? We're planning to start moving CI workers into GCP (again, more than try-ing ;) and I'm wondering if your team will need to do some setup work for us to move a large volume of CI tasks to GCP this year.

Flags: needinfo?(sheehan)

(In reply to Dave House [:dhouse] from comment #6)

Connor, are hg mirrors set up in GCP?

No, not at the moment. :)

We're planning to start moving CI workers into GCP (again, more than try-ing ;) and I'm wondering if your team will need to do some setup work for us to move a large volume of CI tasks to GCP this year.

Yes we will have some things to do in coordination with the migration. We have some tentative plans to try and move hgmo to the cloud this year (another "move to GCP" project that has come up once or twice ;)) so how much setup work I actually need to do will depend on the timeline around that project. I'll follow up with glob on this next week when he returns from holiday PTO.

Flags: needinfo?(sheehan)

Thanks! That's not a problem and could line up well then. I don't think we'll be moving ci workers quickly, and so we should be able to coordinate and move when either the mirrors or cloud! hgmo is ready :)

You need to log in before you can comment on or make changes to this bug.