1749820 - upload clonebundles to more GCP regions

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, task)

Description

[Connor Sheehan [:sheehan]]

Once bug 1749810 lands, we can upload clonebundles to more GCP regions and get stream-clone bundles served from within the same region to tasks in CI, outside of just us-central1. We will need to know which regions we plan to stand up CI tasks in.

:ahal, I've seen you opening a few bugs related to the GCP migration. Do you know (or could you redirect to someone who does know) which regions we plan to run builds/tests/etc in for GCP? If it's too early to say, that's fine too.

Comment 1

[Andrew Halberstadt [:ahal]]

A bit too early to say. We typically try to have a few in-case there's an outage in any given one. We do have some builds that are currently in us-central1. Though I imagine we'll want to add other regions here as well once they become the default.

Fwiw, most test tasks don't clone Gecko, just the source-test kind. So it'll still be important to provide clonebundles in those regions, it just won't necessarily block migrating the majority of our tests.

Comment 2

[Andrew Halberstadt [:ahal]]

Hey Connor, we're almost ready to start running builds in GCP as tier 1.. If you haven't already, could you also upload a clonebundle to the us-west1 region? I think we'll stick with that and us-central1 for now and possible expand to a third region later (if needed).

I'm not sure if you also need to know which zones within a region will be used, but if so lmk.

Comment 3

[Connor Sheehan [:sheehan]]

(In reply to Andrew Halberstadt [:ahal] from comment #2)

Hey Connor, we're almost ready to start running builds in GCP as tier 1.. If you haven't already, could you also upload a clonebundle to the us-west1 region? I think we'll stick with that and us-central1 for now and possible expand to a third region later (if needed).

I'm not sure if you also need to know which zones within a region will be used, but if so lmk.

Sure thing, I'll add us-west1 to the set of GCP regions for clonebundle storage. I don't think zones are required but I'll double check that just to be certain.

Comment 4

[Connor Sheehan [:sheehan]]

Attachment: bundles: add bundles to new `us-west1` GCS bucket (Bug 1749820) r?mhentges,zeid

Adds a new us-west1 Google cloud storage bucket to Terraform with
the same configuration as the us-central1 bucket, and adds an
entry to the list of GCS buckets in generate_hg_s3_bundles.py
so bundles are added to the new bucket on the next iteration.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/4351c386f60d
bundles: add bundles to new us-west1 GCS bucket r=mhentges

Comment 6

[Connor Sheehan [:sheehan]]

This is deployed now, and the bundles should be uploaded to the new bucket by tomorrow.

If we need bundles in a third region we can open a new bug.

Comment 7

[Andrew Halberstadt [:ahal]]

Looks like we're getting failures when builds run in us-west1:
https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=b-gcp&revision=1da0a4e92abb07452653c6f23d04ee396dc5deb0

Connor, do you know if we just need to wait a bit longer for the clonebundle to upload? Or is something else going on here.

Comment 8

[Andrew Halberstadt [:ahal]]

Here's the relevant log:

[vcs 2022-03-17T16:53:28.280Z] executing ['hg', 'robustcheckout', '--sharebase', '/builds/worker/checkouts/hg-store', '--purge', '--config', 'hostsecurity.hg.mozilla.org:fingerprints=sha256:4D:EB:21:6E:35:2F:99:C6:8F:C3:47:9B:57:B8:6C:17:15:8F:86:09:D4:6C:17:1D:87:B0:DE:F9:0E:51:70:FC,sha256:FF:E7:8D:93:E9:56:3C:C0:19:FC:00:4C:18:B9:86:E5:08:E5:10:F5:E2:EA:48:E8:22:D3:A3:3A:CA:99:C3:4C', '--upstream', 'https://hg.mozilla.org/mozilla-unified', '--revision', '1da0a4e92abb07452653c6f23d04ee396dc5deb0', 'https://hg.mozilla.org/mozilla-central', '/builds/worker/checkouts/gecko']
[vcs 2022-03-17T16:53:28.394Z] (using Mercurial 5.8.1)
[vcs 2022-03-17T16:53:28.394Z] ensuring https://hg.mozilla.org/mozilla-central@1da0a4e92abb07452653c6f23d04ee396dc5deb0 is available at /builds/worker/checkouts/gecko
[vcs 2022-03-17T16:53:28.968Z] (cloning from upstream repo https://hg.mozilla.org/mozilla-unified)
[vcs 2022-03-17T16:53:29.052Z] (sharing from new pooled repository 8ba995b74e18334ab3707f27e9eb8f4e37ba3d29)
[vcs 2022-03-17T16:53:30.218Z] applying clone bundle from https://storage.googleapis.com/moz-hg-bundles-gcp-us-west1/mozilla-unified/5fcb88dc0e4a7369d36bac6ba275746760a4aa26.stream-v2.hg
[vcs 2022-03-17T16:53:30.288Z] HTTP error fetching bundle: HTTP Error 403: Forbidden

Comment 9

[Andrew Halberstadt [:ahal]]

Looks like it might have been fixed here:
https://bugzilla.mozilla.org/show_bug.cgi?id=1760052#c2

Comment 10

[Connor Sheehan [:sheehan]]

(In reply to Andrew Halberstadt [:ahal] from comment #9)

Looks like it might have been fixed here:
https://bugzilla.mozilla.org/show_bug.cgi?id=1760052#c2

This should be fixed. Filed bug 1760180 to track having the public ACL in Terraform.