1749810 - switch IP ranges scraper for GCP from DNS to `cloud.json`

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, task)

Description

[Connor Sheehan [:sheehan]]

GCP now publishes it's customer-accessible IP ranges as a JSON object, similarly to how AWS publishes it's ip-ranges.json file. This new method includes a reference to the specific region the block belongs to, allowing us to serve clonebundles from the same region as the request. At the moment we only serve clonebundles from a single region (us-central1) when we notice any request coming from GCP, so this should make clones faster once we start uploading bundles to more GCP regions.

Let's migrate off the DNS-based IP range scraping and use the cloud.json file.

Comment 1

[Connor Sheehan [:sheehan]]

Attachment: clonebundles: parse gcp IP ranges from `cloud.json` instead of DNS (Bug 1749810) r?mhentges,zeid

This commit moves the GCP clonebundles preference code from using
DNS and assuming all requests are coming from us-central1, to
using cloud.json and serving a region-specific bundle where
appropriate.

We remove the recursive DNS query code from the IP scraper and
replace it with a copy of the AWS IP scraper function, with GCP
specific variables, JSON keys and IP object schema.

We change the hgmo extensions clonebundle manifest parsing
code to read the object as a JSON file conforming to the schema
and pass the parsed region value as the filter, instead of the
hard-coded us-central1. We update the hgrc template
to reflect the new file being .json instead of .txt, and
replace the old format test file with a sample cloud.json.
We also update the GCP IP address in test-clonebundles.t
to an IP within a network in the test file.

Since the DNS code has been removed, dnspython is removed as
a dependency of requirements-tools-py3.

We add a new voluptuous checker to assert all required GCP regions
are present in the JSON file. At this time this only includes the
existing us-central1, with more regions to be added in a separate
bug.

Comment 2

[Connor Sheehan [:sheehan]]

Attachment: requirements: bump versions in `requirements-tools-py3.txt` (Bug 1749810) r?mhentges,zeid

Depends on D135904

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/484841ad8174
clonebundles: parse gcp IP ranges from cloud.json instead of DNS r=mhentges
https://hg.mozilla.org/hgcustom/version-control-tools/rev/ecca42926cac
requirements: bump versions in requirements-tools-py3.txt r=mhentges