1518394 - Serve stream clone bundles for requests originating from private IP addresses

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, enhancement)

Description

[Connor Sheehan [:sheehan]]

After making a few try pushes that clone mozilla-unified from
a private hgweb instance running in AWS, I noticed that the
clones are zstd-max format (as seen here, on line 29) and not
packed.hg1 (as seen here, line 30). This is because the code that
forces hosts in AWS to receive streamed bundles is checking the
public IP address against all known AWS IP address blocks, when the
private instances are being accessed from the private IP address
space (hence, they do not register as coming from AWS, and the
fast-but-compressed zstd-max bundle is sent instead). We should
add code to the hgmo extension that causes clones from anywhere
in the private IP address space to receive streamed clone bundles,
under the assumption that any clones coming from private hosts
should be in AWS. Alternatively we could add the known Taskcluster
private IP address blocks to the hgmo extension (or have them
somewhere else we can query) and prioritize streamed clone bundles
for requests originating from those blocks.

Comment 1

[Connor Sheehan [:sheehan]]

I did a little digging through mozilla-central and I found this neat
little hack that lets you query the internal Amazon DNS server (at
http://169.254.169.254/latest/meta-data/placement/availability-zone)
and retrieve the AWS region of the current host. We can leverage
this to determine which region a host is living in, and serve bundles
from the appropriate region using that value. It might be worth
running this query out of band on startup, and writing a file to
disk that we can reference on subsequent clones/pulls.

Comment 2

[Connor Sheehan [:sheehan]]

Even better, we can use the cloud-init instance metadata object. This object contains
some of the metadata collected by cloud-init during first boot as is written
to a common location on hosts in different cloud environments. The AWS
region is included in this object.

Comment 3

[Connor Sheehan [:sheehan]]

Attachment: hgmo: don't split manifest into lines to check for `ec2region` parameter (Bug 1518394) r?smacleod

No need to split lines to correctly parse the manifest for
AWS regions. It also makes my next few patches easier.

Comment 4

[Connor Sheehan [:sheehan]]

Attachment: hgmo: move `stream_clone_cmp` to a standalone function (Bug 1518394) r?smacleod

It is a pure function and doesn't need to be a closure.

Comment 5

[Connor Sheehan [:sheehan]]

Attachment: hgmo: move manifest filter and sort into standalone function (Bug 1518394) r?smacleod

With this functionality moved into a separate function we
can re-use it for use with private AWS mirrors.

Comment 6

[Connor Sheehan [:sheehan]]

Attachment: terraform: move `instance_data.json` object to `/var/hg` on bootstrap (Bug 1518394) r?smacleod

This is essentially a hack around a bug in pre 18.4 versions of
cloud-init where the instance_data.json object would not persist
across reboots.

--
[0] https://bugs.launchpad.net/cloud-init/+bug/1791781

Comment 7

[Connor Sheehan [:sheehan]]

Attachment: hgmo: serve stream clone bundles to private IP address on cloud instances (Bug 1518394) r?smacleod

When Taskcluster hosts pull/clone from private hgweb mirrors in CI,
they do so over a VPC peering connection and do not cross the public
internet. As a result, the IP address exposed to the hg process on
the mirror host sees a private IP address as the source location. This
confuses the clonebundles manifest sorting function, as that function
reads a previously fetched object containing AWS IP blocks for Amazon's
public IP address space. Since the private IPs do not appear in the
object, pulling from the mirror results in the Taskcluster hosts
receiving zstd-max compressed bundles, instead of the faster streamed
clone bundles.

This commit teaches the hgmo extension to serve stream clone bundles
when the client IP address is within the private IP address space and
when the `instance_data.json` object is present on the server. The
`instance_data.json` object contains some useful information gathered
about the instance during first run by cloud-init, including the region
of the host mirror. We grab the region from this object and pass it
to `filter_manifest_for_aws_region` to achieve the same behaviour as
if we received a result from the public AWS IP space.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/54de0d1af317
hgmo: don't split manifest into lines to check for ec2region parameter r=smacleod
https://hg.mozilla.org/hgcustom/version-control-tools/rev/f8673c2b125b
hgmo: move stream_clone_cmp to a standalone function r=smacleod
https://hg.mozilla.org/hgcustom/version-control-tools/rev/c2862b0dbf63
hgmo: move manifest filter and sort into standalone function r=smacleod
https://hg.mozilla.org/hgcustom/version-control-tools/rev/3548753ac274
terraform: move instance_data.json object to /var/hg on bootstrap r=smacleod
https://hg.mozilla.org/hgcustom/version-control-tools/rev/09fa6119e23b
hgmo: serve stream clone bundles to private IP address on cloud instances r=smacleod

Comment 9

[Connor Sheehan [:sheehan]]

Attachment: hgmo: get `Remote-Addr` header from `rawenv` (Bug 1518394) r?smacleod

The Zeus load balancer sets an `X-Cluster-Client-Ip` header
to any request for hg.mo. The alternative header we use to
identify the source IP address, `Remote-Addr`, is set as an
environment variable to the WSGI process that spawns to handle
the HTTP request. The mismatch in locations is due to the
way Mercurial parses incoming HTTP requests - only choosing
to recognize headers beginning with `HTTP_` for inclusion
in the `mercurial.hgweb.wgsiheaders.Headers` object. The
other values set as environment variables are still available
in the `rawenv` container, so this commit switches the value
of `sourceip` to come from that object when required. The
corresponding test is updated to reflect the correct behaviour.

Since http requests to docker images running on a local machine
come from private IP addresses, and the docker images are built
with an `instance_data.json` object into them starting with
commit 09fa6119e23b, making this change causes all requests
in the test suite to default to streamed clone bundles. This
commit also renames the instance data object to a non-standard
name during docker image builds and adds a helper function to
toggle test features for cloud based instances. For now this
simply means renaming the file back to it's standard location.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/84c42cc2838a
hgmo: get `Remote-Addr` header from `rawenv` r=smacleod

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/06a9ba5d8860
terraform: add hg user in cloud config and use quotes around shell commands