1249197 - Advertise S3 URLs to clients in EC2

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, defect)

Description

[Gregory Szorc [:gps]]

Currently, clone bundles / bundleclone manifests expose CDN URLs first and list S3 URLs for clients that prefer them. This means clients must explicitly set some hgrc options to "prefer" the S3 URLs over the CDN.

We want clients in EC2 to use the S3 URLs (assuming they are in a region where we have the bundles uploaded to S3) because S3 is faster and transfers from S3 are free within EC2. If we require clients to explicitly set a config option, someone will inevitably forget and we'll have slower and costlier clones.

Let's IP sniff on the servers and only advertise S3 URLs for the same region an EC2 request is coming from. Everywhere else will continue to get the full manifest with the CDN first.

Comment 1

[Gregory Szorc [:gps]]

Attachment: MozReview Request: ansible/hg-web: install AWS IP ranges JSON file (bug 1249197); r?fubar

We'll need this to examine source IPs to see if they are from AWS.

File obtained from https://ip-ranges.amazonaws.com/ip-ranges.json
and checked in unmodified.

Review commit: https://reviewboard.mozilla.org/r/35567/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/35567/

Comment 2

[Gregory Szorc [:gps]]

Attachment: MozReview Request: ansible/hg-web: install ipaddress Python package (bug 1249197); r?fubar

Python 2 doesn't have a nice API for parsing IP addresses and doing
netmask foo. Python 3.3 introduced the "ipaddress" package. There is a
backport for Python 2, which we install in this commit.

Review commit: https://reviewboard.mozilla.org/r/35569/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/35569/

Comment 3

[Gregory Szorc [:gps]]

Attachment: MozReview Request: hgmo: serve AWS clients bundles from local S3 region (bug 1249197); r?fubar

This commit modifies the hgmo extension to inspect the source IP
address of HTTP requests for bundle clone and clone bundles manifests
and compare against known AWS IP ranges. If the request is coming from
an AWS region that we have S3 URLs for (identified by the presence of
"ec2region=" in the manifest metadata), we filter the manifest to only
advertise URLs in the same region. This guarantees an intra-region
transfer without any client configuration. This is fast and free
since intra-region S3 transfers don't cost anything!

In addition, we also reorder the manifest to advertise the stream clone
bundles first. We recommend clients in automation apply stream clone
bundles (over the default gzip bundles) because they are the fastest
mechanism to clone repositories (a stream clone bundle can apply at
>80 MB/s).

Tests for the new features have been added to the hgserver tests because
we have existing tests that run the code for generating manifests
and we want to use realistic values for testing.

Documentation for the bundle hosting has been updated to reflect the
change in behavior.

Review commit: https://reviewboard.mozilla.org/r/35571/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/35571/

Comment 4

[Kendall Libby [:fubar] (he/him)]

Comment on attachment 8721094 [details]
MozReview Request: ansible/hg-web: install ipaddress Python package (bug 1249197); r?fubar

https://reviewboard.mozilla.org/r/35569/#review32283

lgtm

Comment 5

[Kendall Libby [:fubar] (he/him)]

Comment on attachment 8721095 [details]
MozReview Request: hgmo: serve AWS clients bundles from local S3 region (bug 1249197); r?fubar

https://reviewboard.mozilla.org/r/35571/#review32285

lgtm, but tbh you might want a third set of brains on the python (I'm still new to it and am slightly fried from patching everything)

Comment 6

[Kendall Libby [:fubar] (he/him)]

Comment on attachment 8721093 [details]
MozReview Request: ansible/hg-web: install AWS IP ranges JSON file (bug 1249197); r?fubar

https://reviewboard.mozilla.org/r/35567/#review32287

my only question is: what's the process for determining when the AWS IP ranges have changed and need updating?

Comment 7

[Gregory Szorc [:gps]]

https://reviewboard.mozilla.org/r/35567/#review32287

Yeah, I was trying to figure that out too. I was feeling too lazy to write a CRON. https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html says they send updates to a SNS topic when the file changes. Perhaps I could subscribe and have an e-mail sent to developer-services@ so we know when we need to update. If the frequency is too high we can develop an automated system.

Comment 8

[Gregory Szorc [:gps]]

https://reviewboard.mozilla.org/r/35567/#review32287

developer-services@ is now receiving an e-mail notification when the IP ranges change.

Comment 9

[Dan Minor [:dminor]]

Comment on attachment 8721095 [details]
MozReview Request: hgmo: serve AWS clients bundles from local S3 region (bug 1249197); r?fubar

https://reviewboard.mozilla.org/r/35571/#review32317

::: hgext/hgmo/__init__.py:562
(Diff revision 1)
> +                              proto.req.env.get('REMOTE_ADDR'))

nit: I think pep8 would say this is underindented

::: hgext/hgmo/__init__.py:618
(Diff revision 1)
> +    except Exception:

It would be nice to add a warning here that an exception occurred and we're using the default values instead.

Comment 10

[Kendall Libby [:fubar] (he/him)]

(In reply to Gregory Szorc [:gps] from comment #8)
> https://reviewboard.mozilla.org/r/35567/#review32287
> 
> developer-services@ is now receiving an e-mail notification when the IP
> ranges change.

perfect! now let's not forget why when it doesn't change for 9 months...  ;-)

Comment 11

[Gregory Szorc [:gps]]

https://hg.mozilla.org/hgcustom/version-control-tools/rev/9741cca32ff47892f7ca5e71e680fa586d1554d7
ansible/hg-web: install AWS IP ranges JSON file (bug 1249197); r=fubar

https://hg.mozilla.org/hgcustom/version-control-tools/rev/d4f71a509bacf13ca4ccdcf03ba1a161cb45200b
ansible/hg-web: install ipaddress Python package (bug 1249197); r=fubar

https://hg.mozilla.org/hgcustom/version-control-tools/rev/9cd473d7247e67541b208d7a49beae5f4c3d6d99
hgmo: serve AWS clients bundles from local S3 region (bug 1249197); r=fubar, dminor

Comment 12

[Gregory Szorc [:gps]]

This is deployed to prod.