Closed
Bug 1249197
Opened 6 years ago
Closed 6 years ago
Advertise S3 URLs to clients in EC2
Categories
(Developer Services :: Mercurial: hg.mozilla.org, defect)
Developer Services
Mercurial: hg.mozilla.org
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: gps, Assigned: gps)
References
Details
Attachments
(3 files)
Currently, clone bundles / bundleclone manifests expose CDN URLs first and list S3 URLs for clients that prefer them. This means clients must explicitly set some hgrc options to "prefer" the S3 URLs over the CDN. We want clients in EC2 to use the S3 URLs (assuming they are in a region where we have the bundles uploaded to S3) because S3 is faster and transfers from S3 are free within EC2. If we require clients to explicitly set a config option, someone will inevitably forget and we'll have slower and costlier clones. Let's IP sniff on the servers and only advertise S3 URLs for the same region an EC2 request is coming from. Everywhere else will continue to get the full manifest with the CDN first.
Assignee | ||
Comment 1•6 years ago
|
||
We'll need this to examine source IPs to see if they are from AWS. File obtained from https://ip-ranges.amazonaws.com/ip-ranges.json and checked in unmodified. Review commit: https://reviewboard.mozilla.org/r/35567/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/35567/
Attachment #8721093 -
Flags: review?(klibby)
Assignee | ||
Comment 2•6 years ago
|
||
Python 2 doesn't have a nice API for parsing IP addresses and doing netmask foo. Python 3.3 introduced the "ipaddress" package. There is a backport for Python 2, which we install in this commit. Review commit: https://reviewboard.mozilla.org/r/35569/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/35569/
Attachment #8721094 -
Flags: review?(klibby)
Assignee | ||
Comment 3•6 years ago
|
||
This commit modifies the hgmo extension to inspect the source IP address of HTTP requests for bundle clone and clone bundles manifests and compare against known AWS IP ranges. If the request is coming from an AWS region that we have S3 URLs for (identified by the presence of "ec2region=" in the manifest metadata), we filter the manifest to only advertise URLs in the same region. This guarantees an intra-region transfer without any client configuration. This is fast and free since intra-region S3 transfers don't cost anything! In addition, we also reorder the manifest to advertise the stream clone bundles first. We recommend clients in automation apply stream clone bundles (over the default gzip bundles) because they are the fastest mechanism to clone repositories (a stream clone bundle can apply at >80 MB/s). Tests for the new features have been added to the hgserver tests because we have existing tests that run the code for generating manifests and we want to use realistic values for testing. Documentation for the bundle hosting has been updated to reflect the change in behavior. Review commit: https://reviewboard.mozilla.org/r/35571/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/35571/
Attachment #8721095 -
Flags: review?(klibby)
Updated•6 years ago
|
Attachment #8721094 -
Flags: review?(klibby) → review+
Comment 4•6 years ago
|
||
Comment on attachment 8721094 [details] MozReview Request: ansible/hg-web: install ipaddress Python package (bug 1249197); r?fubar https://reviewboard.mozilla.org/r/35569/#review32283 lgtm
Comment 5•6 years ago
|
||
Comment on attachment 8721095 [details] MozReview Request: hgmo: serve AWS clients bundles from local S3 region (bug 1249197); r?fubar https://reviewboard.mozilla.org/r/35571/#review32285 lgtm, but tbh you might want a third set of brains on the python (I'm still new to it and am slightly fried from patching everything)
Attachment #8721095 -
Flags: review?(klibby) → review+
Comment 6•6 years ago
|
||
Comment on attachment 8721093 [details] MozReview Request: ansible/hg-web: install AWS IP ranges JSON file (bug 1249197); r?fubar https://reviewboard.mozilla.org/r/35567/#review32287 my only question is: what's the process for determining when the AWS IP ranges have changed and need updating?
Attachment #8721093 -
Flags: review?(klibby) → review+
Assignee | ||
Updated•6 years ago
|
Attachment #8721095 -
Flags: review?(dminor)
Assignee | ||
Comment 7•6 years ago
|
||
https://reviewboard.mozilla.org/r/35567/#review32287 Yeah, I was trying to figure that out too. I was feeling too lazy to write a CRON. https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html says they send updates to a SNS topic when the file changes. Perhaps I could subscribe and have an e-mail sent to developer-services@ so we know when we need to update. If the frequency is too high we can develop an automated system.
Assignee | ||
Comment 8•6 years ago
|
||
https://reviewboard.mozilla.org/r/35567/#review32287 developer-services@ is now receiving an e-mail notification when the IP ranges change.
Comment 9•6 years ago
|
||
Comment on attachment 8721095 [details] MozReview Request: hgmo: serve AWS clients bundles from local S3 region (bug 1249197); r?fubar https://reviewboard.mozilla.org/r/35571/#review32317 ::: hgext/hgmo/__init__.py:562 (Diff revision 1) > + proto.req.env.get('REMOTE_ADDR')) nit: I think pep8 would say this is underindented ::: hgext/hgmo/__init__.py:618 (Diff revision 1) > + except Exception: It would be nice to add a warning here that an exception occurred and we're using the default values instead.
Attachment #8721095 -
Flags: review?(dminor) → review+
Comment 10•6 years ago
|
||
(In reply to Gregory Szorc [:gps] from comment #8) > https://reviewboard.mozilla.org/r/35567/#review32287 > > developer-services@ is now receiving an e-mail notification when the IP > ranges change. perfect! now let's not forget why when it doesn't change for 9 months... ;-)
Assignee | ||
Comment 11•6 years ago
|
||
https://hg.mozilla.org/hgcustom/version-control-tools/rev/9741cca32ff47892f7ca5e71e680fa586d1554d7 ansible/hg-web: install AWS IP ranges JSON file (bug 1249197); r=fubar https://hg.mozilla.org/hgcustom/version-control-tools/rev/d4f71a509bacf13ca4ccdcf03ba1a161cb45200b ansible/hg-web: install ipaddress Python package (bug 1249197); r=fubar https://hg.mozilla.org/hgcustom/version-control-tools/rev/9cd473d7247e67541b208d7a49beae5f4c3d6d99 hgmo: serve AWS clients bundles from local S3 region (bug 1249197); r=fubar, dminor
Assignee | ||
Comment 12•6 years ago
|
||
This is deployed to prod.
Assignee: nobody → gps
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•