1490403 - Requesting AWS account for version control services

Status: RESOLVED FIXED

(Cloud Services :: Operations: AWS Account Request, task)

Description

[Connor Sheehan [:sheehan]]

As per the meeting with CloudOps on Monday, September 10, 2018: the version control team would like to request a dedicated AWS account for use in managing and deploying mirror instances of hg.mozilla.org to AWS in the same regions as Firefox CI.

Comment 1

[Gregory Szorc [:gps]]

Note that we have an existing moz-devservices AWS account where we have some hg.mo infra already. Not sure if we should reuse or get a new account. It's probably easier for us to reuse. But we are talking about standing up a new logical service (private hg.mo mirrors for Firefox CI). So a separate account shouldn't be that big of a deal.

Also, we'll need to peer VPC's between this account and Taskcluster's AWS account so TC has access to this private service.

Comment 2

[Hal Wine [:hwine] use NI!]

Moving to AWS is a pretty major change in the security for hg.m.o -- let's update the RRA for hg.m.o -- we can take the "which account" question up there.

I'll get something on the schedule for soon

Comment 3

[Kendall Libby [:fubar] (he/him)]

(In reply to Gregory Szorc [:gps] from comment #1)
> Note that we have an existing moz-devservices AWS account where we have some
> hg.mo infra already. Not sure if we should reuse or get a new account. It's
> probably easier for us to reuse. 

New account. Trust me, you want to take full advantage of CloudOps' AWS model.

Comment 4

[Gregory Szorc [:gps]]

We want to stand up read-only https:// mirrors of hg.mozilla.org for the *private* use of Firefox CI. We want this service to be hosted in the same geographic regions (ideally in same datacenter / AWS region / etc) that Firefox CI runs in.

The service is logically similar to the https://hg.mozilla.org/ service except it would be a) running out of a cloud provider instead of a Mozilla datacenter b) not available on the public Internet.

We will likely want to "peer" whatever networks this new service is in with the Taskcluster VPCs to allow Taskcluster (and presumably only Taskcluster) to access it directly. We will likely create a fake DNS domain and hostname for this service using Route53 (e.g. hg.services.taskcluster-internal) and configure CI in Taskcluster to rewrite https://hg.mozilla.org/ to e.g. https://hg.services.taskcluster-internal so Firefox CI consumes the private instance. We may also create network/firewall rules to prevent Taskcluster workers from hitting the public https://hg.mozilla.org/ endpoint.

As a separate project many months from now, we will likely move hg.mozilla.org out of a Mozilla datacenter and into a cloud provider. But that will be a separate project and is out of scope for this bug.

When Taskcluster workers and Firefox CI start running in GCP, we will want to stand up a private Mercurial mirror in GCP as well. But since this is likely months off and since a) CloudOps doesn't have a good GCP story for non-Kubernetes services yet b) data transfer costs between GCP and Firefox CI in AWS would likely be thousands per month, we've decided to wait on GCP and proceed with establishing the private service in AWS, where Firefox CI is today.

Comment 5

[Gregory Szorc [:gps]]

Oh, despite the bug title, we may not necessarily use a new AWS account. Dustin has offered to host things in the main Taskcluster AWS account. We also have the moz-devservices account. And apparently CloudOps may have accounts for us to use.

The new service will require some special connectivity requirements - including DNS and private service access within MDC1. I'm assuming that means we'll want these services to live in a special VPC. ckolos has configured some dev AWS account already. And dustin leads me to believe that Taskcluster already has some of this connectivity in its account.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

VPN connectivity is, if I remember correctly, per-VPC.  I don't think you can connect a single IPSec tunnel to multiple VPCs.  So you'd probably be best off installing these hosts in a subnet in the existing gecko VPCs.

Here's a quick network layout for the Firefox CI AWS account, with the gory details in bug 1387179.

 * one gecko-workers VPC per region; for each VPC
 * one worker subnet per availability zone
 * one administrative subnet
 * one IPSec tunnel for each (datacenter, region) pair.  Datacenters are [mdc1, releng.scl3]. [*]
 * flows allowed in the netops firewalls:
   * releng jumphosts -> worker subnets for ssh, rdp, and vnc (although this is forbidden by security groups)
   * admin subnets -> KMS server in scl3 [*]
 * all configured in https://github.com/taskcluster/taskcluster-infrastructure/tree/master/dustin-taskcluster-terraform

[*] until next week, when presumably releng.scl3 goes dark.. I haven't heard anything about this

So these mirror hosts could fit comfortably into the admin subnets, and given static IPs netops can add appropriate flows as necessary.

Comment 7

[Kim Moir [:kmoir] ET]

So it looks like we have several options for an account - existing taskcluster account, a new cloudops account, existing moz-devservices account.  I was talking to gps today about this and he mentioned that we should optimize for the least complicated network configuration given our requirements. ckolos do you have any opinion on that given you are familiar with our setup?  I'd like to make a decision on this so we can move forward with implementation.

Comment 8

[Kendall Libby [:fubar] (he/him)]

I think my comment above was in error, because we technically have two devservices AWS accounts; the CloudOps' owned one may be OK to use, but we should definitely NOT use the old one that I manage. :-)

Comment 9

[Chris Kolosiwsky [:ckolos] (ckolos has left the building)]

CloudOps will handle this and send creds to :gps and :sheehan

Comment 10

[Kim Moir [:kmoir] ET]

Thanks ckolos for sending the credentials, we appreciate it.