1509039 - Create Terraform configs for ci-private hgweb environments

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, enhancement)

Description

[Connor Sheehan [:sheehan]]

We want to use an IaC tool to configure the infrastructure for ci-private hgweb mirrors in AWS. We have decided to use Terraform as our tool of choice as it is cloud-agnostic and will allow easy addition of GCP environments without having to use multiple tools.

Comment 1

[Connor Sheehan [:sheehan]]

Attachment: terraform: create terraform directory with `privsubnet` module (Bug 1509039) r?gps

This commit creates a top-level `terraform` directory which will
serve as the root module (with root module files to be added in
a later commit) for all vcs-related Terraform configs going
forward. Within this directory we add a `modules` subdirectory
and populate it with the `privsubnet` module. This module
configures a private subnet within a specified VPC, region
and availability zone. The subnet creates a route table, which
causes all outbound traffic from instances within the subnet to
be routed to a specified NAT gateway (in practice this will be
the NAT gateway of a public subnet in the same AZ).

As this is the first addition of the "terraform" directory,
and Terraform itself creates local state files, we add an entry
to `.hgignore` to avoid tracking unnecessary files.

Comment 2

[Connor Sheehan [:sheehan]]

Attachment: terraform: add `pubsubnet` module (Bug 1509039) r?gps

This commit adds a `pubsubnet` module to the Terraform
configs. The module configures a public subnet within a
specified VPC, and attaches a NAT gateway that can receive
traffic from a private subnet (from the same AZ in practice).
A route table is taken as input, which should include a
rule that sends outbound traffic within the subnet (including
traffic sent from a private subnet through the NAT gateway)
to the internet gateway attached to the VPC.

Comment 3

[Connor Sheehan [:sheehan]]

Attachment: terraform: add a `bastion` module (Bug 1509039) r?gps

This commit adds a `bastion` module for configuring
bastion hosts within a VPC. The module currently only
creates a single host using the Ubuntu 18.04 AMI.
A security group is created which allows only SSH
inbound traffic to the host, and an elastic IP
address is assigned to the host.

Comment 4

[Connor Sheehan [:sheehan]]

Attachment: terraform: add an `hgweb-mirror` module (Bug 1509039) r?gps

This commit adds an `hgweb-mirror` module to the Terraform
configs. This module creates a single EC2 instance intended
to be configured as an hgweb mirror. The new instance is
created using the CentOS7 AMI and are intended to be assigned
to a private subnet within a VPC. The instances use the c4.2xl
instance type and are assigned to security groups via the
`security_group_ids` variable.

Comment 5

[Connor Sheehan [:sheehan]]

Attachment: terraform: add the `ci-only` module (Bug 1509039) r?gps

This commit adds the `ci-only` module, which applies
the config for a VPC containing all CI-only resources
in a single AWS region. The module creates a VPC
and attaches an internet gateway with an assigned
elastic IP address. A route table is created with
a rule to send all public subnet traffic to the
internet gateway. We then create a public and private
subnet for each AZ withing the region, assigning a
/19 CIDR block to each. A bastion host is created
within a public subnet in the VPC to facilitate SSH
access to mirrors in the VPC (for both administrative
access and Ansible deployments). Finally we create a
security group and network ACls for the hosts
and assign relevant security rules to each.

Comment 6

[Connor Sheehan [:sheehan]]

Attachment: terraform: add root module files (Bug 1509039) r?gps

This commit adds the files for the Terraform root module.
When running Terraform commands from a directory, any `.tf`
files in that directory are considered to be within the
`root` module, and all other infrastructure should be
contained within the root module. We add the necessary
module files and configure a single `ci-only` module,
assigned to the `us-west-2` AWS region. As we expand
our infrastructure to new AWS regions or new cloud
providers (GCP, etc), we will simply add new module
entries to the root module (for example add a new
`ci-only` module entry for the us-east-1 region).

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/5af7c52c365a
terraform: create terraform directory with `privsubnet` module r=gps
https://hg.mozilla.org/hgcustom/version-control-tools/rev/51c9d896efb0
terraform: add `pubsubnet` module r=gps
https://hg.mozilla.org/hgcustom/version-control-tools/rev/c560e78724f1
terraform: add a `bastion` module r=gps
https://hg.mozilla.org/hgcustom/version-control-tools/rev/1ab302cec741
terraform: add an `hgweb-mirror` module r=gps
https://hg.mozilla.org/hgcustom/version-control-tools/rev/5843de78b826
terraform: add the `ci-only` module r=gps
https://hg.mozilla.org/hgcustom/version-control-tools/rev/2713b7bee65f
terraform: add root module files r=gps

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/7006560c3f5e
terraform: remove ci-only/outputs.tf

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/4f79353521b2
terraform: remove references to `gps` user 
https://hg.mozilla.org/hgcustom/version-control-tools/rev/3e12e83cfc97
terraform: switch `user_data` from a script to cloud-config

Comment 10

[Connor Sheehan [:sheehan]]

Attachment: terraform: restrict DHCP options set to Mozilla DNS (Bug 1509039) r?bstack

After running some DNS queries in the hg VPC, I realized that
every one queries was hitting the Mozilla DNS server, and external
requests were being resolve non-authoritatively. This commit removes
the Amazon DNS servers from our DHCP options set, so we don't
hit them accidentally and trigger unexpected failures later on.

Comment 11

[Connor Sheehan [:sheehan]]

Attachment: terraform: use an internal load balancer (Bug 1509039) r?bstack

This commit creates an internal ALB resource to be used as
the endpoint for us-west-2.hgmointernal.com. Using an ALB
will allow us to stand up multiple instances across multiple
AZs per region, while avoiding split-horizon DNS. The per-region
hostname (us-west-2.hgmointernal.com, etc) will resolve to
this load balancer which in turn will route traffic to hgweb
mirrors.

Comment 12

[Connor Sheehan [:sheehan]]

Attachment: terraform: use `aws_region` data source instead of passing around variable (Bug 1509039) r?bstack

Rather than create the awsregion variable over and over
again, we can use the aws_region data source.

Comment 13

[Connor Sheehan [:sheehan]]

Attachment: terraform: rename `ci-only-west2` to `ci-only-uw2` (Bug 1509039) r?bstack

"west2" could mean west-2 in some other continent
such as eu-west-2.

Comment 14

[Connor Sheehan [:sheehan]]

Attachment: terraform: remove bastion hosts and allow access to mirrors from MozVPN (Bug 1509039) r?bstack

Now that SSH access to the VPC is limited to MozVPN, there is
no need for the bastion host we were previously using. This
commit removes the bastion module and all associated resources,
and updates the mirrors security group to allow SSH from MozVPN
through the VPN gateway.

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/f171c8834780
terraform: remove bastion hosts and allow access to mirrors from MozVPN r=bstack
https://hg.mozilla.org/hgcustom/version-control-tools/rev/8c91d9d3ee7f
terraform: use `aws_region` data source instead of passing around variable r=bstack
https://hg.mozilla.org/hgcustom/version-control-tools/rev/e39520f4f0d6
terraform: restrict DHCP options set to Mozilla DNS r=bstack
https://hg.mozilla.org/hgcustom/version-control-tools/rev/50025cced335
terraform: use an internal load balancer r=bstack
https://hg.mozilla.org/hgcustom/version-control-tools/rev/3791a58dde8a
terraform: rename `ci-only-west2` to `ci-only-uw2` r=bstack

Comment 16

[Connor Sheehan [:sheehan]]

Attachment: terraform: define security group rules as independent resources (Bug 1509039) r?bstack

Previously we defined security group rules inline with the definition
of the security group itself. This is problematic when we want to
define rules that allow access between hosts in different security
groups (ie allowing traffic to flow between the load balancer and
instances in the balancer's target groups). Terraform fails since
the rule dependencies create a cycle. This cycle can be fixed by
defining the security groups on their own and defining the rules
as aws_security_group_rule resources.

This commit moves all existing rule definitions out of the security
group definition itself and into independent rule resources. A later
commit will add the new rule to allow traffic between resources
in different groups.

Comment 17

[Connor Sheehan [:sheehan]]

Attachment: terraform: add rule to allow traffic between internal load balancer and CI instances (Bug 1509039) r?bstack

This commit adds a new load balancer rule to allow traffic between
instances in the load balancer target group and the load balancer
itself.

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/5bd810508bfc
terraform: define security group rules as independent resources r=bstack
https://hg.mozilla.org/hgcustom/version-control-tools/rev/da500592cfa4
terraform: add rule to allow traffic between internal load balancer and CI instances r=bstack

Comment 19

[Connor Sheehan [:sheehan]]

Attachment: terraform: rename `ci-only` module to `aws-vpc` (Bug 1509039) r?bstack

When I originally created the Terraform configs for the hgmo
AWS account, our design goals were to isolate all resources
for the private hgweb mirrors into their own VPC. After
working towards productionizing these resources over the
last few months, I decided it makes more sense to have
a single VPC for all hg related resources in this region,
and limit logical access to different resources using
security groups. This will allow us to iterate faster and
avoid the work of creating multiple VPCs that link back
to the on-premises Mozilla networks.

Comment 20

[Connor Sheehan [:sheehan]]

Attachment: terraform: make hgweb mirror instances variable (Bug 1509039) r?bstack

Currently we only have a single hgweb mirror stood up,
with the instance type of that mirror hard coded in
to it's Terraform module. This commit makes the instance
type a variable so we can add/reduce the compute power
if necessary.

Comment 21

[Connor Sheehan [:sheehan]]

Attachment: terraform: remove `environment_users` variable (Bug 1509039) r?bstack

We were going to use this to create new users as part
of our user data input. We've switch to a hard-coded
cloud config for now, so this is safe to remove.

Comment 22

[Connor Sheehan [:sheehan]]

Attachment: terraform: upgrade InfluxDB instance to `c4.2xlarge` (Bug 1509039) r?bstack

This is only temporary while I work on redirecting our
Influx data to Mozilla's production instance.

Comment 23

[Connor Sheehan [:sheehan]]

Attachment: terraform: move ELB target group attachment into the `hgweb-mirror` module (Bug 1509039) r?bstack

We will need one attachment per instance - rather than create new
attachments each time, let's just add the attachment to the module
itself.

Comment 24

[Connor Sheehan [:sheehan]]

Attachment: terraform: create new hgweb mirror host (Bug 1509039) r?bstack

This host will be 1 of 2 attached to the internal load
balancer.

Comment 25

[Connor Sheehan [:sheehan]]

Attachment: terraform: rename `test-hgweb-mirror` to `hgweb-a` (Bug 1509039) r?bstack

Same code, but a new instance in AWS.

Comment 26

[Connor Sheehan [:sheehan]]

Attachment: terraform: increase disk space to 150GB on hgweb mirrors (Bug 1509039) r?bstack

We were hitting the edge at 100GB.

Comment 27

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/1340d12236b1
terraform: rename `ci-only` module to `aws-vpc` r=bstack
https://hg.mozilla.org/hgcustom/version-control-tools/rev/51b1d5add43c
terraform: make hgweb mirror instances variable r=bstack
https://hg.mozilla.org/hgcustom/version-control-tools/rev/8219a59625be
terraform: remove `environment_users` variable r=bstack
https://hg.mozilla.org/hgcustom/version-control-tools/rev/2755b890648b
terraform: upgrade InfluxDB instance to `c4.2xlarge` r=bstack
https://hg.mozilla.org/hgcustom/version-control-tools/rev/0c9bfcf4a389
terraform: move ELB target group attachment into the `hgweb-mirror` module r=bstack
https://hg.mozilla.org/hgcustom/version-control-tools/rev/36c8c075447e
terraform: create new hgweb mirror host r=bstack
https://hg.mozilla.org/hgcustom/version-control-tools/rev/b0dedd649093
terraform: rename `test-hgweb-mirror` to `hgweb-a` r=bstack
https://hg.mozilla.org/hgcustom/version-control-tools/rev/0ed956b29df0
terraform: increase disk space to 150GB on hgweb mirrors r=bstack