1147524 - figure out AWS ACL situation for Consul

Status: RESOLVED WORKSFORME

(Socorro :: Infra, task)

Description

[Daniel Maher [:phrawzty]]

Here's the situation: we spin up Consul and those nodes can speak to each other, then we spin up more nodes and those nodes can speak to each other - HOWEVER, the "more nodes" can't speak to the Consul nodes.

We need to figure out why and then fix it.

Comment 1

[Daniel Maher [:phrawzty]]

I can't reproduce the behaviour described in comment #0, which is to say, everything works as expected. :)

Working from the Terraform profiles in socorro-infra, I brought up a Consul cluster (via ASG) and provisioned (read: joined) the Consul *server* processes manually. 

[centos@ip-172-31-28-33]$ consul members
Node              Address             Status  Type    Build  Protocol
ip-172-31-28-33   172.31.28.33:8301   alive   server  0.5.0  2
ip-172-31-39-159  172.31.39.159:8301  alive   server  0.5.0  2
ip-172-31-39-160  172.31.39.160:8301  alive   server  0.5.0  2

K/V for test:
[centos@ip-172-31-28-33 ~]$ curl -XPUT http://localhost:8500/v1/kv/foo -d 'bar'


I then brought up (via ASG) the Processors cluster (only) and manually joined the *client* processes manually:

[centos@ip-172-31-29-233 ~]$ consul members
Node              Address             Status  Type    Build  Protocol
ip-172-31-29-233  172.31.29.233:8301  alive   client  0.5.0  2
ip-172-31-38-230  172.31.38.230:8301  alive   client  0.5.0  2
ip-172-31-29-234  172.31.29.234:8301  alive   client  0.5.0  2


Finally, from one of the *client* nodes above, I joined the two Consul clusters into a raft:

[centos@ip-172-31-38-230 consul]$ sudo consul join 172.31.28.33
Successfully joined cluster by contacting 1 nodes.
[centos@ip-172-31-38-230 consul]$ consul members
Node              Address             Status  Type    Build  Protocol
ip-172-31-38-230  172.31.38.230:8301  alive   client  0.5.0  2
ip-172-31-29-233  172.31.29.233:8301  alive   client  0.5.0  2
ip-172-31-29-234  172.31.29.234:8301  alive   client  0.5.0  2
ip-172-31-39-159  172.31.39.159:8301  alive   server  0.5.0  2
ip-172-31-39-160  172.31.39.160:8301  alive   server  0.5.0  2
ip-172-31-28-33   172.31.28.33:8301   alive   server  0.5.0  2

Verify test K/V:
[centos@ip-172-31-29-233 ~]$ curl localhost:8500/v1/kv/foo
[{"CreateIndex":15,"ModifyIndex":17,"LockIndex":0,"Key":"foo","Flags":0,"Value":"YmFy"}]

Comment 2

[Daniel Maher [:phrawzty]]

The Consul Terraform config has rules in aws_security_group.private_to_consul__consul that follow the form:

    ingress {
        from_port = 8300
        to_port = 8302
        protocol = "tcp"
        cidr_blocks = [
            "172.31.0.0/16"
        ]
    }

This behaves as expected, which is to say, instances in our VPC in the 172.31.0.0/16 block can access those ports / protocols - even instances spun up in other ASGs or whatever.  The important thing is that everything is in the same VPC and correct netblock.

The Infra Terraform config has rules in aws_security_group.private_to_private__any that follow the form:

    ingress {
        from_port = "-1"
        to_port = "-1"
        protocol = "icmp"
        cidr_blocks = [
            "172.31.0.0/16"
        ]
    }

I chose the ICMP rule specifically because it illustrates the behaviour of ingress in the context of security groups.  The above rule allows the Infra nodes to ping each other; however, the Infra nodes cannot ping the Consul nodes (and the Consul nodes cannot ping each other).  This is true even though all nodes are in the same VPC and netblock.  This is because aws_security_group.private_to_private__any is applied to the Infra nodes only.