1300088 - Create ssh flow from buildduty-tools to buildbot masters

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: NetOps, task)

Description

[Alin Selagea [:aselagea]]

We'd like to move the script that restarts the buildbot-masters to the buildduty-tools machine. We'd need ssh access to the masters for that. 

Flows needed:
    - 10.132.51.74 --> 10.26.68.0/24 (bb_releng_scl3)
    - 10.132.51.74 --> 10.132.68.0/24 (bb_releng_usw2)
    - 10.132.51.74 --> 10.134.68.0/24 (bb_releng_use1)

Comment 1

[Dave Curado :dcurado]

working on this.

Comment 2

[Dave Curado :dcurado]

For this security policy (aka flow)

    - 10.132.51.74 --> 10.132.68.0/24 (bb_releng_usw2)

Those appear to be in the same VPC, and therefor the packets won't ricochet through SCL3.
i.e. the packets will remain within the VPC in AWS.  
If that doesn't work now, my guess is that the security_group on one or more of the instances involved need to be modified.

Comment 3

[Dave Curado :dcurado]

For this security policy (aka flow)

    - 10.132.51.74 --> 10.134.68.0/24 (bb_releng_use1)

Those are two different VPCs, and the packets between them will bounce through the releng firewall in SCL3.
The security policy on the firewall currently says, "let anything through".

So, again, if this doesn't work now, I would guess that security_group(s) on one or more of the instances involved need to be modified.

Comment 4

[Dave Curado :dcurado]

For the last one:

 - 10.132.51.74 --> 10.26.68.0/24 (bb_releng_scl3)

I modified the existing policy to include the new server:

Policy: vpc-bb--ssh, action-type: permit, State: enabled, Index: 1205, Scope Policy: 0
  Policy Type: Configured
  Sequence number: 1
  From zone: vpc, To zone: bb
  Source addresses:
    buildduty-tools: 10.132.51.74/32 <--------
    us-west-2-68: 10.132.68.0/24
    us-east-1-68: 10.134.68.0/24
  Destination addresses:
    any-ipv4: 0.0.0.0/0
    any-ipv6: ::/0
  Application: junos-ssh
    IP protocol: tcp, ALG: 0, Inactivity timeout: 86400
      Source port range: [0-0]
      Destination port range: [22-22]

Here's the diff:
[e-mail]> show configuration | compare rollback 1
[edit security policies from-zone vpc to-zone bb policy vpc-bb--ssh match]
-      source-address aws-buildbot-masters;
+      source-address [ aws-buildbot-masters buildduty-tools ];
[edit security zones security-zone vpc address-book]
       address cruncher-aws.srv.releng.usw2 { ... }
+      address buildduty-tools 10.132.51.74/32;

Comment 5

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: 1.png

Dave could you check please to see that the security policy between:
    - 10.132.51.74 (buildduty-tools) --> 10.26.68.0/24 (bb_releng_scl3)
    - 10.132.51.74 (buildduty-tools) --> 10.132.68.0/24 (bb_releng_usw2)
    - 10.132.51.74 (buildduty-tools) --> 10.134.68.0/24 (bb_releng_use1)
will  be the same as the security policy between:
    - 10.134.68.32 (dev-master2.bb.releng.use1.mozilla.com) --> 10.26.68.0/24 (bb_releng_scl3)
    - 10.134.68.32 (dev-master2.bb.releng.use1.mozilla.com) --> 10.132.68.0/24 (bb_releng_usw2)
    - 10.134.68.32 (dev-master2.bb.releng.use1.mozilla.com) --> 10.134.68.0/24 (bb_releng_use1)

Or at least these to be the same:

buildbot-http
ping
ssh


Now these configs are different(check 1.png that was attached).

Comment 6

[Dave Curado :dcurado]

OK, so the original request was to allow ssh through.
But now you would like http as well, right?
(ping is always allowed, it's a global policy.)

Thanks in advance for any clarification.

Comment 7

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Yes we also need http because our script require to access the web interface for each master to be able to initiate a clean shutdown.
So we will need for this server(10.132.51.74 (buildduty-tools)) to have also security policy of buildbot-http and ssh set with all buildbot-masters.

Thanks

Comment 8

[Dave Curado :dcurado]

OK, hopefully this will do what you are asking for:
Policy: vpc-bb--ssh, action-type: permit, State: enabled, Index: 1205, Scope Policy: 0
  Policy Type: Configured
  Sequence number: 1
  From zone: vpc, To zone: bb
  Source addresses:
    buildduty-tools: 10.132.51.74/32
    us-west-2-68: 10.132.68.0/24
    us-east-1-68: 10.134.68.0/24
  Destination addresses:
    any-ipv4: 0.0.0.0/0
    any-ipv6: ::/0
  Application: junos-ssh
    IP protocol: tcp, ALG: 0, Inactivity timeout: 86400
      Source port range: [0-0]
      Destination port range: [22-22]
  Application: buildbot-http
    IP protocol: tcp, ALG: 0, Inactivity timeout: 86400
      Source port range: [0-0]
      Destination port range: [8001-8006]
    IP protocol: tcp, ALG: 0, Inactivity timeout: 86400
      Source port range: [0-0]
      Destination port range: [8101-8106]
    IP protocol: tcp, ALG: 0, Inactivity timeout: 86400
      Source port range: [0-0]
      Destination port range: [8201-8206]
    IP protocol: tcp, ALG: 0, Inactivity timeout: 86400
      Source port range: [0-0]
      Destination port range: [8000-8999]

Here's the diff:

[e-mail]> show configuration | compare rollback 1
[edit security policies from-zone vpc to-zone bb policy vpc-bb--ssh match]
-      application junos-ssh;
+      application [ junos-ssh buildbot-http ];

Comment 9

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: firewall policy.png

Nothing was changed in the configuration,as can be seen, the server buildbot-tools is able to connect to any buildbot-master and his web interface from scl3 but is not able to connect to bb_releng_usw2(10.132.68.0/24) or bb_releng_use1 (10.134.68.0/24).
Please also check the connection via ssh and to the web interface from buildbot-tools server to the buildbot-masters after configuration is done to make sure is working.

Comment 10

[Alin Selagea [:aselagea]]

Attachment: bug1300088_cloud-tools

Okay, so we'll need to add some entries for the buildduty-tools machine in the inbound rules definition for the buildbot master security group.

Comment 11

[Jake Watkins [:dividehex]]

Comment on attachment 8789423 [details] [review]
bug1300088_cloud-tools

lgtm

Comment 12

[Dave Curado :dcurado]

Follow up from netops:
If something here is still not the way you need it to be, and you believe it is a firewall issue, please make it as clear as possible.  i.e. use FQDN and port numbers.  When you use terms like "buildbot-master"
or what-have-you, that doesn't mean anything to me.  

As well, the way we do firewall changes is to get a clear description of what you need, then make the change, and set the bug to the unconfirmed state.  You then verify that it works as you need, and hopefully, close the bug.  I may or may not have access to the machines involved, and IMHO it just makes more sense for us to say, "we think it should work now, can you confirm?"

Thanks,
Dave

Comment 13

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Thank you for the advices Dave
Everything is working fine now.