Create ssh flow from buildduty-tools to buildbot masters

RESOLVED FIXED

Status

Infrastructure & Operations
NetOps
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: aselagea, Assigned: dcurado)

Tracking

Details

Attachments

(3 attachments)

(Reporter)

Description

2 years ago
We'd like to move the script that restarts the buildbot-masters to the buildduty-tools machine. We'd need ssh access to the masters for that. 

Flows needed:
    - 10.132.51.74 --> 10.26.68.0/24 (bb_releng_scl3)
    - 10.132.51.74 --> 10.132.68.0/24 (bb_releng_usw2)
    - 10.132.51.74 --> 10.134.68.0/24 (bb_releng_use1)
(Assignee)

Comment 1

2 years ago
working on this.
Assignee: network-operations → dcurado
Status: NEW → ASSIGNED
(Assignee)

Comment 2

2 years ago
For this security policy (aka flow)

    - 10.132.51.74 --> 10.132.68.0/24 (bb_releng_usw2)

Those appear to be in the same VPC, and therefor the packets won't ricochet through SCL3.
i.e. the packets will remain within the VPC in AWS.  
If that doesn't work now, my guess is that the security_group on one or more of the instances involved need to be modified.
(Assignee)

Comment 3

2 years ago
For this security policy (aka flow)

    - 10.132.51.74 --> 10.134.68.0/24 (bb_releng_use1)

Those are two different VPCs, and the packets between them will bounce through the releng firewall in SCL3.
The security policy on the firewall currently says, "let anything through".

So, again, if this doesn't work now, I would guess that security_group(s) on one or more of the instances involved need to be modified.
(Assignee)

Comment 4

2 years ago
For the last one:

 - 10.132.51.74 --> 10.26.68.0/24 (bb_releng_scl3)

I modified the existing policy to include the new server:

Policy: vpc-bb--ssh, action-type: permit, State: enabled, Index: 1205, Scope Policy: 0
  Policy Type: Configured
  Sequence number: 1
  From zone: vpc, To zone: bb
  Source addresses:
    buildduty-tools: 10.132.51.74/32 <--------
    us-west-2-68: 10.132.68.0/24
    us-east-1-68: 10.134.68.0/24
  Destination addresses:
    any-ipv4: 0.0.0.0/0
    any-ipv6: ::/0
  Application: junos-ssh
    IP protocol: tcp, ALG: 0, Inactivity timeout: 86400
      Source port range: [0-0]
      Destination port range: [22-22]

Here's the diff:
dcurado@fw1.ops.releng.scl3.mozilla.net> show configuration | compare rollback 1
[edit security policies from-zone vpc to-zone bb policy vpc-bb--ssh match]
-      source-address aws-buildbot-masters;
+      source-address [ aws-buildbot-masters buildduty-tools ];
[edit security zones security-zone vpc address-book]
       address cruncher-aws.srv.releng.usw2 { ... }
+      address buildduty-tools 10.132.51.74/32;
Status: ASSIGNED → UNCONFIRMED
Ever confirmed: false
Created attachment 8788399 [details]
1.png

Dave could you check please to see that the security policy between:
    - 10.132.51.74 (buildduty-tools) --> 10.26.68.0/24 (bb_releng_scl3)
    - 10.132.51.74 (buildduty-tools) --> 10.132.68.0/24 (bb_releng_usw2)
    - 10.132.51.74 (buildduty-tools) --> 10.134.68.0/24 (bb_releng_use1)
will  be the same as the security policy between:
    - 10.134.68.32 (dev-master2.bb.releng.use1.mozilla.com) --> 10.26.68.0/24 (bb_releng_scl3)
    - 10.134.68.32 (dev-master2.bb.releng.use1.mozilla.com) --> 10.132.68.0/24 (bb_releng_usw2)
    - 10.134.68.32 (dev-master2.bb.releng.use1.mozilla.com) --> 10.134.68.0/24 (bb_releng_use1)

Or at least these to be the same:

buildbot-http
ping
ssh


Now these configs are different(check 1.png that was attached).
Flags: needinfo?(dcurado)
(Assignee)

Comment 6

2 years ago
OK, so the original request was to allow ssh through.
But now you would like http as well, right?
(ping is always allowed, it's a global policy.)

Thanks in advance for any clarification.
Flags: needinfo?(dcurado)
Yes we also need http because our script require to access the web interface for each master to be able to initiate a clean shutdown.
So we will need for this server(10.132.51.74 (buildduty-tools)) to have also security policy of buildbot-http and ssh set with all buildbot-masters.

Thanks
Flags: needinfo?(dcurado)
(Assignee)

Comment 8

2 years ago
OK, hopefully this will do what you are asking for:
Policy: vpc-bb--ssh, action-type: permit, State: enabled, Index: 1205, Scope Policy: 0
  Policy Type: Configured
  Sequence number: 1
  From zone: vpc, To zone: bb
  Source addresses:
    buildduty-tools: 10.132.51.74/32
    us-west-2-68: 10.132.68.0/24
    us-east-1-68: 10.134.68.0/24
  Destination addresses:
    any-ipv4: 0.0.0.0/0
    any-ipv6: ::/0
  Application: junos-ssh
    IP protocol: tcp, ALG: 0, Inactivity timeout: 86400
      Source port range: [0-0]
      Destination port range: [22-22]
  Application: buildbot-http
    IP protocol: tcp, ALG: 0, Inactivity timeout: 86400
      Source port range: [0-0]
      Destination port range: [8001-8006]
    IP protocol: tcp, ALG: 0, Inactivity timeout: 86400
      Source port range: [0-0]
      Destination port range: [8101-8106]
    IP protocol: tcp, ALG: 0, Inactivity timeout: 86400
      Source port range: [0-0]
      Destination port range: [8201-8206]
    IP protocol: tcp, ALG: 0, Inactivity timeout: 86400
      Source port range: [0-0]
      Destination port range: [8000-8999]

Here's the diff:

dcurado@fw1.ops.releng.scl3.mozilla.net> show configuration | compare rollback 1
[edit security policies from-zone vpc to-zone bb policy vpc-bb--ssh match]
-      application junos-ssh;
+      application [ junos-ssh buildbot-http ];
Flags: needinfo?(dcurado)
Created attachment 8789263 [details]
firewall policy.png

Nothing was changed in the configuration,as can be seen, the server buildbot-tools is able to connect to any buildbot-master and his web interface from scl3 but is not able to connect to bb_releng_usw2(10.132.68.0/24) or bb_releng_use1 (10.134.68.0/24).
Please also check the connection via ssh and to the web interface from buildbot-tools server to the buildbot-masters after configuration is done to make sure is working.
Flags: needinfo?(dcurado)
(Reporter)

Comment 10

2 years ago
Created attachment 8789423 [details] [review]
bug1300088_cloud-tools

Okay, so we'll need to add some entries for the buildduty-tools machine in the inbound rules definition for the buildbot master security group.
Flags: needinfo?(dcurado)
Attachment #8789423 - Flags: review?(jwatkins)
Comment on attachment 8789423 [details] [review]
bug1300088_cloud-tools

lgtm
Attachment #8789423 - Flags: review?(jwatkins) → review+
Attachment #8789423 - Flags: checked-in+
(Assignee)

Comment 12

2 years ago
Follow up from netops:
If something here is still not the way you need it to be, and you believe it is a firewall issue, please make it as clear as possible.  i.e. use FQDN and port numbers.  When you use terms like "buildbot-master"
or what-have-you, that doesn't mean anything to me.  

As well, the way we do firewall changes is to get a clear description of what you need, then make the change, and set the bug to the unconfirmed state.  You then verify that it works as you need, and hopefully, close the bug.  I may or may not have access to the machines involved, and IMHO it just makes more sense for us to say, "we think it should work now, can you confirm?"

Thanks,
Dave
Thank you for the advices Dave
Everything is working fine now.
Status: UNCONFIRMED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.