Closed
Bug 1058225
Opened 10 years ago
Closed 10 years ago
Restrict incoming flows in AWS SG's, remove scl3 -> aws rules from fw1.releng.scl3
Categories
(Infrastructure & Operations :: RelOps: General, task)
Infrastructure & Operations
RelOps: General
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: dustin, Assigned: dustin)
References
Details
(Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/4191] )
Attachments
(9 files, 1 obsolete file)
1.32 KB,
text/plain
|
Details | |
6.86 KB,
patch
|
rail
:
review+
dustin
:
checked-in+
|
Details | Diff | Splinter Review |
447 bytes,
patch
|
dustin
:
review+
dustin
:
checked-in+
|
Details | Diff | Splinter Review |
2.25 KB,
patch
|
rail
:
review+
dustin
:
checked-in+
|
Details | Diff | Splinter Review |
9.21 KB,
patch
|
rail
:
review+
dustin
:
checked-in+
|
Details | Diff | Splinter Review |
15.60 KB,
patch
|
rail
:
review+
dustin
:
checked-in+
|
Details | Diff | Splinter Review |
1.89 KB,
patch
|
rail
:
review+
dustin
:
checked-in+
|
Details | Diff | Splinter Review |
7.68 KB,
patch
|
dustin
:
checked-in+
|
Details | Diff | Splinter Review |
5.82 KB,
patch
|
rail
:
review+
dustin
:
checked-in+
|
Details | Diff | Splinter Review |
Rules for flows from scl3 (outside of releng.scl3) to the releng AWS networks are currently governed *both* by rules on fw1.releng.scl3 and in AWS security groups.
This is hard to reason about, and breaks the Mozilla practice of filtering traffic only at its destination or, more rarely, source.
We should modify the security groups to limit the IP ranges appropriately for incoming traffic, and then remove the now-redundant "from-zone dc to-zone aws" policies.
See bug 1044035.
Comment 1•10 years ago
|
||
Hi Dustin -- I will gladly work with you on this from the netops side.
As you point out, on the firewalls, when packets are going to another security zone with
the same data center or a remote data center, we put security policies on the destination side.
(seems to me that we should treat VPC instances at AWS as remote data centers too, but
that's just me...)
Assignee | ||
Comment 2•10 years ago
|
||
That's the idea here. I think things are as they are because of the weird history of the releng-in-AWS project, not for any technical reason.
Updated•10 years ago
|
Assignee: relops → dustin
Comment 3•10 years ago
|
||
A Pivotal Tracker story has been created for this Bug: https://www.pivotaltracker.com/story/show/80405588
Comment 4•10 years ago
|
||
Dustin J. Mitchell added a comment in Pivotal Tracker:
Adjust AWS SG's
test_buildmasters.py: # BUG: AWS flows don't limit http from AWS slaves to AWS masters
test_buildmasters.py: # BUG: AWS flows don't limit http from AWS slaves to AWS masters
test_buildmasters.py: # BUG: AWS is too permissive here
test_buildmasters.py: # BUG 1078296: AWS masters don't have SSH access to scl3 individual masters
Bug 1058225 - Restrict incoming flows in AWS SG's, remove scl3 -> aws rules from fw1.releng.scl3
test_puppetmasters.py:@skip("Flows from outside releng.scl3 to AWS are not represented properly; bug 1058225")
test_puppetmasters.py:@skip("Flows from outside releng.scl3 to AWS are not represented properly; bug 1058225")
test_buildmasters.py: # TODO: bug 1058225: denies on fw1.releng.scl3 for other_mozilla -> releng_aws are not visible to fwunit
Updated•10 years ago
|
Whiteboard: [kanban:engops:https://kanbanize.com/ctrl_board/6/378]
Updated•10 years ago
|
Whiteboard: [kanban:engops:https://kanbanize.com/ctrl_board/6/378] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/957] [kanban:engops:https://kanbanize.com/ctrl_board/6/378]
Updated•10 years ago
|
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/957] [kanban:engops:https://kanbanize.com/ctrl_board/6/378] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/962] [kanban:engops:https://kanbanize.com/ctrl_board/6/378]
Updated•10 years ago
|
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/962] [kanban:engops:https://kanbanize.com/ctrl_board/6/378] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/964] [kanban:engops:https://kanbanize.com/ctrl_board/6/378]
Updated•10 years ago
|
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/964] [kanban:engops:https://kanbanize.com/ctrl_board/6/378] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/378] [kanban:engops:https://kanbanize.com/ctrl_board/6/378]
Assignee | ||
Comment 5•10 years ago
|
||
Python script to print out security groups; the results for the current set of security groups in cloud-tools is
Region: us-east-1
ID: sg-18a07677
Name: windows
rules:
allow 445/tcp from any
allow 3389/tcp from any
allow 5900/tcp from any
rules_egress:
allow None/any to any
Region: us-east-1
ID: sg-31e8185e
Name: buildbot-master
rules:
allow 9301/tcp from any
allow 8001/tcp from any
allow 9302/tcp from any
allow 8201/tcp from any
allow 5666/tcp from any
allow 22/tcp from any
allow 9101/tcp from any
allow 8101/tcp from any
allow 9001/tcp from any
allow any/icmp from any
allow 9201/tcp from any
rules_egress:
allow None/any to any
Region: us-east-1
ID: sg-718b1214
Name: try
rules:
allow 5900/tcp from 10.22.248.0/22 10.22.75.6/31 10.22.252.0/22 10.26.48.25/32 10.26.48.17/32 10.26.48.16/32 10.26.48.23/32 10.26.48.43/32
allow 22/tcp from 10.22.248.0/22 10.22.75.6/31 10.22.252.0/22 10.26.48.25/32 10.26.48.17/32 10.26.48.16/32 10.26.48.23/32 10.26.48.43/32
allow any/icmp from 10.22.248.0/22 10.22.75.6/31 10.22.252.0/22 10.26.48.25/32 10.26.48.17/32 10.26.48.16/32 10.26.48.23/32 10.26.48.43/32
rules_egress:
allow None/any to any
Region: us-east-1
ID: sg-e758e982
Name: build
rules:
allow 5900/tcp from 10.22.248.0/22 10.22.252.0/22 10.26.48.25/32 10.26.48.17/32 10.26.48.16/32 10.26.48.23/32 10.26.48.43/32 10.22.75.6/31
allow 22/tcp from 10.22.248.0/22 10.22.252.0/22 10.26.48.25/32 10.26.48.17/32 10.26.48.16/32 10.26.48.23/32 10.26.48.43/32 10.22.75.6/31
allow any/icmp from 10.22.248.0/22 10.22.252.0/22 10.26.48.25/32 10.26.48.17/32 10.26.48.16/32 10.26.48.23/32 10.26.48.43/32 10.22.75.6/31
rules_egress:
allow None/any to any
Region: us-east-1
ID: sg-f0f1239f
Name: tests
rules:
allow 5900/tcp from 10.26.48.17/32 10.26.48.16/32 10.26.48.25/32 10.26.48.43/32 10.22.252.0/22 10.22.248.0/22 10.26.48.23/32 10.22.75.6/31
allow 22/tcp from 10.26.48.17/32 10.22.252.0/22 10.26.48.43/32 10.26.48.16/32 10.22.248.0/22 10.26.48.25/32 10.26.48.23/32 10.22.75.6/31
allow any/icmp from 10.22.252.0/22 10.26.48.16/32 10.26.48.25/32 10.26.48.17/32 10.26.48.43/32 10.22.248.0/22 10.26.48.23/32 10.22.75.6/31
rules_egress:
allow None/any to any
Region: us-east-1
ID: sg-f3927c9c
Name: default
rules:
allow 22/tcp from any
allow any/icmp from any
rules_egress:
allow None/any to any
Region: us-west-2
ID: sg-3aaa095f
Name: try
rules:
allow 5900/tcp from 10.22.248.0/22 10.22.75.6/31 10.22.252.0/22 10.26.48.25/32 10.26.48.17/32 10.26.48.16/32 10.26.48.23/32 10.26.48.43/32
allow 22/tcp from 10.22.248.0/22 10.22.75.6/31 10.22.252.0/22 10.26.48.25/32 10.26.48.17/32 10.26.48.16/32 10.26.48.23/32 10.26.48.43/32
allow any/icmp from 10.22.248.0/22 10.22.75.6/31 10.22.252.0/22 10.26.48.25/32 10.26.48.17/32 10.26.48.16/32 10.26.48.23/32 10.26.48.43/32
rules_egress:
allow None/any to any
Region: us-west-2
ID: sg-84beade6
Name: windows
rules:
allow 445/tcp from any
allow 3389/tcp from any
allow 5900/tcp from any
rules_egress:
allow None/any to any
Region: us-west-2
ID: sg-8b9f7ce4
Name: tests
rules:
allow 5900/tcp from 10.22.248.0/22 10.22.252.0/22 10.26.48.17/32 10.26.48.16/32 10.26.48.43/32 10.26.48.25/32 10.26.48.23/32 10.22.75.6/31
allow 22/tcp from 10.22.248.0/22 10.22.252.0/22 10.26.48.17/32 10.26.48.16/32 10.26.48.43/32 10.26.48.25/32 10.26.48.23/32 10.22.75.6/31
allow any/icmp from 10.22.248.0/22 10.22.252.0/22 10.26.48.17/32 10.26.48.16/32 10.26.48.43/32 10.26.48.25/32 10.26.48.23/32 10.22.75.6/31
rules_egress:
allow None/any to any
Region: us-west-2
ID: sg-932e33ff
Name: buildbot-master
rules:
allow 9301/tcp from any
allow 8001/tcp from any
allow 9302/tcp from any
allow 8201/tcp from any
allow 5666/tcp from any
allow 22/tcp from any
allow 9101/tcp from any
allow 8101/tcp from any
allow 9001/tcp from any
allow any/icmp from any
allow 9201/tcp from any
rules_egress:
allow None/any to any
Region: us-west-2
ID: sg-d5617cb9
Name: default
rules:
allow 22/tcp from any
allow any/icmp from any
rules_egress:
allow None/any to any
Region: us-west-2
ID: sg-f5ca0690
Name: build
rules:
allow 5900/tcp from 10.22.248.0/22 10.22.252.0/22 10.26.48.25/32 10.26.48.17/32 10.26.48.16/32 10.26.48.23/32 10.26.48.43/32 10.22.75.6/31
allow 22/tcp from 10.22.248.0/22 10.22.252.0/22 10.26.48.25/32 10.26.48.17/32 10.26.48.16/32 10.26.48.23/32 10.26.48.43/32 10.22.75.6/31
allow any/icmp from 10.22.248.0/22 10.22.252.0/22 10.26.48.25/32 10.26.48.17/32 10.26.48.16/32 10.26.48.23/32 10.26.48.43/32 10.22.75.6/31
rules_egress:
allow None/any to any
Rail, what do you think is the best way to modify these? Alter the SG's in place? Cretae new SG's and update cloud-tools? When would either option take effect?
Flags: needinfo?(rail)
Assignee | ||
Comment 6•10 years ago
|
||
..or should we invent a way to program these via cloud-tools?
Comment 7•10 years ago
|
||
If you update/add rules in http://hg.mozilla.org/build/cloud-tools/file/5d32265cd9f1/configs/securitygroups.yml that should work too. The script using that config is idempotent and will replace the rules in place.
Will this help?
Flags: needinfo?(rail)
Assignee | ||
Comment 8•10 years ago
|
||
Hah, well, I feel dumb. Per irc, the applies-to in securitygroups.yml can be ignored, and the fact that the config/<hotstype> files refer to sg's by id isn't important here (although it would confuse things if we had to wipe our config and start over). So, I'll make a patch.
Assignee | ||
Comment 9•10 years ago
|
||
I'm writing rules for
19 - cruncher.build.mozilla.org
20 - slaveapi1.srv.releng.scl3.mozilla.com
21 - slaveapi-dev1.srv.releng.scl3.mozilla.com
22 - 10.22.75.6/31
23 - 10.22.252.0/22
24 - 10.22.248.0/22
25 - dev-master1.build.mozilla.org
26 - aws-manager1.srv.releng.scl3.mozilla.com
and it's worth noting from that set that the dev-master1 flow actually doesn't work (it's blocked by the SRX). The 10.22 IPs are admin1a/b, scl3-vpn, and the cloud VLANs, respectively. The latter is gone now.
Assignee | ||
Comment 10•10 years ago
|
||
Here are the rules to replicate:
https://infra.etherpad.mozilla.org/1067178
(sorry for those without access)
Assignee | ||
Comment 11•10 years ago
|
||
Bug 1058225: attempt to (mostly) replicate the current AWS config
This replaces the specific source IPs for ping with 0.0.0.0/0, which is
what we do for the rest of Mozilla (ever notice you can ping anything?).
It also removes the openstack network, gets the netmask right for the
VPN, and adds some comments to identify subnets.
This is based on clicking around the AWS UI. If this is OK, how should I go about landing it safely?
Once it's landed, I'll submit a few patches further limiting access based on what's currently configured in the SRXes, until I'm confident that everything's represented in the security groups. Then we can remove the rules from the SRXes.
Attachment #8518532 -
Flags: review?(rail)
Updated•10 years ago
|
Attachment #8518532 -
Flags: review?(rail) → review+
Assignee | ||
Updated•10 years ago
|
Attachment #8518532 -
Flags: checked-in+
Assignee | ||
Comment 12•10 years ago
|
||
ugh, aws_manage_securitygroups.py is really buggy. Although it asks to confirm each change, it then makes different changes:
tests - Delete rule ('inbound', u'icmp', u'-1', u'-1') to set([u'10.22.248.0/22', u'10.22.75.6/31', u'10.22.252.0/22', u'10.26.48.25/32', u'10.26.48.17/32', u'10.26.48.16/32', u'10.26.48.23/32', u'10.26.48.43/32']) (y/N) y
2014-11-07 12:03:36,376 - tests - removing rule for ('inbound', u'icmp', u'-1', u'-1') to set(['10.22.240.0/20'])
So, that sucks.
Assignee | ||
Comment 13•10 years ago
|
||
I edited up the bits it modified by hand, so we should be good for now. There are still a bunch of security groups to go, though.
Updated•10 years ago
|
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/378] [kanban:engops:https://kanbanize.com/ctrl_board/6/378] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/378]
Assignee | ||
Comment 14•10 years ago
|
||
I found port 8201 used in one region and not the others.
Attachment #8524080 -
Flags: review?(rail)
Assignee | ||
Comment 15•10 years ago
|
||
Comment on attachment 8524080 [details] [diff] [review]
add-8201.patch
r+ from nthomas in rc
Attachment #8524080 -
Flags: review?(rail)
Attachment #8524080 -
Flags: review+
Attachment #8524080 -
Flags: checked-in+
Assignee | ||
Comment 16•10 years ago
|
||
Attachment #8524136 -
Flags: review?(rail)
Assignee | ||
Comment 17•10 years ago
|
||
This should help as things get longer :)
Attachment #8524152 -
Flags: review?(rail)
Assignee | ||
Comment 18•10 years ago
|
||
It turns out includes within includes are useful..
Attachment #8524152 -
Attachment is obsolete: true
Attachment #8524152 -
Flags: review?(rail)
Attachment #8524161 -
Flags: review?(rail)
Assignee | ||
Comment 19•10 years ago
|
||
This one needs some serious squinting at! I based this on a read of all flows from scl3 to the VPC (in the SRX configuration), intersected with existing allowed flows in scl3, plus my understanding of what intra-AWS flows should look like.
Specific notes:
- admin-access contains more hosts than had access before, and allows any port,
not just ssh
- observium and infra-puppetize are new, but don't hurt
- the puppet security group has been removed (bug 1022368)
- incoming access on port 5900 doesn't work now, so it has been removed
- blobber access is now limited to slave VLANs, and only HTTPS was permitted by the Junipers
so only HTTPS is listed here
- incoming access to buildbot masters was refactored to include both traffic over the VPC
and internal traffic from buildslaves
If it helps, here's the output of the script (with everything answered 'n'):
> 2014-11-17 19:28:26,725 - Working in regions set(['us-west-2', 'us-east-1', 'us-west-1'])
> 2014-11-17 19:28:26,725 - Loading groups for us-west-2
> 2014-11-17 19:28:27,351 - Loading groups for us-east-1
> 2014-11-17 19:28:27,710 - Loading groups for us-west-1
> 2014-11-17 19:28:28,252 - Working in us-west-2
> 2014-11-17 19:28:28,252 - Looking for sg tests
> 2014-11-17 19:28:28,252 - Found SecurityGroup:tests
> tests - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N)
> tests - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.22.75.36/32'])? (y/N)
> tests - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N)
> tests - Delete rule ('inbound', u'tcp', u'5900', u'5900') to set([u'10.22.75.6/31', u'10.26.48.25/32', u'10.26.48.17/32', u'10.26.48.16/32', u'10.26.48.23/32', u'10.22.240.0/20', u'10.26.48.43/32']) (y/N)
> tests - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'10.22.75.6/31', u'10.22.240.0/20']) (y/N)
> 2014-11-17 19:28:30,763 - Working in us-east-1
> 2014-11-17 19:28:30,764 - Looking for sg tests
> 2014-11-17 19:28:30,764 - Found SecurityGroup:tests
> tests - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N)
> tests - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.22.75.36/32'])? (y/N)
> tests - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N)
> tests - Delete rule ('inbound', u'tcp', u'5900', u'5900') to set([u'10.22.75.6/31', u'10.26.48.25/32', u'10.26.48.17/32', u'10.26.48.16/32', u'10.26.48.23/32', u'10.22.240.0/20', u'10.26.48.43/32']) (y/N)
> tests - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'10.22.75.6/31', u'10.22.240.0/20']) (y/N)
> 2014-11-17 19:28:33,644 - Working in us-west-1
> 2014-11-17 19:28:33,644 - Looking for sg tests
> 2014-11-17 19:28:33,644 - Found SecurityGroup:tests
> tests - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N)
> tests - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.22.75.36/32'])? (y/N)
> tests - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N)
> tests - Delete rule ('inbound', u'tcp', u'5900', u'5900') to set([u'10.22.75.6/31', u'10.26.48.25/32', u'10.26.48.17/32', u'10.26.48.16/32', u'10.26.48.23/32', u'10.22.240.0/20', u'10.26.48.43/32']) (y/N)
> tests - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'10.22.75.6/31', u'10.22.240.0/20']) (y/N)
> 2014-11-17 19:28:33,962 - Working in us-west-2
> 2014-11-17 19:28:33,963 - Looking for sg buildbot-master
> 2014-11-17 19:28:33,963 - Found SecurityGroup:buildbot-master
> buildbot-master - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.26.48.24/32', '10.26.48.26/32', '10.26.48.30/32', '10.132.49.154/32', '10.26.48.19/32', '10.26.48.25/32', '10.134.49.77/32', '10.26.48.59/32', '10.26.48.36/32', '10
> .26.48.52/32', '10.132.49.29/32', '10.134.49.144/32', '10.26.48.57/32', '10.132.49.158/32', '10.132.50.54/32', '10.134.68.0/24', '10.26.68.0/24', '10.134.48.7/32', '10.26.48.51/32', '10.132.48.17/32', '10.134.49.94/32', '10.132.48.136/32', '10.132.50.189/32', '10.26.48.40/32', '10.26.48.38/32', '10.26.48.55/32', '10.132.50.240/32', '10.134.48.122/32', '10.132.49.181/32', '10.132.49.94/32', '10.26.48.56/32', '10.26.48.54/32', '10.26.48.20/32', '10.134.48.86/32', '10.26.48.37/32', '10.26.48.28/32', '10.132.48.18/32', '10.26.48.23/32', '10.132.48.16/32', '10.132.68.0/24', '10.132.49.117/32', '10.134.48.8/32', '10.134.49.70/32', '10.134.48.9/32', '10.134.48.31/32', '10.26.48.17/32', '10.26.48.39/32', '10.26.48.27/32', '10.134.48.10/32', '10.134.48.234/32', '10.26.48.16/32', '10.26.48.43/32', '10.22.75.36/32', '10.26.48.53/32', '10.134.48.40/32', '10.134.48.106/32', '10.26.48.32/32', '10.132.50.247/32', '10.132.50.89/32', '10.134.49.111/32'])? (y/N)
> buildbot-master - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N)
> buildbot-master - Add rule for ('inbound', 'tcp', '8000', '8999') to set(['10.26.48.24/32', '10.26.48.26/32', '10.26.48.30/32', '10.132.49.154/32', '10.26.48.19/32', '10.134.49.77/32', '10.26.48.59/32', '10.26.48.36/32', '10.26.48.52/32', '10.132.49.29/32', '10.134.49.144/32', '10.26.48.57/32', '10.132.49.158/32', '10.132.50.54/32', '10.134.68.0/24', '10.26.68.0/24', '10.134.48.7/32', '10.26.48.51/32', '10.132.48.17/32', '10.134.49.94/32', '10.132.48.136/32', '10.132.50.189/32', '10.26.48.40/32', '10.26.48.55/32', '10.132.50.240/32', '10.134.48.122/32', '10.132.49.181/32', '10.26.48.56/32', '10.26.48.54/32', '10.26.48.20/32', '10.134.48.86/32', '10.26.48.37/32', '10.26.48.28/32', '10.132.48.18/32', '10.26.48.23/32', '10.132.48.16/32', '10.132.68.0/24', '10.132.49.117/32', '10.134.48.8/32', '10.134.49.70/32', '10.134.48.9/32', '10.134.48.31/32', '10.26.48.39/32', '10.26.48.27/32', '10.134.48.10/32', '10.134.48.234/32', '10.26.48.38/32', '10.26.48.43/32', '10.132.49.94/32', '10.26.48.53/32', '10.134.48.40/32', '10.134.48.106/32', '10.26.48.32/32', '10.132.50.247/32', '10.132.50.89/32', '10.134.49.111/32'])? (y/N)
> buildbot-master - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N)
> buildbot-master - Add rule for ('inbound', 'tcp', '9000', '9999') to set(['10.26.36.0/22', '10.26.48.24/32', '10.26.48.53/32', '10.26.48.26/32', '10.134.68.0/24', '10.132.49.154/32', '10.26.48.19/32', '10.134.49.77/32', '10.134.156.0/22', '10.132.56.0/22', '10.26.48.59/32', '10.26.48.36/32', '10.26.48.52/32', '10.132.49.29/32', '10.132.64.0/22', '10.134.49.144/32', '10.26.48.57/32', '10.132.49.158/32', '10.132.50.54/32', '10.26.48.30/32', '10.26.68.0/24', '10.134.48.7/32', '10.26.48.51/32', '10.132.48.17/32', '10.134.49.94/32', '10.132.48.136/32', '10.132.50.189/32', '10.26.48.40/32', '10.26.48.39/32', '10.134.48.106/32', '10.26.48.55/32', '10.132.50.240/32', '10.134.48.122/32', '10.132.49.181/32', '10.26.128.0/17', '10.26.48.56/32', '10.26.48.54/32', '10.134.52.0/22', '10.26.48.20/32', '10.134.48.86/32', '10.26.48.37/32', '10.26.48.28/32', '10.26.88.0/21', '10.132.48.18/32', '10.132.48.16/32', '10.132.68.0/24', '10.132.49.117/32', '10.134.48.8/32', '10.26.56.0/22', '10.134.49.70/32', '10.134.48.9/32', '10.132.52.0/22', '10.134.48.31/32', '10.134.64.0/22', '10.26.44.0/22', '10.26.52.0/22', '10.132.156.0/22', '10.26.48.27/32', '10.134.48.10/32', '10.134.48.234/32', '10.26.48.38/32', '10.26.48.43/32', '10.132.49.94/32', '10.134.56.0/22', '10.134.48.40/32', '10.26.40.0/22', '10.26.48.32/32', '10.132.50.247/32', '10.132.50.89/32', '10.134.49.111/32'])? (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'9301', u'9301') to set([u'0.0.0.0/0']) (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'8202', u'8202') to set([u'0.0.0.0/0']) (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'5666', u'5666') to set([u'0.0.0.0/0']) (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'9001', u'9001') to set([u'0.0.0.0/0']) (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'8201', u'8201') to set([u'0.0.0.0/0']) (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'0.0.0.0/0']) (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'9302', u'9302') to set([u'0.0.0.0/0']) (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'8001', u'8001') to set([u'0.0.0.0/0']) (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'8101', u'8101') to set([u'0.0.0.0/0']) (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'9101', u'9101') to set([u'0.0.0.0/0']) (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'9201', u'9201') to set([u'0.0.0.0/0']) (y/N)
> 2014-11-17 19:28:33,963 - No interface filters to apply, skipping.
> 2014-11-17 19:28:33,963 - No interface filters to apply, skipping.
> 2014-11-17 19:28:33,963 - Working in us-east-1
> 2014-11-17 19:28:33,964 - Looking for sg buildbot-master
> 2014-11-17 19:28:33,964 - Found SecurityGroup:buildbot-master
> buildbot-master - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.26.48.24/32', '10.26.48.26/32', '10.26.48.30/32', '10.132.49.154/32', '10.26.48.19/32', '10.26.48.25/32', '10.134.49.77/32', '10.26.48.59/32', '10.26.48.36/32', '10.26.48.52/32', '10.132.49.29/32', '10.134.49.144/32', '10.26.48.57/32', '10.132.49.158/32', '10.132.50.54/32', '10.134.68.0/24', '10.26.68.0/24', '10.134.48.7/32', '10.26.48.51/32', '10.132.48.17/32', '10.134.49.94/32', '10.132.48.136/32', '10.132.50.189/32', '10.26.48.40/32', '10.26.48.38/32', '10.26.48.55/32', '10.132.50.240/32', '10.134.48.122/32', '10.132.49.181/32', '10.132.49.94/32', '10.26.48.56/32', '10.26.48.54/32', '10.26.48.20/32', '10.134.48.86/32', '10.26.48.37/32', '10.26.48.28/32', '10.132.48.18/32', '10.26.48.23/32', '10.132.48.16/32', '10.132.68.0/24', '10.132.49.117/32', '10.134.48.8/32', '10.134.49.70/32', '10.134.48.9/32', '10.134.48.31/32', '10.26.48.17/32', '10.26.48.39/32', '10.26.48.27/32', '10.134.48.10/32', '10.134.48.234/32', '10.26.48.16/32', '10.26.48.43/32', '10.22.75.36/32', '10.26.48.53/32', '10.134.48.40/32', '10.134.48.106/32', '10.26.48.32/32', '10.132.50.247/32', '10.132.50.89/32', '10.134.49.111/32'])? (y/N)
> buildbot-master - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N)
> buildbot-master - Add rule for ('inbound', 'tcp', '8000', '8999') to set(['10.26.48.24/32', '10.26.48.26/32', '10.26.48.30/32', '10.132.49.154/32', '10.26.48.19/32', '10.134.49.77/32', '10.26.48.59/32', '10.26.48.36/32', '10.26.48.52/32', '10.132.49.29/32', '10.134.49.144/32', '10.26.48.57/32', '10.132.49.158/32', '10.132.50.54/32', '10.134.68.0/24', '10.26.68.0/24', '10.134.48.7/32', '10.26.48.51/32', '10.132.48.17/32', '10.134.49.94/32', '10.132.48.136/32', '10.132.50.189/32', '10.26.48.40/32', '10.26.48.55/32', '10.132.50.240/32', '10.134.48.122/32', '10.132.49.181/32', '10.26.48.56/32', '10.26.48.54/32', '10.26.48.20/32', '10.134.48.86/32', '10.26.48.37/32', '10.26.48.28/32', '10.132.48.18/32', '10.26.48.23/32', '10.132.48.16/32', '10.132.68.0/24', '10.132.49.117/32', '10.134.48.8/32', '10.134.49.70/32', '10.134.48.9/32', '10.134.48.31/32', '10.26.48.39/32', '10.26.48.27/32', '10.134.48.10/32', '10.134.48.234/32', '10.26.48.38/32', '10.26.48.43/32', '10.132.49.94/32', '10.26.48.53/32', '10.134.48.40/32', '10.134.48.106/32', '10.26.48.32/32', '10.132.50.247/32', '10.132.50.89/32', '10.134.49.111/32'])? (y/N)
> buildbot-master - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N)
> buildbot-master - Add rule for ('inbound', 'tcp', '9000', '9999') to set(['10.26.36.0/22', '10.26.48.24/32', '10.26.48.53/32', '10.26.48.26/32', '10.134.68.0/24', '10.132.49.154/32', '10.26.48.19/32', '10.134.49.77/32', '10.134.156.0/22', '10.132.56.0/22', '10.26.48.59/32', '10.26.48.36/32', '10.26.48.52/32', '10.132.49.29/32', '10.132.64.0/22', '10.134.49.144/32', '10.26.48.57/32', '10.132.49.158/32', '10.132.50.54/32', '10.26.48.30/32', '10.26.68.0/24', '10.134.48.7/32', '10.26.48.51/32', '10.132.48.17/32', '10.134.49.94/32', '10.132.48.136/32', '10.132.50.189/32', '10.26.48.40/32', '10.26.48.39/32', '10.134.48.106/32', '10.26.48.55/32', '10.132.50.240/32', '10.134.48.122/32', '10.132.49.181/32', '10.26.128.0/17', '10.26.48.56/32', '10.26.48.54/32', '10.134.52.0/22', '10.26.48.20/32', '10.134.48.86/32', '10.26.48.37/32', '10.26.48.28/32', '10.26.88.0/21', '10.132.48.18/32', '10.132.48.16/32', '10.132.68.0/24', '10.132.49.117/32', '10.134.48.8/32', '10.26.56.0/22', '10.134.49.70/32', '10.134.48.9/32', '10.132.52.0/22', '10.134.48.31/32', '10.134.64.0/22', '10.26.44.0/22', '10.26.52.0/22', '10.132.156.0/22', '10.26.48.27/32', '10.134.48.10/32', '10.134.48.234/32', '10.26.48.38/32', '10.26.48.43/32', '10.132.49.94/32', '10.134.56.0/22', '10.134.48.40/32', '10.26.40.0/22', '10.26.48.32/32', '10.132.50.247/32', '10.132.50.89/32', '10.134.49.111/32'])? (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'9301', u'9301') to set([u'0.0.0.0/0']) (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'8202', u'8202') to set([u'0.0.0.0/0']) (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'5666', u'5666') to set([u'0.0.0.0/0']) (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'9001', u'9001') to set([u'0.0.0.0/0']) (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'8201', u'8201') to set([u'0.0.0.0/0']) (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'0.0.0.0/0']) (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'9302', u'9302') to set([u'0.0.0.0/0']) (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'8001', u'8001') to set([u'0.0.0.0/0']) (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'8101', u'8101') to set([u'0.0.0.0/0']) (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'9101', u'9101') to set([u'0.0.0.0/0']) (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'9201', u'9201') to set([u'0.0.0.0/0']) (y/N)
> 2014-11-17 19:28:33,964 - No interface filters to apply, skipping.
> 2014-11-17 19:28:33,964 - No interface filters to apply, skipping.
> 2014-11-17 19:28:33,964 - Working in us-west-1
> 2014-11-17 19:28:33,964 - Looking for sg buildbot-master
> 2014-11-17 19:28:33,964 - Found SecurityGroup:buildbot-master
> buildbot-master - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.26.48.24/32', '10.26.48.26/32', '10.26.48.30/32', '10.132.49.154/32', '10.26.48.19/32', '10.26.48.25/32', '10.134.49.77/32', '10.26.48.59/32', '10.26.48.36/32', '10.26.48.52/32', '10.132.49.29/32', '10.134.49.144/32', '10.26.48.57/32', '10.132.49.158/32', '10.132.50.54/32', '10.134.68.0/24', '10.26.68.0/24', '10.134.48.7/32', '10.26.48.51/32', '10.132.48.17/32', '10.134.49.94/32', '10.132.48.136/32', '10.132.50.189/32', '10.26.48.40/32', '10.26.48.38/32', '10.26.48.55/32', '10.132.50.240/32', '10.134.48.122/32', '10.132.49.181/32', '10.132.49.94/32', '10.26.48.56/32', '10.26.48.54/32', '10.26.48.20/32', '10.134.48.86/32', '10.26.48.37/32', '10.26.48.28/32', '10.132.48.18/32', '10.26.48.23/32', '10.132.48.16/32', '10.132.68.0/24', '10.132.49.117/32', '10.134.48.8/32', '10.134.49.70/32', '10.134.48.9/32', '10.134.48.31/32', '10.26.48.17/32', '10.26.48.39/32', '10.26.48.27/32', '10.134.48.10/32', '10.134.48.234/32', '10.26.48.16/32', '10.26.48.43/32', '10.22.75.36/32', '10.26.48.53/32', '10.134.48.40/32', '10.134.48.106/32', '10.26.48.32/32', '10.132.50.247/32', '10.132.50.89/32', '10.134.49.111/32'])? (y/N)
> buildbot-master - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N)
> buildbot-master - Add rule for ('inbound', 'tcp', '8000', '8999') to set(['10.26.48.24/32', '10.26.48.26/32', '10.26.48.30/32', '10.132.49.154/32', '10.26.48.19/32', '10.134.49.77/32', '10.26.48.59/32', '10.26.48.36/32', '10.26.48.52/32', '10.132.49.29/32', '10.134.49.144/32', '10.26.48.57/32', '10.132.49.158/32', '10.132.50.54/32', '10.134.68.0/24', '10.26.68.0/24', '10.134.48.7/32', '10.26.48.51/32', '10.132.48.17/32', '10.134.49.94/32', '10.132.48.136/32', '10.132.50.189/32', '10.26.48.40/32', '10.26.48.55/32', '10.132.50.240/32', '10.134.48.122/32', '10.132.49.181/32', '10.26.48.56/32', '10.26.48.54/32', '10.26.48.20/32', '10.134.48.86/32', '10.26.48.37/32', '10.26.48.28/32', '10.132.48.18/32', '10.26.48.23/32', '10.132.48.16/32', '10.132.68.0/24', '10.132.49.117/32', '10.134.48.8/32', '10.134.49.70/32', '10.134.48.9/32', '10.134.48.31/32', '10.26.48.39/32', '10.26.48.27/32', '10.134.48.10/32', '10.134.48.234/32', '10.26.48.38/32', '10.26.48.43/32', '10.132.49.94/32', '10.26.48.53/32', '10.134.48.40/32', '10.134.48.106/32', '10.26.48.32/32', '10.132.50.247/32', '10.132.50.89/32', '10.134.49.111/32'])? (y/N)
> buildbot-master - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N)
> buildbot-master - Add rule for ('inbound', 'tcp', '9000', '9999') to set(['10.26.36.0/22', '10.26.48.24/32', '10.26.48.53/32', '10.26.48.26/32', '10.134.68.0/24', '10.132.49.154/32', '10.26.48.19/32', '10.134.49.77/32', '10.134.156.0/22', '10.132.56.0/22', '10.26.48.59/32', '10.26.48.36/32', '10.26.48.52/32', '10.132.49.29/32', '10.132.64.0/22', '10.134.49.144/32', '10.26.48.57/32', '10.132.49.158/32', '10.132.50.54/32', '10.26.48.30/32', '10.26.68.0/24', '10.134.48.7/32', '10.26.48.51/32', '10.132.48.17/32', '10.134.49.94/32', '10.132.48.136/32', '10.132.50.189/32', '10.26.48.40/32', '10.26.48.39/32', '10.134.48.106/32', '10.26.48.55/32', '10.132.50.240/32', '10.134.48.122/32', '10.132.49.181/32', '10.26.128.0/17', '10.26.48.56/32', '10.26.48.54/32', '10.134.52.0/22', '10.26.48.20/32', '10.134.48.86/32', '10.26.48.37/32', '10.26.48.28/32', '10.26.88.0/21', '10.132.48.18/32', '10.132.48.16/32', '10.132.68.0/24', '10.132.49.117/32', '10.134.48.8/32', '10.26.56.0/22', '10.134.49.70/32', '10.134.48.9/32', '10.132.52.0/22', '10.134.48.31/32', '10.134.64.0/22', '10.26.44.0/22', '10.26.52.0/22', '10.132.156.0/22', '10.26.48.27/32', '10.134.48.10/32', '10.134.48.234/32', '10.26.48.38/32', '10.26.48.43/32', '10.132.49.94/32', '10.134.56.0/22', '10.134.48.40/32', '10.26.40.0/22', '10.26.48.32/32', '10.132.50.247/32', '10.132.50.89/32', '10.134.49.111/32'])? (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'9301', u'9301') to set([u'0.0.0.0/0']) (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'8202', u'8202') to set([u'0.0.0.0/0']) (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'5666', u'5666') to set([u'0.0.0.0/0']) (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'9001', u'9001') to set([u'0.0.0.0/0']) (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'8201', u'8201') to set([u'0.0.0.0/0']) (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'0.0.0.0/0']) (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'9302', u'9302') to set([u'0.0.0.0/0']) (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'8001', u'8001') to set([u'0.0.0.0/0']) (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'8101', u'8101') to set([u'0.0.0.0/0']) (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'9101', u'9101') to set([u'0.0.0.0/0']) (y/N)
> buildbot-master - Delete rule ('inbound', u'tcp', u'9201', u'9201') to set([u'0.0.0.0/0']) (y/N)
> 2014-11-17 19:28:33,967 - No interface filters to apply, skipping.
> 2014-11-17 19:28:33,967 - No interface filters to apply, skipping.
> 2014-11-17 19:28:33,967 - Working in us-west-2
> 2014-11-17 19:28:33,967 - Looking for sg nagios
> 2014-11-17 19:28:33,967 - Found SecurityGroup:nagios
> nagios - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.22.75.36/32'])? (y/N)
> nagios - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N)
> nagios - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N)
> nagios - Delete rule ('inbound', u'tcp', u'443', u'443') to set([u'0.0.0.0/0']) (y/N)
> nagios - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'0.0.0.0/0']) (y/N)
> nagios - Delete rule ('inbound', u'tcp', u'80', u'80') to set([u'0.0.0.0/0']) (y/N)
> 2014-11-17 19:28:33,967 - No interface filters to apply, skipping.
> 2014-11-17 19:28:33,967 - No interface filters to apply, skipping.
> 2014-11-17 19:28:33,967 - Working in us-east-1
> 2014-11-17 19:28:33,967 - Looking for sg nagios
> 2014-11-17 19:28:33,967 - Found SecurityGroup:nagios
> nagios - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.22.75.36/32'])? (y/N)
> nagios - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N)
> nagios - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N)
> nagios - Delete rule ('inbound', u'tcp', u'443', u'443') to set([u'0.0.0.0/0']) (y/N)
> nagios - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'0.0.0.0/0']) (y/N)
> nagios - Delete rule ('inbound', u'tcp', u'80', u'80') to set([u'0.0.0.0/0']) (y/N)
> 2014-11-17 19:28:33,968 - No interface filters to apply, skipping.
> 2014-11-17 19:28:33,968 - No interface filters to apply, skipping.
> 2014-11-17 19:28:33,968 - Working in us-west-1
> 2014-11-17 19:28:33,968 - Looking for sg nagios
> 2014-11-17 19:28:33,968 - Found SecurityGroup:nagios
> nagios - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.22.75.36/32'])? (y/N)
> nagios - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N)
> nagios - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N)
> nagios - Delete rule ('inbound', u'tcp', u'443', u'443') to set([u'0.0.0.0/0']) (y/N)
> nagios - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'0.0.0.0/0']) (y/N)
> nagios - Delete rule ('inbound', u'tcp', u'80', u'80') to set([u'0.0.0.0/0']) (y/N)
> 2014-11-17 19:28:33,968 - No interface filters to apply, skipping.
> 2014-11-17 19:28:33,968 - No interface filters to apply, skipping.
> 2014-11-17 19:28:33,968 - Working in us-west-2
> 2014-11-17 19:28:33,968 - Looking for sg try
> 2014-11-17 19:28:33,968 - Found SecurityGroup:try
> try - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N)
> try - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.22.75.36/32'])? (y/N)
> try - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N)
> try - Delete rule ('inbound', u'tcp', u'5900', u'5900') to set([u'10.22.75.6/31', u'10.26.48.25/32', u'10.26.48.17/32', u'10.26.48.16/32', u'10.26.48.23/32', u'10.22.240.0/20', u'10.26.48.43/32']) (y/N)
> try - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'10.22.75.6/31', u'10.22.240.0/20']) (y/N)
> 2014-11-17 19:28:34,374 - Working in us-east-1
> 2014-11-17 19:28:34,374 - Looking for sg try
> 2014-11-17 19:28:34,375 - Found SecurityGroup:try
> try - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N)
> try - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.22.75.36/32'])? (y/N)
> try - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N)
> try - Delete rule ('inbound', u'tcp', u'5900', u'5900') to set([u'10.22.75.6/31', u'10.26.48.25/32', u'10.26.48.17/32', u'10.26.48.16/32', u'10.26.48.23/32', u'10.22.240.0/20', u'10.26.48.43/32']) (y/N)
> try - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'10.22.75.6/31', u'10.22.240.0/20']) (y/N)
> 2014-11-17 19:28:35,024 - Working in us-west-1
> 2014-11-17 19:28:35,024 - Looking for sg try
> 2014-11-17 19:28:35,025 - Found SecurityGroup:try
> try - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N)
> try - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.22.75.36/32'])? (y/N)
> try - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N)
> try - Delete rule ('inbound', u'tcp', u'5900', u'5900') to set([u'10.22.75.6/31', u'10.26.48.25/32', u'10.26.48.17/32', u'10.26.48.16/32', u'10.26.48.23/32', u'10.22.240.0/20', u'10.26.48.43/32']) (y/N)
> try - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'10.22.75.6/31', u'10.22.240.0/20']) (y/N)
> 2014-11-17 19:28:35,315 - Working in us-west-2
> 2014-11-17 19:28:35,315 - Looking for sg build
> 2014-11-17 19:28:35,315 - Found SecurityGroup:build
> build - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N)
> build - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.22.75.36/32'])? (y/N)
> build - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N)
> build - Delete rule ('inbound', u'tcp', u'5900', u'5900') to set([u'10.22.75.6/31', u'10.26.48.25/32', u'10.26.48.17/32', u'10.26.48.16/32', u'10.26.48.23/32', u'10.22.240.0/20', u'10.26.48.43/32']) (y/N)
> build - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'10.22.75.6/31', u'10.22.240.0/20']) (y/N)
> 2014-11-17 19:28:35,871 - Working in us-east-1
> 2014-11-17 19:28:35,871 - Looking for sg build
> 2014-11-17 19:28:35,871 - Found SecurityGroup:build
> build - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N)
> build - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.22.75.36/32'])? (y/N)
> build - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N)
> build - Delete rule ('inbound', u'tcp', u'5900', u'5900') to set([u'10.22.75.6/31', u'10.26.48.25/32', u'10.26.48.17/32', u'10.26.48.16/32', u'10.26.48.23/32', u'10.22.240.0/20', u'10.26.48.43/32']) (y/N)
> build - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'10.22.75.6/31', u'10.22.240.0/20']) (y/N)
> 2014-11-17 19:28:36,630 - Working in us-west-1
> 2014-11-17 19:28:36,630 - Looking for sg build
> 2014-11-17 19:28:36,630 - Found SecurityGroup:build
> build - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N)
> build - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.22.75.36/32'])? (y/N)
> build - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N)
> build - Delete rule ('inbound', u'tcp', u'5900', u'5900') to set([u'10.22.75.6/31', u'10.26.48.25/32', u'10.26.48.17/32', u'10.26.48.16/32', u'10.26.48.23/32', u'10.22.240.0/20', u'10.26.48.43/32']) (y/N)
> build - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'10.22.75.6/31', u'10.22.240.0/20']) (y/N)
> 2014-11-17 19:28:36,861 - Working in us-west-2
> 2014-11-17 19:28:36,861 - Looking for sg blobber
> 2014-11-17 19:28:36,862 - Found SecurityGroup:blobber
> blobber - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.22.75.36/32'])? (y/N)
> blobber - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N)
> blobber - Add rule for ('inbound', 'tcp', '443', '443') to set(['10.26.36.0/22', '10.132.64.0/22', '10.26.128.0/17', '10.134.64.0/22', '10.26.52.0/22', '10.134.52.0/22', '10.26.44.0/22', '10.132.52.0/22', '10.134.156.0/22', '10.132.156.0/22', '10.26.88.0/21', '10.134.56.0/22', '10.132.56.0/22', '10.26.56.0/22', '10.26.40.0/22'])? (y/N)
> blobber - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N)
> blobber - Delete rule ('inbound', u'tcp', u'443', u'443') to set([u'10.0.0.0/8']) (y/N)
> blobber - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'0.0.0.0/0']) (y/N)
> blobber - Delete rule ('inbound', u'tcp', u'80', u'80') to set([u'10.0.0.0/8']) (y/N)
> 2014-11-17 19:28:36,862 - No interface filters to apply, skipping.
> 2014-11-17 19:28:36,862 - No interface filters to apply, skipping.
> 2014-11-17 19:28:36,862 - Working in us-east-1
> 2014-11-17 19:28:36,862 - Looking for sg blobber
> 2014-11-17 19:28:36,862 - Found SecurityGroup:blobber
> blobber - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.22.75.36/32'])? (y/N)
> blobber - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N)
> blobber - Add rule for ('inbound', 'tcp', '443', '443') to set(['10.26.36.0/22', '10.132.64.0/22', '10.26.128.0/17', '10.134.64.0/22', '10.26.52.0/22', '10.134.52.0/22', '10.26.44.0/22', '10.132.52.0/22', '10.134.156.0/22', '10.132.156.0/22', '10.26.88.0/21', '10.134.56.0/22', '10.132.56.0/22', '10.26.56.0/22', '10.26.40.0/22'])? (y/N)
> blobber - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N)
> blobber - Delete rule ('inbound', u'tcp', u'443', u'443') to set([u'10.0.0.0/8']) (y/N)
> blobber - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'0.0.0.0/0']) (y/N)
> blobber - Delete rule ('inbound', u'tcp', u'80', u'80') to set([u'10.0.0.0/8']) (y/N)
> 2014-11-17 19:28:36,863 - No interface filters to apply, skipping.
> 2014-11-17 19:28:36,863 - No interface filters to apply, skipping.
> 2014-11-17 19:28:36,863 - Working in us-west-1
> 2014-11-17 19:28:36,863 - Looking for sg blobber
> 2014-11-17 19:28:36,863 - Found SecurityGroup:blobber
> blobber - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.22.75.36/32'])? (y/N)
> blobber - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N)
> blobber - Add rule for ('inbound', 'tcp', '443', '443') to set(['10.26.36.0/22', '10.132.64.0/22', '10.26.128.0/17', '10.134.64.0/22', '10.26.52.0/22', '10.134.52.0/22', '10.26.44.0/22', '10.132.52.0/22', '10.134.156.0/22', '10.132.156.0/22', '10.26.88.0/21', '10.134.56.0/22', '10.132.56.0/22', '10.26.56.0/22', '10.26.40.0/22'])? (y/N)
> blobber - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N)
> blobber - Delete rule ('inbound', u'tcp', u'443', u'443') to set([u'10.0.0.0/8']) (y/N)
> blobber - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'0.0.0.0/0']) (y/N)
> blobber - Delete rule ('inbound', u'tcp', u'80', u'80') to set([u'10.0.0.0/8']) (y/N)
> 2014-11-17 19:28:36,863 - No interface filters to apply, skipping.
> 2014-11-17 19:28:36,863 - No interface filters to apply, skipping.
Attachment #8524193 -
Flags: review?(rail)
Updated•10 years ago
|
Attachment #8524136 -
Flags: review?(rail) → review+
Updated•10 years ago
|
Attachment #8524161 -
Flags: review?(rail) → review+
Updated•10 years ago
|
Attachment #8524193 -
Flags: review?(rail) → review+
Assignee | ||
Updated•10 years ago
|
Attachment #8524193 -
Flags: checked-in+
Assignee | ||
Updated•10 years ago
|
Attachment #8524161 -
Flags: checked-in+
Assignee | ||
Updated•10 years ago
|
Attachment #8524136 -
Flags: checked-in+
Assignee | ||
Comment 20•10 years ago
|
||
So midway through applying that, I ran into
RulesPerSecurityGroupLimitExceeded:
The maximum number of rules per security group has been reached
I had been hoping that "Rules" in http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Appendix_Limits.html referred to, you know, rules. It doesn't -- it refers to the total number of *grants* in the security group, across all rules.
I'd like to do two things:
1. Raise "Rules per security group" to 125 (reducing "Security groups per network interface" to 2), which needs the account rep's interaction; and
2. Try to combine some rules into aggregates; slaveapi{-dev,}1 is the first one I see.
Assignee | ||
Comment 21•10 years ago
|
||
Attachment #8524693 -
Flags: review?(rail)
Assignee | ||
Comment 22•10 years ago
|
||
Can you connect me with the account rep for #1?
Flags: needinfo?(catlee)
Assignee | ||
Comment 23•10 years ago
|
||
I filed case 1271429571 with Amazon to raise the limits, but also managed to hack things to get under 50 temporarily (although the result is substantially too open)
Flags: needinfo?(catlee)
Comment 24•10 years ago
|
||
Attachment #8524693 -
Flags: review?(rail) → review+
Assignee | ||
Comment 25•10 years ago
|
||
The part where all of srv.releng.* gets unfettered access to buildmasters makes me deeply sad, so let's hope AWS comes through soon!
Attachment #8524726 -
Flags: checked-in+
Assignee | ||
Comment 26•10 years ago
|
||
attachment 8524726 [details] [diff] [review] is applied for usw1/usw2, but *not* use1. Nothing has exploded yet, but I'll give it a while longer before applying for use1. So at worst we're on a single region until we can revert the SG's for usw2.
Assignee | ||
Comment 27•10 years ago
|
||
Comment on attachment 8524693 [details] [diff] [review]
bug1058225-sg-size.patch
https://hg.mozilla.org/build/cloud-tools/rev/5153c591f335
Attachment #8524693 -
Flags: checked-in+
Assignee | ||
Comment 28•10 years ago
|
||
use1 is in place now too. Now to un-restrict flows on the SRX.
Assignee | ||
Updated•10 years ago
|
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Assignee | ||
Comment 29•10 years ago
|
||
We got the limit raised to 125, so we can put some more accurate flows into the SG.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Assignee | ||
Comment 30•10 years ago
|
||
Attachment #8526943 -
Flags: review?(rail)
Updated•10 years ago
|
Attachment #8526943 -
Flags: review?(rail) → review+
Updated•10 years ago
|
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/378]
Updated•10 years ago
|
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/4191]
Assignee | ||
Comment 31•10 years ago
|
||
Comment on attachment 8526943 [details] [diff] [review]
bug1058225-expand.patch
Landed in the repo, and deployed to usw2 only - I'll do use1 once everything seems clear.
https://hg.mozilla.org/build/cloud-tools/rev/202bb8eea212
Attachment #8526943 -
Flags: checked-in+
Assignee | ||
Comment 32•10 years ago
|
||
remainder is now landed.
Status: REOPENED → RESOLVED
Closed: 10 years ago → 10 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•