1392820 - Roll out hostbased firewall ssh/vnc logging to SCL3 for anything under puppet control

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Jake Watkins [:dividehex]]

Similar to bug 1387251, let's start rolling out hostbased firewalls to systems in scl3.

We'll start with:
log-aggregator[1,2,3].srv.releng.scl3.mozilla.com
aws-manager[1,2].srv.releng.scl3.mozilla.com
rejh[1,2].srv.releng.scl3.mozilla.com

Comment 1

[Jake Watkins [:dividehex]]

Attachment: Enable firewalling for scl3 logaggrogators, rejh and aws_managers

Comment 2

[:dhouse]

Comment on attachment 8900015 [details] [diff] [review]
Enable firewalling for scl3 logaggrogators, rejh and aws_managers

Review of attachment 8900015 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good. Thanks for including scl3 for rejh.

Comment 3

[Jake Watkins [:dividehex]]

Comment on attachment 8900015 [details] [diff] [review]
Enable firewalling for scl3 logaggrogators, rejh and aws_managers

https://hg.mozilla.org/build/puppet/rev/a577dc609736004d1d80a7a2f908f3f24cbde0dd https://hg.mozilla.org/build/puppet/rev/6645c10ba7b3b8ae974d886c7096212f45fa8bab

Comment 4

[Jake Watkins [:dividehex]]

Attachment: Enable fw on scl3 try builders

Comment 5

[Jake Watkins [:dividehex]]

Comment on attachment 8900483 [details] [diff] [review]
Enable fw on scl3 try builders

https://hg.mozilla.org/build/puppet/rev/0edceedaf667afaac3049d3947aeb4b46b5bd2fd
https://hg.mozilla.org/build/puppet/rev/d33a7ad64bf8e8cdf169966b133fa43ddc7941e0

Comment 6

[Jake Watkins [:dividehex]]

Attachment: Enable fw on slaveapi

Comment 7

[Jake Watkins [:dividehex]]

Comment on attachment 8900497 [details] [diff] [review]
Enable fw on slaveapi

https://hg.mozilla.org/build/puppet/rev/93a8dc181306305d8fbc029761ed954f0a5d8bd7
https://hg.mozilla.org/build/puppet/rev/9981f37469ce9e8a59ddae3a66b9f5bb0fafad83

Comment 8

[Jake Watkins [:dividehex]]

Attachment: enable fw on scl3 osx taskcluster workers

This has already been enabled on the tc workers in mcd1 which have been running in production now since Fri.

Comment 9

[Jake Watkins [:dividehex]]

Attachment: enabled fw on scl3 tc workers

Comment 10

[Jake Watkins [:dividehex]]

Attachment: enable fw on scl3 osx taskcluster workers

<patch fail> Let's try that one more time.

Comment 11

[Jake Watkins [:dividehex]]

Comment on attachment 8901955 [details] [diff] [review]
enable fw on scl3 osx taskcluster workers

https://hg.mozilla.org/build/puppet/rev/2cd116207ccccb1f9af9b52ba01b4918323dda8e
https://hg.mozilla.org/build/puppet/rev/de5ee50cc78be2dc19d2f4211955ba72f901211b

Comment 12

[Jake Watkins [:dividehex]]

Attachment: enable fw on scl3 buildbot test slaves (osx and linux)

Comment 13

[Jake Watkins [:dividehex]]

Comment on attachment 8903255 [details] [diff] [review]
enable fw on scl3 buildbot test slaves (osx and linux)

https://hg.mozilla.org/build/puppet/rev/74aece25a299256e44e74d30a588bc1ddb6c3047
https://hg.mozilla.org/build/puppet/rev/d6c605254b58356ca056446e0cb5849b8e19293e

Comment 14

[Jake Watkins [:dividehex]]

Attachment: Enable fw on sl3 osx bb build slaves

Comment 15

[Jake Watkins [:dividehex]]

Attachment: enable fw on releng-puppet1.srv.releng.scl3

Comment 16

[Jake Watkins [:dividehex]]

Comment on attachment 8904663 [details] [diff] [review]
Enable fw on sl3 osx bb build slaves

https://hg.mozilla.org/build/puppet/rev/bda2d22e107123f06aaffa6ca73477652b7be111
https://hg.mozilla.org/build/puppet/rev/b21ba89f9fd02c139915d6ef433cc532c136e4c9

Comment 17

[Jake Watkins [:dividehex]]

Comment on attachment 8905102 [details] [diff] [review]
enable fw on releng-puppet1.srv.releng.scl3

https://hg.mozilla.org/build/puppet/rev/34d09bb5f68a87626166a278ac79cdfb299bcda8
https://hg.mozilla.org/build/puppet/rev/9d24fb3f4e4cdd2d1232bab3f2ef858105d9c0dc

Comment 18

[Jake Watkins [:dividehex]]

Attachment: enable fw on releng-puppet2.srv.releng.scl3

Also enabled ssh logging

Comment 19

[Jake Watkins [:dividehex]]

Comment on attachment 8908263 [details] [diff] [review]
enable fw on releng-puppet2.srv.releng.scl3

https://hg.mozilla.org/build/puppet/rev/b119af979a48dc492818320d923f5c2372815f05
https://hg.mozilla.org/build/puppet/rev/8a362ea7d804cb91f7f250d3da771c49b05daef7

Comment 20

[Jake Watkins [:dividehex]]

Pushed a quick fix; wrong roll name.

https://hg.mozilla.org/build/puppet/rev/ae81e2f6d90fb476b2c9c0ecc460e584498459f9
https://hg.mozilla.org/build/puppet/rev/da4ae117d427c6f4dafe60b5a33c3f3218c1aba5

diff --git a/modules/fw/manifests/profiles/distinguished_puppetmaster.pp b/modules/fw/manifests/profiles/distinguished_puppetmaster.pp
--- a/modules/fw/manifests/profiles/distinguished_puppetmaster.pp
+++ b/modules/fw/manifests/profiles/distinguished_puppetmaster.pp
@@ -4,25 +4,25 @@

 class fw::profiles::distinguished_puppetmaster {

     case $::fqdn {
         /.*\.mdc1\.mozilla\.com/: {
             include ::fw::roles::bacula_from_mdc1_bacula_host

             include ::fw::roles::puppetmaster_from_all_releng
-            include ::fw::roles::puppetmaster_sync_from_all_releng
+            include ::fw::roles::puppetmaster_sync_from_all_puppetmasters
             include ::fw::roles::ssh_from_anywhere_logging
             include ::fw::roles::nrpe_from_nagios
         }
         /.*\.scl3\.mozilla\.com/: {
             include ::fw::roles::bacula_from_scl3_bacula_host

             include ::fw::roles::puppetmaster_from_all_releng
-            include ::fw::roles::puppetmaster_sync_from_all_releng
+            include ::fw::roles::puppetmaster_sync_from_all_puppetmasters
             include ::fw::roles::ssh_from_anywhere_logging
             include ::fw::roles::nrpe_from_nagios
         }
         default: {
             # Silently skip other DCs
         }
     }
 }

Comment 21

[Jake Watkins [:dividehex]]

Attachment: Add buildbot_master profile and allow for port ranges

Comment 22

[:dhouse]

Comment on attachment 8909568 [details] [diff] [review]
Add buildbot_master profile and allow for port ranges

The use of port ranges looks good; nicely applied to pf and iptables. And I think the "8000-8999" format is more clear than "8000:8999"

Comment 23

[Jake Watkins [:dividehex]]

Comment on attachment 8909568 [details] [diff] [review]
Add buildbot_master profile and allow for port ranges

https://hg.mozilla.org/build/puppet/rev/200f7c483295c00be113a9911be4e6af6c48c182
https://hg.mozilla.org/build/puppet/rev/c02c95fa8c139b4052e823e2e332bd14eca35506

Comment 24

[Jake Watkins [:dividehex]]

Comment on attachment 8909568 [details] [diff] [review]
Add buildbot_master profile and allow for port ranges

:catlee, before I start rolling out the buildbot master profile to the scl3 masters, could you take a look at the profile and make sure is correct?

Particularly, apps.pp, networks.pp, buildbot_master_from_various.pp and buildbot_master.pp

It was modeled after the aws security groups but I'm would like to get a second pair of eyes on it for a sanity check.

ps. feel free to pass the buck to anyone else :-)

Comment 25

[Jake Watkins [:dividehex]]

Attachment: make sure bb slaves and tc generic workers allow slaveapi ssh access

Comment 26

[Justin Wood (:Callek)]

Comment on attachment 8910037 [details] [diff] [review]
make sure bb slaves and tc generic workers allow slaveapi ssh access

Review of attachment 8910037 [details] [diff] [review]:
-----------------------------------------------------------------

I am pretty sure this is all that is needed for slaveapi ssh access's. We can do more if needed in followups.

Comment 27

[Chris AtLee [:catlee]]

Comment on attachment 8909568 [details] [diff] [review]
Add buildbot_master profile and allow for port ranges

Review of attachment 8909568 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm.

I did find the 'all_bb' name confusing. Would calling it 'all_bb_masters' make more sense?

Comment 28

[Jake Watkins [:dividehex]]

Comment on attachment 8910037 [details] [diff] [review]
make sure bb slaves and tc generic workers allow slaveapi ssh access

https://hg.mozilla.org/build/puppet/rev/5686d22ce729d7e907c9ae795358a87bc1706f47
https://hg.mozilla.org/build/puppet/rev/16891afffc2b6798116ebf4a6f3eda31886975ad

Comment 29

[Jake Watkins [:dividehex]]

(In reply to Chris AtLee [:catlee] from comment #27)
> Comment on attachment 8909568 [details] [diff] [review]
> Add buildbot_master profile and allow for port ranges
> 
> Review of attachment 8909568 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> lgtm.
> 
> I did find the 'all_bb' name confusing. Would calling it 'all_bb_masters'
> make more sense?

Totally would make sense

Comment 30

[Jake Watkins [:dividehex]]

Attachment: rename $all_bb to $all_bb_masters for clarity

Comment 31

[Jake Watkins [:dividehex]]

Attachment: Enable fw on scl3 buildbot masters w/allow all by default

Comment 32

[Jake Watkins [:dividehex]]

Comment on attachment 8911251 [details] [diff] [review]
Enable fw on scl3 buildbot masters w/allow all by default

https://hg.mozilla.org/build/puppet/rev/95c2840017d418bef7fbe5f056175e4bfbcc3b70
https://hg.mozilla.org/build/puppet/rev/424135f9df89751683261609f37bb0ae3ea81822

Comment 33

[Jake Watkins [:dividehex]]

Attachment: Enable logging of ssh connections that fall to default deny

This will help us find ssh connections being denied as we move forward with enforcing jumphost usage.

Comment 34

[Jake Watkins [:dividehex]]

Comment on attachment 8911924 [details] [diff] [review]
Enable logging of ssh connections that fall to default deny

https://hg.mozilla.org/build/puppet/rev/cb5b596189631a9dbb68c3e5b09408f0e739ca98
https://hg.mozilla.org/build/puppet/rev/15bede441a8c05ac44309e304279304db83a0b13

Comment 35

[Jake Watkins [:dividehex]]

Attachment: Remove firewall training wheels on subset of scl3 bb masters

Comment 36

[Jake Watkins [:dividehex]]

Attachment: Enabled default deny policy on scl3 bb masters

This patch can be reverted if things go south.  It will put back the 'allow all default' policy.

Comment 37

[Jake Watkins [:dividehex]]

Comment on attachment 8914891 [details] [diff] [review]
Enabled default deny policy on scl3 bb masters

https://hg.mozilla.org/build/puppet/rev/a463dbda4f352e023b121afec8f3da86e2a65142
https://hg.mozilla.org/build/puppet/rev/ee2d9e867dc53cc0fc6b1172e57befc067b30cda

Comment 38

[Jake Watkins [:dividehex]]

Calling this R/F