disable password authentication for SSH on the following hosts

RESOLVED FIXED

Status

Infrastructure & Operations
RelOps: Puppet
RESOLVED FIXED
2 years ago
a year ago

People

(Reporter: claudijd, Assigned: dividehex)

Tracking

Details

Attachments

(2 attachments)

As part of a pentest of BMO/PuppetAgain/BuildBotMasters, we determined that password authentication is enabled for the following hosts, which makes them a target for bruteforcing attacks when a weak credential is used.  I'm not sure if password authentication is needed for these hosts, but I would think by now they should be all pubkey auth.

Also, asking around for some peer review, in the case password auth is required on any of these systems.

releng-puppet1.srv.releng.scl3.mozilla.com
releng-puppet2.srv.releng.scl3.mozilla.com
releng-puppet1.srv.releng.use1.mozilla.com
releng-puppet1.srv.releng.usw2.mozilla.com
buildbot-master01.bb.releng.use1.mozilla.com
buildbot-master02.bb.releng.use1.mozilla.com
buildbot-master03.bb.releng.use1.mozilla.com
buildbot-master04.bb.releng.usw2.mozilla.com
buildbot-master05.bb.releng.usw2.mozilla.com
buildbot-master06.bb.releng.usw2.mozilla.com
buildbot-master07.bb.releng.usw2.mozilla.com
buildbot-master08.bb.releng.use1.mozilla.com
buildbot-master51.bb.releng.use1.mozilla.com
buildbot-master52.bb.releng.use1.mozilla.com
buildbot-master53.bb.releng.usw2.mozilla.com
buildbot-master54.bb.releng.usw2.mozilla.com
buildbot-master67.bb.releng.use1.mozilla.com
buildbot-master68.bb.releng.usw2.mozilla.com
buildbot-master68.bb.releng.usw2.mozilla.com
buildbot-master70.bb.releng.use1.mozilla.com
buildbot-master71.bb.releng.use1.mozilla.com
buildbot-master72.bb.releng.usw2.mozilla.com
buildbot-master73.bb.releng.usw2.mozilla.com
buildbot-master74.bb.releng.usw2.mozilla.com
buildbot-master74.bb.releng.usw2.mozilla.com
buildbot-master76.bb.releng.use1.mozilla.com
buildbot-master77.bb.releng.use1.mozilla.com
buildbot-master78.bb.releng.usw2.mozilla.com
buildbot-master79.bb.releng.usw2.mozilla.com
buildbot-master81.bb.releng.scl3.mozilla.com
buildbot-master82.bb.releng.scl3.mozilla.com
buildbot-master83.bb.releng.scl3.mozilla.com
buildbot-master84.bb.releng.scl3.mozilla.com
buildbot-master85.bb.releng.scl3.mozilla.com
buildbot-master86.bb.releng.scl3.mozilla.com
buildbot-master87.bb.releng.scl3.mozilla.com
buildbot-master91.bb.releng.usw2.mozilla.com
buildbot-master91.bb.releng.usw2.mozilla.com
buildbot-master103.bb.releng.scl3.mozilla.com
buildbot-master104.bb.releng.scl3.mozilla.com
buildbot-master105.bb.releng.scl3.mozilla.com
buildbot-master106.bb.releng.scl3.mozilla.com
buildbot-master107.bb.releng.scl3.mozilla.com
buildbot-master108.bb.releng.scl3.mozilla.com
buildbot-master109.bb.releng.scl3.mozilla.com
buildbot-master110.bb.releng.scl3.mozilla.com
buildbot-master111.bb.releng.scl3.mozilla.com
buildbot-master112.bb.releng.scl3.mozilla.com
buildbot-master113.bb.releng.use1.mozilla.com
buildbot-master114.bb.releng.use1.mozilla.com
buildbot-master117.bb.releng.use1.mozilla.com
buildbot-master115.bb.releng.usw2.mozilla.com
buildbot-master116.bb.releng.usw2.mozilla.com
buildbot-master118.bb.releng.usw2.mozilla.com
buildbot-master119.bb.releng.scl3.mozilla.com
buildbot-master120.bb.releng.use1.mozilla.com
buildbot-master121.bb.releng.use1.mozilla.com
buildbot-master122.bb.releng.usw2.mozilla.com
buildbot-master123.bb.releng.usw2.mozilla.com
buildbot-master124.bb.releng.use1.mozilla.com
buildbot-master125.bb.releng.usw2.mozilla.com
buildbot-master126.bb.releng.scl3.mozilla.com
buildbot-master127.bb.releng.scl3.mozilla.com
buildbot-master128.bb.releng.use1.mozilla.com
buildbot-master129.bb.releng.usw2.mozilla.com
buildbot-master130.bb.releng.use1.mozilla.com
buildbot-master131.bb.releng.usw2.mozilla.com
buildbot-master132.bb.releng.scl3.mozilla.com
buildbot-master133.bb.releng.scl3.mozilla.com
buildbot-master134.bb.releng.scl3.mozilla.com
buildbot-master135.bb.releng.scl3.mozilla.com
buildbot-master136.bb.releng.scl3.mozilla.com
buildbot-master137.bb.releng.use1.mozilla.com
buildbot-master138.bb.releng.use1.mozilla.com
buildbot-master139.bb.releng.usw2.mozilla.com
buildbot-master140.bb.releng.usw2.mozilla.com
buildbot-master141.bb.releng.use1.mozilla.com
buildbot-master142.bb.releng.usw2.mozilla.com
dev-master2.bb.releng.use1.mozilla.com
Summary: disable password authentication on the following hosts → disable password authentication for SSH on the following hosts
It would also be extra special if these hosts subscribed to the OpenSSH guidelines published here (https://wiki.mozilla.org/Security/Guidelines/OpenSSH), but really, disabling password auth and requiring pubkey auth to avoid bruteforce/guessing attacks is the biggest value proposition here.
Assignee: relops → jwatkins
(Assignee)

Comment 2

2 years ago
:catlee, would it be problematic to apply this across all posix hosts?  Is there any places were automation would be using password login?  What about loaners hosts/instances?

The only automation, I could think would be attempting password login would be slaveapi.

If it is too much of an impact to apply broadly (to include slaves), then we could applied different ssh configs in that case.
Flags: needinfo?(catlee)
I'd love to do this everywhere. Check with Callek to see what slaveapi needs.
Flags: needinfo?(catlee) → needinfo?(bugspam.Callek)
:catlee|:dividehex - what do you think about me doing a small local password audit against a couple sample servers from the above list?

Like maybe, the following...

- dev-master2.bb.releng.use1.mozilla.com
- buildbot-master81.bb.releng.scl3.mozilla.com
- releng-puppet2.srv.releng.scl3.mozilla.com

Process would include grabbing /etc/passwd and /etc/shadow and running a light crack to see if there are any credentials there that we might want to clean up?
Flags: needinfo?(jwatkins)
Flags: needinfo?(catlee)
(In reply to Chris AtLee [:catlee] from comment #3)
> I'd love to do this everywhere. Check with Callek to see what slaveapi needs.

Slaveapi does a tiered vector of reboots:

* login as cltbld via password auth and `sudo reboot`
* login as root via password auth and do a `sudo reboot` (I forget if sudo is done)
* use ipmi/pdu rebooting if available.

It tries each, and if it fails for some reason moves onto the next (fails includes not seeing it go down and then back up).

So I think we'd be ok without root password auth, BUT it would slow slaveapi's actions down a bit, and if pdu/ipmi ports/info is wrong we could end up with a higher chance of rebooting the wrong system.
Flags: needinfo?(bugspam.Callek)
(In reply to Justin Wood (:Callek) from comment #5)
> (In reply to Chris AtLee [:catlee] from comment #3)
> > I'd love to do this everywhere. Check with Callek to see what slaveapi needs.
> 
> Slaveapi does a tiered vector of reboots:
> 
> * login as cltbld via password auth and `sudo reboot`
> * login as root via password auth and do a `sudo reboot` (I forget if sudo
> is done)

why can't these be done using ssh key authentication?
Flags: needinfo?(catlee)
(In reply to Chris AtLee [:catlee] from comment #6)
> (In reply to Justin Wood (:Callek) from comment #5)
> > (In reply to Chris AtLee [:catlee] from comment #3)
> > > I'd love to do this everywhere. Check with Callek to see what slaveapi needs.
> > 
> > Slaveapi does a tiered vector of reboots:
> > 
> > * login as cltbld via password auth and `sudo reboot`
> > * login as root via password auth and do a `sudo reboot` (I forget if sudo
> > is done)
> 
> why can't these be done using ssh key authentication?

python's paramiko doesn't support that (afaik), I remember some issues trying to establish the ssh-agent to the proc properly. But to do so would require some devoted hacking on slaveapi, which isn't exactly a clean architecture nor does it have any test code attached :(
(Assignee)

Comment 8

2 years ago
(In reply to Jonathan Claudius [:claudijd] (use NEEDINFO) from comment #4)
> :catlee|:dividehex - what do you think about me doing a small local password
> audit against a couple sample servers from the above list?
> 
> Like maybe, the following...
> 
> - dev-master2.bb.releng.use1.mozilla.com
> - buildbot-master81.bb.releng.scl3.mozilla.com
> - releng-puppet2.srv.releng.scl3.mozilla.com
> 
> Process would include grabbing /etc/passwd and /etc/shadow and running a
> light crack to see if there are any credentials there that we might want to
> clean up?

I have don't have problem with that, although, there shouldn't be any regular user accounts with passwords set.  You will find a few generic accounts such as 'buildduty' and 'cltbld' with passwords.
Flags: needinfo?(jwatkins)
:dividehex - would you be able to assist me with gathering the /etc/shadow samples from each of those hosts? I would ask that it be GPG encrypted when sent to avoid it being compromised in transit.
Flags: needinfo?(jwatkins)
(Assignee)

Comment 10

2 years ago
As discussed via irc, I have sent the passwd/shadow files(gpg encrypted) of aws-manager1 to :claudijd.  aws-manager1 contains most of the generic user accounts which should give coverage of the attack surface.
Flags: needinfo?(jwatkins)
I attempted to crack the 3 password hashes provided in the supplied sample, which included hashes for users cltbld, buildduty, and root.  I used three dictionary sets; (1) A small custom list with out product names (10 seeds), (2) the fast-track list (~150 seeds), and (3) the 500 worst passwords list (500 seeds).  Each list was run through with John-the-Ripper (a password cracking tool) with rules enabled to "l33tify" and provide additional permutations from the original source dictionaries.  Using this process, I was unable to crack the passwords for these accounts.  John-the-ripper also performed some hash detection capabilities and it was torn between these being "sha512crypt-opencl" or just "sha512crypt" formated hashes.  I attempted both formats to be on the safe side.

It is my opinion, based on this black-box testing, that the credentials for these are not easily guessable by an attacker who would need to brute-force them actively with the service.  For those who know the actual passwords for those accounts, please ensure they are using strong passwords in the case I simply didn't get the right permutation to successfully guess.  I would also like to note that having these accounts with the same password across multiple systems effectively puts each system in the same security domain (ie. if one of these systems is owned in some way, said attacker can re-use those credentials and access additional systems trivially).

Many thanks to :dividehex for providing the hashes to be tested.  I have since deleted all unencrypted files from my workstation as a safety precaution.
> based on this black-box testing

Wouldn't it be easier to make sure the password for these accounts are longer than 24 characters with a mix of letters and numbers? Can anyone confirm that it is the case today?
(In reply to Julien Vehent [:ulfr] from comment #12)
> > based on this black-box testing
> 
> Wouldn't it be easier to make sure the password for these accounts are
> longer than 24 characters with a mix of letters and numbers? Can anyone
> confirm that it is the case today?

Yes, I would agree. Would love to see someone confirm that.
(In reply to Julien Vehent [:ulfr] from comment #12)
> > based on this black-box testing
> 
> Wouldn't it be easier to make sure the password for these accounts are
> longer than 24 characters with a mix of letters and numbers? Can anyone
> confirm that it is the case today?

What is the basis of this password requirement? Is this documented anywhere?
I'm not sure if we have a "policy" for machine password, but surely we can agree that a weak password is a bad idea, and pick something reasonably complex.
(In reply to Julien Vehent [:ulfr] from comment #15)
> I'm not sure if we have a "policy" for machine password, but surely we can
> agree that a weak password is a bad idea, and pick something reasonably
> complex.

No disagreement here. I'm just wondering where "24" came from as opposed to 8/12/40 characters.
Taking a step back here - can we proceed with disabling pw authentication on just these hosts now, given that slaveapi has additional complications that mean we can't apply this to all posix hosts?

Updated

2 years ago
Flags: needinfo?(jwatkins)
(Assignee)

Comment 18

2 years ago
Created attachment 8832946 [details] [diff] [review]
bug1323457-disable-passwd-auth.patch

This patch disables ssh password authentication and prevents direct root logins to all puppet masters and buildbot masters.  It also fixes a typo preventing the actual implementation.
Flags: needinfo?(jwatkins)
Attachment #8832946 - Flags: review?(catlee)

Updated

2 years ago
Attachment #8832946 - Flags: review?(catlee) → review+
(Assignee)

Comment 20

2 years ago
Created attachment 8833065 [details] [diff] [review]
remove extra_authorized_key

Since root no longer allows logins there is not need to add user ssh keys to the root authorized key.  This is causing puppet to fail.
Attachment #8833065 - Flags: review?(catlee)

Updated

2 years ago
Attachment #8833065 - Flags: review?(catlee) → review+
(Assignee)

Updated

2 years ago
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.