As part of a pentest of BMO/PuppetAgain/BuildBotMasters, we determined that password authentication is enabled for the following hosts, which makes them a target for bruteforcing attacks when a weak credential is used. I'm not sure if password authentication is needed for these hosts, but I would think by now they should be all pubkey auth. Also, asking around for some peer review, in the case password auth is required on any of these systems. releng-puppet1.srv.releng.scl3.mozilla.com releng-puppet2.srv.releng.scl3.mozilla.com releng-puppet1.srv.releng.use1.mozilla.com releng-puppet1.srv.releng.usw2.mozilla.com buildbot-master01.bb.releng.use1.mozilla.com buildbot-master02.bb.releng.use1.mozilla.com buildbot-master03.bb.releng.use1.mozilla.com buildbot-master04.bb.releng.usw2.mozilla.com buildbot-master05.bb.releng.usw2.mozilla.com buildbot-master06.bb.releng.usw2.mozilla.com buildbot-master07.bb.releng.usw2.mozilla.com buildbot-master08.bb.releng.use1.mozilla.com buildbot-master51.bb.releng.use1.mozilla.com buildbot-master52.bb.releng.use1.mozilla.com buildbot-master53.bb.releng.usw2.mozilla.com buildbot-master54.bb.releng.usw2.mozilla.com buildbot-master67.bb.releng.use1.mozilla.com buildbot-master68.bb.releng.usw2.mozilla.com buildbot-master68.bb.releng.usw2.mozilla.com buildbot-master70.bb.releng.use1.mozilla.com buildbot-master71.bb.releng.use1.mozilla.com buildbot-master72.bb.releng.usw2.mozilla.com buildbot-master73.bb.releng.usw2.mozilla.com buildbot-master74.bb.releng.usw2.mozilla.com buildbot-master74.bb.releng.usw2.mozilla.com buildbot-master76.bb.releng.use1.mozilla.com buildbot-master77.bb.releng.use1.mozilla.com buildbot-master78.bb.releng.usw2.mozilla.com buildbot-master79.bb.releng.usw2.mozilla.com buildbot-master81.bb.releng.scl3.mozilla.com buildbot-master82.bb.releng.scl3.mozilla.com buildbot-master83.bb.releng.scl3.mozilla.com buildbot-master84.bb.releng.scl3.mozilla.com buildbot-master85.bb.releng.scl3.mozilla.com buildbot-master86.bb.releng.scl3.mozilla.com buildbot-master87.bb.releng.scl3.mozilla.com buildbot-master91.bb.releng.usw2.mozilla.com buildbot-master91.bb.releng.usw2.mozilla.com buildbot-master103.bb.releng.scl3.mozilla.com buildbot-master104.bb.releng.scl3.mozilla.com buildbot-master105.bb.releng.scl3.mozilla.com buildbot-master106.bb.releng.scl3.mozilla.com buildbot-master107.bb.releng.scl3.mozilla.com buildbot-master108.bb.releng.scl3.mozilla.com buildbot-master109.bb.releng.scl3.mozilla.com buildbot-master110.bb.releng.scl3.mozilla.com buildbot-master111.bb.releng.scl3.mozilla.com buildbot-master112.bb.releng.scl3.mozilla.com buildbot-master113.bb.releng.use1.mozilla.com buildbot-master114.bb.releng.use1.mozilla.com buildbot-master117.bb.releng.use1.mozilla.com buildbot-master115.bb.releng.usw2.mozilla.com buildbot-master116.bb.releng.usw2.mozilla.com buildbot-master118.bb.releng.usw2.mozilla.com buildbot-master119.bb.releng.scl3.mozilla.com buildbot-master120.bb.releng.use1.mozilla.com buildbot-master121.bb.releng.use1.mozilla.com buildbot-master122.bb.releng.usw2.mozilla.com buildbot-master123.bb.releng.usw2.mozilla.com buildbot-master124.bb.releng.use1.mozilla.com buildbot-master125.bb.releng.usw2.mozilla.com buildbot-master126.bb.releng.scl3.mozilla.com buildbot-master127.bb.releng.scl3.mozilla.com buildbot-master128.bb.releng.use1.mozilla.com buildbot-master129.bb.releng.usw2.mozilla.com buildbot-master130.bb.releng.use1.mozilla.com buildbot-master131.bb.releng.usw2.mozilla.com buildbot-master132.bb.releng.scl3.mozilla.com buildbot-master133.bb.releng.scl3.mozilla.com buildbot-master134.bb.releng.scl3.mozilla.com buildbot-master135.bb.releng.scl3.mozilla.com buildbot-master136.bb.releng.scl3.mozilla.com buildbot-master137.bb.releng.use1.mozilla.com buildbot-master138.bb.releng.use1.mozilla.com buildbot-master139.bb.releng.usw2.mozilla.com buildbot-master140.bb.releng.usw2.mozilla.com buildbot-master141.bb.releng.use1.mozilla.com buildbot-master142.bb.releng.usw2.mozilla.com dev-master2.bb.releng.use1.mozilla.com
Summary: disable password authentication on the following hosts → disable password authentication for SSH on the following hosts
It would also be extra special if these hosts subscribed to the OpenSSH guidelines published here (https://wiki.mozilla.org/Security/Guidelines/OpenSSH), but really, disabling password auth and requiring pubkey auth to avoid bruteforce/guessing attacks is the biggest value proposition here.
:catlee, would it be problematic to apply this across all posix hosts? Is there any places were automation would be using password login? What about loaners hosts/instances? The only automation, I could think would be attempting password login would be slaveapi. If it is too much of an impact to apply broadly (to include slaves), then we could applied different ssh configs in that case.
I'd love to do this everywhere. Check with Callek to see what slaveapi needs.
Flags: needinfo?(catlee) → needinfo?(bugspam.Callek)
:catlee|:dividehex - what do you think about me doing a small local password audit against a couple sample servers from the above list? Like maybe, the following... - dev-master2.bb.releng.use1.mozilla.com - buildbot-master81.bb.releng.scl3.mozilla.com - releng-puppet2.srv.releng.scl3.mozilla.com Process would include grabbing /etc/passwd and /etc/shadow and running a light crack to see if there are any credentials there that we might want to clean up?
(In reply to Chris AtLee [:catlee] from comment #3) > I'd love to do this everywhere. Check with Callek to see what slaveapi needs. Slaveapi does a tiered vector of reboots: * login as cltbld via password auth and `sudo reboot` * login as root via password auth and do a `sudo reboot` (I forget if sudo is done) * use ipmi/pdu rebooting if available. It tries each, and if it fails for some reason moves onto the next (fails includes not seeing it go down and then back up). So I think we'd be ok without root password auth, BUT it would slow slaveapi's actions down a bit, and if pdu/ipmi ports/info is wrong we could end up with a higher chance of rebooting the wrong system.
(In reply to Justin Wood (:Callek) from comment #5) > (In reply to Chris AtLee [:catlee] from comment #3) > > I'd love to do this everywhere. Check with Callek to see what slaveapi needs. > > Slaveapi does a tiered vector of reboots: > > * login as cltbld via password auth and `sudo reboot` > * login as root via password auth and do a `sudo reboot` (I forget if sudo > is done) why can't these be done using ssh key authentication?
(In reply to Chris AtLee [:catlee] from comment #6) > (In reply to Justin Wood (:Callek) from comment #5) > > (In reply to Chris AtLee [:catlee] from comment #3) > > > I'd love to do this everywhere. Check with Callek to see what slaveapi needs. > > > > Slaveapi does a tiered vector of reboots: > > > > * login as cltbld via password auth and `sudo reboot` > > * login as root via password auth and do a `sudo reboot` (I forget if sudo > > is done) > > why can't these be done using ssh key authentication? python's paramiko doesn't support that (afaik), I remember some issues trying to establish the ssh-agent to the proc properly. But to do so would require some devoted hacking on slaveapi, which isn't exactly a clean architecture nor does it have any test code attached :(
(In reply to Jonathan Claudius [:claudijd] (use NEEDINFO) from comment #4) > :catlee|:dividehex - what do you think about me doing a small local password > audit against a couple sample servers from the above list? > > Like maybe, the following... > > - dev-master2.bb.releng.use1.mozilla.com > - buildbot-master81.bb.releng.scl3.mozilla.com > - releng-puppet2.srv.releng.scl3.mozilla.com > > Process would include grabbing /etc/passwd and /etc/shadow and running a > light crack to see if there are any credentials there that we might want to > clean up? I have don't have problem with that, although, there shouldn't be any regular user accounts with passwords set. You will find a few generic accounts such as 'buildduty' and 'cltbld' with passwords.
:dividehex - would you be able to assist me with gathering the /etc/shadow samples from each of those hosts? I would ask that it be GPG encrypted when sent to avoid it being compromised in transit.
As discussed via irc, I have sent the passwd/shadow files(gpg encrypted) of aws-manager1 to :claudijd. aws-manager1 contains most of the generic user accounts which should give coverage of the attack surface.
I attempted to crack the 3 password hashes provided in the supplied sample, which included hashes for users cltbld, buildduty, and root. I used three dictionary sets; (1) A small custom list with out product names (10 seeds), (2) the fast-track list (~150 seeds), and (3) the 500 worst passwords list (500 seeds). Each list was run through with John-the-Ripper (a password cracking tool) with rules enabled to "l33tify" and provide additional permutations from the original source dictionaries. Using this process, I was unable to crack the passwords for these accounts. John-the-ripper also performed some hash detection capabilities and it was torn between these being "sha512crypt-opencl" or just "sha512crypt" formated hashes. I attempted both formats to be on the safe side. It is my opinion, based on this black-box testing, that the credentials for these are not easily guessable by an attacker who would need to brute-force them actively with the service. For those who know the actual passwords for those accounts, please ensure they are using strong passwords in the case I simply didn't get the right permutation to successfully guess. I would also like to note that having these accounts with the same password across multiple systems effectively puts each system in the same security domain (ie. if one of these systems is owned in some way, said attacker can re-use those credentials and access additional systems trivially). Many thanks to :dividehex for providing the hashes to be tested. I have since deleted all unencrypted files from my workstation as a safety precaution.
> based on this black-box testing Wouldn't it be easier to make sure the password for these accounts are longer than 24 characters with a mix of letters and numbers? Can anyone confirm that it is the case today?
(In reply to Julien Vehent [:ulfr] from comment #12) > > based on this black-box testing > > Wouldn't it be easier to make sure the password for these accounts are > longer than 24 characters with a mix of letters and numbers? Can anyone > confirm that it is the case today? Yes, I would agree. Would love to see someone confirm that.
(In reply to Julien Vehent [:ulfr] from comment #12) > > based on this black-box testing > > Wouldn't it be easier to make sure the password for these accounts are > longer than 24 characters with a mix of letters and numbers? Can anyone > confirm that it is the case today? What is the basis of this password requirement? Is this documented anywhere?
I'm not sure if we have a "policy" for machine password, but surely we can agree that a weak password is a bad idea, and pick something reasonably complex.
(In reply to Julien Vehent [:ulfr] from comment #15) > I'm not sure if we have a "policy" for machine password, but surely we can > agree that a weak password is a bad idea, and pick something reasonably > complex. No disagreement here. I'm just wondering where "24" came from as opposed to 8/12/40 characters.
Taking a step back here - can we proceed with disabling pw authentication on just these hosts now, given that slaveapi has additional complications that mean we can't apply this to all posix hosts?
Created attachment 8832946 [details] [diff] [review] bug1323457-disable-passwd-auth.patch This patch disables ssh password authentication and prevents direct root logins to all puppet masters and buildbot masters. It also fixes a typo preventing the actual implementation.
Attachment #8832946 - Flags: review?(catlee)
Comment on attachment 8832946 [details] [diff] [review] bug1323457-disable-passwd-auth.patch https://hg.mozilla.org/build/puppet/rev/cb71986f16043723a6dcf98d76cb7bbfccc18bbe https://hg.mozilla.org/build/puppet/rev/7e23bcff88aa301d59891b8f633747dcc5f93c80
Attachment #8832946 - Flags: checked-in+
Created attachment 8833065 [details] [diff] [review] remove extra_authorized_key Since root no longer allows logins there is not need to add user ssh keys to the root authorized key. This is causing puppet to fail.
Attachment #8833065 - Flags: review?(catlee)
Comment on attachment 8833065 [details] [diff] [review] remove extra_authorized_key https://hg.mozilla.org/build/puppet/rev/da443f61a364c199afbcbb8303687030c2464225
Attachment #8833065 - Flags: checked-in+
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.