685575 - Do LDAP authentication on dev-master01 with keys from puppet, not ssh-lpk

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

I set this machine up before I knew that ssh-lpk was the old, bad, deprecated way.  And, I think, before there was a new, good, preferred way.  The result is that when LDAP has problems, the machine is inaccessible - bug 685570.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Ah, I forgot that this is using the releng puppet manifests.

The way we've done this in IT is to autogenerate a set of user resources and SSH keys in the puppet manifests, based on a crontab that pulls from LDAP.  That way an LDAP outage hurts neither puppet nor, more importantly, running systems.

I think the best way to do this in releng will be including the public keys directly, manually.  Does that make sense to y'all, too?

Comment 2

[[e-mail] (:bhearsum)]

I've got no real preference, personally.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

So rather than put usernames, uids, and public keys in a public hg repo, it probably makes more sense to treat this similar to the secrets.pp.template file.  In fact, I'll probably add it directly to secrets.pp, just so we don't have to worry about carrying around another non-versioned file.  Does that sound OK?

Comment 4

[Amy Rich [:arr] [:arich]]

I actually thinks it makes more sense to split the two files because you're constantly auto-generating one of them from an outside source.  You don't want to accidentally zorch the other non-autogenerated secrets if something should go wrong with the ldap lookup.

Since this file does not get checked into the repo, we'll need to add some extra logic to make sure that we're not doing a puppet run with a partial or corrupted file (ldap timeout, partial transfer, etc).  I think writing to a temp file, doing a compare to see if too many lines changed, then moving the temp file into place might be a help.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

I tried porting the script IT uses, and it's too much trouble to get right - particularly given that the puppet masters aren't puppetized.

So I'm just going to organize this as I said in comment 3.  To be clear, that's not dynamically generated.

We can probably (definitely) do this with the new puppet-2.7.1 infra, but that's still proof-of-concept.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: m685575-puppet-manifests-p1-r1.patch

So puppet, uh, "fixed itself" again.  After we talked at lunch, I went back to try to reproduce the problems I was having with the clientbucket.  Works fine.

To pull dev-master01 back from the brink, I manually replaced /etc/ldap.conf, /etc/ssh/sshd_config, /etc/nsswitch.conf, and /etc/pam.d/system-auth-ac with files from another master.  I did *not* change the version of SSH -- the LPK-enabled version should (and seems to) work fine.  I also disabled puppetd, since it will revert these changes until the patch is landed, and lock us all out again.

So once this lands, I'll need to restart puppetd.

Comment 7

[[e-mail] (:bhearsum)]

Comment on attachment 560047 [details] [diff] [review]
m685575-puppet-manifests-p1-r1.patch

Review of attachment 560047 [details] [diff] [review]:
-----------------------------------------------------------------

I think you forgot to include the userlogins module in this patch. Could you also add a sample user/key to secrets.pp.template?

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 560047 [details] [diff] [review]
m685575-puppet-manifests-p1-r1.patch

Review of attachment 560047 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/userlogins/manifests/init.pp
@@ +1,1 @@
> +# The userids and ssh keys for these users are defined in secrets.pp.

This is the userlogins module..

I'll add the example, though.

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: m685575-puppet-manifests-p1-r2.patch

Comment 10

[[e-mail] (:bhearsum)]

Comment on attachment 560657 [details] [diff] [review]
m685575-puppet-manifests-p1-r2.patch

Review of attachment 560657 [details] [diff] [review]:
-----------------------------------------------------------------

Sorry, I'm not sure how I missed the userlogins module before!

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

Landed and deployed with the corresponding secrets.pp changes.

The template was incorrect in the patch, so I fixed it up in rev 9e75ff75a9e1.

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

I also used 'yum downgrade' to downgrade back to the normal version of puppet.  This helpfully removed a bunch of stuff, which i then re-added:

[root@dev-master01 ~]# yum downgrade openssh
Loaded plugins: fastestmirror
Setting up Downgrade Process
Loading mirror speeds from cached hostfile
 * base: centos.mirrors.hoobly.com
 * epel: linux.mirrors.es.net
 * extras: centos.mirror.facebook.net
 * updates: centos.mirror.facebook.net
Resolving Dependencies
--> Running transaction check
---> Package openssh.x86_64 0:4.3p2-72.el5_7.5 set to be updated
---> Package openssh.x86_64 0:5.2p1-1.rhel5 set to be erased
--> Processing Dependency: openssh = 5.2p1-1.rhel5 for package: openssh-server
--> Processing Dependency: openssh = 5.2p1-1.rhel5 for package: openssh-clients
--> Running transaction check
---> Package openssh-clients.x86_64 0:5.2p1-1.rhel5 set to be erased
--> Processing Dependency: openssh-clients for package: git
---> Package openssh-server.x86_64 0:5.2p1-1.rhel5 set to be erased
--> Running transaction check
---> Package git.x86_64 0:1.7.4.1-1.el5 set to be erased
--> Processing Dependency: git = 1.7.4.1-1.el5 for package: perl-Git
--> Running transaction check
---> Package perl-Git.x86_64 0:1.7.4.1-1.el5 set to be erased
--> Processing Dependency: /usr/bin/ssh for package: nagios-plugins-by_ssh
--> Restarting Dependency Resolution with new changes.
--> Running transaction check
---> Package nagios-plugins-by_ssh.x86_64 0:1.4.15-2.el5 set to be erased
--> Processing Dependency: nagios-plugins-by_ssh for package: nagios-plugins-all
--> Running transaction check
---> Package nagios-plugins-all.x86_64 0:1.4.15-2.el5 set to be erased
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================================================================================================================================
 Package                                                                      Arch                                                          Version                                                                    Repository                                                        Size
==============================================================================================================================================================================================================================================================================================
Downgrading:
 openssh                                                                      x86_64                                                        4.3p2-72.el5_7.5                                                           updates                                                          289 k
Removing for dependencies:
 git                                                                          x86_64                                                        1.7.4.1-1.el5                                                              installed                                                         11 M
 nagios-plugins-all                                                           x86_64                                                        1.4.15-2.el5                                                               installed                                                         0.0
 nagios-plugins-by_ssh                                                        x86_64                                                        1.4.15-2.el5                                                               installed                                                         56 k
 openssh-clients                                                              x86_64                                                        5.2p1-1.rhel5                                                              installed                                                        3.1 M
 openssh-server                                                               x86_64                                                        5.2p1-1.rhel5                                                              installed                                                        1.8 M
 perl-Git                                                                     x86_64                                                        1.7.4.1-1.el5                                                              installed                                                         35 k

Transaction Summary
==============================================================================================================================================================================================================================================================================================
Remove        6 Package(s)
Reinstall     0 Package(s)
Downgrade     1 Package(s)

Total download size: 289 k

[root@dev-master01 ~]# yum install nagios-plugins-all git openssh-server openssh-clients
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: centos.mirrors.hoobly.com
 * epel: linux.mirrors.es.net
 * extras: centos.mirror.facebook.net
 * updates: centos.mirror.facebook.net
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package git.x86_64 0:1.7.4.1-1.el5 set to be updated
--> Processing Dependency: perl-Git = 1.7.4.1-1.el5 for package: git
--> Processing Dependency: perl(Git) for package: git
---> Package nagios-plugins-all.x86_64 0:1.4.15-2.el5 set to be updated
--> Processing Dependency: nagios-plugins-by_ssh for package: nagios-plugins-all
---> Package openssh-clients.x86_64 0:4.3p2-72.el5_7.5 set to be updated
---> Package openssh-server.x86_64 0:4.3p2-72.el5_7.5 set to be updated
--> Running transaction check
---> Package nagios-plugins-by_ssh.x86_64 0:1.4.15-2.el5 set to be updated
---> Package perl-Git.x86_64 0:1.7.4.1-1.el5 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================================================================================================================================
 Package                                                                       Arch                                                           Version                                                                   Repository                                                       Size
==============================================================================================================================================================================================================================================================================================
Installing:
 git                                                                           x86_64                                                         1.7.4.1-1.el5                                                             epel                                                            4.5 M
 nagios-plugins-all                                                            x86_64                                                         1.4.15-2.el5                                                              epel                                                             10 k
 openssh-clients                                                               x86_64                                                         4.3p2-72.el5_7.5                                                          updates                                                         452 k
 openssh-server                                                                x86_64                                                         4.3p2-72.el5_7.5                                                          updates                                                         278 k
Installing for dependencies:
 nagios-plugins-by_ssh                                                         x86_64                                                         1.4.15-2.el5                                                              epel                                                             37 k perl-Git                                                                      x86_64                                                         1.7.4.1-1.el5                                                             epel                                                             28 k

Transaction Summary
==============================================================================================================================================================================================================================================================================================
Install       6 Package(s)
Upgrade       0 Package(s)

Total download size: 5.3 M



I need to do the same on releng-mirror01, but it's currently out of space.

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

OK, I did the relevant un-horking in comments 6 and 12.

releng-mirror01 was out of disk space (and had been alerting as such for a while), which may have been what precipitated this.  I blew away the seamonkey dir and disabled the rsync for the moment.

Puppet was not set up to run at boot, so I fixed that and started it back up.  It put the proper SSH keys in place, and re-enabled the rsync.  So it will fill up again soon.