569567 - Make slaves use authorized_keys for passwordless ssh

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Rail Aliiev [:rail]]

It will be easier if we set up the slaves so, that they can be accessed by ssh without entering a password.

authorized_keys can be managed by Puppet.

Not sure if we need to use the same method for root user.

Comment 1

[[e-mail] (:bhearsum)]

Are you looking for RelEng'ers public SSH keys to be in authorized_keys, or cltbld/ffxbld?

Comment 2

[Chris AtLee [:catlee]]

Yes, for RelEng'ers public SSH keys

Comment 3

[John Ford [:jhford] CET/CEST Berlin Time]

This sounds like a great idea (i'd love it!).  Maybe we can also have a set of basic config files so we have a known baseline to work from with common tools (.vimrc, .screenrc, .ssh/config .emacs.el are some that come to mind).

If someone has time to take this on feel free to take it :)

Comment 4

[Rail Aliiev [:rail]]

I spend a lot of time today playing with expect and finally managed to run ssh commands on slaves without manual interaction. But the resulting code is not maintainable. So, probably, it would be better to spend some time and get this bug fixed. It's mine. :)

Comment 5

[Rail Aliiev [:rail]]

Attachment: Dump ssh keys from LDAP

A simple ssh key dumper. I tested it on build-vpn.m.c.
Usage: perl ldap_dump.pl ldap_password (grep bindpw /etc/ldap.conf | awk '{print $2}')

It may be used to keep the authorized_keys file up to date.

Comment 6

[Rail Aliiev [:rail]]

BTW,

> if (/^sshPublicKey::? (.+)$/){

jford and bhearsum have two semicolons in LDAP for sshPublicKey, so we have "::?" here. FYI.

Comment 7

[John Ford [:jhford] CET/CEST Berlin Time]

instead of pulling in from LDAP, could we just build a static file that we maintain and deploy that file?

Comment 8

[Rail Aliiev [:rail]]

(In reply to comment #7)
> instead of pulling in from LDAP, could we just build a static file that we
> maintain and deploy that file?

Sure. The script should be used for initial file creation and on manual (on demand) updates, just to reduce manual work.

Comment 9

[Rail Aliiev [:rail]]

Attachment: Puppet manifest

Comment 10

[Rail Aliiev [:rail]]

Attachment: Initial authorized_keys2

Initial authorized_keys2 generated by the ldap_dump.pl tool.

I deployed it in staging using the following script:

---------------
#!/bin/bash
for platform in \
  centos5-i686  centos5-x86_64  fedora12-i686  fedora12-x86_64; do
    target=/N/staging/$platform/build/local/home/cltbld/.ssh
    mkdir -p $target
    cp authorized_keys2 $target
done

for platform in \
  darwin10-i386  darwin9-i386; do
    target=/N/staging/$platform/build/local/Users/cltbld/.ssh
    mkdir -p $target
    cp authorized_keys2 $target
done
---------------

I tested the attached puppet manifest in staging and it worked for moz2-linux-slave51, darwin9 04 and 68, darwin10 04 and linux64 build slaves.

Comment 11

[[e-mail] (:bhearsum)]

Comment on attachment 451501 [details] [diff] [review]
Puppet manifest

What's the rational for putting this in $local_fileroot rather than $puppet_fileroot? Are there differences between the staging and production versions?

Also, any reason we shouldn't apply this to the test machines, too?

Comment 12

[Rail Aliiev [:rail]]

(In reply to comment #11)
> What's the rational for putting this in $local_fileroot rather than
> $puppet_fileroot?

We can move it to $puppet_fileroot, sure.

> Are there differences between the staging and production
> versions?

No. I don't think that we're going to have different files.
 
> Also, any reason we shouldn't apply this to the test machines, too?

Do you mean adding the same entries to talos_fedora.pp and talos_osx.pp?

Comment 13

[Rail Aliiev [:rail]]

Attachment: Initial authorized_keys2

LDAP returns busted keys for bhearsum and jford. I manually fixed their keys.

Should we ask each person to confirm his/her key or a review would be enough?

Comment 14

[Armen [:armenzg]]

My key matches correctly. Thanks Rail!

Comment 15

[Rail Aliiev [:rail]]

Attachment: Puppet manifest

* s/${local_fileroot}/${platform_fileroot}/g
* Added the same entries to os/talos_*.pp

The following shell script could be used:
-----
for level in staging production; do
    for type in build test; do
        for os in \
          centos5-i686  centos5-x86_64  fedora12-i686  fedora12-x86_64; do
            target=/N/$level/$os/$type/home/cltbld/.ssh
            mkdir -p $target
            cp authorized_keys2 $target
        done
        for os in darwin10-i386  darwin9-i386; do
            target=/N/$level/$os/$type/Users/cltbld/.ssh
            mkdir -p $target
            cp authorized_keys2 $target
        done
    done
done
-----

Comment 16

[[e-mail] (:bhearsum)]

Comment on attachment 452200 [details] [diff] [review]
Puppet manifest

Seems like we're going towards sharing our staging envs with devs...so we'll have to go back to using ${local_fileroot} here.

Comment 17

[Rail Aliiev [:rail]]

Attachment: Puppet manifest

A little bit simplified version.

packages/authorized_keys.pp:
${local_fileroot}/home/cltbld/.ssh/authorized_keys2 could be used instead of ${local_fileroot}/${home}/cltbld/.ssh/authorized_keys2 to simplify file management, but the current one looks more obvious, IMO.

base/nodes.pp:

I had to "include authorized_keys" twice for "centos" and "darwin-build" because "cltbld" user is managed by cltbld.pp on build slaves.

Comment 18

[[e-mail] (:bhearsum)]

Comment on attachment 466236 [details] [diff] [review]
Puppet manifest

Why not put the authorized_keys include in the "slave" node definition? Did that not work because of where the file/httproots get defined?

I think the ~/.ssh part needs to go elsewhere. build/cltbld.pp already defines it for Linux build machines, not sure where the best place to put it for others' is.

Comment 19

[Rail Aliiev [:rail]]

Attachment: Puppet manifests

Comment 20

[Rail Aliiev [:rail]]

Attachment: authorized_keys

Updated authorized_keys (Alice has been removed).

Comment 21

[Rail Aliiev [:rail]]

Comment on attachment 477288 [details] [diff] [review]
Puppet manifests

http://hg.mozilla.org/build/puppet-manifests/rev/510a1d7d9902

Comment 22

[Rail Aliiev [:rail]]

The feature is being deployed and seen on some of the hosts. I tried to login to some of the updated hosts with different roles and platforms and they allowed me in without entering cltbld password.

Feel free to reopen if you find a host asking for a password.