463113 - ssh batch mode

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P4)

Description

[Chris AtLee [:catlee]]

ssh / scp should be run with batch mode on the buildbot slaves.  Batch mode prevents ssh asking for a password if no public key is accepted.  Right now if the keys aren't set up properly we wait 20 minutes before killing off the process.

Comment 1

[Armen [:armenzg]]

what is to be run in batch mode?
sh -C "scp other arguments"?

Comment 2

[Chris AtLee [:catlee]]

scp has a '-B' option, or you can use the generic -o 'BatchMode=yes' format for scp or ssh.

This just prevents ssh / scp from asking for a password.

Comment 3

[Armen [:armenzg]]

with this as I understand it does make the step fail if the keys are not set properly in that slave rather than timing out. is this correct?

Comment 4

[Chris AtLee [:catlee]]

(In reply to comment #3)
> with this as I understand it does make the step fail if the keys are not set
> properly in that slave rather than timing out. is this correct?

yes, that's right.

Comment 5

[Chris Cooper [:coop] (he/him)]

Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.

Comment 6

[Chris AtLee [:catlee]]

Fixable via a ~/.ssh/config setting?

BatchMode "yes"

or

PasswordAuthentication "no"

Comment 7

[Chris AtLee [:catlee]]

Attachment: Sync up config/known_hosts on other platforms

We'll need to audit known_hosts at some point.  I copied from staging slaves, and deleted a bunch of miscellaneous hosts from known_hosts.

I added 'BatchMode yes' to the top of all .ssh/config files.

Comment 8

[[e-mail] (:bhearsum)]

Comment on attachment 468424 [details] [diff] [review]
Sync up config/known_hosts on other platforms

Looks fine to me. Sorry for misleading you on the mac-should-work-in-cltbld.pp thing.

Comment 9

[Chris AtLee [:catlee]]

Comment on attachment 468424 [details] [diff] [review]
Sync up config/known_hosts on other platforms

changeset:   217:b581524de8a6

Comment 10

[Chris AtLee [:catlee]]

Attachment: OPSI packages for ssh config/known hosts

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 471974 [details] [diff] [review]
OPSI packages for ssh config/known hosts

>diff --git a/ssh-config-staging/CLIENT_DATA/config b/ssh-config-staging/CLIENT_DATA/config
>+IdentityFile /home/cltbld/.ssh/id_dsa

cltbld@WIN32-SLAVE20 ~
$ ls /home/cltbld/.ssh/
ls: /home/cltbld/.ssh/: No such file or directory

Need to translate the paths for win32. Looks like we have this sort of thing at the moment:
IdentityFile "/c/Documents and Settings/cltbld/.ssh/id_dsa"

Same for ssh-config/CLIENT_DATA/config. Did you test making connections after deploying the staging package ?

>diff --git a/ssh-config/CLIENT_DATA/config b/ssh-config/CLIENT_DATA/config
>+Host cvs.mozilla.org
>+IdentityFile /home/cltbld/.ssh/cvs
>+User stgbld

I've lost track of what stgbld is for, but this seems wrong production. Somehow it has no effect, eg 
 cvs -d :ext:[e-mail]:/cvsroot co -r UPDATE_PACKAGING_R12 -d build mozilla/tools/patcher
still works on the linux boxes this has been deployed to.

Otherwise looks fine, r+ if you fix the paths in config.

Comment 12

[Chris AtLee [:catlee]]

(In reply to comment #11)
> Comment on attachment 471974 [details] [diff] [review]
> OPSI packages for ssh config/known hosts
> 
> >diff --git a/ssh-config-staging/CLIENT_DATA/config b/ssh-config-staging/CLIENT_DATA/config
> >+IdentityFile /home/cltbld/.ssh/id_dsa
> 
> cltbld@WIN32-SLAVE20 ~
> $ ls /home/cltbld/.ssh/
> ls: /home/cltbld/.ssh/: No such file or directory
> 
> Need to translate the paths for win32. Looks like we have this sort of thing at
> the moment:
> IdentityFile "/c/Documents and Settings/cltbld/.ssh/id_dsa"
> 
> Same for ssh-config/CLIENT_DATA/config. Did you test making connections after
> deploying the staging package ?

mw32-ix-slave01 had /home/cltbld/.ssh referenced in its original ssh config...which looks like it wasn't working.

> 
> >diff --git a/ssh-config/CLIENT_DATA/config b/ssh-config/CLIENT_DATA/config
> >+Host cvs.mozilla.org
> >+IdentityFile /home/cltbld/.ssh/cvs
> >+User stgbld
> 
> I've lost track of what stgbld is for, but this seems wrong production. Somehow
> it has no effect, eg 
>  cvs -d :ext:[e-mail]:/cvsroot co -r UPDATE_PACKAGING_R12 -d
> build mozilla/tools/patcher
> still works on the linux boxes this has been deployed to.

I also don't know what it's for.  Should I remove it altogether?

Comment 13

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: idea for mozilla-central patch

if the main consumers of the upload and uploadsymbols targets are automated systems we could make the default action be to upload in batch mode regardless of client options.

(untested patch)

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #12)
> I also don't know what it's for.  Should I remove it altogether?

Perhaps we should, after looking at all the cvs references in buildbotcustom. I bet we're using :ext:cltbld@cvs just about everywhere, but haven't checked how that interacts with the .ssh/config. A random spot check of production machines finds cvs.pub present, with cltbld@staging-build-console as the generating host. Goes against our policy to not mix production and staging keys.

Comment 15

[Chris AtLee [:catlee]]

(In reply to comment #14)
> (In reply to comment #12)
> > I also don't know what it's for.  Should I remove it altogether?
> 
> Perhaps we should, after looking at all the cvs references in buildbotcustom. I
> bet we're using :ext:cltbld@cvs just about everywhere, but haven't checked how
> that interacts with the .ssh/config. A random spot check of production machines
> finds cvs.pub present, with cltbld@staging-build-console as the generating
> host. Goes against our policy to not mix production and staging keys.

If we're using explicit username options to CVS, those override the user in .ssh/config.

I'll remove them from the opsi configs.

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

The two modifications we talked about (path to home dir, dropping staging keys) aren't implemented anywhere on staging-opsi. Removing blocking on downtime bug

Comment 17

[Chris AtLee [:catlee]]

Attachment: OPSI packages for ssh config/known hosts

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 476840 [details] [diff] [review]
OPSI packages for ssh config/known hosts

>diff --git a/ssh-config-staging/CLIENT_DATA/config b/ssh-config-staging/CLIENT_DATA/config
>@@ -0,0 +1,5 @@
>+BatchMode yes
>+IdentityFile "/c/Documents and Settings/cltbld/.ssh/id_dsa"

I think I'd prefer to use ~ instead of "/c/Documents and Settings/cltbld/", since it should allow us to use the same config on all platforms. I tested that it works on win32-slave04.

Comment 19

[[e-mail] (:bhearsum)]

Comment on attachment 476840 [details] [diff] [review]
OPSI packages for ssh config/known hosts

changeset:   63:d9a4f7770b6a

Comment 20

[[e-mail] (:bhearsum)]

Set to deploy on all 32-bit windows build machines (staging and production).

Comment 21

[Chris AtLee [:catlee]]

We're all done here.