1045547 - See if we can get cygwin sshd working on our windows infrastructure

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Pete Moore [:pmoore][:pete]]

In a past life, I set up Windows servers to run sshd using cygwin.

Here is a modified form of the installation script I used:
https://github.com/petemoore/myscrapbook/blob/master/setup_cygwin.bat

This installed cygwin on a remote machine, and then copied a bash script over to the remote machine, and ran it, to complete the installation step:
https://github.com/petemoore/myscrapbook/blob/master/setup.sh

It relies on PsTools:
http://technet.microsoft.com/en-us/sysinternals/bb896649.aspx

This was set up for around 300 Windows 2003 servers, and worked reliably. I have not tested on other Windows versions. However, I'd like to see if I can also get cygwin/sshd working for the Windows OS versions we have at Mozilla, as the current sshd we have is rather limited.

For example, I cannot run a remote command, such as:

pmoore@Elisandra:~/git/mozilla master $ ssh cltbld@b-2008-ix-0082 "ls '/c/Program Files/Microsoft Visual Studio 10.0'" </dev/null
Connection to b-2008-ix-0082 closed by remote host.
pmoore@Elisandra:~/git/mozilla master $ 

This, and other problems, should be resolved by using cygwin.

Comment 1

[Justin Wood (:Callek)]

personally I don't *want* cygwin on these hosts, its management is both hard and cumbersome. And iirc there are some issues with its dll's as to how they interact with MSYS dll's.

That said, even if all-is-well, the path magic for cygwin is quite removed from the path magic for MSYS. e.g. cygwin is /cygdrive/c/foo/bar and iirc uses magic mount points. while Msys is /c/foo/bar and doesn't use magic mount points.

All that said, we absolutely need a better and more reliant ssh daemon on windows, and I'd be willing to sacrifice the useability concerns for that stability if no other options exist.

Comment 2

[Pete Moore [:pmoore][:pete]]

So, it worked!

Installed cygwin on b-2008-sm-0023.

Remote commands working, e.g.:

pmoore@Elisandra:~/git/tibco-config master $ ssh cltbld@b-2008-sm-0023 'ls; echo hello; hostname'
readme.txt
hello
B-2008-SM-0023
pmoore@Elisandra:~/git/tibco-config master $ 

scp also working...

pmoore@Elisandra:~ $ scp ship-it-config cltbld@b-2008-sm-0023:.
ship-it-config                                                                                                                                  100% 4324     4.2KB/s   00:00    
pmoore@Elisandra:~ $ ssh cltbld@b-2008-sm-0023 'ls ship*'
ship-it-config
pmoore@Elisandra:~ $

Comment 3

[Pete Moore [:pmoore][:pete]]

Incidentally, to go back to the example command I could not run in comment 0, here are the results when using cygwin sshd:

pmoore@Elisandra:~ $ ssh cltbld@b-2008-sm-0023 "ls '/c/Program Files/Microsoft Visual Studio 10.0'" </dev/null
Common7
pmoore@Elisandra:~ $

Comment 4

[Pete Moore [:pmoore][:pete]]

\o/

Comment 5

[Pete Moore [:pmoore][:pete]]

Until now I believe the objection to using cygwin was a dependency on MozillaBuild for the build process, namely:

https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Windows_Prerequisites#Install_build_prerequisites

However, on this page in the notes it says:

"If you have cygwin installed, make sure that the MozillaBuild directories come before any cygwin directories in the search path (use echo $PATH to see your search path)."

which suggests it might be possible to have both installed without problems.

The question is whether we can get builds to build with the correct PATH, but allow cygwin sshd to run, and our automation to still work.

Considerations:

1) I'm not sure if it is possible to put MozillaBuild directories in the PATH before cygwin for the shells that are spawned by the sshd process when connections are made. For interactive shells this should no doubt be trivial, but for non-interactive ones I'm not sure.
2) I believe the buildbot masters communicate using PeanutButter / PB protocol with the slaves, not over ssh, so maybe builds are unaffected by the ssh daemon, so long as it doesn't appear in the PATH in place of MozillaBuild tools for the buildbot slave processes.

I've reopened this bug as I'd like to test making builds via buildbot with b-2008-sm-0023 now that it has cygwin ssh daemon installed and running on it, and also testing how well we can interact with the shell over ssh (although it is unlikely we need to run builds via ssh interactive sessions).

Comment 6

[Pete Moore [:pmoore][:pete]]

Worth investigation: http://seasonofcode.com/posts/msys-and-ssh-server-under-windows-7.html (thanks Callek for the tip)

Comment 7

[Pete Moore [:pmoore][:pete]]

Working for TaskCluster: https://github.com/taskcluster/generic-worker/blob/b0786db002921d3ec7d40053ac2e6799679ab290/worker_types/win2012r2/userdata#L84-L101