1261899 - Update OpenSSH in MozillaBuild

Status: RESOLVED FIXED

(Firefox Build System :: MozillaBuild, task)

Description

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

MozillaBuild 2.2.0 ships with OpenSSH_5.4p1, OpenSSL 1.0.0 29 Mar 2010, which doesn't accept any of the configurations given in https://wiki.mozilla.org/Security/Guidelines/OpenSSH#Modern .

We need to upgrade it such that it's compatible with our OpenSSH guidelines, or we need to aggressively switch to MSYS2.

Comment 1

[Ryan VanderMeulen [:RyanVM]]

My understanding is that there are a few options:
1) Use the Microsoft OpenSSH package that was recently released. Probably the best option from the standpoint of how easily it'll Just Work (though I haven't actually tested it to confirm). However, the big drawback as I understand it is that it uses the native Windows crypto APIs rather than OpenSSL. I have no idea if that's a dealbreaker or not, but I'm assuming our security folks would have opinions on that.
2) Use the OpenSSH package from http://www.mls-software.com/opensshd.html. AFAICT, it's the only recent OpenSSH+OpenSSL package out there for Windows besides msys2. It comes with the drawbacks of coming from an unknown source (which IIRC, "someone" here expressed concerns about in the past) and that it ships with its own cygwin DLLs. I have no idea how that would play out in practice with our copy of MSYS.
3) Compile a newer OpenSSH from source that can play nicely with MSYS1. I suspect this isn't going to happen due to the supporting toolchain being too old for it.
4) Switch to msys2.

If everyone's OK with option #1, I could give that a try and see how it works. Should require the least amount of effort to stand up.
#2 could also probably be done without too much effort, but it would require some exploration and testing not to mention some level of vetting that the source is trustworthy.
#3 is almost certainly a nonstarter due to the aforementioned toolchain issues, but I'm open to it if someone's willing to do the legwork to look into it with further detail.
#4 is almost certainly going to require more time investment than I can afford to give at this point in time. I would strongly encourage an interested party to take ownership of MozillaBuild if they want to pursue switching to msys2. I will happily do what I can to support those efforts, but I can't be the one leading it at this point.

Comment 2

[Ryan VanderMeulen [:RyanVM]]

I had a chance to discuss option #1 with jbryner today and he was of the opinion that the security implications are probably not a big deal, since we'll be able to update via Windows Update if there's an issue with Windows' crypto APIs (instead of needing to pick up an updated OpenSSL package and respin MozillaBuild). And if the OS is insecure, nothing else is going to help us anyway.

So now that I've got his blessing, I'm going to proceed with investigating OpenSSH for Windows as a short-term solution.

Comment 3

[Adam Gashlin (he/him) [:agashlin] (ex-moco)]

One reason an update is desirable is to support the new OpenSSH private key format (generated via ssh-keygen -o), which was introduced in OpenSSH 6.5 and uses a much more secure key derivation mechanism. See [1], which is titled "The default OpenSSH key encryption is worse than plaintext".

[1] https://latacora.singles/2018/08/03/the-default-openssh.html

Comment 4

[Gregory Szorc [:gps]]

Windows 10 ships OpenSSH after the April 2018 update. If possible, we should consider removing OpenSSH from MozillaBuild and rely on the version installed by Windows.

That being said, I'm not sure if all of the tools in MozillaBuild will "just work" with ssh.exe provided by Windows. I would think they would, as the Windows version is literally OpenSSH (7.6p1) and shouldn't vary much from what we ship in MozillaBuild.

But obvious this would require using a modern Windows install. I can't recall if we care about supporting older Windows [10] versions in MozillaBuild.

Comment 5

[Ryan VanderMeulen [:RyanVM]]

I'm not sure if the NSIS installer has the macros to detect specific Win10 sub-versions. Doubtful we can do much at runtime to stop users on older versions. That said, our own telemetry already shows >70% of our Win10 users running 1803 anyway. I think the bigger potential issue is having to drop support for Win7 also.

Comment 6

[Adam Gashlin (he/him) [:agashlin] (ex-moco)]

We could add C:\Windows\System32\OpenSSH early in the path to prefer those SSH binaries if they exist.

I was just trying this out now and it doesn't work, though, there is something preventing MSYS from seeing that directory.

Comment 7

[Adam Gashlin (he/him) [:agashlin] (ex-moco)]

Still digging into this, but to save others some time:
This seems to be an issue with MSYS listing the directory, not permissions as I'd thought. readdir() is getting corrupted names which just happen to include corrupting the OpenSSH entry. There's an "OposHost.exe" showing up which doesn't exist at around the place where it would appear. I've traced it back as far as the FindNextFileW call the Windows API does inside of FindNextFileA but still not sure where it's ultimately coming from. One this is for sure, it's strangely reproducible.

Comment 8

[Adam Gashlin (he/him) [:agashlin] (ex-moco)]

Talk about overthinking it, the whole MSYS environment is 32 bit on 64 bit Windows, so C:\Windows\System32 points at C:\Windows\SysWOW64. It can find the dir just fine at C:\Windows\Sysnative\OpenSSH

The Python binaries are 64-bit, so if I prefix PATH with /c/Windows/System32/OpenSSH Mercurial will find it. I changed the line in /etc/profile.d/profile-extravars.sh:

> PATH="/c/Windows/System32/OpenSSH:/c/Windows/Sysnative/OpenSSH:/local/bin:$MSYS_MOZBUILD/bin:$MSYS_MOZBUILD/kdiff3:$MSYS_MOZBUILD/node-v8.11.1-win-x64:$MSYS_MOZBUILD/nsis-3.01:$MSYS_MOZBUILD/python:$MSYS_MOZBUILD/python/Scripts:$MSYS_MOZBUILD/python3:$MSYS_MOZBUILD/python3/Scripts:$PATH"

Then I hit "unable to start ssh-agent service, error :1058" when starting the shell. This is Windows' ERROR_SERVICE_DISABLED, so I went into Services.msc and enabled the OpenSSH Authentication Agent service (changed Startup type to Manual).

I also had an issue: "remote: Bad owner or permissions on C:\\Users\\Adam Gashlin/.ssh/config". Managed to fix that by removing permissions for another user on the system for the .ssh directory, through Windows' Properties->Security->Advanced, I had to disable inheritance and remove access by that other user. Single-user setups probably won't have that issue?

As an added bonus the terminal doesn't hang when closing without killing ssh-agent, because the new ssh-agent just starts the service and exits immediately!

---

So there's a few steps to take, but then ssh-add works with my new-format keys and hg over ssh can use the agent. To summarize:

1. Prefix PATH with /c/Windows/System32/OpenSSH:/c/Windows/Sysnative/OpenSSH: in /etc/profile.d/profile-extravars.sh (this is something that could be included in MozillaBuild)
2. If desired, enable the OpenSSH Authentication Agent service
3. Fix any permissions issues around ~/.ssh

Comment 9

[Jeff Muizelaar [:jrmuizel]]

Any update on this?

Comment 10

[Hector Zhao [:hectorz]]

(In reply to Adam Gashlin (he/him) [:agashlin] from comment #3)

One reason an update is desirable is to support the new OpenSSH private key
format (generated via ssh-keygen -o), which was introduced in OpenSSH 6.5
and uses a much more secure key derivation mechanism. See [1], which is
titled "The default OpenSSH key encryption is worse than plaintext".

[1] https://latacora.singles/2018/08/03/the-default-openssh.html

I've recently generated a new key in Git Bash on Windows, which was in this newer format. And I was getting a rather misleading "Bad passphrase" error when trying to use it within MozillaBuild.

Comment 12

[Kagami Rosylight [:saschanaz] (they/them)]

Can we add C:/Windows/Sysnative/OpenSSH to the path so that it can be opt-in (since it's only available when enabled as comment #8 says)?

Comment 13

[Randell Jesup [:jesup] (needinfo me)]

Note: OpenSSH in the current MozillaBuild package is so old I can't connect to my Fedora 34 system

Comment 14

[Beth Rennie [:beth] (she/her)]

(In reply to Kagami :saschanaz from comment #12)

Can we add C:/Windows/Sysnative/OpenSSH to the path so that it can be opt-in (since it's only available when enabled as comment #8 says)?

If you are using 64-bit Python, we also need to add C:/Windows/System32/OpenSSH to the path so that hg can see (because SysNative is only visible to 32-bit executables).

(In reply to Randell Jesup [:jesup] (needinfo me) from comment #13)

Note: OpenSSH in the current MozillaBuild package is so old I can't connect to my Fedora 34 system

You should switch to the SSH provided by Windows by adding /c/Windows/SysNative/OpenSSH and /c/Windows/System32/OpenSSH to your path and removing the ssh binaries in msys. I've been using it for ages and it works great. You may need

Comment 15

[Mitchell Hentges [:mhentges] 🦀]

This will likely be solved by embracing the SSH provided by Windows itself, rather than shipping a copy.

Comment 16

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1261899: Use Windows builtin OpenSSH by default

Rather than shipping OpenSSH and manually managing its agent, lean on
Windows' built-in SSH tooling which was released as part of
Windows 10 1809.

Accordingly, since Windows' OpenSSH doesn't read MSYS2's
/etc/ssh_config, remove it. This should be ok for two reasons:

• Now, we're going to always be using an up-to-date SSH version.
• Modern SSH versions have sufficiently-good defaults.

Note that this means that, for older Windows Server instances, they'll
need to manually install OpenSSH if they need ssh.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-build/rev/0340612f40b2
Use Windows builtin OpenSSH by default r=glandium