1303766 - Configure Mercurial to use a trusted CA certificate file

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Alin Selagea [:aselagea]]

While testing the new hg version (3.9.1), I noticed that cloning the /tools/ repo fails (OS X) or displays a related warning (Ubuntu) as the fingerprints for the servers mentioned in the bug description have changed (see https://dxr.mozilla.org/build-central/source/puppet/modules/mercurial/templates/mercurial.ini.erb#89) 
 
However, cloning went fine when adding the correct entries in puppet and running puppet agent.

Comment 1

[Alin Selagea [:aselagea]]

I'm not sure if the fingerprint for ftp-ssl.mozilla.org has also changed (couldn't find a way to determine that).
@gps: any thoughts on this?

Comment 2

[Gregory Szorc [:gps]]

Pinning fingerprints for domains not under our control is likely not a game we want to play since those certificates can change at any time. Also, server operators may have multiple certificates installed for a hostname.

I would configure Mercurial to use a trusted CA cert file ([web] cacerts=/path/to/file) and only pin fingerprints for hostnames under Mozilla's control.

Comment 3

[Gregory Szorc [:gps]]

1. Download https://curl.haxx.se/ca/cacert.pem, which is Firefox's CA bundle in concatenated x509 format.
2. Install said file somewhere on hosts
3. Add the following to /etc/mercurial/hgrc

[web]
cacerts = /path/to/cacert.pem

In addition, the global hgrc should pin the hg.mozilla.org fingerprint per the instructions in bug 1147548 comment 12.

Comment 4

[Bill Gianopoulos [:WG9s]]

(In reply to Gregory Szorc [:gps] from comment #2)
> Pinning fingerprints for domains not under our control is likely not a game
> we want to play since those certificates can change at any time. Also,
> server operators may have multiple certificates installed for a hostname.
> 
> I would configure Mercurial to use a trusted CA cert file ([web]
> cacerts=/path/to/file) and only pin fingerprints for hostnames under
> Mozilla's control.

This would also cause issues at certificate renewal time.

Comment 5

[Gregory Szorc [:gps]]

We want to pin the certificate for hg.mozilla.org because that is more secure than trusting N CAs. We don't want to take the chance that Firefox source code delivery can be man-in-the-middled by a rogue CA.

Mercurial 3.8+ supports pinning multiple fingerprints per hostname. For hg.mozilla.org, we announced the new fingerprint several weeks before the new cert was installed (see bug 1147548 comment 12).

So there should be no problem pinning [multiple] fingerprints for hg.mozilla.org.

Comment 6

[Bill Gianopoulos [:WG9s]]

(In reply to Gregory Szorc [:gps] from comment #5)
> We want to pin the certificate for hg.mozilla.org because that is more
> secure than trusting N CAs. We don't want to take the chance that Firefox
> source code delivery can be man-in-the-middled by a rogue CA.
> 
> Mercurial 3.8+ supports pinning multiple fingerprints per hostname. For
> hg.mozilla.org, we announced the new fingerprint several weeks before the
> new cert was installed (see bug 1147548 comment 12).
> 
> So there should be no problem pinning [multiple] fingerprints for
> hg.mozilla.org.

Pinning fingerprints for things like the automatic update etc seem just fine to me.  pinning fingerprints for normal web browsing would seem to me to open up more issues than it solves.

Comment 7

[Gregory Szorc [:gps]]

This bug only concerns Firefox CI, not something that ships in Firefox.

Comment 8

[Bill Gianopoulos [:WG9s]]

(In reply to Gregory Szorc [:gps] from comment #7)
> This bug only concerns Firefox CI, not something that ships in Firefox.

OK. Sorry, I mis-understood.

Comment 9

[Bill Gianopoulos [:WG9s]]

(In reply to Bill Gianopoulos [:WG9s] from comment #8)
> (In reply to Gregory Szorc [:gps] from comment #7)
> > This bug only concerns Firefox CI, not something that ships in Firefox.
> 
> OK. Sorry, I mis-understood.

(and I can't resist saying this reminds me of an old song by the animals!)

Comment 10

[Alin Selagea [:aselagea]]

Attachment: add_CA_bundle.patch

Adding the CA bundle to puppet configs.

Comment 11

[Alin Selagea [:aselagea]]

Attachment: bug_1303766.patch

Comment 12

[Alin Selagea [:aselagea]]

I applied the above changes to my puppet environment and pinned several slaves to it (CentOS, Ubuntu, OS X - 10.6, 10.7 and 10.10.5). The hg upgrade process went fine in each case. I even tried cloning the /tools repo and that also worked fine.

Comment 13

[Alin Selagea [:aselagea]]

Attachment: bug_1303766.patch

Actually, I think we need to migrate cacert.pem wherever mercurial is installed (not only on slaves), so I updated the patch to account for that. I tested it on buildbot-master69 (which is out of production) and it looks fine.

Comment 14

[Alin Selagea [:aselagea]]

Also, once the work on this bug is done, is there a reason not to go on and land the patches to upgrade HG to version 3.9.1 on CentOS, Ubuntu and OS X? Or should we wait for Windows to be ready too?

Comment 15

[Gregory Szorc [:gps]]

(In reply to Alin Selagea [:aselagea][:buildduty] from comment #14)
> Also, once the work on this bug is done, is there a reason not to go on and
> land the patches to upgrade HG to version 3.9.1 on CentOS, Ubuntu and OS X?
> Or should we wait for Windows to be ready too?

If a platform is ready to be upgraded, I say move forward with it: don't wait.

Comment 16

[Gregory Szorc [:gps]]

Comment on attachment 8793289 [details] [diff] [review]
add_CA_bundle.patch

Review of attachment 8793289 [details] [diff] [review]:
-----------------------------------------------------------------

rs=me

Comment 17

[Gregory Szorc [:gps]]

Comment on attachment 8793354 [details] [diff] [review]
bug_1303766.patch

Review of attachment 8793354 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/mercurial/manifests/cacert.pp
@@ +7,5 @@
> +class mercurial::cacert {
> +    case $operatingsystem {
> +        CentOS,Ubuntu,Darwin: {
> +            file {
> +                '/usr/local/etc/cacert':

Since this is used by Mercurial, perhaps you should put the file in /etc/mercurial?

::: modules/mercurial/templates/hgrc.erb
@@ +37,4 @@
>  prefers = stream=revlogv1
>  <% end -%>
>  
> +[hostsecurity]

You can't make this change until Mercurial 3.9 is deployed. I don't see Mercurial 3.9 being deployed in this patch.

Comment 18

[Alin Selagea [:aselagea]]

Attachment: bug_1303766_v2.patch

Sorry, I was trying to rush things a bit :-)
Attached a new version of the patch.

Comment 19

[Gregory Szorc [:gps]]

Comment on attachment 8793369 [details] [diff] [review]
bug_1303766_v2.patch

Review of attachment 8793369 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/mercurial/manifests/cacert.pp
@@ +1,5 @@
> +# This Source Code Form is subject to the terms of the Mozilla Public
> +# License, v. 2.0. If a copy of the MPL was not distributed with this
> +# file, You can obtain one at http://mozilla.org/MPL/2.0/.
> +
> +# install cacert.pem and place it in /usr/local/etc/cacert/ (Ubuntu, CentOS and Darwin)

This comment needs updated.

Comment 20

[Amy Rich [:arr] [:arich]]

I've checked in the cert bundle:

remote:   https://hg.mozilla.org/build/puppet/rev/c60ca7cd6316
remote:   https://hg.mozilla.org/build/puppet/rev/c1a9f27c3db6

Comment 21

[Amy Rich [:arr] [:arich]]

checked in config to add cacert.pem on OS X and linux machines (location comment updated):

remote:   https://hg.mozilla.org/build/puppet/rev/da673dc6966d
remote:   https://hg.mozilla.org/build/puppet/rev/6bb4755a035b (fix trailing whitespace bustage from the patch)
remote:   https://hg.mozilla.org/build/puppet/rev/11a968bdda8e

Comment 22

[Amy Rich [:arr] [:arich]]

(In reply to Gregory Szorc [:gps] from comment #2)

Does cloudfront control the SSL cert, or do we?

Comment 23

[Alin Selagea [:aselagea]]

Attachment: fix_mock_env.patch

Switching the location of the CA bundle to /builds/mercurial-certs/ for Linux servers due to build failures ("Abort: could not find web.cacerts: /etc/mercurial/cacert/cacert.pem").

Comment 24

[Justin Wood (:Callek)]

Comment on attachment 8794823 [details] [diff] [review]
fix_mock_env.patch

afaik this should fix it, but we certainly need to regen the ami's.

Comment 25

[Alin Selagea [:aselagea]]

Comment on attachment 8794823 [details] [diff] [review]
fix_mock_env.patch

http://hg.mozilla.org/build/puppet/rev/ead550cce5e5
http://hg.mozilla.org/build/puppet/rev/76995acb56c5

Comment 27

[Alin Selagea [:aselagea]]

Attachment: fix_mock_env_2.patch

Ensuring we have a /builds directory on all the machines where we copy the cert bundle as it is missing on some of them.

Comment 28

[Jake Watkins [:dividehex]]

Comment on attachment 8794896 [details] [diff] [review]
fix_mock_env_2.patch

Review of attachment 8794896 [details] [diff] [review]:
-----------------------------------------------------------------

I'm fine with this but for consistency sake you could just include dirs::builds

Comment 29

[Alin Selagea [:aselagea]]

Attachment: fix_mock_env_3.patch

Comment 30

[Alin Selagea [:aselagea]]

Attachment: fix_mock_env_3.patch

Updated patch.

Comment 31

[Alin Selagea [:aselagea]]

Comment on attachment 8794918 [details] [diff] [review]
fix_mock_env_3.patch

Fixed the syntax error.