Upgrade hg.m.o SSL certificate to SHA2

RESOLVED FIXED

Status

defect
--
major
RESOLVED FIXED
5 years ago
11 months ago

People

(Reporter: Atoll, Assigned: fubar, NeedInfo)

Tracking

(Blocks 1 bug)

Details

Attachments

(3 attachments)

Per bug 1068715, the hg.mozilla.org SSL certificate is currently signed with SHA1 and needs to be reissued as SHA2.

The Digicert portal offers a one-click 'Rekey' option that uses the previous CSR to issue the new certificate w/o revoking the old one, HOWEVER, the fingerprint of the certificate will change when it is put into place.

Mercurial 1.7 adds CA verification as an alternative to the "must accept each individual certificate" model, but presumably some developer communication will be required.

Once the new cert is generated, 'openssl x509 -in new.crt -noout -fingerprint' will emit the new fingerprint, and this can presumably be verified on hg staging as well.

:fubar, you can reach the Rekey option at https://www.digicert.com/enterprise/sha1-sunset.php
Note: if the https fingerprint is going to change, we need to update developers and releng to ensure it's not a surprise. (some apps ping to hg.m.o cert)

I.e. we'd like to publish new fingerprint about 2 weeks in advance of actual certificate cutover.
First, Mercurial doesn't allow specifying multiple fingerprints for the same hostname. So, rolling this out if clients have set fingerprints could be painful.

Mercurial does support specifying a CA cert file. We should arguably be using this everywhere we're doing fingerprint pinning so transitions can be simpler. It does open the door for a CA issuing a new certificate for hg.mozilla.org behind our backs. Not sure how paranoid we are.

I'm a Mercurial core contributor and there at 15 days left to submit patches for Mercurial 3.4. If you want it to do something more intelligent with certs, ask me and I can probably write a patch.
OpenSSH handles this by publishing 'future fingerprints' at the server, so clients can learn them prior to an event. If Mercurial were to support that sort of feature, it would simplify things tremendously for clients that (for whatever reason) do not use CA verification.

http://blog.djm.net.au/2015/02/key-rotation-in-openssh-68.html

I'm not sure there's anything we can do to make this transition easier on users than notifying them ahead of time (with both the old and new fingerprints in the announcement).
It would be *nice* if we asked users to switch to CA verification, using their system-wide certs file, rather than using certificate pinning, but that will only help if they agree with our request and do so, which is not everyone.
Duplicate of this bug: 1238938
Group: infra
(In reply to Richard Soderberg [:atoll] from comment #4)
> It would be *nice* if we asked users to switch to CA verification, using
> their system-wide certs file, rather than using certificate pinning, but
> that will only help if they agree with our request and do so, which is not
> everyone.

Python's CA verification in versions < 2.7.9 is broken in many ways. This is why Mercurial issues a warning on Python < 2.7.9 when cert fingerprints aren't pinned in the Mercurial config. Mercurial versions 3.4 and newer (I think it is 3.4 - could be 3.3) support the new Python 2.7.9 APIs that give more control over the TLS settings and CA verification methods. If you are running 2.7.9+ and hg 3.4+, Mercurial establishes sane SSL/CA defaults (like disabling SSL and requiring TLS) and won't issue a warning if the cert fingerprint isn't pinned (it will verify the cert against your system's trusted CA list).

The ideal solution is to upgrade the world to Python 2.7.11+ and Mercurial 3.6.3+ (current latest stable versions) and remove the cert pinning, as it is no longer necessary except for the truly paranoid.

Glancing at Firefox automation logs, it doesn't appear we even pin the hg.mozilla.org fingerprint everywhere. So perhaps this isn't as big as a problem as we think it is (ignoring the part where consumers not pinning aren't getting terrific transport-level encryption *sigh*).
Blocks: 450645
hg.mozilla.org's existing certificate expires on September 28.

Per IRL discussions a few weeks ago, we don't think we have any clients in automation that don't support SHA2 certs (namely XP), so it should be safe to roll out a SHA2 cert to replace the SHA1 cert.

We explicitly want to avoid multiple cert changes, as many systems have the cert fingerprint pinned and changing everything is a non-trivial amount of work. So going straight from the expiring SHA1 cert to a SHA2 cert is highly preferred.
new certificate generated and loaded onto zeus (but not yet served), as hg.mozilla.org-new

subject= /C=US/ST=California/L=Mountain View/O=Mozilla Foundation/CN=hg.mozilla.org
notBefore=Aug 29 00:00:00 2016 GMT
notAfter=Nov  2 12:00:00 2018 GMT
SHA256 Fingerprint=8E:AD:F7:6A:EB:44:06:15:ED:F3:E4:69:A6:64:60:37:2D:FF:98:88:37:BF:D7:B8:40:84:01:48:9C:26:CE:D9
SHA1 Fingerprint=73:7F:EF:AB:68:0F:49:3F:88:91:F0:B7:06:69:FD:8F:F2:55:C9:56

let me know when clients have the new fingerprint pinned and it's safe to swap certs on zeus
Assignee: nobody → klibby
fubar: could you please upload the x509 public cert to this bug?
Flags: needinfo?(klibby)
Posted file hg_mozilla_org.crt
(per request, cert on zeus is instead named hg.mozilla.org-sha2)
Flags: needinfo?(klibby)
Depends on: 1298948
Depends on: 1298976
Here is how the transition looks for various Mercurial versions that pin the fingerprint.

<3.8
====

Mercurial 3.7 and older support pinning a single SHA-1 fingerprint per host. If you are running Mercurial 3.7 and pin fingerprints, there will be a window where your client refuses to connect to hg.mozilla.org between when the server cert changes and when the client config is updated.

If you are running Mercurial <3.8, *do not disable fingerprint pinning* as Mercurial does not validate certificate CAs by default on these versions, so removing fingerprint pinning effectively removes connection security.

Today
-----

  [hostfingerprints]
  hg.mozilla.org = AF:27:B9:34:47:4E:E5:98:01:F6:83:2B:51:C9:AA:D8:DF:FB:1A:27

After Server Transition
-----------------------

  [hostfingerprints]
  hg.mozilla.org = 73:7F:EF:AB:68:0F:49:3F:88:91:F0:B7:06:69:FD:8F:F2:55:C9:56

3.8
===

3.8 allows pinning multiple SHA-1 fingerprints. Simply define both the existing and new fingerprint now and the server transition should be seamless.

Today
-----

  [hostfingerprints]
  hg.mozilla.org = 73:7F:EF:AB:68:0F:49:3F:88:91:F0:B7:06:69:FD:8F:F2:55:C9:56, AF:27:B9:34:47:4E:E5:98:01:F6:83:2B:51:C9:AA:D8:DF:FB:1A:27

After Server Transition
-----------------------

  [hostfingerprints]
  hg.mozilla.org = 73:7F:EF:AB:68:0F:49:3F:88:91:F0:B7:06:69:FD:8F:F2:55:C9:56

3.9
===

3.9 replaces [hostfingerprints] with [hostsecurity]. It also allows specifying the pinned fingerprint as a SHA-256 hash (instead of SHA-1). SHA-256 is a stronger hash, so it should be used instead of SHA-1.

Today
-----

  [hostsecurity]
  hg.mozilla.org:fingerprints = sha256:8E:AD:F7:6A:EB:44:06:15:ED:F3:E4:69:A6:64:60:37:2D:FF:98:88:37:BF:D7:B8:40:84:01:48:9C:26:CE:D9, sha256:81:3D:75:69:E3:76:F8:5B:31:1E:92:C9:CF:56:23:F6:4B:C2:82:77:E3:63:FB:7F:28:65:D0:9A:88:FB:BE:B7

After Server Transition
-----------------------

  [hostsecurity]
  hg.mozilla.org:fingerprints = sha256:8E:AD:F7:6A:EB:44:06:15:ED:F3:E4:69:A6:64:60:37:2D:FF:98:88:37:BF:D7:B8:40:84:01:48:9C:26:CE:D9
The server transition has been scheduled for 2016-09-21T1700+0000. That's 10 AM PDT on September 21.
Comment on attachment 8786083 [details]
docs: update hg.mozilla.org cert fingerprints (bug 1147548);

https://reviewboard.mozilla.org/r/75054/#review73130

::: docs/vcs-server-info.asc:39
(Diff revision 1)
>  
> +This certificate expires on 2016-09-28.
>  -----BEGIN PGP SIGNATURE-----
>  Version: GnuPG v1
>  
> -iQIcBAEBAgAGBQJW/hhyAAoJEMOR9zuaWl0rLcsQAJ/Vz4N1AItuKmct43L65QlR
> +iQIcBAEBAgAGBQJXxJp2AAoJEMOR9zuaWl0rzEgP/27Oji+US6obblMuAmMmDtpU

I'm getting 'BAD signature' while trying to verify this?

$ gpg --verify cert.gpg hg_mozilla_org.crt
gpg: Signature made Mon Aug 29 16:26:30 2016 EDT using RSA key ID 9A5A5D2B
gpg: BAD signature from "Gregory Szorc <gps@mozilla.com>" [unknown]

$ gpg -k gps@mozilla.com
pub   4096R/9A5A5D2B 2014-09-04
uid       [ unknown] Gregory Szorc <gps@mozilla.com>
uid       [  full  ] Gregory Szorc <gszorc@mozilla.com>
sub   4096R/922BEDC4 2014-09-04
Duplicate of this bug: 1302398
This is still showing as expiring. The certificate expires in 47 hours.

Raising this ticket to major, needinfo :fubar
Severity: normal → major
Flags: needinfo?(klibby)
:fubar is on PTO. The replacement is scheduled for 1000 PDT today.
Flags: needinfo?(klibby) → needinfo?(gps)
Depends on: 1305405
Flags: needinfo?(gps)
Keywords: leave-open
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/15b2d8f281ef
ansible/hg-web: update pinned fingerprints for hg.mozilla.org
hg.mozilla.org's fingerprint has been flipped.
Comment on attachment 8786083 [details]
docs: update hg.mozilla.org cert fingerprints (bug 1147548);

https://reviewboard.mozilla.org/r/75054/#review73130

> I'm getting 'BAD signature' while trying to verify this?
> 
> $ gpg --verify cert.gpg hg_mozilla_org.crt
> gpg: Signature made Mon Aug 29 16:26:30 2016 EDT using RSA key ID 9A5A5D2B
> gpg: BAD signature from "Gregory Szorc <gps@mozilla.com>" [unknown]
> 
> $ gpg -k gps@mozilla.com
> pub   4096R/9A5A5D2B 2014-09-04
> uid       [ unknown] Gregory Szorc <gps@mozilla.com>
> uid       [  full  ] Gregory Szorc <gszorc@mozilla.com>
> sub   4096R/922BEDC4 2014-09-04

I'm able to get a valid signature now

jwatkins@redridgemountains] ~$ gpg --verify Downloads/vcs-server-info.asc
gpg: Signature made Mon Sep 26 09:09:57 2016 PDT using RSA key ID 9A5A5D2B
gpg: Good signature from "Gregory Szorc <gps@mozilla.com>" [unknown]
gpg:                 aka "Gregory Szorc <gszorc@mozilla.com>" [undefined]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: C38B 9F6D A388 09CA 08DA  A5D2 C391 F73B 9A5A 5D2B
Comment on attachment 8786083 [details]
docs: update hg.mozilla.org cert fingerprints (bug 1147548);

https://reviewboard.mozilla.org/r/75054/#review79782

Lgtm.  Signature is valid although I have not verified and signed :gps' gpg key.
Attachment #8786083 - Flags: review?(jwatkins) → review+
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/e4515b3671c8
docs: update hg.mozilla.org cert fingerprints ; r=dividehex
Comment on attachment 8794878 [details]
Bug 1147548 - Update hg fingerprint in merge day scripts;

https://reviewboard.mozilla.org/r/81130/#review79784

These are usually run on a local system, or a loaner-esque system in aws closer to hg.m.o itself. So the fingerprint update here is welcome
Attachment #8794878 - Flags: review+
The certificate has been swapped. I'm not sure what other use this bug has other than to track the landing of the merge day script, which will happen when autoland is reopened. So closing.
Status: NEW → RESOLVED
Closed: 3 years ago
Keywords: leave-open
Resolution: --- → FIXED
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/b0a1038a5797
Update hg fingerprint in merge day scripts; r=Callek
Comment on attachment 8794878 [details]
Bug 1147548 - Update hg fingerprint in merge day scripts;

https://reviewboard.mozilla.org/r/81130/#review81050

these are specific to b2g merge day process. we should just delete these files.
Attachment #8794878 - Flags: review?(catlee) → review-
(In reply to Chris AtLee [:catlee] from comment #29)

This commit landed already. The b2g foo will be deleted. Soon, hopefully. I've got part 1 in bug 1306641 already :)
(In reply to Gregory Szorc [:gps] from comment #30)
> (In reply to Chris AtLee [:catlee] from comment #29)
> 
> This commit landed already. The b2g foo will be deleted. Soon, hopefully.
> I've got part 1 in bug 1306641 already :)

and yes, sadly I glossed over the "this is b2g" piece when reviewing.
Duplicate of this bug: 1277162
Flags: needinfo?(history)
Blocks: 1495464
You need to log in before you can comment on or make changes to this bug.