1495464 - hg.mozilla.org certificate renewal work

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, enhancement)

Description

[Connor Sheehan [:sheehan]]

The certificate for hg.mozilla.org is set to expire on November 2nd, 2018. According to Kendall from #vcs: 

> web.ops will automatically create one in their comp. we need to have them (or me, as last resort) renew it, add it to the load balancers *WITHOUT* enabling it, and then give us the new fingerprint a week or two in advance
> then we need to communicate out to various lists that we're updating it (assuming the fp changes)
> not sure if we're still pinning the fp anywhere in automation; if so we'll need to file bugs to update

A quick search of v-c-t shows we are pinning the fingerprint in the configwizard extension, so they are set in users' `hostsecurity` hgrc section. We will need  to update the fingerprint there and anywhere else we find in the next few weeks.

Comment 1

[Connor Sheehan [:sheehan]]

The Taskcluster secret `project/taskcluster/gecko/hgfingerprint` contains the fingerprint, and will need to be updated accordingly as well.

Comment 2

[Kim Moir [:kmoir] ET]

From #vcs

tomprince> Probably should update the fallback fingerprint in run-task (after the cert has changed)

Comment 3

[Gregory Szorc [:gps]]

Bug 1147548 is the last time we rotated the cert and contains a lot of history.

Here are some emails I sent out at the time:

---
hg.mozilla.org's existing x509/SSL certificate expires in a few weeks.

We plan on swapping the certificate on September 21 around 1700 UTC (1000 PDT).

Bug 1147548 tracks the work.

A new certificate has already been issued. Instructions on how to configure your client to pin the fingerprint are in bug 1147548.

The new certificate is pretty similar to the old one. The main difference is it is using SHA-256 for signatures instead of SHA-1.

Mercurial 3.8+ supports defining multiple fingerprints for a single host. This means you can add the new fingerprint before the server change and you don't have to do anything when the server transitions to the new cert.

If you are running Mercurial 3.7 or earlier and you have the fingerprint pinned, you'll need to update your systems when the server changes or you'll get fingerprint mismatches and Mercurial will refuse to connect until you do (i.e. you will have service disruption). *Do not remove fingerprint pinning on these versions without my review otherwise you risk disabling connection security nearly completely.*

If you are upgrading Mercurial to get the ability to pin multiple fingerprints and avoid a service disruption, I *highly* recommend upgrading to Mercurial 3.9 (3.9.1 will be released September 1), as 3.9 contains much better defaults when it comes to connection security (https://www.mercurial-scm.org/wiki/SecureConnections).

If you need help upgrading your systems, modifying the Mercurial config, or want my review, I'll be happy to assist. Just ask.

A more broad announcement will be made at a later date. For now, I want to get the message out to those running automated systems.
---

---
Gregory Szorc <[e-mail]>
	
Thu, Sep 22, 2016, 1:57 PM
	
to dev-platform, Firefox
hg.mozilla.org's x509 server certificate (AKA an "SSL certificate") expires next week.

A new certificate has already been issued and it is scheduled to be swapped in around 2016-09-26T17:00Z (Monday September 26 10:00 PDT). The transition may be delayed to avoid downtime in automation, which hasn't fully prepared for the change yet.

The only major change to the certificate is it is using SHA-256 for signatures. This is known to not work with ancient software (such as Windows XP SP2). We don't anticipate any major problems with this, however.

If you pin the host fingerprint in your Mercurial config file, you'll need to install a new fingerprint or Mercurial will refuse to connect once the certificate is swapped. The fingerprint of the new certificate and Mercurial config snippets for configuring it are available at https://bugzilla.mozilla.org/show_bug.cgi?id=1147548#c12.

It's worth noting that Mercurial 3.8+ supports pinning multiple fingerprints per host. So, if you install the new fingerprint today, you don't need to take action when the server certificate is swapped next week.

If you notice any problems after the cert change, please make noise in #vcs on IRC.
---

---
The certificate has been flipped.

New hashes are:

sha1:73:7f:ef:ab:68:0f:49:3f:88:91:f0:b7:06:69:fd:8f:f2:55:c9:56
sha256:8e:ad:f7:6a:eb:44:06:15:ed:f3:e4:69:a6:64:60:37:2d:ff:98:88:37:bf:d7:b8:40:84:01:48:9c:26:ce:d9

You can pin these in your hgrc via:

# Mercurial 3.9+

[hostsecurity]
hg.mozilla.org:fingerprints = sha256:8e:ad:f7:6a:eb:44:06:15:ed:f3:e4:69:a6:64:60:37:2d:ff:98:88:37:bf:d7:b8:40:84:01:48:9c:26:ce:d9

# Mercurial <= 3.8

[hostfingerprints]
hg.mozilla.org = 73:7f:ef:ab:68:0f:49:3f:88:91:f0:b7:06:69:fd:8f:f2:55:c9:56

Please make noise in #vcs or #releng if you see breakage.
---

Our procedure for doing the flip is:

1) Generate a new x509 certificate (but don't activate it yet)
2) Modify all parts of CI that pin the certificate to recognize the fingerprint of the newly-generated certificate (Mercurial 3.9+ - which should be running everywhere) supports pinning multiple fingerprints.
3) Send out announcement emails containing the transition time, the fingerprint of the new certificate, and instructions for pinning the new certificate (likely a link to a bug comment)
4) Wait until transition time
5) Install the new certificate in the load balancers

Last time, we generated the new cert and sent out emails and change notices ~1 month before the cert flip. We're within that 1 month window already. So I think we should generate the new cert ASAP and send out the notifications and get bugs on filed soon after. Ideally by middle of next week.

Looks like last time fubar created the new cert. So needinfo fubar to do that again or delegate to someone who can.

Again, the new certificate won't get installed until closer to expiration. Maybe on October 30 or so.

Comment 4

[Gregory Szorc [:gps]]

Forgot to set needinfo on last comment.

Comment 5

[Kendall Libby [:fubar] (he/him)]

While I can generate new certs, generally we should leave it to WebOps; they've got a system to notify them of upcoming renewals and automating the renewal. I've moved bug 1496800 over to the correct component.

Comment 6

[Gregory Szorc [:gps]]

The new certificate has been issued in bug 1496800.

Its fingerprints are as follows:

sha1:   1c:a5:7d:a1:28:db:78:f6:52:4d:c0:e6:38:9b:08:43:ec:1f:ef:64
sha256: 17:38:aa:92:0b:84:3e:aa:8e:52:52:e9:4c:2f:98:a9:0e:bf:6c:3e:e9:15:ff:0a:29:80:f7:06:02:5b:e8:48

We plan to swap in the new cert on 2018-10-31 around 1700 UTC / 1000 PDT.

Between now and the swap, Mercurial's fingerprint pinning should be configured as follows:

Mercurial 3.9+
--------------

[hostsecurity]
hg.mozilla.org:fingerprints = sha256:17:38:aa:92:0b:84:3e:aa:8e:52:52:e9:4c:2f:98:a9:0e:bf:6c:3e:e9:15:ff:0a:29:80:f7:06:02:5b:e8:48, sha256:8e:ad:f7:6a:eb:44:06:15:ed:f3:e4:69:a6:64:60:37:2d:ff:98:88:37:bf:d7:b8:40:84:01:48:9c:26:ce:d9

Mercurial 3.8 (which should not be used anymore but just in case)
-----------------------------------------------------------------

[hostfingerprints]
hg.mozilla.org = 1c:a5:7d:a1:28:db:78:f6:52:4d:c0:e6:38:9b:08:43:ec:1f:ef:64, 73:7f:ef:ab:68:0f:49:3f:88:91:f0:b7:06:69:fd:8f:f2:55:c9:56

Mercurial <= 3.7
----------------

(no change - <= 3.7 does not support pinning multiple fingerprints)

After the new certificate is installed, you can drop the old certificate fingerprint from the config.

Mercurial 3.9+
--------------

[hostsecurity]
hg.mozilla.org:fingerprints = sha256: 17:38:aa:92:0b:84:3e:aa:8e:52:52:e9:4c:2f:98:a9:0e:bf:6c:3e:e9:15:ff:0a:29:80:f7:06:02:5b:e8:48

Mercurial <=3.8
---------------

[hostfingerprints]
hg.mozilla.org = 1c:a5:7d:a1:28:db:78:f6:52:4d:c0:e6:38:9b:08:43:ec:1f:ef:64

Comment 7

[Gregory Szorc [:gps]]

Please be careful of whitespace when copying comment #6: the fingerprints should be a comma-delimited list on the same line in the file.

Comment 8

[Gregory Szorc [:gps]]

Attachment: ansible/hg-web: add new hg.mo certificate fingerprint to hgrc (bug 1495464); r?sheehan

Comment 9

[Gregory Szorc [:gps]]

Attachment: configwizard: add new hg.mo certificate fingerprint (bug 1495464); r?sheehan

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/a90572853508
ansible/hg-web: add new hg.mo certificate fingerprint to hgrc ; r=sheehan
https://hg.mozilla.org/hgcustom/version-control-tools/rev/e5c5c4c1b3ea
configwizard: add new hg.mo certificate fingerprint ; r=sheehan

Comment 11

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Attachment: GitHub Pull Request

update the mercurial.ini fingerprints on windows taskcluster workers

Comment 12

[Pete Moore [:pmoore][:pete]]

Comment on attachment 9021436 [details] [diff] [review]
GitHub Pull Request

Review of attachment 9021436 [details] [diff] [review]:
-----------------------------------------------------------------

Looks perfect! Thanks Rob. :)

Comment 13

[Pete Moore [:pmoore][:pete]]

(no idea why my review caused the mime type changes etc - I just reviewed in Splinter as normal...)

Comment 14

[Rob Thijssen [:grenade (EET/UTC+0300)]]

merged: https://github.com/mozilla-releng/OpenCloudConfig/commit/4fd026c

Comment 15

[Pete Moore [:pmoore][:pete]]

(In reply to Pete Moore [:pmoore][:pete] from comment #13)
> (no idea why my review caused the mime type changes etc - I just reviewed in
> Splinter as normal...)

Ah of course, it was a github review request, and the `[review]` link took me to Splinter review...

*sigh*

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/f83b578fcafb
docs: update standalone file with certificate information

Comment 17

[Gregory Szorc [:gps]]

Attachment: configwizard: update SHA-1 certificate, purge old certificate (bug 1495464); r?sheehan

We couldn't update the SHA-1 fingerprint until the certificate was live
because Mercurial versions using the SHA-1 fingerprint can't pin multiple
fingerprints.

We also remove the old SHA-256 fingerprint because it is no longer live.

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/7891d8c4a282
configwizard: update SHA-1 certificate, purge old certificate ; r=sheehan