Deploy Mercurial 3.9 to automation

RESOLVED FIXED

Status

Release Engineering
General Automation
RESOLVED FIXED
2 years ago
a year ago

People

(Reporter: gps, Assigned: aselagea)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

(Reporter)

Description

2 years ago
Generally speaking, Firefox automation is running Mercurial 3.7.3. (We last upgraded in bug 1229494.)

Mercurial 3.8 and 3.9 contain feature enhancements which additionally benefit security.

* Mercurial 3.8 supports pinning multiple certificate fingerprints per hostname. This means servers can rotate certificates without having to coordinate client updates at exactly the same time.

* Mercurial 3.9 requires TLS 1.1+ by default

* Mercurial 3.9 requires validated server certificates by default

* Mercurial 3.9 allows pinning CA certificates on a per-hostname basis

See more on Mercurial's enhanced connection security in version 3.9 at https://www.mercurial-scm.org/wiki/SecureConnections.

We should deploy Mercurial 3.9 (likely 3.9.1 which will be released this Friday) to automation.

If we deploy Mercurial 3.8+ to automation *before September 21*, automation can be configured to seamlessly transition to the new hg.mozilla.org certificate being installed on that date (tracked in bug 1147548). Otherwise, automation will likely incur some downtime on September 21 during the window where the hg.mozilla.org certificate is replaced.
(Reporter)

Comment 1

2 years ago
Mercurial 3.9.1 was released today.

Coop: can you please find an owner for this?

Per the initial comment, if we do this before September 21, buildbot automation can transition seamlessly when hg.mozilla.org's SSL cert is replaced on that date.
Flags: needinfo?(coop)

Comment 2

2 years ago
(In reply to Gregory Szorc [:gps] from comment #1)
> Mercurial 3.9.1 was released today.
> 
> Coop: can you please find an owner for this?
> 
> Per the initial comment, if we do this before September 21, buildbot
> automation can transition seamlessly when hg.mozilla.org's SSL cert is
> replaced on that date.

I *think* we have sufficient docs on this now that buildduty should be able to handle the upgrade (possibly not on Windows). I will ask them to pick this up at our meeting tomorrow.
Flags: needinfo?(coop)
(Assignee)

Updated

2 years ago
Assignee: nobody → aselagea
(Assignee)

Updated

a year ago
Depends on: 1302373
(Assignee)

Updated

a year ago
Depends on: 1302374
(Assignee)

Updated

a year ago
Depends on: 1302375
(Assignee)

Updated

a year ago
Depends on: 1302376
(Assignee)

Comment 3

a year ago
The new(In reply to Gregory Szorc [:gps] from comment #1)
 
> Per the initial comment, if we do this before September 21, buildbot
> automation can transition seamlessly when hg.mozilla.org's SSL cert is
> replaced on that date.

I don't think the upgrade process will be finished by then. If possible, could you please help with some directions on what measures we should take to prevent unwanted issues when the SSL cert for hg.mozilla.org is replaced?
Flags: needinfo?(gps)
(Assignee)

Updated

a year ago
Depends on: 1303766
ryanvm: are there plans to release a new MozillaBuild shortly with hg 3.9? no worries if not, but if there are it's a simpler upgrade for TaskCluster Windows workers.
Flags: needinfo?(ryanvm)
No. MozillaBuild is on hiatus in favor of msys2. I'd be happy to spin you a custom build w/ newer hg if it helps, though. Would be easy for me to do.
Flags: needinfo?(ryanvm)
if mozilla-build isn't going to get regular updates, i'll bite the bullet now and build support into the ami generators to update components individually (including hg).
(Reporter)

Comment 7

a year ago
The Mercurial in the latest release of MozillaBuild can be updated by running `pip install --upgrade Mercurial`.

Although you don't want to be relying on pypi.python.org in automation. It should be simple enough to vendor the Mercurial .whl file into our PyPI mirror or wherever.
(Reporter)

Comment 8

a year ago
Bug 1147548 comment #12 documents what Mercurial config files should contain to support the server certificate migration.

Note that unless Mercurial 3.8+ is deployed to automation, switching the hg.mozilla.org certificate will result in downtime in automation where we're pinning the certificate fingerprint until the pinned fingerprint can be updated to the new one.

Also, we should be pinning the hg.mozilla.org fingerprint everywhere in automation for security reasons. I consider failure to pin the hg.mozilla.org fingerprint a bug, regardless of what Mercurial version is running.
Flags: needinfo?(gps)
(Assignee)

Comment 9

a year ago
It looks that HG 3.9+ will display a warning message when the TLS version used is 1.0, regardless the Python version. While TLS 1.1+ is only available starting with Python 2.7.9 (with some exceptions), I don't think our builders and testers do cloning operations using a python version >= 2.7.9 (could be wrong though).

While I'm not seeing this as a real issue, we started receiving some mails related to the above mentioned warning message for cruncher-aws server after the hg upgrade. I think the easiest fix would be to do something like:

[hostsecurity]
disabletls10warning = true
Flags: needinfo?(gps)
(Reporter)

Comment 10

a year ago
The warning is harmless.

I'd prefer you leave it as a constant reminder that our Python used in automation is insecure. Perhaps seeing it enough times will finally spur someone into fixing it.
Flags: needinfo?(gps)

Updated

a year ago
Depends on: 1304813
(In reply to Gregory Szorc [:gps] from comment #10)
> The warning is harmless.
> 
> I'd prefer you leave it as a constant reminder that our Python used in
> automation is insecure. Perhaps seeing it enough times will finally spur
> someone into fixing it.

I agree but NOT if they are coming in every 15m.  Can we at least file a bug to upgrade python on cruncher and then maybe silence the check?
(Assignee)

Comment 12

a year ago
Upgrading Python will likely not solve the issue here. For example, t-yosemite-r7 testers have Python 2.7.10 installed and HG 3.9.1 will still throw this warning when cloning a repo (tested on such a loaner). I suspect we have set TLS 1.0 as the default security technology somewhere, but I'm not able to find that bit. So unless we also change that, upgrading the Python version will not get rid of this message.

Besides the alert coming for cruncher, the buildbot masters also log this warning when they do a reconfig (that usually happens hourly) ==> extra amount of logs, which are then sent to Papertrail (we don't receive alerts for those, though). 
https://papertrailapp.com/groups/1223224/events?q=program%3Amaybe_reconfig.sh&focus=715870300549713922

So using "disabletls10warning" option would likely be the easiest way to silence the warnings, yet I do agree that a reminder for the insecure Python version we are using in automation is useful.
(Assignee)

Comment 13

a year ago
Created attachment 8794166 [details] [diff] [review]
bug_1298976.patch

Patch to replace the fingerprints for Linux and OS X servers (already upgraded to HG 3.9.1), we can remove the "IF" loop once the work for Windows is also done.
Attachment #8794166 - Flags: review?(arich)
Comment on attachment 8794166 [details] [diff] [review]
bug_1298976.patch

Review of attachment 8794166 [details] [diff] [review]:
-----------------------------------------------------------------

If we're removing the ftp-ssl.mozilla.org fingerprint for one set of OSes, we should remove it for all to be consistent. It's not clear to me that that's the best route to take, though. For this patch, how about we just modify the hg one for linux/osx and leave ftp-ssl alone till we can have a larger discussion?
Attachment #8794166 - Flags: review?(arich) → review+
(Assignee)

Comment 15

a year ago
Created attachment 8794185 [details] [diff] [review]
bug_1298976_v2.patch

Okay, this would be the updated patch so that we don't touch ftp-ssl.m.o yet.
(Assignee)

Updated

a year ago
Depends on: 1305077
(Reporter)

Comment 17

a year ago
(In reply to Alin Selagea [:aselagea][:buildduty] from comment #12)
> Upgrading Python will likely not solve the issue here. For example,
> t-yosemite-r7 testers have Python 2.7.10 installed and HG 3.9.1 will still
> throw this warning when cloning a repo (tested on such a loaner). I suspect
> we have set TLS 1.0 as the default security technology somewhere, but I'm
> not able to find that bit. So unless we also change that, upgrading the
> Python version will not get rid of this message.

OS X is special. The Python provided by Apple (/usr/bin/python) doesn't support TLS 1.1+. Even on OS X Sierra. If you want a secure Python on OS X, you need to install it from elsewhere or compile from source. https://github.com/Homebrew/homebrew-core/issues/3541 has more info.
(In reply to Amy Rich [:arr] [:arich] from comment #14)
> If we're removing the ftp-ssl.mozilla.org fingerprint for one set of OSes,
> we should remove it for all to be consistent. It's not clear to me that
> that's the best route to take, though. For this patch, how about we just
> modify the hg one for linux/osx and leave ftp-ssl alone till we can have a
> larger discussion?

This is the right call, there are still a few jobs that use bundles via ftp-ssl.m.o. Bug 1247169 covers cleaning that up.
nthomas: Sorry, which is the right call, leaving it in or removing it? :}
Flags: needinfo?(nthomas)
Keeping ftp-ssl.m.o in .hgrc for now.
Flags: needinfo?(nthomas)
(Assignee)

Comment 21

a year ago
Mercurial 3.9.1 has been deployed to automation, marking this as fixed.
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.