1078442 - Run Mercurial from Python 2.7

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, defect)

Description

[Gregory Szorc [:gps]]

We're currently running Mercurial (hgweb and hgssh) from Python 2.6. Python 2.6 has been end-of-lifed since October 2013. RedHat and others keep it on life support in order to fulfill support obligations.

As I write more and more code for Mozilla's Mercurial needs, I find myself having to spend more and more time ensuring my code works with Python 2.6. It is slowing me down and resulting in uglier, harder-to-maintain code. Python 2.6 is wasting my time and increasing technical debt.

Also, Python 2.7 is faster than Python 2.6. It is silly for us to continue to operate a service with known scaling and performance concerns on a platform that is known to be slower than another available.

I see little downside to running 2.7 in production. Others, like Facebook, do it. Let's please upgrade the Mercurial servers to Python 2.7 so we no longer have to support 2.6.

As an implementation detail, it isn't necessary to install 2.7 as the system Python. It should be fine to install 2.7 in userland and have mod_wsgi, scripts, etc simply invoke the userland Python and not the system one.

Comment 1

[Gregory Szorc [:gps]]

I booted up some Docker containers and started an `hg serve` process against a) Python 2.6 + Mercurial 3.3.2 configured just like production b) Python 2.7.8 + Mercurial 3.3.2 installed in userland. Relevant numbers are below. I ran each test multiple times and took the fastest time. I took steps to ensure page cache and I/O wait should not be a factor in these tests.

Python 2.6.6
Server User Time       187.47
Server System Time      30.61
Server Total CPU Time  218.08
Client Wall Time       334.88

Python 2.7.8
Server User Time       187.42
Server System Time      17.45
Server Total CPU Time  204.87
Client Wall Time       336.34

So 2.7.8 uses ~13.2s less CPU time per clone on the server. Or, 6%.

Not the greatest savings, but savings.

Now, not having to worry about 2.6 compatibility is reason enough to do the upgrade. A little extra performance is just a nice bonus.

Comment 2

[Gregory Szorc [:gps]]

We swung the hgweb machines over to Python 2.7 as part of running WSGI processes out of a virtualenv. However, we're still running Mercurial from Python 2.6 and the system Mercurial package on hgssh. We'll want to move hgssh over to Python 2.7.

Comment 3

[Gregory Szorc [:gps]]

Attachment: MozReview Request: hgserver: move pash to hgserver/; r?fubar

hgserver: move pash to hgserver/; r?fubar

Trying to consolidate everything into hgserver/.

Comment 4

[Gregory Szorc [:gps]]

Attachment: MozReview Request: hgserver: fix up formatting of hg_helper.py; r?fubar

hgserver: fix up formatting of hg_helper.py; r?fubar

Mostly whitespace. Some other nits reported by pyflakes addressed as
well.

This should mostly be a rubber stamp review.

Comment 5

[Gregory Szorc [:gps]]

Attachment: MozReview Request: hgserver: parameterize hg binary; r?fubar

hgserver: parameterize hg binary; r?fubar

We want to run Mercurial out of a virtualenv so we can use
a non-system package and Python 2.7. Let's paramaterize the path to the
hg binary to make this easier.

Comment 6

[Gregory Szorc [:gps]]

Attachment: MozReview Request: ansible/hg-ssh: add a virtualenv for pash; r?fubar

ansible/hg-ssh: add a virtualenv for pash; r?fubar

The best practice for virtualenvs is to have one for everything.

We establish a new virtualenv that will soon be used by pash. As part of
this, we need to add the openldap-devel package to the system so we can
build the Python LDAP package from source.

Comment 7

[Gregory Szorc [:gps]]

Attachment: MozReview Request: hgserver: run hg from Python 2.7 virtualenv in pash (bug 1078442); r?fubar

hgserver: run hg from Python 2.7 virtualenv in pash (bug 1078442); r?fubar

Now that we have a Python 2.7 virtualenv, all we need to do is have
things use it.

We simply change pash to reference the `hg` in the virtualenv to have it
use the different Mercurial installation.

As part of this, we switch from Mercurial 3.4.1 to 3.4.2. Normally I
don't like doing multiple upgrades at once. But we already run 3.4.2 on
hgweb without problems and there is nothing in the changelog from 3.4.1
that warrants concern. We would have upgraded to 3.4.2 a long time ago
had an RPM been produced.

Comment 8

[Kendall Libby [:fubar] (he/him)]

Comment on attachment 8650262 [details]
MozReview Request: ansible/hg-ssh: add a virtualenv for pash; r?fubar

https://reviewboard.mozilla.org/r/16589/#review14863

Ship It!

Comment 9

[Kendall Libby [:fubar] (he/him)]

Comment on attachment 8650259 [details]
MozReview Request: hgserver: move pash to hgserver/; r?fubar

https://reviewboard.mozilla.org/r/16583/#review14865

Ship It!

Comment 10

[Kendall Libby [:fubar] (he/him)]

Comment on attachment 8650260 [details]
MozReview Request: hgserver: fix up formatting of hg_helper.py; r?fubar

https://reviewboard.mozilla.org/r/16585/#review14867

Ship It!

Comment 11

[Kendall Libby [:fubar] (he/him)]

Comment on attachment 8650261 [details]
MozReview Request: hgserver: parameterize hg binary; r?fubar

https://reviewboard.mozilla.org/r/16587/#review14871

Ship It!

Comment 12

[Kendall Libby [:fubar] (he/him)]

Comment on attachment 8650263 [details]
MozReview Request: hgserver: run hg from Python 2.7 virtualenv in pash (bug 1078442); r?fubar

https://reviewboard.mozilla.org/r/16591/#review14873

Ship It!

Comment 13

[Gregory Szorc [:gps]]

https://hg.mozilla.org/hgcustom/version-control-tools/rev/01d8654be13f85e9fcdf0211b301c59e15f4ef0a
hgserver: run hg from Python 2.7 virtualenv in pash (bug 1078442); r=fubar

Comment 14

[Gregory Szorc [:gps]]

This is deployed and seems to be working OK. I confirmed there are python2.7 processes on the server now.

Aside from a few replication-related processes still using /usr/bin/hg and Python 2.6 (will be cleaned up shortly over in bug 1196612), hg.mozilla.org is for all intents and purposes running on Python 2.7.9.