1307757 - Deploy python 2.7.12 to releng servers

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Chris Cooper [:coop] (he/him)]

As a follow-up to bug 1304813, I want us to deploy python 2.7.12 to our admin nodes, namely buildduty-tools and cruncher-aws. 

We continue to get errors like this when performing hg ops using the existing python version (2.7.3):

"warning: connecting to hg.mozilla.org using legacy security technology (TLS 1.0); see https://mercurial-scm.org/wiki/SecureConnections for more info"

We'll need to recreate the venvs for the various scripts to take advantage of the new python version once it's installed. Puppet _may_ be able to do this for us automatically.

Comment 1

[Alin Selagea [:aselagea]]

Attachment: bug_1307757.patch

Created a source rpm for Python 2.7.12 and, starting from it, the actual rpm for x86_64. Next, I deployed them to /data/repos/yum/custom/mozilla-python2712 on releng-puppet2 and tested for a custom loaner similar to the "buildduty-tools" and "cruncher-aws" machines that we want to upgrade the Python version for.

I initially tried to create separate manifests for these two machines, but we have many dependencies for the "packages::mozilla::python27" manifest, so I think the easiest way would be to simply add an IF statement and only upgrade to the newer version if the hostname corresponds to one of the two machines mentioned above.

Comment 2

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8802174 [details] [diff] [review]
bug_1307757.patch

Review of attachment 8802174 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/packages/manifests/mozilla/python27.pp
@@ +42,4 @@
>  
>              case $::operatingsystem {
>                  CentOS: {
> +                    if $hostname in [ 'buildduty-tools', 'cruncher-aws' ] {

Is there a machine class attribute we could use for this rather than specifying these by name? 

At the very least, let's add a comment above this section to call out why we're making the exception for these hosts.

Comment 3

[Alin Selagea [:aselagea]]

Comment on attachment 8802174 [details] [diff] [review]
bug_1307757.patch

http://hg.mozilla.org/build/puppet/rev/58e5c9f14551
http://hg.mozilla.org/build/puppet/rev/0a4bd48aff4b

Comment 4

[Alin Selagea [:aselagea]]

Checked the two servers and noticed that the Python version inside the virtual envs has been updated to 2.7.12. However, the warning about the insecure connection is still triggered:

"warning: connecting to hg.cdn.mozilla.net using legacy security technology (TLS 1.0); see https://mercurial-scm.org/wiki/SecureConnections for more info"

@gps: Any idea what further changes are needed here?

Comment 5

[Kim Moir [:kmoir] ET]

Alin: You mentioned in the buildduty meeting today that you were having errors when the puppet command was run manually on the vs. automatically?  Could you post these logs so we can take a look?

Comment 6

[Gregory Szorc [:gps]]

When you run `hg debuginstall`, does it say it is using Python 2.7.12?

If so, you can check if Python has modern TLS support by running the following code in a Python interactive REPL:

  $ python2.7
  >>> import ssl
  >>> print('\n'.join(k for k in dir(ssl) if k.startswith('PROTOCOL_')))
  PROTOCOL_SSLv23
  PROTOCOL_TLSv1
  PROTOCOL_TLSv1_1
  PROTOCOL_TLSv1_2

If you don't see the 1.1 and 1.2 symbols, Python wasn't compiled properly. (It was probably compiled against an ancient OpenSSL version.)

Comment 7

[Gregory Szorc [:gps]]

To state it explicitly, Mercurial needs to be built against Python 2.7.12. When you build Mercurial, it includes the absolute path of the Python it was built with in the shebang.

Depending on what OS we're talking about, I may already have a .rpm or .deb suitable to use in automation.

Comment 8

[Chris AtLee [:catlee]]

Yeah, it looks like the python2.7.12 rpm was built against an old openssl version.

[[e-mail] ~]$ hg debuginstall
checking encoding (UTF-8)...
checking Python executable (/tools/python27/bin/python2.7)
checking Python version (2.7.12)
checking Python lib (/tools/python27/lib/python2.7)...
checking Mercurial version (3.9.1)
checking Mercurial custom build ()
checking module policy (c)
checking installed modules (/tools/python27-mercurial/lib/python2.7/site-packages/mercurial)...
checking templates (/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/templates)...
checking default template (/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/templates/map-cmdline.default)
checking commit editor... (vi)
checking username (Mozilla Release Engineering <[e-mail]>)
no problems detected
[[e-mail] ~]$ /tools/python27/bin/python2.7
Python 2.7.12 (default, Oct  7 2016, 07:20:13)
[GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> print('\n'.join(k for k in dir(ssl) if k.startswith('PROTOCOL_')))
PROTOCOL_SSLv2
PROTOCOL_SSLv23
PROTOCOL_SSLv3
PROTOCOL_TLSv1

Comment 9

[Alin Selagea [:aselagea]]

That seems to be the case - I used rpmpackager1 to build the rpm. The openssl version is indeed old:

[[e-mail] python-2.7.12]$ openssl version
OpenSSL 1.0.1e-fips 11 Feb 2013

Comment 10

[Alin Selagea [:aselagea]]

(In reply to Kim Moir [:kmoir] from comment #5)
> Alin: You mentioned in the buildduty meeting today that you were having
> errors when the puppet command was run manually on the vs. automatically? 
> Could you post these logs so we can take a look?

That happens on buildduty-tools machine. The script that is run by the cronjob in /etc/cron.d/ looks like this:
"/usr/local/bin/proxy_reset_env.sh && /usr/bin/puppet agent --enable >/dev/null 2>/dev/null && /usr/bin/puppet agent --onetime --no-daemonize --logdest=syslog >/dev/null 2>&1"

If I manually run it (as root), an e-mail with the errors that you can see in the attachment will be sent. It looks like it tries to update some packages to the latest version but it fails to do so.

Comment 11

[Alin Selagea [:aselagea]]

Continued investigating this and found the following:

[[e-mail] ~]$ /usr/local/bin/python2.7 -c "import ssl; print ssl.OPENSSL_VERSION"
OpenSSL 1.0.0-fips 29 Mar 2010

So the openssl version referenced by python is even older that expected. The ssl module is imported from the following location:

>>> import ssl
>>> import os.path
>>> print os.path.abspath(ssl.__file__)
/tools/python27/lib/python2.7/ssl.pyc

Taking a look at ssl.py script, we can see that we extract the OPENSSL_VERSION parameter from a binary file:
=> "from _ssl import OPENSSL_VERSION_NUMBER, OPENSSL_VERSION_INFO, OPENSSL_VERSION"

>>> import _ssl
>>> import os.path
>>> print os.path.abspath(_ssl.__file__)
/tools/python27/lib/python2.7/lib-dynload/_ssl.so

I tried compiling the python rpm against "OpenSSL 1.0.1e-fips 11 Feb 2013" and even "OpenSSL 1.0.2j  26 Sep 2016", yet I still got the old openssl version being referenced by python.

Comment 12

[Alin Selagea [:aselagea]]

Deployed the rpm compiled against "OpenSSL 1.0.1e-fips 11 Feb 2013" (which supports TLSv1.1 and TLSv1.2) to the puppet masters, but had to manually re-install python on buildduty-tools and cruncher. Tried to clone the /tools repo => everything went smoothly.
 
Alerts coming for cruncher-aws have also stopped.

(allthethings)[[e-mail] allthethings]$ python -c "import ssl; print('\n'.join(k for k in dir(ssl) if k.startswith('PROTOCOL_')))"
PROTOCOL_SSLv2
PROTOCOL_SSLv23
PROTOCOL_SSLv3
PROTOCOL_TLSv1
PROTOCOL_TLSv1_1
PROTOCOL_TLSv1_2

The openssl version it now points is the same as the one it was compiled against.
(allthethings)[[e-mail] allthethings]$ python -c "import ssl; print ssl.OPENSSL_VERSION"
OpenSSL 1.0.1e-fips 11 Feb 2013