1547903 - Fix the python situation on mac workers

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Posix OS, defect)

Description

[Mike Hommey [:glandium]]

As per https://phabricator.services.mozilla.com/D28024:

The macos workers have two python 2.7 installed: one in /usr/bin, and
one in /usr/local/bin. For some reason, the one in /usr/local/bin is
broken wrt SSL.

Comment 1

[Kendall Libby [:fubar] (he/him)]

Dragos, can you fix this? I don't recall offhand why we have two versions installed, but it should be easy to either remove one or fix the SSL bits.

Comment 2

[Dragos Crisan [:dragrom]]

Looking on a worker, on /usr/bin we have:

[[e-mail] ~]# /usr/bin/python2.7 --version
Python 2.7.10

and on /usr/local/bin:

[[e-mail] ~]# /usr/local/bin/python2.7 --version
Python 2.7.3

The python from /usr/local/bin was installed by us from puppet:
[[e-mail] ~]# ls -l /usr/local/bin/python2.7
lrwxr-xr-x 1 root wheel 29 20 Dec 11:21 /usr/local/bin/python2.7 -> /tools/python27/bin/python2.7

and the python2.7 from /usr/bin was installed with OSX:

[[e-mail] ~]# ls -l /usr/bin/python2.7
lrwxr-xr-x 1 root wheel 75 8 Sep 2015 /usr/bin/python2.7 -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7

I tried to import ssl library on /usr/local/bin/python2.7 and it works without errors:

[[e-mail] ~]# /usr/local/bin/python2.7
Python 2.7.3 (default, Oct 14 2014, 13:07:22)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.54)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

import ssl
help(ssl)

glandium:Can you help me to reproduce the issue?

Comment 3

[Mike Hommey [:glandium]]

Try doing some https requests with urllib2.urlopen. Unfortunately, the tasks I had on try expired, so I don't have the logs anymore to tell what server failed. Presumably, though, it should be either s3 or hg.

Comment 4

[Mike Hommey [:glandium]]

Python 3.7 also has problems with https:
https://taskcluster-artifacts.net/QxiNhaOATMW3hDmGHbUBSg/0/public/logs/live_backing.log

Comment 5

[Dragos Crisan [:dragrom]]

Python 3.7.1 require openssl grater that 1.0.1. On OSX 10.10 we have now installed OpenSSL 0.9.8.

We need to upgrade OpenSSL to 1.0.2 or test python 3.7.1 on OSX 10.14 (Mojave)

Comment 6

[Mike Hommey [:glandium]]

(In reply to Dragos Crisan [:dragrom] from comment #5)

We need to upgrade OpenSSL to 1.0.2 or test python 3.7.1 on OSX 10.14 (Mojave)

ITYM we need to upgrade OpenSSL to 1.0.2 or downgrade python to 3.6 on the 10.10 workers.

Comment 7

[Dragos Crisan [:dragrom]]

Installed openssl 1.0.2l and built python 3.7.1 with ssl support:

[[e-mail] ~]# python3
Python 3.7.1 (default, May 23 2019, 13:23:22)
[Clang 6.0 (clang-600.0.54)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

import ssl

For the moment, all changes are made on staging worker and PR created. If everything will be OK, Monday morning I'll merge booth PR's

Comment 8

[Dragos Crisan [:dragrom]]

Attachment: Build and install python 3.7.1 with ssl support

Comment 10

[Dragos Crisan [:dragrom]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=19c31fb907fa673539e12aef8207c4cb9875d710&selectedJob=247981608

Comment 11

[Mike Hommey [:glandium]]

(In reply to Dragos Crisan [:dragrom] from comment #10)

https://treeherder.mozilla.org/#/jobs?repo=try&revision=19c31fb907fa673539e12aef8207c4cb9875d710&selectedJob=247981608

You want to apply at least bug 1534578 to make this worthwhile, and even then, you probably wouldn't trigger something that fails due to missing ssl without bug 1525373.
I created a manual task that uses python 3.7 on the gecko-t-osx1010-beta worker type, and it still failed:

Download failed: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1051)>

https://tools.taskcluster.net/groups/B_um3U0nSPWiFuVunM6eog/tasks/B_um3U0nSPWiFuVunM6eog/runs/0/logs/public%2Flogs%2Flive_backing.log

This is, however a different error from before, where it was:

Download failed: <urlopen error unknown url type: https>

https://tools.taskcluster.net/groups/UZXT3jrISruGZuo--vYTNA/tasks/UZXT3jrISruGZuo--vYTNA/runs/0/logs/public%2Flogs%2Flive_backing.log

So we're a step in the right direction.

Comment 12

[Mike Hommey [:glandium]]

See https://stackoverflow.com/a/53615877

Comment 13

[Dragos Crisan [:dragrom]]

Tomorrow I'll try this: https://bugs.python.org/issue28182

Comment 14

[Mike Hommey [:glandium]]

How did it go?

Comment 15

[Dragos Crisan [:dragrom]]

I locally reproduced the issue:

[[e-mail] python37]# python3
Python 3.7.1 (default, May 23 2019, 13:23:22)
[Clang 6.0 (clang-600.0.54)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

import urllib.request; urllib.request.urlopen("https://self-signed.badssl.com/")
Traceback (most recent call last):
File "/tools/python37/lib/python3.7/urllib/request.py", line 1317, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/tools/python37/lib/python3.7/http/client.py", line 1229, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/tools/python37/lib/python3.7/http/client.py", line 1275, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/tools/python37/lib/python3.7/http/client.py", line 1224, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/tools/python37/lib/python3.7/http/client.py", line 1016, in _send_output
self.send(msg)
File "/tools/python37/lib/python3.7/http/client.py", line 956, in send
self.connect()
File "/tools/python37/lib/python3.7/http/client.py", line 1392, in connect
server_hostname=server_hostname)
File "/tools/python37/lib/python3.7/ssl.py", line 412, in wrap_socket
session=session
File "/tools/python37/lib/python3.7/ssl.py", line 853, in _create
self.do_handshake()
File "/tools/python37/lib/python3.7/ssl.py", line 1117, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1051)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/tools/python37/lib/python3.7/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/tools/python37/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/tools/python37/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/tools/python37/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/tools/python37/lib/python3.7/urllib/request.py", line 1360, in https_open
context=self._context, check_hostname=self._check_hostname)
File "/tools/python37/lib/python3.7/urllib/request.py", line 1319, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1051)>

I have some issues to build python with the patch provided by python, but today I'll have a result

Comment 16

[Mike Hommey [:glandium]]

Note that the issue you linked to was fixed in python 3.7, so the patch would fail to apply because it's already applied...

Comment 17

[Dragos Crisan [:dragrom]]

installed certificates on python 3.7 on all staging workers gecko-t-osx-1010-beta

I retriggered https://tools.taskcluster.net/groups/B_um3U0nSPWiFuVunM6eog/tasks/Zl_XnTViRlSf4LcZ6janVQ/runs/0/logs/public%2Flogs%2Flive_backing.log. Looks like the ssl error is fixed, but the task failed.

Mike, can you have a look and run another python37 tasks?

Comment 18

[Mike Hommey [:glandium]]

The task failed because it was a manual task that doesn't provide its declared artifacts. It ran all it had to run successfully. You should be able to deploy safely on gecko-t-osx-1010.