1457996 - upgrade signing servers to latest python version

Status: RESOLVED FIXED

(Release Engineering :: General, enhancement)

Description

[[e-mail] (:bhearsum)]

Signing servers are currently running Python 2.7.3 and we need to upgrade them to the latest version.

I did a test of this on a not-yet-in-production signing server, and it seems to break the clients. For example, I get this error when I try to get a token:
Traceback (most recent call last):
  File "get_token.py", line 26, in <module>
    options.password, options.slave_ip, options.duration))
  File "/home/bhearsum/tools/lib/python/signing/client.py", line 41, in get_token
    return urllib2.urlopen(r).read()
  File "/tools/python27/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/tools/python27/lib/python2.7/urllib2.py", line 400, in open
    response = self._open(req, data)
  File "/tools/python27/lib/python2.7/urllib2.py", line 418, in _open
    '_open', req)
  File "/tools/python27/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/home/bhearsum/tools/lib/python/signing/client.py", line 251, in https_open
    return self.do_open(self.specialized_conn_class, req)
  File "/tools/python27/lib/python2.7/urllib2.py", line 1177, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 8] _ssl.c:504: EOF occurred in violation of protocol>


If I adjust the code to stop forcing TLSv1, I get a different error:
Traceback (most recent call last):
  File "get_token.py", line 26, in <module>
    options.password, options.slave_ip, options.duration))
  File "/home/bhearsum/tools/lib/python/signing/client.py", line 41, in get_token
    return urllib2.urlopen(r).read()
  File "/tools/python27/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/tools/python27/lib/python2.7/urllib2.py", line 400, in open
    response = self._open(req, data)
  File "/tools/python27/lib/python2.7/urllib2.py", line 418, in _open
    '_open', req)
  File "/tools/python27/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/home/bhearsum/tools/lib/python/signing/client.py", line 251, in https_open
    return self.do_open(self.specialized_conn_class, req)
  File "/tools/python27/lib/python2.7/urllib2.py", line 1177, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 104] Connection reset by peer>


I suspect we'll need some code changes before we can do this upgrade.

Comment 1

[[e-mail] (:bhearsum)]

It appears that the upgraded signing servers don't have any available ciphers, according to the "sslscan" tool:
$ ./sslscan signing13.srv.releng.mdc1.mozilla.com:9110
Version: 1.11.11-rbsec-15-g8fee8a3-static
OpenSSL 1.0.2-chacha (1.0.2g-dev)

Connected to 10.49.48.30

Testing SSL server signing13.srv.releng.mdc1.mozilla.com on port 9110 using SNI name signing13.srv.releng.mdc1.mozilla.com

  TLS Fallback SCSV:
Connection failed - unable to determine TLS Fallback SCSV support

  TLS renegotiation:
Session renegotiation not supported

  TLS Compression:
Compression disabled

  Heartbleed:
TLS 1.2 not vulnerable to heartbleed
TLS 1.1 not vulnerable to heartbleed
TLS 1.0 not vulnerable to heartbleed

  Supported Server Cipher(s):


vs. not upgraded:

$ ./sslscan signing13.srv.releng.mdc1.mozilla.com:9110
Version: 1.11.11-rbsec-15-g8fee8a3-static
OpenSSL 1.0.2-chacha (1.0.2g-dev)

Connected to 10.49.48.30

Testing SSL server signing13.srv.releng.mdc1.mozilla.com on port 9110 using SNI name signing13.srv.releng.mdc1.mozilla.com

  TLS Fallback SCSV:
Server supports TLS Fallback SCSV

  TLS renegotiation:
Secure session renegotiation supported

  TLS Compression:
Compression disabled

  Heartbleed:
TLS 1.2 not vulnerable to heartbleed
TLS 1.1 not vulnerable to heartbleed
TLS 1.0 not vulnerable to heartbleed

  Supported Server Cipher(s):
Preferred TLSv1.2  256 bits  AES256-GCM-SHA384            
Accepted  TLSv1.2  256 bits  AES256-SHA256                
Accepted  TLSv1.2  256 bits  AES256-SHA                   
Accepted  TLSv1.2  256 bits  CAMELLIA256-SHA              
Accepted  TLSv1.2  128 bits  AES128-GCM-SHA256            
Accepted  TLSv1.2  128 bits  AES128-SHA256                
Accepted  TLSv1.2  128 bits  AES128-SHA                   
Accepted  TLSv1.2  128 bits  SEED-SHA                     
Accepted  TLSv1.2  128 bits  CAMELLIA128-SHA              
Accepted  TLSv1.2  128 bits  IDEA-CBC-SHA                 
Accepted  TLSv1.2  128 bits  RC4-SHA                      
Accepted  TLSv1.2  128 bits  RC4-MD5                      
Accepted  TLSv1.2  112 bits  DES-CBC3-SHA                 
Preferred TLSv1.1  256 bits  AES256-SHA                   
Accepted  TLSv1.1  256 bits  CAMELLIA256-SHA              
Accepted  TLSv1.1  128 bits  AES128-SHA                   
Accepted  TLSv1.1  128 bits  SEED-SHA                     
Accepted  TLSv1.1  128 bits  CAMELLIA128-SHA              
Accepted  TLSv1.1  128 bits  IDEA-CBC-SHA                 
Accepted  TLSv1.1  128 bits  RC4-SHA                      
Accepted  TLSv1.1  128 bits  RC4-MD5                      
Accepted  TLSv1.1  112 bits  DES-CBC3-SHA                 
Preferred TLSv1.0  256 bits  AES256-SHA                   
Accepted  TLSv1.0  256 bits  CAMELLIA256-SHA              
Accepted  TLSv1.0  128 bits  AES128-SHA                   
Accepted  TLSv1.0  128 bits  SEED-SHA                     
Accepted  TLSv1.0  128 bits  CAMELLIA128-SHA              
Accepted  TLSv1.0  128 bits  IDEA-CBC-SHA                 
Accepted  TLSv1.0  128 bits  RC4-SHA                      
Accepted  TLSv1.0  128 bits  RC4-MD5                      
Accepted  TLSv1.0  112 bits  DES-CBC3-SHA                 
Preferred SSLv3    256 bits  AES256-SHA                   
Accepted  SSLv3    256 bits  CAMELLIA256-SHA              
Accepted  SSLv3    128 bits  AES128-SHA                   
Accepted  SSLv3    128 bits  SEED-SHA                     
Accepted  SSLv3    128 bits  CAMELLIA128-SHA              
Accepted  SSLv3    128 bits  IDEA-CBC-SHA                 
Accepted  SSLv3    128 bits  RC4-SHA                      
Accepted  SSLv3    128 bits  RC4-MD5                      
Accepted  SSLv3    112 bits  DES-CBC3-SHA                 

  SSL Certificate:
Signature Algorithm: sha256WithRSAEncryption
RSA Key Strength:    2048

Subject:  signing4.srv.releng.scl3.mozilla.com
Altnames: DNS:signing4.srv.releng.scl3.mozilla.com, DNS:signing5.srv.releng.scl3.mozilla.com, DNS:signing6.srv.releng.scl3.mozilla.com, DNS:mac-v2-signing1.srv.releng.scl3.mozilla.com, DNS:mac-v2-signing2.srv.releng.scl3.mozilla.com, DNS:mac-v2-signing3.srv.releng.scl3.mozilla.com, DNS:mac-v2-signing4.srv.releng.scl3.mozilla.com, DNS:mac-v2-signing6.srv.releng.scl3.mozilla.com, DNS:mac-v2-signing7.srv.releng.scl3.mozilla.com
Issuer:   signing4.srv.releng.scl3.mozilla.com

Not valid before: Aug 30 19:09:00 2017 GMT
Not valid after:  Aug 30 19:09:00 2018 GMT


---

I've been fiddling around ssl parameters in https://hg.mozilla.org/build/tools/file/tip/lib/python/signing/server.py#l714, changing both ssl_version and ciphers -- nothing seems to make a difference. My suspicion is that this has something to do with the SSLv2 deprecation (https://bugs.python.org/issue20207) that happened between 2.7.3 and 2.7.14, but it's hard to be sure. Need to dive into this more still.

Comment 2

[[e-mail] (:bhearsum)]

I did a bit more debugging today and couldn't find a way to make the signing server work with Python 2.7.14 and the current gevent version. However, after upgrading gevent and greenlet the SSL issues disappeared. Since we want to upgrade dependencies anyways, I think it would be best to just upgrade gevent and greenlet before python rather than trying to fuss with older versions and a newer Python.

I tried out the latest gevent+greenlet with both 2.7.3 and 2.7.14, and other than a minor issue with a header and some double logging, it seems to work fine -- I'll have a patch to fix these in a bit.

So, the plan of action here is:
1) Land fixes needed for gevent upgrade
2) Upgrade gevent and greenlet
3) Upgrade Python

Comment 3

[[e-mail] (:bhearsum)]

Attachment: fix header issue and duplicate logging

Newer versions of gevent require headers to be strings, not ints -- I also added general exception logging for do_GET so the next person doing debugging doesn't spend hours trying to find an actual error :(.

The logging part is a bit weird, as described in the comment. There's probably a better fix, but not worth pursuing at this time IMO.

Comment 4

[[e-mail] (:bhearsum)]

Attachment: upgrade gevent and greenlet

I tested this patch on Linux -- we don't have a viable place to test on Mac. When this lands, I plan to pin an instance of each platform to my environment to use as canaries.

Comment 5

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum (:bhearsum) from comment #2)
> So, the plan of action here is:
> 1) Land fixes needed for gevent upgrade
> 2) Upgrade gevent and greenlet
> 3) Upgrade Python

I'm not 100% sure when this will happen. Maybe next week, if releases are going smoothly, but I might also wait until the following week to ensure we're really out of the way of 60.0...

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/build/puppet/rev/b792d08aa6ec
upgrade gevent and greenlet on signing servers. r=catlee

Comment 7

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum (:bhearsum) from comment #2)
> So, the plan of action here is:
> 1) Land fixes needed for gevent upgrade
> 2) Upgrade gevent and greenlet

These two steps are done now! Everything seems okay so far.

Comment 8

[[e-mail] (:bhearsum)]

This is done, except for signing6 (which is pinned to jlorenzo's env), and the mdc1/mdc2 servers (which aren't in production, and I consider them out of scope) by way of https://bugzilla.mozilla.org/attachment.cgi?id=8974775&action=edit.

Johan, can you close this bug out once signing6 has been restarted with Python 2.7.15?

Comment 9

[[e-mail] (:bhearsum)]

I'm slightly concerned about mac-v2-signing1, which appears to have shut down its nightly signing server on its own:
2018-05-14 19:08:46,902 - DEBUG - Pending: {}
2018-05-14 19:08:46,902 - DEBUG - Cleaning up...
2018-05-14 19:11:31,055 - INFO - pid 2947 exiting normally
2018-05-14 19:11:31,056 - INFO - exiting

There was nobody logged onto the machine when this happened, nor did Puppet do anything that was logged. We should watch out for this on Linux & Mac signing servers in case it's an issue with the new Gevent and/or Python.

Comment 10

[Johan Lorenzo [:jlorenzo]]

(In reply to Ben Hearsum (:bhearsum) from comment #8)
> Johan, can you close this bug out once signing6 has been restarted with
> Python 2.7.15?

Discussed over IRC. This is done now. I rebased my puppet env on top of the latest build/puppet. I did the same for build/tools. 

This was tested to be working on https://tools.taskcluster.net/groups/F95X-e9JTVCyMIuLKOwLVA/tasks/cj7DKkvCRcmbTg9KSOsXpg/runs/10