1202966 - OS X 10.10.5 python (2.7.10) fails to verify puppet certs

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Amy Rich [:arr] [:arich]]

The puppetization process no longer works with OS X 10.10.5 since python has been upgraded to 2.7.10. When it attempts to download the certs, it dumps the following stack trace:

Traceback (most recent call last):
  File "<stdin>", line 11, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1240, in https_open
    context=self._context)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1197, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 8] nodename nor servname provided, or not known>
Failed to get certificates; re-trying after delay


This might be indicative of the tightening of verification in 2.7.9 and above: https://stackoverflow.com/questions/27804710/python-urllib2-ssl-error/27826829#27826829

Comment 1

[Amy Rich [:arr] [:arich]]

Go ahead and use t-yosemite-r7-0004 to debug.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

I'm getting

Contacting puppet server puppet
Traceback (most recent call last):
  File "<stdin>", line 11, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1240, in https_open
    context=self._context)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1197, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>

I think the error is that it's attempting to verify the certificate at all -- we don't have the root cert on the host (this is how we *get* the root cert).  Previous versions of Python didn't validate by default, but 2.7.9 and above do (http://legacy.python.org/dev/peps/pep-0476/).

This will re-create the original (non-verifying) behavior:

context = ssl._create_unverified_context()
sslhandler = urllib2.HTTPSHandler(context=context)
opener = urllib2.build_opener(sslhandler, handler)

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

The better option is to bake the root cert into the script so that we can actually verify we're talking to the puppetmaster.  This isn't *strictly* necessary -- if we're MITM'd, then the attacker is impersonating the master, and thus doesn't have access to any of the interesting data on the master.  In other words, a MITM gets you about as much access as a try job.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: MozReview Request: Bug 1202966: skip SSL verification on Python-2.7.9+; r?arr

Bug 1202966: skip SSL verification on Python-2.7.9+; r?arr

Comment 5

[Amy Rich [:arr] [:arich]]

Comment on attachment 8658687 [details]
MozReview Request: Bug 1202966: skip SSL verification on Python-2.7.9+; r?arr

We'll need to make this same change on windows, since that's not handled through the puppetize.sh script.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

remote:   https://hg.mozilla.org/build/puppet/rev/3619c261dc9b
remote:   https://hg.mozilla.org/build/puppet/rev/5de5a7e870f5

This needs to be deployed manually to deploystudio, but will be automatically deployed for linux re-images.  Windows uses VBS, so no change there.

We can open a new bug to verify the cert, but I don't consider it a high priority.