1071659 - [traceback] unable to escalate questions to zendesk (SSLHandshakeError)

Status: RESOLVED FIXED

(support.mozilla.org :: Questions, task, P1)

Description

[Ricky Rosario [:rrosario, :r1cky]]

Task kitsune.questions.tasks.escalate_question with id ea65f1e0-39ae-4696-bf44-dc429683ad7d raised exception:
"SSLHandshakeError(SSLError(1, '_ssl.c:492: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed'),)"


Task was called with args: (1021070L,) kwargs: {}.

The contents of the full traceback was:

Traceback (most recent call last):
  File "/data/www/support.mozilla.com/kitsune/virtualenv/lib/python2.6/site-packages/celery/task/trace.py", line 233, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/data/www/support.mozilla.com/kitsune/virtualenv/lib/python2.6/site-packages/celery/task/trace.py", line 420, in __protected_call__
    return self.run(*args, **kwargs)
  File "/data/www/support.mozilla.org/kitsune/kitsune/sumo/decorators.py", line 109, in _timeit
    result = f(*args, **kwargs)
  File "/data/www/support.mozilla.org/kitsune/kitsune/questions/tasks.py", line 195, in escalate_question
    tags=[t.slug for t in question.tags.all()])
  File "/data/www/support.mozilla.org/kitsune/kitsune/questions/marketplace.py", line 63, in submit_ticket
    ticket_url = zendesk.create_ticket(data=new_ticket)
  File "/data/www/support.mozilla.com/kitsune/virtualenv/lib/python2.6/site-packages/zendesk/zendesk.py", line 205, in call
    headers=self.headers
  File "/data/www/support.mozilla.com/kitsune/virtualenv/lib/python2.6/site-packages/httplib2/__init__.py", line 1570, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/data/www/support.mozilla.com/kitsune/virtualenv/lib/python2.6/site-packages/httplib2/__init__.py", line 1317, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/data/www/support.mozilla.com/kitsune/virtualenv/lib/python2.6/site-packages/httplib2/__init__.py", line 1252, in _conn_request
    conn.connect()
  File "/data/www/support.mozilla.com/kitsune/virtualenv/lib/python2.6/site-packages/httplib2/__init__.py", line 1044, in connect
    raise SSLHandshakeError(e)
SSLHandshakeError: [Errno 1] _ssl.c:492: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

Comment 1

[Ricky Rosario [:rrosario, :r1cky]]

This error started when we switched to peep. The code works locally for me.

We suspect it is something related to the compilation of pyopenssl.

Comment 2

[Michael Cooper [:mythmon]]

We are installing the same version of pyOpenSSL via peep that we were installing before. I'm not exactly sure how it was set up before. Ricky mentioned that we had a special version of the pyOpenSSL RPM that we built that was made for us. I tried to figure out what was different about that compared to the standard RHEL one, but my yum-fu was not strong enough to learn anything.

cturra: Is there anything special about the package pyOpenSSLmoz.x86_64 compared to pyOpenSSL.x86_64 besides the version bump from 0.10 to 0.13?

Comment 3

[Chris Turra [:cturra]]

(In reply to Mike Cooper [:mythmon] from comment #2)
> 
> cturra: Is there anything special about the package pyOpenSSLmoz.x86_64
> compared to pyOpenSSL.x86_64 besides the version bump from 0.10 to 0.13?

nothing that i am aware of. i created this package a while back with fpm from the following source:

http://pypi.python.org/packages/source/p/pyOpenSSL/pyOpenSSL-0.13.tar.gz#md5=767bca18a71178ca353dff9e10941929

Comment 4

[Michael Cooper [:mythmon]]

That is the exact same URL that we are installing from now. Additionally, I downloaded it and checked the md5sum matches what is in the URL, and that the peep hash of the downloaded file (which is sha256 encoded as base64) matches what we have in our requirements file.

In other words, I'm very sure that we downloaded the same in our peep installation as in the RPM.

My next idea was that they were linked different at install time. To test this, I compared the filenames output from the ldd command on all of the .so files found in pyOpenSSL both in our virtualenv, and on the system. That didn't reveal any differences. In particular, I did:


$ ldd /usr/lib64/python2.6/site-packages/OpenSSL/*.so | grep '=> /' | awk '{print $3}' | sort | uniq > system
$ ldd /data/support/src/support.mozilla.org/kitsune/virtualenv/lib/python2.6/site-packages/OpenSSL/*.so | grep '=> /' | awk '{print $3}' | sort | uniq > venv
$ diff system venv

No differences were found.

As far as I can tell, we have exactly the same pyOpenSSL in the virtualenv as we do on the system.

Comment 5

[Michael Cooper [:mythmon]]

PS: All of the above was done on supportadm.private.phx1.

Comment 6

[Michael Cooper [:mythmon]]

We think that the problem is actually a different version of httplib2. We had 0.6 in the vendor library before, but installed 0.8 with peep. 0.6 isn't available anymore. We upgraded it to 0.9, which seems to have fixed the problem.

PR: https://github.com/mozilla/kitsune/pull/2121
Commit: https://github.com/mozilla/kitsune/commit/91891c20bf32d8aa60f4f3e0e14a0a4332221bf5

I'll deploy this to prod. If it actually fixes the problem we can close this bug.

Comment 7

[Michael Cooper [:mythmon]]

I haven't seen any more errors like this.

Comment 8

[Ricky Rosario [:rrosario, :r1cky]]

making this 1pt.