If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

[traceback] unable to escalate questions to zendesk (SSLHandshakeError)

RESOLVED FIXED in 2014Q3

Status

support.mozilla.org
Questions
P1
normal
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: rrosario, Assigned: mythmon)

Tracking

unspecified
2014Q3

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: u=sumo-team c=questions p=1 s=2014.17)

(Reporter)

Description

3 years ago
Task kitsune.questions.tasks.escalate_question with id ea65f1e0-39ae-4696-bf44-dc429683ad7d raised exception:
"SSLHandshakeError(SSLError(1, '_ssl.c:492: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed'),)"


Task was called with args: (1021070L,) kwargs: {}.

The contents of the full traceback was:

Traceback (most recent call last):
  File "/data/www/support.mozilla.com/kitsune/virtualenv/lib/python2.6/site-packages/celery/task/trace.py", line 233, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/data/www/support.mozilla.com/kitsune/virtualenv/lib/python2.6/site-packages/celery/task/trace.py", line 420, in __protected_call__
    return self.run(*args, **kwargs)
  File "/data/www/support.mozilla.org/kitsune/kitsune/sumo/decorators.py", line 109, in _timeit
    result = f(*args, **kwargs)
  File "/data/www/support.mozilla.org/kitsune/kitsune/questions/tasks.py", line 195, in escalate_question
    tags=[t.slug for t in question.tags.all()])
  File "/data/www/support.mozilla.org/kitsune/kitsune/questions/marketplace.py", line 63, in submit_ticket
    ticket_url = zendesk.create_ticket(data=new_ticket)
  File "/data/www/support.mozilla.com/kitsune/virtualenv/lib/python2.6/site-packages/zendesk/zendesk.py", line 205, in call
    headers=self.headers
  File "/data/www/support.mozilla.com/kitsune/virtualenv/lib/python2.6/site-packages/httplib2/__init__.py", line 1570, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/data/www/support.mozilla.com/kitsune/virtualenv/lib/python2.6/site-packages/httplib2/__init__.py", line 1317, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/data/www/support.mozilla.com/kitsune/virtualenv/lib/python2.6/site-packages/httplib2/__init__.py", line 1252, in _conn_request
    conn.connect()
  File "/data/www/support.mozilla.com/kitsune/virtualenv/lib/python2.6/site-packages/httplib2/__init__.py", line 1044, in connect
    raise SSLHandshakeError(e)
SSLHandshakeError: [Errno 1] _ssl.c:492: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
(Reporter)

Comment 1

3 years ago
This error started when we switched to peep. The code works locally for me.

We suspect it is something related to the compilation of pyopenssl.
(Assignee)

Comment 2

3 years ago
We are installing the same version of pyOpenSSL via peep that we were installing before. I'm not exactly sure how it was set up before. Ricky mentioned that we had a special version of the pyOpenSSL RPM that we built that was made for us. I tried to figure out what was different about that compared to the standard RHEL one, but my yum-fu was not strong enough to learn anything.

cturra: Is there anything special about the package pyOpenSSLmoz.x86_64 compared to pyOpenSSL.x86_64 besides the version bump from 0.10 to 0.13?
Assignee: nobody → mcooper
Flags: needinfo?(cturra)
(In reply to Mike Cooper [:mythmon] from comment #2)
> 
> cturra: Is there anything special about the package pyOpenSSLmoz.x86_64
> compared to pyOpenSSL.x86_64 besides the version bump from 0.10 to 0.13?

nothing that i am aware of. i created this package a while back with fpm from the following source:

http://pypi.python.org/packages/source/p/pyOpenSSL/pyOpenSSL-0.13.tar.gz#md5=767bca18a71178ca353dff9e10941929
Flags: needinfo?(cturra)
(Assignee)

Comment 4

3 years ago
That is the exact same URL that we are installing from now. Additionally, I downloaded it and checked the md5sum matches what is in the URL, and that the peep hash of the downloaded file (which is sha256 encoded as base64) matches what we have in our requirements file.

In other words, I'm very sure that we downloaded the same in our peep installation as in the RPM.

My next idea was that they were linked different at install time. To test this, I compared the filenames output from the ldd command on all of the .so files found in pyOpenSSL both in our virtualenv, and on the system. That didn't reveal any differences. In particular, I did:


$ ldd /usr/lib64/python2.6/site-packages/OpenSSL/*.so | grep '=> /' | awk '{print $3}' | sort | uniq > system
$ ldd /data/support/src/support.mozilla.org/kitsune/virtualenv/lib/python2.6/site-packages/OpenSSL/*.so | grep '=> /' | awk '{print $3}' | sort | uniq > venv
$ diff system venv

No differences were found.

As far as I can tell, we have exactly the same pyOpenSSL in the virtualenv as we do on the system.
(Assignee)

Comment 5

3 years ago
PS: All of the above was done on supportadm.private.phx1.
(Assignee)

Comment 6

3 years ago
We think that the problem is actually a different version of httplib2. We had 0.6 in the vendor library before, but installed 0.8 with peep. 0.6 isn't available anymore. We upgraded it to 0.9, which seems to have fixed the problem.

PR: https://github.com/mozilla/kitsune/pull/2121
Commit: https://github.com/mozilla/kitsune/commit/91891c20bf32d8aa60f4f3e0e14a0a4332221bf5

I'll deploy this to prod. If it actually fixes the problem we can close this bug.
(Assignee)

Comment 7

3 years ago
I haven't seen any more errors like this.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
(Reporter)

Comment 8

3 years ago
making this 1pt.
Whiteboard: u=sumo-team c=questions p= s=2014.17 → u=sumo-team c=questions p=1 s=2014.17
You need to log in before you can comment on or make changes to this bug.