Certificate validation failure when connecting to amqp://USER:PASSWORD@pulse.mozilla.org :5671/?ssl=1
Categories
(Webtools :: Pulse, defect)
Tracking
(Not tracked)
People
(Reporter: marco, Unassigned)
Details
Jan 26 10:33:59 bugbug app/web.1: 2021-01-26 10:33:58,860:WARNING:kombu.mixins:Broker connection error, trying again in 32.0 seconds: SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1124)').
Jan 26 10:33:59 bugbug app/web.1: Traceback (most recent call last):
Jan 26 10:33:59 bugbug app/web.1: File "/usr/local/lib/python3.8/site-packages/kombu/utils/functional.py", line 325, in retry_over_time
Jan 26 10:33:59 bugbug app/web.1: return fun(*args, **kwargs)
Jan 26 10:33:59 bugbug app/web.1: File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 866, in _connection_factory
Jan 26 10:33:59 bugbug app/web.1: self._connection = self._establish_connection()
Jan 26 10:33:59 bugbug app/web.1: File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 801, in _establish_connection
Jan 26 10:33:59 bugbug app/web.1: conn = self.transport.establish_connection()
Jan 26 10:33:59 bugbug app/web.1: File "/usr/local/lib/python3.8/site-packages/kombu/transport/pyamqp.py", line 128, in establish_connection
Jan 26 10:33:59 bugbug app/web.1: conn.connect()
Jan 26 10:33:59 bugbug app/web.1: File "/usr/local/lib/python3.8/site-packages/amqp/connection.py", line 323, in connect
Jan 26 10:33:59 bugbug app/web.1: self.transport.connect()
Jan 26 10:33:59 bugbug app/web.1: File "/usr/local/lib/python3.8/site-packages/amqp/transport.py", line 114, in connect
Jan 26 10:33:59 bugbug app/web.1: self._init_socket(
Jan 26 10:33:59 bugbug app/web.1: File "/usr/local/lib/python3.8/site-packages/amqp/transport.py", line 225, in _init_socket
Jan 26 10:33:59 bugbug app/web.1: self._setup_transport()
Jan 26 10:33:59 bugbug app/web.1: File "/usr/local/lib/python3.8/site-packages/amqp/transport.py", line 405, in _setup_transport
Jan 26 10:33:59 bugbug app/web.1: self.sock.do_handshake()
Jan 26 10:33:59 bugbug app/web.1: File "/usr/local/lib/python3.8/ssl.py", line 1309, in do_handshake
Jan 26 10:33:59 bugbug app/web.1: self._sslobj.do_handshake()
Jan 26 10:33:59 bugbug app/web.1: ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1124)
Comment 1•3 years ago
|
||
I'm still able to connect here, and the cert appears to remain valid:
Validity
Not Before: Mar 2 00:00:00 2020 GMT
Not After : Mar 24 12:00:00 2021 GMT
nothing else on that cert has changed.
The issuer:
Issuer: C = US, O = DigiCert Inc, CN = DigiCert SHA2 Secure Server CA
is valid until March 2023. Is this maybe a client-side issue, using an old or empty set of CA certs?
Reporter | ||
Comment 2•3 years ago
|
||
(In reply to Dustin J. Mitchell [:dustin] (he/him) from comment #1)
I'm still able to connect here, and the cert appears to remain valid:
Validity Not Before: Mar 2 00:00:00 2020 GMT Not After : Mar 24 12:00:00 2021 GMT
nothing else on that cert has changed.
The issuer:
Issuer: C = US, O = DigiCert Inc, CN = DigiCert SHA2 Secure Server CA
is valid until March 2023. Is this maybe a client-side issue, using an old or empty set of CA certs?
I'm hitting this both on Heroku and locally (not in a Docker container), so this seems unlikely.
STR:
git clone git@github.com:mozilla/bugbug.git
python -m pip install -r http_service/requirements.txt
cd http_service
export PULSE_USER=your_username
export PULSE_PASSWORD=your_password
export BUGBUG_HTTP_SERVER=https://bugbug.herokuapp.com
PYTHONPATH=.. python -m bugbug_http.listener
Comment 3•3 years ago
|
||
This seems to work fine:
import socket
import certifi
import ssl
hostname = 'pulse.mozilla.org'
context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
context.load_verify_locations(certifi.where())
with socket.create_connection((hostname, 5671)) as sock:
with context.wrap_socket(sock, server_hostname=hostname) as ssock:
print(ssock.version())
but removing that load_verify_locations
gives the same error as above.
Comment 4•3 years ago
|
||
There was a deploy about 15h ago, and errors began about 12h ago on a Heroku dyno restart. So if this was a remote change, it occurred from 12-36h ago. I'll file an issue with CloudAMQP.
Comment 5•3 years ago
|
||
update: it looks like the previous Heroku slug works, so this is likely something on the client side -- so no issue filed (yet)
Comment 6•3 years ago
|
||
It looks like this is https://github.com/celery/py-amqp/issues/349 so I think downgrading to the previous amqp release will fix it.
Reporter | ||
Comment 7•3 years ago
|
||
Description
•