Closed Bug 1688840 Opened 3 years ago Closed 3 years ago

Certificate validation failure when connecting to amqp://USER:PASSWORD@pulse.mozilla.org:5671/?ssl=1

Categories

(Webtools :: Pulse, defect)

Product:
Webtools
Component:
Pulse
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: marco, Unassigned)

Details

Jan 26 10:33:59 bugbug app/web.1: 2021-01-26 10:33:58,860:WARNING:kombu.mixins:Broker connection error, trying again in 32.0 seconds: SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1124)'). 
Jan 26 10:33:59 bugbug app/web.1: Traceback (most recent call last): 
Jan 26 10:33:59 bugbug app/web.1:   File "/usr/local/lib/python3.8/site-packages/kombu/utils/functional.py", line 325, in retry_over_time 
Jan 26 10:33:59 bugbug app/web.1:     return fun(*args, **kwargs) 
Jan 26 10:33:59 bugbug app/web.1:   File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 866, in _connection_factory 
Jan 26 10:33:59 bugbug app/web.1:     self._connection = self._establish_connection() 
Jan 26 10:33:59 bugbug app/web.1:   File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 801, in _establish_connection 
Jan 26 10:33:59 bugbug app/web.1:     conn = self.transport.establish_connection() 
Jan 26 10:33:59 bugbug app/web.1:   File "/usr/local/lib/python3.8/site-packages/kombu/transport/pyamqp.py", line 128, in establish_connection 
Jan 26 10:33:59 bugbug app/web.1:     conn.connect() 
Jan 26 10:33:59 bugbug app/web.1:   File "/usr/local/lib/python3.8/site-packages/amqp/connection.py", line 323, in connect 
Jan 26 10:33:59 bugbug app/web.1:     self.transport.connect() 
Jan 26 10:33:59 bugbug app/web.1:   File "/usr/local/lib/python3.8/site-packages/amqp/transport.py", line 114, in connect 
Jan 26 10:33:59 bugbug app/web.1:     self._init_socket( 
Jan 26 10:33:59 bugbug app/web.1:   File "/usr/local/lib/python3.8/site-packages/amqp/transport.py", line 225, in _init_socket 
Jan 26 10:33:59 bugbug app/web.1:     self._setup_transport() 
Jan 26 10:33:59 bugbug app/web.1:   File "/usr/local/lib/python3.8/site-packages/amqp/transport.py", line 405, in _setup_transport 
Jan 26 10:33:59 bugbug app/web.1:     self.sock.do_handshake() 
Jan 26 10:33:59 bugbug app/web.1:   File "/usr/local/lib/python3.8/ssl.py", line 1309, in do_handshake 
Jan 26 10:33:59 bugbug app/web.1:     self._sslobj.do_handshake() 
Jan 26 10:33:59 bugbug app/web.1: ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1124)

I'm still able to connect here, and the cert appears to remain valid:

        Validity                                                                                        
            Not Before: Mar  2 00:00:00 2020 GMT                                                        
            Not After : Mar 24 12:00:00 2021 GMT

nothing else on that cert has changed.

The issuer:

        Issuer: C = US, O = DigiCert Inc, CN = DigiCert SHA2 Secure Server CA

is valid until March 2023. Is this maybe a client-side issue, using an old or empty set of CA certs?

(In reply to Dustin J. Mitchell [:dustin] (he/him) from comment #1)

I'm still able to connect here, and the cert appears to remain valid:

        Validity                                                                                        
            Not Before: Mar  2 00:00:00 2020 GMT                                                        
            Not After : Mar 24 12:00:00 2021 GMT

nothing else on that cert has changed.

The issuer:

        Issuer: C = US, O = DigiCert Inc, CN = DigiCert SHA2 Secure Server CA

is valid until March 2023. Is this maybe a client-side issue, using an old or empty set of CA certs?

I'm hitting this both on Heroku and locally (not in a Docker container), so this seems unlikely.

STR:

  1. git clone git@github.com:mozilla/bugbug.git
  2. python -m pip install -r http_service/requirements.txt
  3. cd http_service
  4. export PULSE_USER=your_username
  5. export PULSE_PASSWORD=your_password
  6. export BUGBUG_HTTP_SERVER=https://bugbug.herokuapp.com
  7. PYTHONPATH=.. python -m bugbug_http.listener

This seems to work fine:

import socket
import certifi
import ssl 

hostname = 'pulse.mozilla.org'
context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
context.load_verify_locations(certifi.where())                                                          

with socket.create_connection((hostname, 5671)) as sock:
    with context.wrap_socket(sock, server_hostname=hostname) as ssock:
        print(ssock.version())

but removing that load_verify_locations gives the same error as above.

There was a deploy about 15h ago, and errors began about 12h ago on a Heroku dyno restart. So if this was a remote change, it occurred from 12-36h ago. I'll file an issue with CloudAMQP.

Assignee: nobody → dustin

update: it looks like the previous Heroku slug works, so this is likely something on the client side -- so no issue filed (yet)

It looks like this is https://github.com/celery/py-amqp/issues/349 so I think downgrading to the previous amqp release will fix it.

Assignee: dustin → nobody

Fixed by https://github.com/mozilla/bugbug/commit/918561e9e7eb8d11180632e8ba121e20ec9b9ce4.

Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.