1563261 - [traceback] "ConnectionError(("bad handshake: Error..." in /monitoring/heartbeat/

Status: RESOLVED FIXED

(Socorro :: Webapp, defect, P1)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

From sentry: https://sentry.prod.mozaws.net/operations/socorro-new-stage/issues/6005834/

Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]
  File "urllib3/contrib/pyopenssl.py", line 472, in wrap_socket
    cnx.do_handshake()
  File "OpenSSL/SSL.py", line 1915, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "OpenSSL/SSL.py", line 1647, in _raise_ssl_error
    _raise_current_error()
  File "OpenSSL/_util.py", line 54, in exception_from_error_queue
    raise exception_type(errors)

SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",)
  File "urllib3/connectionpool.py", line 603, in urlopen
    chunked=chunked)
  File "urllib3/connectionpool.py", line 344, in _make_request
    self._validate_conn(conn)
  File "urllib3/connectionpool.py", line 843, in _validate_conn
    conn.connect()
  File "urllib3/connection.py", line 370, in connect
    ssl_context=context)
  File "urllib3/util/ssl_.py", line 355, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "urllib3/contrib/pyopenssl.py", line 478, in wrap_socket
    raise ssl.SSLError('bad handshake: %r' % e)

SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",)
  File "elasticsearch/connection/http_urllib3.py", line 78, in perform_request
    response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
  File "urllib3/connectionpool.py", line 641, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "urllib3/util/retry.py", line 344, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "urllib3/packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "urllib3/connectionpool.py", line 603, in urlopen
    chunked=chunked)
  File "urllib3/connectionpool.py", line 344, in _make_request
    self._validate_conn(conn)
  File "urllib3/connectionpool.py", line 843, in _validate_conn
    conn.connect()
  File "urllib3/connection.py", line 370, in connect
    ssl_context=context)
  File "urllib3/util/ssl_.py", line 355, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "urllib3/contrib/pyopenssl.py", line 478, in wrap_socket
    raise ssl.SSLError('bad handshake: %r' % e)

SSLError: ConnectionError(("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",)) caused by: SSLError(("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",))
  File "django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "crashstats/crashstats/utils.py", line 212, in wrapper
    response = f(request, *args, **kw)
  File "crashstats/monitoring/views.py", line 119, in healthcheck
    return dockerflow_heartbeat(request)
  File "crashstats/crashstats/utils.py", line 212, in wrapper
    response = f(request, *args, **kw)
  File "crashstats/monitoring/views.py", line 102, in dockerflow_heartbeat
    es.info()  # will raise an error if there's a problem with the cluster
  File "elasticsearch/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "elasticsearch/client/__init__.py", line 201, in info
    _, data = self.transport.perform_request('GET', '/', params=params)
  File "elasticsearch/transport.py", line 307, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "elasticsearch/connection/http_urllib3.py", line 83, in perform_request
    raise SSLError('N/A', str(e), e)

This started happening after we upgraded urllib3 to 1.25.3 a couple of days ago.

This bug covers fixing this.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

The webapp builds the Elasticsearch connection differently than the crashstorage connection context:

https://github.com/mozilla-services/socorro/blob/051ff0fb812f8d3a36970c1d64b7411375484c23/webapp-django/crashstats/monitoring/views.py#L99-L101

vs.

https://github.com/mozilla-services/socorro/blob/051ff0fb812f8d3a36970c1d64b7411375484c23/socorro/external/es/connection_context.py#L109-L114

I think we should fix the webapp one to use RequestHttpConnection.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 4978: bug 1563261: fix /monitoring/heartbeat/ Elasticsearch connection

Comment 3

[John Whitlock [:jwhitlock]]

jwhitlock merged PR #4978: "bug 1563261: fix /monitoring/heartbeat/ Elasticsearch connection" in 25a05a9.

Comment 4

[John Whitlock [:jwhitlock]]

The fix worked on https://crash-stats.allizom.org/monitoring/healthcheck/. It will be deployed with 2019.07.03-2, so that https://crash-stats.mozilla.org/monitoring/healthcheck/ will work again.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

Both the stage and prod healthcheck endpoints work now, so I'm marking this as FIXED.