Closed Bug 1510000 Opened 7 years ago Closed 7 years ago

CERTIFICATE_VERIFY_FAILED errors on stage from connections to Heroku Redis

Categories

(Tree Management :: Treeherder: Infrastructure, defect, P1)

Product:
Tree Management
Component:
Treeherder: Infrastructure
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: emorley, Assigned: emorley)

Details

Attempting to log into Treeherder stage fails with an HTTP 500 response from Treeherder's API. Checking New Relic there is: redis.exceptions:ConnectionError: Error 1 connecting to ec2-54-156-186-43.compute-1.amazonaws.com:6750. [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:726) (https://rpm.newrelic.com/accounts/677903/applications/14179733/filterable_errors?tw%5Bend%5D=1543263701&tw%5Bstart%5D=1543261901#/show/acf4c17e-f1b8-11e8-9c53-0242ac110008_0_7094/stack_trace?top_facet=transactionUiName&primary_facet=error.class&barchart=barchart) The Heroku Redis dashboard for that instance says "Available", however all the metrics charts are reading zero: https://data.heroku.com/datastores/afc93b4b-d6fb-4ba1-a3e6-d41c8b69f8d1 I'll file a Heroku support ticket.
Ah it's due to the Redis 3 upgrade - buried in the "other changes" section rather than the "breaking changes" one even though it is breaking: > ... > * ssl_cert_reqs now has a default value of 'required' by default. This > should make connecting to a remote Redis server over SSL more secure. This breaks us since Heroku Redis doesn't support TLS certificate verification (they are self-signed). See: https://github.com/andymccurdy/redis-py/blob/fdd40a58db6e6d6b6b929f52eab91c49be411a98/CHANGES#L79-L81 https://github.com/andymccurdy/redis-py/pull/1017 It looks like django-redis doesn't expose an option to override this (https://github.com/niwinz/django-redis/issues/353), so for now we should just roll back to redis-py 2.x.
Commit pushed to master at https://github.com/mozilla/treeherder https://github.com/mozilla/treeherder/commit/60f2dee13b80d1142099bdefbb7bdf847a771464 Revert "Update redis from 2.10.6 to 3.0.1 (#4296)" This reverts commit 3295915d57966baca24f2483d0978a7442f275d8, due to bug 1510000.
I've reverted the redis-py update, which has fixed the issue (comment 2 was me reverting django-redis accidentally instead due to copying the wrong SHA, which I've cleaned up on master).
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
I've filed another Heroku ticket requesting a status update on them stopping using self-signed certificates with Heroku Redis (last similar status request was on Sept 2017): https://help.heroku.com/tickets/667970
Issue worked around and redis-py updated in: https://github.com/mozilla/treeherder/pull/4310
You need to log in before you can comment on or make changes to this bug.