1397726 - Memcachier stunnel TLS certificate has expired causing all memcache requests to fail

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P1)

Description

[Ed Morley [:emorley]]

In bug 1397716 MySQL CPU usage is off the charts due to memcached not working. Using the debugging method I came up with in [1], the logs show:

2017.09.07 12:40:19 LOG6[0]: SNI: sending servername: XXXXX.XXXXX.us-east-5.heroku.prod.memcachier.com
2017.09.07 12:40:19 LOG6[0]: Certificate accepted at depth=1: C=US, ST=CA, O=MemCachier Inc., CN=MemCachier Root CA
2017.09.07 12:40:19 LOG4[0]: CERT: Pre-verification error: certificate has expired

This means their TLS certificate has expired. Again! (Previous time in [2])

I've now filed:
https://github.com/memcachier/memcachier-tls-buildpack/issues/16

Plus opened this support ticket:
https://memcachier.zendesk.com/hc/en-us/requests/2038

This really isn't ideal. I'm even more inclined now to just ditch memcached (given the crappy C libary/python packages and other bugs) and move to redis (bug 1384518).

[1] https://github.com/memcachier/memcachier-tls-buildpack/issues/12
[2] https://github.com/memcachier/memcachier-tls-buildpack/issues/10

Comment 1

[Ed Morley [:emorley]]

Their initial reply earlier:

"""
Many apologies about this. We'll definitely put monitoring in place once we've renewed the certificates. I've messaged Amit and David to get the CA certificate to generate new SSL certificates. I'll let you know as soon as it's done.
"""

Followed by just now:

"""
I've just restarted the proxy servers in all the clusters where I think Mozilla has caches, and they'll have picked up new certificates. Can you take a look to see if things are working again? If not, could you let me know the cache IDs you're having trouble with and I'll make sure that the clusters for those caches are good.

Sorry for the disruption. I've added setting up some monitoring for this to my to-do list...
"""

Looks good now, eg:

$ openssl s_client -showcerts -connect XXXXX.XXXXX.us-east-5.heroku.prod.memcachier.com:11219 < /dev/null 2> /dev/null | openssl x509 -noout -enddate
notAfter=Sep  7 13:46:38 2018 GMT

I've restarted all three Heroku apps and verified the error messages are no longer present in Papertrail. 

I'll close this out and leave bug 1397716 open to track various improvements we can make to reduce impact.