1334087 - Errors after resetting treeherder-prototype

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P1)

Description

[Ed Morley [:emorley]]

Bug 1331397 is becoming more involved, so breaking this out.

There were CloudAMQP authentication errors (looking at the logs these starting ~1 week ago - perhaps CloudAMQP tried to auto-rotate the password but the env var change didn't apply since the app was in a broken state due to DB reset?).

I've fixed this by re-rotating the CloudAMQP password.

However now I'm seeing errors of form:

[2017-01-26 12:56:24,562: ERROR/Worker-2] MemcachedError: error 3 from memcached_get(:1:error-summary-71424578): (0x5728df0) CONNECTION FAILURE(Connection reset by peer),  host: localhost:11211 -> libmemcached/io.cc:466 
Traceback (most recent call last): 
  File "/app/.heroku/python/lib/python2.7/site-packages/django_pylibmc/memcached.py", line 130, in get 
    return super(PyLibMCCache, self).get(key, default, version) 
  File "/app/.heroku/python/lib/python2.7/site-packages/django/core/cache/backends/memcached.py", line 76, in get 
    val = self._cache.get(key) 
  File "/app/.heroku/python/lib/python2.7/site-packages/newrelic-2.78.0.57/newrelic/api/datastore_trace.py", line 77, in _nr_datastore_trace_wrapper_ 
    return wrapped(*args, **kwargs) 
ConnectionError: error 3 from memcached_get(:1:error-summary-71424578): (0x5728df0) CONNECTION FAILURE(Connection reset by peer),  host: localhost:11211 -> libmemcached/io.cc:466

Comment 1

[Ed Morley [:emorley]]

I tried:
* restarting all dynos
* flushing the cache via the memcachier admin panel
* pressing the "switch clusters" button on the memcachier admin panel
* using "add new credentials" etc

Nothing worked, so I just deleted the Memcachier addon from the treeherder-prototype app and re-added it, which fixed the problem. (Since it's just a cache with minimal settings, nothing to really lose by deleting it.)

I've re-added the New Relic key via the Memcachier admin panel to make the new instance report to our New relic account like the old one did.

Everything appears to be working fine now.