1382229 - Elasticsearch instance failures should not block collectstatic or app start

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P3)

Description

[Ed Morley [:emorley]]

If ELASTICSEARCH_URL is set, Django tries to connect to the Elasticsearch instance immediately at startup - even for unrelated commands like collectstatic. 

In bug 1378361 this meant ES cluster issues blocked the app deploy (which runs collectstatic) - even though the compile phase is not supposed to touch external resources (that's what release phase is for).

Before we consider adding the Elasticsearch addon back to prototype/stage/prod in bug 1340552 we must make sure the connection only occurs on demand, like for the Django MySQL parts.

STR:
1) vagrant up && vagrant ssh
2) sudo service elasticsearch stop
3) ./manage.py collectstatic

Expected:
Collectstatic works

Actual:

  File "/home/vagrant/treeherder/treeherder/model/models.py", line 27, in <module>
    from .search import (TestFailureLine,
  File "/home/vagrant/treeherder/treeherder/model/search.py", line 204, in <module>
    connection = _init()
  File "/home/vagrant/treeherder/treeherder/model/search.py", line 194, in _init
    indices = connection.indices.get("*")
  File "/home/vagrant/python/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 73, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/home/vagrant/python/lib/python2.7/site-packages/elasticsearch/client/indices.py", line 135, in get
    feature), params=params)
  File "/home/vagrant/python/lib/python2.7/site-packages/elasticsearch/transport.py", line 312, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/home/vagrant/python/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 123, in perform_request
    raise ConnectionError('N/A', str(e), e)
elasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7fc3abae89d0>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7fc3abae89d0>: Failed to establish a new connection: [Errno 111] Connection refused)

Comment 1

[George Hickman]

This got fixed when we moved to switched to using the lower level ES library in https://github.com/mozilla/treeherder/pull/3472