Elasticsearch instance failures should not block collectstatic or app start

RESOLVED FIXED

Status

P3
normal
RESOLVED FIXED
a year ago
8 months ago

People

(Reporter: emorley, Assigned: ghickman)

Tracking

(Blocks: 2 bugs)

Details

If ELASTICSEARCH_URL is set, Django tries to connect to the Elasticsearch instance immediately at startup - even for unrelated commands like collectstatic. 

In bug 1378361 this meant ES cluster issues blocked the app deploy (which runs collectstatic) - even though the compile phase is not supposed to touch external resources (that's what release phase is for).

Before we consider adding the Elasticsearch addon back to prototype/stage/prod in bug 1340552 we must make sure the connection only occurs on demand, like for the Django MySQL parts.

STR:
1) vagrant up && vagrant ssh
2) sudo service elasticsearch stop
3) ./manage.py collectstatic

Expected:
Collectstatic works

Actual:

  File "/home/vagrant/treeherder/treeherder/model/models.py", line 27, in <module>
    from .search import (TestFailureLine,
  File "/home/vagrant/treeherder/treeherder/model/search.py", line 204, in <module>
    connection = _init()
  File "/home/vagrant/treeherder/treeherder/model/search.py", line 194, in _init
    indices = connection.indices.get("*")
  File "/home/vagrant/python/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 73, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/home/vagrant/python/lib/python2.7/site-packages/elasticsearch/client/indices.py", line 135, in get
    feature), params=params)
  File "/home/vagrant/python/lib/python2.7/site-packages/elasticsearch/transport.py", line 312, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/home/vagrant/python/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 123, in perform_request
    raise ConnectionError('N/A', str(e), e)
elasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7fc3abae89d0>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7fc3abae89d0>: Failed to establish a new connection: [Errno 111] Connection refused)
Component: Treeherder → Treeherder: Log Parsing & Classification
(Assignee)

Updated

8 months ago
Assignee: nobody → ghickman
Status: NEW → ASSIGNED
(Assignee)

Comment 1

8 months ago
This got fixed when we moved to switched to using the lower level ES library in https://github.com/mozilla/treeherder/pull/3472
Status: ASSIGNED → RESOLVED
Last Resolved: 8 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.