1377064 - Sporadic ES ReadTimeout errors in Travis

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Peter Bengtsson [:peterbe]]

The change that triggered this master travis build was totally unrelated.

Just in case that Travis build disappears (see URL), the error was:

 socorro/unittest/external/es/base.py::IntegrationTestSuperSearch::test_get_with_too_many 

Traceback (most recent call last):

  File "/home/travis/build/mozilla-services/socorro/socorro/unittest/external/es/base.py", line 871, in health_check

    request_timeout=1

  File "/home/travis/virtualenv/python2.7.11/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 69, in _wrapped

    return func(*args, params=params, **kwargs)

  File "/home/travis/virtualenv/python2.7.11/lib/python2.7/site-packages/elasticsearch/client/cluster.py", line 30, in health

    'health', index), params=params)

  File "/home/travis/virtualenv/python2.7.11/lib/python2.7/site-packages/elasticsearch/transport.py", line 307, in perform_request

    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)

  File "/home/travis/virtualenv/python2.7.11/lib/python2.7/site-packages/elasticsearch/connection/http_requests.py", line 70, in perform_request

    raise ConnectionTimeout('TIMEOUT', str(e), e)

ConnectionTimeout: ConnectionTimeout caused by - ReadTimeout(HTTPConnectionPool(host='localhost', port=9200): Read timed out. (read timeout=1))

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

I don't think I've seen these with Circle, yet. Circle runs our tests with services in containers which is different than how Travis does.

So, maybe this will go away or reduce a lot between either the ES upgrade or switching to Circle.

Comment 2

[Peter Bengtsson [:peterbe]]

Happened again:
https://travis-ci.org/mozilla-services/socorro/builds/254978976?utm_source=github_status&utm_medium=notification
https://travis-ci.org/mozilla-services/socorro/builds/254973282?utm_source=github_status&utm_medium=notification

These worked when they were PRs but not when tested with master. The commits are totally unrelated.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

(In reply to Will Kahn-Greene [:willkg] ET needinfo? me from comment #1)
> I don't think I've seen these with Circle, yet. Circle runs our tests with
> services in containers which is different than how Travis does.

I've seen these with Circle, too.

> So, maybe this will go away or reduce a lot between either the ES upgrade or
> switching to Circle.

I think this is still a possibility.

Comment 4

[Peter Bengtsson [:peterbe]]

PR https://github.com/mozilla-services/socorro/pull/3868

Comment 5

[[github robot]]

Commit pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/09f0c570bfc6053f5da26feaac645a4bd17c5e9a
fixes bug 1377064 - longer patience with unittest healthcheck (#3868)