829847 - [traceback][stage] TimeoutError: Request timed out after 2.000000 seconds

Status: VERIFIED WORKSFORME

(Participation Infrastructure :: Phonebook, defect)

Description

[Matt Brandt [:mbrandt]]

I apologize if this is a dupe of another bug. Off and on throughout the previous year we've been seeing this traceback at spurious moments.

They historically have occurred during searches; http://mozillians.allizom.org/en-US/search. Do we need to up a timeout variable?

Traceback (most recent call last):

  File "/data/www/mozillians.allizom.org/mozillians/vendor/src/django/django/core/handlers/base.py", line 111, in get_response
    response = callback(request, *callback_args, **callback_kwargs)

  File "/data/www/mozillians.allizom.org/mozillians/vendor/src/django/django/contrib/auth/decorators.py", line 20, in _wrapped_view
    return view_func(request, *args, **kwargs)

  File "/data/www/mozillians.allizom.org/mozillians/apps/phonebook/views.py", line 38, in wrapped
    return f(request, *args, **kwargs)

  File "/data/www/mozillians.allizom.org/mozillians/apps/phonebook/views.py", line 179, in search
    if len(profiles) == 1 and not groups:

  File "/data/www/mozillians.allizom.org/mozillians/vendor/src/elasticutils/elasticutils/__init__.py", line 434, in __len__
    return len(self._do_search())

  File "/data/www/mozillians.allizom.org/mozillians/vendor/src/elasticutils/elasticutils/__init__.py", line 614, in _do_search
    hits = self.raw()

  File "/data/www/mozillians.allizom.org/mozillians/vendor/src/elasticutils/elasticutils/contrib/django/__init__.py", line 153, in raw
    hits = super(S, self).raw()

  File "/data/www/mozillians.allizom.org/mozillians/vendor/src/elasticutils/elasticutils/__init__.py", line 655, in raw
    hits = es.search(qs, self.get_indexes(), self.get_doctypes())

  File "/data/www/mozillians.allizom.org/mozillians/vendor/src/pyes/pyes/es.py", line 613, in search
    return self._query_call("_search", query, indexes, doc_types, **query_params)

  File "/data/www/mozillians.allizom.org/mozillians/vendor/src/pyes/pyes/es.py", line 240, in _query_call
    response = self._send_request('GET', path, body, querystring_args)

  File "/data/www/mozillians.allizom.org/mozillians/vendor/src/pyes/pyes/es.py", line 197, in _send_request
    response = self.connection.execute(request)

  File "/data/www/mozillians.allizom.org/mozillians/vendor/src/pyes/pyes/connection_http.py", line 166, in _client_call
    return getattr(conn.client, attr)(*args, **kwargs)

  File "/data/www/mozillians.allizom.org/mozillians/vendor/src/pyes/pyes/connection_http.py", line 58, in execute
    response = self.client.urlopen(Method._VALUES_TO_NAMES[request.method], uri, body=request.body, headers=request.headers)

  File "/data/www/mozillians.allizom.org/mozillians/vendor/src/pyes/pyes/urllib3/connectionpool.py", line 286, in urlopen
    raise TimeoutError("Request timed out after %f seconds" % self.timeout)

TimeoutError: Request timed out after 2.000000 seconds

Comment 1

[Giorgos Logiotatidis [:giorgos]]

This is django failing to connect to ElasticSearch server. Maybe we can up the timeout variable, but I'll first have a chat with ElasticSearch gurus.

Thanks for reporting Matt!

Comment 2

[Giorgos Logiotatidis [:giorgos]]

I noticed that this traceback is from stage. 

Matt, do you face the same issue on prod too?

Comment 3

[Matt Brandt [:mbrandt]]

(In reply to Giorgos Logiotatidis [:giorgos] from comment #2)
> Matt, do you face the same issue on prod too?

asfaik we're only receiving tracebacks of this sort from stage (none from dev and prod).

Comment 4

[Andrei Hajdukewycz [:sancus]]

Is this still happening, Matt?

Did you discover anything interesting about timeouts from any gurus, Giorgos?

Comment 5

[Matt Brandt [:mbrandt]]

(In reply to Andrei Hajdukewycz [:sancus] from comment #4)
> Is this still happening, Matt?
> 
> Did you discover anything interesting about timeouts from any gurus, Giorgos?

:-) I have not spoken with The Guru (aka Giorgos) however I haven't seen any new tracebacks. I would be happy resolving this worksforme and reopening in the bug in the future if we start to see the tracebacks with frequency again.

What do you think?

Comment 6

[Giorgos Logiotatidis [:giorgos]]

IIRC mozillians.org prod and maybe stage were to be moved to a new ElasticSearch cluster to fight these problems. Maybe that's complete, maybe not, in any case I suggest that since this is about stage and no new tracebacks received we close it as WFM.

Thanks!

Comment 7

[Andrei Hajdukewycz [:sancus]]

Okay, calling this WFM for now.

Comment 8

[Matt Brandt [:mbrandt]]

Lovely! Thank you gentlemen.

Bumping to verified.