762096 - Any search returns a 500 error

Status: VERIFIED FIXED

(Participation Infrastructure :: Phonebook, defect)

Description

[Matt Brandt [:mbrandt]]

Performing a search on dev and stage returns a 500 error.

Steps to reproduce:
1. with a vouched account perform a search (ex. https://mozillians.allizom.org/en-US/search?q=staff)

Expected:
Results are returned

Actual:
[10:43:15.651] GET https://mozillians.allizom.org/en-US/search?q=staff [HTTP/1.1 500 INTERNAL SERVER ERROR 1479ms]

Traceback (most recent call last):

 File "/data/www/mozillians.allizom.org/mozillians/vendor/src/django/django/core/handlers/base.py", line 111, in get_response
   response = callback(request, *callback_args, **callback_kwargs)

 File "/data/www/mozillians.allizom.org/mozillians/vendor/src/django/django/contrib/auth/decorators.py", line 19, in _wrapped_view
   return view_func(request, *args, **kwargs)

 File "/data/www/mozillians.allizom.org/mozillians/apps/phonebook/views.py", line 33, in wrapped
   return f(request, *args, **kwargs)

 File "/data/www/mozillians.allizom.org/mozillians/apps/phonebook/views.py", line 151, in search
   people = paginator.page(page)

 File "/data/www/mozillians.allizom.org/mozillians/vendor/src/django/django/core/paginator.py", line 37, in page
   number = self.validate_number(number)

 File "/data/www/mozillians.allizom.org/mozillians/vendor/src/django/django/core/paginator.py", line 28, in validate_number
   if number > self.num_pages:

 File "/data/www/mozillians.allizom.org/mozillians/vendor/src/django/django/core/paginator.py", line 60, in _get_num_pages
   if self.count == 0 and not self.allow_empty_first_page:

 File "/data/www/mozillians.allizom.org/mozillians/vendor/src/django/django/core/paginator.py", line 48, in _get_count
   self._count = self.object_list.count()

 File "/data/www/mozillians.allizom.org/mozillians/vendor/src/elasticutils/elasticutils/__init__.py", line 257, in count
   return self[:0].raw()['hits']['total']

 File "/data/www/mozillians.allizom.org/mozillians/vendor/src/elasticutils/elasticutils/__init__.py", line 384, in raw
   hits = es.search(qs, index, self.type._meta.db_table)

 File "/data/www/mozillians.allizom.org/mozillians/vendor/src/pyes/pyes/es.py", line 613, in search
   return self._query_call("_search", query, indexes, doc_types, **query_params)

 File "/data/www/mozillians.allizom.org/mozillians/vendor/src/pyes/pyes/es.py", line 240, in _query_call
   response = self._send_request('GET', path, body, querystring_args)

 File "/data/www/mozillians.allizom.org/mozillians/vendor/src/pyes/pyes/es.py", line 197, in _send_request
   response = self.connection.execute(request)

 File "/data/www/mozillians.allizom.org/mozillians/vendor/src/pyes/pyes/connection_http.py", line 166, in _client_call
   return getattr(conn.client, attr)(*args, **kwargs)

 File "/data/www/mozillians.allizom.org/mozillians/vendor/src/pyes/pyes/connection_http.py", line 58, in execute
   response = self.client.urlopen(Method._VALUES_TO_NAMES[request.method], uri, body=request.body, headers=request.headers)

 File "/data/www/mozillians.allizom.org/mozillians/vendor/src/pyes/pyes/urllib3/connectionpool.py", line 286, in urlopen
   raise TimeoutError("Request timed out after %f seconds" % self.timeout)

TimeoutError: Request timed out after 1.000000 seconds

Comment 1

[James Socol [:jsocol, :james]]

This is an ElasticSearch timeout. CCing IT folks and will ping in IRC.

Comment 2

[Brandon Burton [:solarce]]

This was actually a VLAN issue, with the switch over to the new Xeon Seamicros, the IP -stage was trying to use to get to ES was wrong.

I've updated /etc/hosts and confirmed search works now.

Comment 3

[Matt Brandt [:mbrandt]]

Most excellent. Thank you solarce - bumping to verified.

[17:56:57.027] GET https://mozillians.allizom.org/en-US/search?q=staff [HTTP/1.1 200 OK 1709ms]

Comment 4

[Matt Brandt [:mbrandt]]

Sorry I spoke to soon - does /etc/hosts need to be updated on dev as well? I get a similar traceback (comment 0) from dev when I run a search.

https://mozillians-dev.allizom.org/en-US/search?q=staff

Comment 5

[Brandon Burton [:solarce]]

Same issue, fixing now

Comment 6

[Brandon Burton [:solarce]]

https://mozillians-dev.allizom.org/en-US/search?q=staff works for me now, how's it look for you?

Comment 7

[James Socol [:jsocol, :james]]

Works for me, thanks!

Comment 8

[Matt Brandt [:mbrandt]]

Verified - thanks solarce.