917612 - [prod] SUMO search is throwing a 503

Status: VERIFIED FIXED

(support.mozilla.org :: Search, defect, P1)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

https://support.mozilla.org/en-US/search?esab=a&q=Firefox+crashed is suddenly throwing a 503.

Comment 1

[Stephen Donner [:stephend] Not actively reading bugmail]

From Errormill:

(500, u'SearchPhaseExecutionException[Failed to execute phase [query], total failure; shardFailures {[QgMsRmO7TZ-lPsrT_3ZSdw][sumo_sumo-20130913][1]: RemoteTransportException[[elasticsearch5_phx1][inet[/10.8.81.137:9300]][search/phase/query]]; nested: SearchParseException[[sumo_sumo-20130913][1]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"or": [{"and": [{"term": {"model

Stacktrace (most recent call last):

  File "kitsune/search/views.py", line 362, in search
    num_results = min(searcher.count(), settings.SEARCH_MAX_RESULTS)
  File "elasticutils/__init__.py", line 1252, in count
    return self[:0].raw()['hits']['total']
  File "elasticutils/__init__.py", line 1232, in raw
    doc_type=self.get_doctypes())
  File "pyelasticsearch/client.py", line 96, in decorate
    return func(*args, query_params=query_params, **kwargs)
  File "pyelasticsearch/client.py", line 591, in search
    return self._search_or_count('_search', query, **kwargs)
  File "pyelasticsearch/client.py", line 568, in _search_or_count
    query_params=query_params)
  File "pyelasticsearch/client.py", line 255, in send_request
    self._raise_exception(resp, prepped_response)
  File "pyelasticsearch/client.py", line 269, in _raise_exception
    raise error_class(response.status_code, error_message)

https://errormill.mozilla.org/support/support/group/107778/

Comment 2

[Ricky Rosario [:rrosario, :r1cky]]

Thanks for reporting. This looks related to our recent changes to the mapping and subsequent reindexing.

I rolled back one commit to https://github.com/mozilla/kitsune/commit/c9f858193dd750657b4197a6b47da235a7a27b69 and search is back (on a slightly stale index). Now I am rebuilding the index and we'll try again.

Comment 3

[Ricky Rosario [:rrosario, :r1cky]]

OK, I recreated the index and reshipped master and things are back to normal. YAY!

Comment 4

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED.