Closed Bug 1119015 Opened 10 years ago Closed 10 years ago

fix get API with ES 1.2.4

Categories

(Input Graveyard :: Search, defect, P1)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: willkg, Assigned: willkg)

Details

(Whiteboard: u=user c=api p=1 s=input.2015q1)

We have ES 1.2.4 on the dev ES cluster which we use for the Input -dev and -stage environments. If you go to: https://input.allizom.org/api/v1/feedback/ You get an HTTP 500: Traceback (most recent call last): File "/data/www/input.allizom.org/input/vendor/src/django/django/core/handlers/base.py", line 112, in get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/data/www/input.allizom.org/venv/lib/python2.7/site-packages/newrelic-2.38.0.31/newrelic/hooks/framework_django.py", line 497, in wrapper return wrapped(*args, **kwargs) File "/data/www/input.allizom.org/input/vendor/src/django/django/views/decorators/csrf.py", line 57, in wrapped_view return view_func(*args, **kwargs) File "/data/www/input.allizom.org/input/fjord/base/utils.py", line 457, in decorated_func response = f(request, *args, **kwargs) File "/data/www/input.allizom.org/input/fjord/feedback/api_views.py", line 143, in _feedback_api_router return public_feedback_api_view(request) File "/data/www/input.allizom.org/input/vendor/src/django/django/views/generic/base.py", line 69, in view return self.dispatch(request, *args, **kwargs) File "/data/www/input.allizom.org/venv/lib/python2.7/site-packages/newrelic-2.38.0.31/newrelic/hooks/component_djangorestframework.py", line 27, in _nr_wrapper_APIView_dispatch_ return wrapped(*args, **kwargs) File "/data/www/input.allizom.org/input/vendor/src/django/django/views/decorators/csrf.py", line 57, in wrapped_view return view_func(*args, **kwargs) File "/data/www/input.allizom.org/input/vendor/src/django-rest-framework/rest_framework/views.py", line 327, in dispatch response = self.handle_exception(exc) File "/data/www/input.allizom.org/input/vendor/src/django-rest-framework/rest_framework/views.py", line 324, in dispatch response = handler(request, *args, **kwargs) File "/data/www/input.allizom.org/input/fjord/feedback/api_views.py", line 97, in get responses = models.ResponseMappingType.reshape(search[:maximum]) File "/data/www/input.allizom.org/input/fjord/feedback/models.py", line 423, in reshape for result in results File "/data/www/input.allizom.org/input/vendor/src/elasticutils/elasticutils/__init__.py", line 1615, in __iter__ return iter(self._do_search()) File "/data/www/input.allizom.org/input/vendor/src/elasticutils/elasticutils/__init__.py", line 1431, in _do_search response = self.raw() File "/data/www/input.allizom.org/input/vendor/src/elasticutils/elasticutils/__init__.py", line 1510, in raw **extra_search_kwargs) File "/data/www/input.allizom.org/input/vendor/src/elasticsearch-py/elasticsearch/client/utils.py", line 68, in _wrapped return func(*args, params=params, **kwargs) File "/data/www/input.allizom.org/input/vendor/src/elasticsearch-py/elasticsearch/client/__init__.py", line 440, in search params=params, body=body) File "/data/www/input.allizom.org/input/vendor/src/elasticsearch-py/elasticsearch/transport.py", line 284, in perform_request status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout) File "/data/www/input.allizom.org/input/vendor/src/elasticsearch-py/elasticsearch/connection/http_urllib3.py", line 55, in perform_request self._raise_error(response.status, raw_data) File "/data/www/input.allizom.org/input/vendor/src/elasticsearch-py/elasticsearch/connection/base.py", line 97, in _raise_error raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info) RequestError: TransportError(400, u'SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[s8HM99PgT12L_9gKv2RRbg][stage-inputindex][4]: RemoteTransportException[[elasticsearch1_dev_phx1][inet[/10.8.81.181:9300]][search/phase/query]]; nested: SearchParseException[[stage-inputindex][4]: from[-1],size[-1],sort[<custom:"created": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@6ca4ae62>!]: Parse Failure [Failed to parse source [{"sort": [{"created": "desc"}], "fields": [["id", "happy", "api", "url_domain", "has_email", "description", "category", "description_bigrams", "user_agent", "product", "version", "platform", "locale", "source", "campaign", "organic", "created"]], "size": 1000}]]]; nested: ElasticsearchParseException[Expected field name but got END_ARRAY "fields"]; }{[l7m1bjB-RC-AbQ4Srp0eLQ][stage-inputindex][3]: SearchParseException[[stage-inputindex][3]: from[-1],size[-1],sort[<custom:"created": org. elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@20978f4>!]: Parse Failure [Failed to parse source [{"sort": [{"created": "desc"}], "fields": [["id", "happy", "api", "url_domain", "has_email", "description", "category", "description_bigrams", "user_agent", "product", "version", "platform", "locale", "source", "campaign", "organic", "created"]], "size": 1000}]]]; nested: ElasticsearchParseException[Expected field name but got END_ARRAY "fields"]; }{[7P21zPibSNGvmWl952lotA][stage-inputindex][2]: RemoteTransportException[[elasticsearch3_dev_phx1][inet[/10.8.81.183:9300]][search/phase/query]]; nested: SearchParseException[[stage-inputindex][2]: from[-1],size[-1],sort[<custom:"created": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@1f07f1e4>!]: Parse Failure [Failed to parse source [{"sort": [{"created": "desc"}], "fields": [["id", "happy", "api", "url_domain", "has_email", "description", "category", "description_bigrams", "u ser_agent", "product", "version", "platform", "locale", "source", "campaign", "organic", "created"]], "size": 1000}]]]; nested: ElasticsearchParseException[Expected field name but got END_ARRAY "fields"]; }{[s8HM99PgT12L_9gKv2RRbg][stage-inputindex][1]: RemoteTransportException[[elasticsearch1_dev_phx1][inet[/10.8.81.181:9300]][search/phase/query]]; nested: SearchParseException[[stage-inputindex][1]: from[-1],size[-1],sort[<custom:"created": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@492666b8>!]: Parse Failure [Failed to parse source [{"sort": [{"created": "desc"}], "fields": [["id", "happy", "api", "url_domain", "has_email", "description", "category", "description_bigrams", "user_agent", "product", "version", "platform", "locale", "source", "campaign", "organic", "created"]], "size": 1000}]]]; nested: ElasticsearchParseException[Expected field name but got END_ARRAY "fields"]; }{[7P21zPibSNGvmWl952lotA][stage-inputindex][0]: RemoteTransport Exception[[elasticsearch3_dev_phx1][inet[/10.8.81.183:9300]][search/phase/query]]; nested: SearchParseException[[stage-inputindex][0]: from[-1],size[-1],sort[<custom:"created": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@424f80e>!]: Parse Failure [Failed to parse source [{"sort": [{"created": "desc"}], "fields": [["id", "happy", "api", "url_domain", "has_email", "description", "category", "description_bigrams", "user_agent", "product", "version", "platform", "locale", "source", "campaign", "organic", "created"]], "size": 1000}]]]; nested: ElasticsearchParseException[Expected field name but got END_ARRAY "fields"]; }]') Need to look into what's going on. Seems like it's having problems with the "sort" portion which might mean it's an ElasticUtils problem.
Grabbing this to work on tomorrow.
Assignee: nobody → willkg
Running the test suite kicks up 17 errors. We can track down the issues using that.
Depends on: 1119304
Depends on: 1119305
Depends on: 1119326
No longer depends on: 1119326
No longer depends on: 1119304
No longer depends on: 1119305
Status: NEW → ASSIGNED
Whiteboard: u=user c=api p= s=input.2015q1 → u=user c=api p=1 s=input.2015q1
The reason there's all this "depends on"/"does not depends on" stuff in this bug is because I accidentally blocked a bunch of bugs on this bug, but got the bug number wrong. Landed in: https://github.com/mozilla/fjord/commit/de4621e524f959d5615393a88e494b9b02deca09 Will push on Monday.
Had to push today because next week is release week. Pushed it to production. Verified it works now on -stage (where we have ES 1.2.4) and -prod (where we have ES 0.90). Marking as FIXED.
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Product: Input → Input Graveyard
You need to log in before you can comment on or make changes to this bug.