770037 - [traceback] Celery SearchPhaseExecutionException for mkt.stats.tasks.index_finance_total_by_currency

Status: RESOLVED FIXED

(Marketplace Graveyard :: Statistics, defect, P1)

Description

[Christopher Van Wiemeersch [:cvan]]

We've been getting this traceback every morning when `index_latest_mkt_stats` (https://github.com/mozilla/zamboni/blob/master/scripts/crontab/crontab.tpl#L70) runs:

Task mkt.stats.tasks.index_finance_total_by_currency with id 7f0e07cf-2e05-4398-a4b9-006f53c6ad28 raised exception:
'SearchPhaseExecutionException(u\'Failed to execute phase [query], total failure; shardFailures {[RCHLDB4pQTOAKAPlFkdhDA][addons][2]: RemoteTransportException[[Mekano][inet[/10.8.81.68:9300]][search/phase/query]]; nested: SearchParseException[[addons][2]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"count": 4}}, {"term": {"currency": "usd"}}, {"term": {"revenue": 3.96}}, {"term": {"refunds": 5}}, {"term": {"addon": 368172}}]}, "fields": ["id", "count"]}]]]; nested: NumberFormatException[For input string: "3.96"]; }{[UJSzB8XZQjyV_y9kkfGrkw][addons][3]: SearchParseException[[addons][3]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"count": 4}}, {"term": {"currency": "usd"}}, {"term": {"revenue": 3.96}}, {"term": {"refunds": 5}}, {"term": {"addon": 368172}}]}, "fields": ["id", "count"]}]]]; nested: NumberFormatException[For input string: "3.96"]; }{[UJSzB8XZQjyV_y9kkfGrkw][addons][0]: SearchParseException[[addons][0]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"count": 4}}, {"term": {"currency": "usd"}}, {"term": {"revenue": 3.96}}, {"term": {"refunds": 5}}, {"term": {"addon": 368172}}]}, "fields": ["id", "count"]}]]]; nested: NumberFormatException[For input string: "3.96"]; }{[UJSzB8XZQjyV_y9kkfGrkw][addons][1]: SearchParseException[[addons][1]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"count": 4}}, {"term": {"currency": "usd"}}, {"term": {"revenue": 3.96}}, {"term": {"refunds": 5}}, {"term": {"addon": 368172}}]}, "fields": ["id", "count"]}]]]; nested: NumberFormatException[For input string: "3.96"]; }{[uWWAHmxVR6mCeaFTdEqezw][addons][4]: RemoteTransportException[[Sawyer, Sam][inet[/10.8.81.65:9300]][search/phase/query]]; nested: SearchParseException[[addons][4]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"count": 4}}, {"term": {"currency": "usd"}}, {"term": {"revenue": 3.96}}, {"term": {"refunds": 5}}, {"term": {"addon": 368172}}]}, "fields": ["id", "count"]}]]]; nested: NumberFormatException[For input string: "3.96"]; }\',)'


Task was called with args: [[368172L, 368160L, 368158L, 368156L, 368152L, 368149L, 368147L, 368136L, 368127L, 368111L, 368090L]] kwargs: {'task_name': 'mkt.stats.tasks.index_finance_total_by_currency', 'task_id': '7f0e07cf-2e05-4398-a4b9-006f53c6ad28', 'loglevel': 20, 'delivery_info': {'routing_key': u'celery', 'exchange': u'celery'}, 'task_is_eager': False, 'logfile': '/var/log/celeryd-marketplace-prod.log', 'task_retries': 2}.

The contents of the full traceback was:

Traceback (most recent call last):
 File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/celery/execute/trace.py", line 153, in trace_task
   R = retval = task(*args, **kwargs)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/nuggets/celeryutils.py", line 35, in wrapped
   return fun(*args, **kw)
 File "/data/www/addons.mozilla.org/zamboni/mkt/stats/tasks.py", line 98, in index_finance_total_by_currency
   index_finance_total_by_currency.retry(args=[addons], exc=exc)
 File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/celery/app/task/__init__.py", line 556, in retry
   maybe_reraise()
 File "/data/www/addons.mozilla.org/zamboni/mkt/stats/tasks.py", line 94, in index_finance_total_by_currency
   if not already_indexed(Contribution, data):
 File "/data/www/addons.mozilla.org/zamboni/mkt/stats/tasks.py", line 278, in already_indexed
   return list(model.search().filter(**data).values_dict(data.keys()[0]))
 File "/data/www/addons.mozilla.org/zamboni/apps/amo/search.py", line 208, in __iter__
   return iter(self._do_search())
 File "/data/www/addons.mozilla.org/zamboni/apps/amo/search.py", line 184, in _do_search
   hits = self.raw()
 File "/data/www/addons.mozilla.org/zamboni/apps/amo/search.py", line 199, in raw
   hits = es.search(qs, self.index, self.type._meta.db_table)
 File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/pyes/es.py", line 841, in search
   return self._query_call("_search", body, indexes, doc_types, **query_params)
 File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/pyes/es.py", line 251, in _query_call
   response = self._send_request('GET', path, body, querystring_args)
 File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/pyes/es.py", line 223, in _send_request
   raise_if_error(response.status, decoded)
 File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/pyes/convert_errors.py", line 68, in raise_if_error
   raise excClass(msg, status, result)
SearchPhaseExecutionException: Failed to execute phase [query], total failure; shardFailures {[RCHLDB4pQTOAKAPlFkdhDA][addons][2]: RemoteTransportException[[Mekano][inet[/10.8.81.68:9300]][search/phase/query]]; nested: SearchParseException[[addons][2]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"count": 4}}, {"term": {"currency": "usd"}}, {"term": {"revenue": 3.96}}, {"term": {"refunds": 5}}, {"term": {"addon": 368172}}]}, "fields": ["id", "count"]}]]]; nested: NumberFormatException[For input string: "3.96"]; }{[UJSzB8XZQjyV_y9kkfGrkw][addons][3]: SearchParseException[[addons][3]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"count": 4}}, {"term": {"currency": "usd"}}, {"term": {"revenue": 3.96}}, {"term": {"refunds": 5}}, {"term": {"addon": 368172}}]}, "fields": ["id", "count"]}]]]; nested: NumberFormatException[For input string: "3.96"]; }{[UJSzB8XZQjyV_y9kkfGrkw][addons][0]: SearchParseException[[addons][0]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"count": 4}}, {"term": {"currency": "usd"}}, {"term": {"revenue": 3.96}}, {"term": {"refunds": 5}}, {"term": {"addon": 368172}}]}, "fields": ["id", "count"]}]]]; nested: NumberFormatException[For input string: "3.96"]; }{[UJSzB8XZQjyV_y9kkfGrkw][addons][1]: SearchParseException[[addons][1]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"count": 4}}, {"term": {"currency": "usd"}}, {"term": {"revenue": 3.96}}, {"term": {"refunds": 5}}, {"term": {"addon": 368172}}]}, "fields": ["id", "count"]}]]]; nested: NumberFormatException[For input string: "3.96"]; }{[uWWAHmxVR6mCeaFTdEqezw][addons][4]: RemoteTransportException[[Sawyer, Sam][inet[/10.8.81.65:9300]][search/phase/query]]; nested: SearchParseException[[addons][4]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"count": 4}}, {"term": {"currency": "usd"}}, {"term": {"revenue": 3.96}}, {"term": {"refunds": 5}}, {"term": {"addon": 368172}}]}, "fields": ["id", "count"]}]]]; nested: NumberFormatException[For input string: "3.96"]; }

Comment 1

[Kevin Ngo [:ngoke]]

https://github.com/mozilla/zamboni/commit/3e543a666f39694f1267db17144aad05240b6b65

Comment 2

[Christopher Van Wiemeersch [:cvan]]

Still happening. Just got 75 tracebacks for this:

Date: 	July 12, 2012 8:44:09 PM PDT

Task mkt.stats.tasks.index_finance_daily with id f26bfd69-a4dd-4754-9841-1b673c297e49 raised exception:
'SearchPhaseExecutionException(u\'Failed to execute phase [query], total failure; shardFailures {[6g7X35lVRZu74QMX30-0Bw][addons][0]: SearchParseException[[addons][0]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-10T00:00:00"}}, {"term": {"count": 28}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 2108}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[5BMXD4OjQRaZ5gUMNphgyg][addons][2]: RemoteTransportException[[Aftershock][inet[/10.8.81.68:9300]][search/phase/query]]; nested: SearchParseException[[addons][2]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-10T00:00:00"}}, {"term": {"count": 28}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 2108}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[5BMXD4OjQRaZ5gUMNphgyg][addons][4]: RemoteTransportException[[Aftershock][inet[/10.8.81.68:9300]][search/phase/query]]; nested: SearchParseException[[addons][4]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-10T00:00:00"}}, {"term": {"count": 28}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 2108}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[5BMXD4OjQRaZ5gUMNphgyg][addons][3]: RemoteTransportException[[Aftershock][inet[/10.8.81.68:9300]][search/phase/query]]; nested: SearchParseException[[addons][3]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-10T00:00:00"}}, {"term": {"count": 28}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 2108}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[6g7X35lVRZu74QMX30-0Bw][addons][1]: SearchParseException[[addons][1]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-10T00:00:00"}}, {"term": {"count": 28}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 2108}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }\',)'


Task was called with args: [[5270587L, 5270585L, 5270583L, 5270581L, 5270579L, 5270577L, 5270575L, 5270573L, 5270571L, 5270569L, 5270567L, 5270565L, 5270563L, 5270561L, 5270559L, 5270557L, 5270555L, 5270553L, 5270551L, 5270549L, 5270547L, 5270545L, 5270543L, 5270541L, 5270539L, 5270537L, 5270535L, 5270533L, 5270531L, 5270529L, 5270527L, 5270525L, 5270523L, 5270521L, 5270519L, 5270517L, 5270515L, 5270513L, 5270511L, 5270509L, 5270507L, 5270505L, 5270503L, 5270501L, 5270499L, 5270497L, 5270495L, 5270493L, 5270491L, 5270489L]] kwargs: {'task_name': 'mkt.stats.tasks.index_finance_daily', 'task_id': 'f26bfd69-a4dd-4754-9841-1b673c297e49', 'loglevel': 20, 'delivery_info': {'routing_key': u'celery', 'exchange': u'celery'}, 'task_is_eager': False, 'logfile': '/var/log/celeryd-marketplace-prod.log', 'task_retries': 2}.

The contents of the full traceback was:

Traceback (most recent call last):
 File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/celery/execute/trace.py", line 153, in trace_task
   R = retval = task(*args, **kwargs)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/nuggets/celeryutils.py", line 35, in wrapped
   return fun(*args, **kw)
 File "/data/www/addons.mozilla.org/zamboni/mkt/stats/tasks.py", line 142, in index_finance_daily
   index_finance_daily.retry(args=[ids], exc=exc)
 File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/celery/app/task/__init__.py", line 556, in retry
   maybe_reraise()
 File "/data/www/addons.mozilla.org/zamboni/mkt/stats/tasks.py", line 137, in index_finance_daily
   if not already_indexed(Contribution, data):
 File "/data/www/addons.mozilla.org/zamboni/mkt/stats/tasks.py", line 356, in already_indexed
   return list(model.search().filter(**data).values_dict(data.keys()[0]))
 File "/data/www/addons.mozilla.org/zamboni/apps/amo/search.py", line 208, in __iter__
   return iter(self._do_search())
 File "/data/www/addons.mozilla.org/zamboni/apps/amo/search.py", line 184, in _do_search
   hits = self.raw()
 File "/data/www/addons.mozilla.org/zamboni/apps/amo/search.py", line 199, in raw
   hits = es.search(qs, self.index, self.type._meta.db_table)
 File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/pyes/es.py", line 841, in search
   return self._query_call("_search", body, indexes, doc_types, **query_params)
 File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/pyes/es.py", line 251, in _query_call
   response = self._send_request('GET', path, body, querystring_args)
 File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/pyes/es.py", line 223, in _send_request
   raise_if_error(response.status, decoded)
 File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/pyes/convert_errors.py", line 68, in raise_if_error
   raise excClass(msg, status, result)
SearchPhaseExecutionException: Failed to execute phase [query], total failure; shardFailures {[6g7X35lVRZu74QMX30-0Bw][addons][0]: SearchParseException[[addons][0]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-10T00:00:00"}}, {"term": {"count": 28}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 2108}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[5BMXD4OjQRaZ5gUMNphgyg][addons][2]: RemoteTransportException[[Aftershock][inet[/10.8.81.68:9300]][search/phase/query]]; nested: SearchParseException[[addons][2]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-10T00:00:00"}}, {"term": {"count": 28}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 2108}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[5BMXD4OjQRaZ5gUMNphgyg][addons][4]: RemoteTransportException[[Aftershock][inet[/10.8.81.68:9300]][search/phase/query]]; nested: SearchParseException[[addons][4]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-10T00:00:00"}}, {"term": {"count": 28}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 2108}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[5BMXD4OjQRaZ5gUMNphgyg][addons][3]: RemoteTransportException[[Aftershock][inet[/10.8.81.68:9300]][search/phase/query]]; nested: SearchParseException[[addons][3]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-10T00:00:00"}}, {"term": {"count": 28}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 2108}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[6g7X35lVRZu74QMX30-0Bw][addons][1]: SearchParseException[[addons][1]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-10T00:00:00"}}, {"term": {"count": 28}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 2108}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }

Comment 3

[Christopher Van Wiemeersch [:cvan]]

Got 165 tracebacks in two days. We should fix this.

Comment 4

[Kevin Ngo [:ngoke]]

https://github.com/mozilla/zamboni/commit/63455757d67d270a661e97c440cc899b8dd5983b

Comment 5

[Christopher Van Wiemeersch [:cvan]]

92 celery tracebacks at 8:44 PM today:

Task mkt.stats.tasks.index_finance_daily with id 8a70d2f3-a305-4fc9-ac16-e79cb3ffeb11 raised exception:
'SearchPhaseExecutionException(u\'Failed to execute phase [query], total failure; shardFailures {[3nuG5ezdR7iEbeeJQ2fDeQ][addons][4]: RemoteTransportException[[Goldbug][inet[/10.8.81.67:9300]][search/phase/query]]; nested: SearchParseException[[addons][4]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-01T00:00:00"}}, {"term": {"count": 1}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 427}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[Q1vdm4SUShCy6TxlDAK3Vg][addons][3]: SearchParseException[[addons][3]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-01T00:00:00"}}, {"term": {"count": 1}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 427}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[Q1vdm4SUShCy6TxlDAK3Vg][addons][2]: SearchParseException[[addons][2]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-01T00:00:00"}}, {"term": {"count": 1}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 427}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[Q1vdm4SUShCy6TxlDAK3Vg][addons][1]: SearchParseException[[addons][1]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-01T00:00:00"}}, {"term": {"count": 1}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 427}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[ejqupHrYTMm7PbAUAfr-ZQ][addons][0]: RemoteTransportException[[Blitzschlag, Baron Von][inet[/10.8.81.68:9300]][search/phase/query]]; nested: SearchParseException[[addons][0]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-01T00:00:00"}}, {"term": {"count": 1}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 427}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }\',)'


Task was called with args: [[5259873L, 5259861L, 5259857L, 5259853L, 5259845L, 5259843L, 5259827L, 5259815L, 5259795L, 5259779L, 5259771L, 5259741L, 5259703L, 5259697L, 5259671L, 5259647L, 5259573L, 5259559L, 5259503L, 5259443L, 5259437L, 5259429L, 5259397L, 5259335L, 5259331L, 5259297L, 5259281L, 5259279L, 5259273L, 5258777L, 5258769L, 5258763L, 5258747L, 5258721L, 5258717L, 5258713L, 5258683L, 5258679L, 5258641L, 5258621L, 5258581L, 5258573L, 5258561L, 5258539L, 5258527L, 5258517L, 5258505L, 5258491L, 5258451L, 5258447L]] kwargs: {'task_name': 'mkt.stats.tasks.index_finance_daily', 'task_id': '8a70d2f3-a305-4fc9-ac16-e79cb3ffeb11', 'loglevel': 20, 'delivery_info': {'routing_key': u'celery', 'exchange': u'celery'}, 'task_is_eager': False, 'logfile': '/var/log/celeryd-marketplace-prod.log', 'task_retries': 2}.

The contents of the full traceback was:

Traceback (most recent call last):
 File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/celery/execute/trace.py", line 153, in trace_task
   R = retval = task(*args, **kwargs)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/nuggets/celeryutils.py", line 35, in wrapped
   return fun(*args, **kw)
 File "/data/www/addons.mozilla.org/zamboni/mkt/stats/tasks.py", line 143, in index_finance_daily
   index_finance_daily.retry(args=[ids], exc=exc)
 File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/celery/app/task/__init__.py", line 556, in retry
   maybe_reraise()
 File "/data/www/addons.mozilla.org/zamboni/mkt/stats/tasks.py", line 138, in index_finance_daily
   if not already_indexed(Contribution, data):
 File "/data/www/addons.mozilla.org/zamboni/mkt/stats/tasks.py", line 357, in already_indexed
   return list(model.search().filter(**data).values_dict(data.keys()[0]))
 File "/data/www/addons.mozilla.org/zamboni/apps/amo/search.py", line 207, in __iter__
   return iter(self._do_search())
 File "/data/www/addons.mozilla.org/zamboni/apps/amo/search.py", line 183, in _do_search
   hits = self.raw()
 File "/data/www/addons.mozilla.org/zamboni/apps/amo/search.py", line 198, in raw
   hits = es.search(qs, self.index, self.type._meta.db_table)
 File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/pyes/es.py", line 841, in search
   return self._query_call("_search", body, indexes, doc_types, **query_params)
 File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/pyes/es.py", line 251, in _query_call
   response = self._send_request('GET', path, body, querystring_args)
 File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/pyes/es.py", line 223, in _send_request
   raise_if_error(response.status, decoded)
 File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/pyes/convert_errors.py", line 68, in raise_if_error
   raise excClass(msg, status, result)
SearchPhaseExecutionException: Failed to execute phase [query], total failure; shardFailures {[3nuG5ezdR7iEbeeJQ2fDeQ][addons][4]: RemoteTransportException[[Goldbug][inet[/10.8.81.67:9300]][search/phase/query]]; nested: SearchParseException[[addons][4]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-01T00:00:00"}}, {"term": {"count": 1}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 427}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[Q1vdm4SUShCy6TxlDAK3Vg][addons][3]: SearchParseException[[addons][3]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-01T00:00:00"}}, {"term": {"count": 1}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 427}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[Q1vdm4SUShCy6TxlDAK3Vg][addons][2]: SearchParseException[[addons][2]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-01T00:00:00"}}, {"term": {"count": 1}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 427}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[Q1vdm4SUShCy6TxlDAK3Vg][addons][1]: SearchParseException[[addons][1]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-01T00:00:00"}}, {"term": {"count": 1}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 427}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[ejqupHrYTMm7PbAUAfr-ZQ][addons][0]: RemoteTransportException[[Blitzschlag, Baron Von][inet[/10.8.81.68:9300]][search/phase/query]]; nested: SearchParseException[[addons][0]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-01T00:00:00"}}, {"term": {"count": 1}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 427}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }

Comment 6

[Kevin Ngo [:ngoke]]

let's try this 

https://github.com/mozilla/zamboni/commit/67f7629d3b70d3406c6796ef5c0be97c5eb26925

Comment 7

[Christopher Van Wiemeersch [:cvan]]

104 tracebacks again:

ask mkt.stats.tasks.index_finance_daily with id 4e06ed1e-de81-426e-92b3-5a13192c9024 raised exception:
'SearchPhaseExecutionException(u\'Failed to execute phase [query], total failure; shardFailures {[-ik3Ji4eTBabTGX4GdVVmg][addons][4]: RemoteTransportException[[Zaladane][inet[/10.8.81.65:9300]][search/phase/query]]; nested: SearchParseException[[addons][4]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-24T00:00:00"}}, {"term": {"count": 41}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 3677}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[-ik3Ji4eTBabTGX4GdVVmg][addons][2]: RemoteTransportException[[Zaladane][inet[/10.8.81.65:9300]][search/phase/query]]; nested: SearchParseException[[addons][2]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-24T00:00:00"}}, {"term": {"count": 41}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 3677}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[-ik3Ji4eTBabTGX4GdVVmg][addons][0]: RemoteTransportException[[Zaladane][inet[/10.8.81.65:9300]][search/phase/query]]; nested: SearchParseException[[addons][0]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-24T00:00:00"}}, {"term": {"count": 41}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 3677}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[rVPV38a_SZGxd1934DOhvw][addons][1]: RemoteTransportException[[Kaluu][inet[/10.8.81.67:9300]][search/phase/query]]; nested: SearchParseException[[addons][1]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-24T00:00:00"}}, {"term": {"count": 41}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 3677}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[rVPV38a_SZGxd1934DOhvw][addons][3]: RemoteTransportException[[Kaluu][inet[/10.8.81.67:9300]][search/phase/query]]; nested: SearchParseException[[addons][3]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-24T00:00:00"}}, {"term": {"count": 41}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 3677}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }\',)'


Task was called with args: [[5287963L, 5287961L, 5287959L, 5287957L, 5287955L, 5287953L, 5287951L, 5287949L, 5287947L, 5287945L, 5287943L, 5287941L, 5287939L, 5287937L, 5287935L, 5287933L, 5287931L, 5287929L, 5287927L, 5287925L, 5287923L, 5287921L, 5287919L, 5287917L, 5287915L, 5287913L, 5287911L, 5287909L, 5287907L, 5287905L, 5287903L, 5287901L, 5287899L, 5287897L, 5287895L, 5287893L, 5287891L, 5287889L, 5287887L, 5287885L, 5287883L, 5287881L, 5287879L, 5287877L, 5287875L, 5287873L, 5287871L, 5287869L, 5287867L, 5287865L]] kwargs: {'task_name': 'mkt.stats.tasks.index_finance_daily', 'task_id': '4e06ed1e-de81-426e-92b3-5a13192c9024', 'loglevel': 20, 'delivery_info': {'routing_key': u'celery', 'exchange': u'celery'}, 'task_is_eager': False, 'logfile': '/var/log/celeryd-marketplace-prod.log', 'task_retries': 2}.

The contents of the full traceback was:

Traceback (most recent call last):
 File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/celery/execute/trace.py", line 153, in trace_task
   R = retval = task(*args, **kwargs)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/nuggets/celeryutils.py", line 35, in wrapped
   return fun(*args, **kw)
 File "/data/www/addons.mozilla.org/zamboni/mkt/stats/tasks.py", line 143, in index_finance_daily
   index_finance_daily.retry(args=[ids], exc=exc)
 File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/celery/app/task/__init__.py", line 556, in retry
   maybe_reraise()
 File "/data/www/addons.mozilla.org/zamboni/mkt/stats/tasks.py", line 138, in index_finance_daily
   if not already_indexed(Contribution, data):
 File "/data/www/addons.mozilla.org/zamboni/mkt/stats/tasks.py", line 357, in already_indexed
   return list(model.search().filter(**data).values_dict(data.keys()[0]))
 File "/data/www/addons.mozilla.org/zamboni/apps/amo/search.py", line 207, in __iter__
   return iter(self._do_search())
 File "/data/www/addons.mozilla.org/zamboni/apps/amo/search.py", line 183, in _do_search
   hits = self.raw()
 File "/data/www/addons.mozilla.org/zamboni/apps/amo/search.py", line 198, in raw
   hits = es.search(qs, self.index, self.type._meta.db_table)
 File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/pyes/es.py", line 841, in search
   return self._query_call("_search", body, indexes, doc_types, **query_params)
 File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/pyes/es.py", line 251, in _query_call
   response = self._send_request('GET', path, body, querystring_args)
 File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/pyes/es.py", line 223, in _send_request
   raise_if_error(response.status, decoded)
 File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/pyes/convert_errors.py", line 68, in raise_if_error
   raise excClass(msg, status, result)
SearchPhaseExecutionException: Failed to execute phase [query], total failure; shardFailures {[-ik3Ji4eTBabTGX4GdVVmg][addons][4]: RemoteTransportException[[Zaladane][inet[/10.8.81.65:9300]][search/phase/query]]; nested: SearchParseException[[addons][4]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-24T00:00:00"}}, {"term": {"count": 41}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 3677}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[-ik3Ji4eTBabTGX4GdVVmg][addons][2]: RemoteTransportException[[Zaladane][inet[/10.8.81.65:9300]][search/phase/query]]; nested: SearchParseException[[addons][2]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-24T00:00:00"}}, {"term": {"count": 41}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 3677}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[-ik3Ji4eTBabTGX4GdVVmg][addons][0]: RemoteTransportException[[Zaladane][inet[/10.8.81.65:9300]][search/phase/query]]; nested: SearchParseException[[addons][0]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-24T00:00:00"}}, {"term": {"count": 41}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 3677}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[rVPV38a_SZGxd1934DOhvw][addons][1]: RemoteTransportException[[Kaluu][inet[/10.8.81.67:9300]][search/phase/query]]; nested: SearchParseException[[addons][1]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-24T00:00:00"}}, {"term": {"count": 41}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 3677}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }{[rVPV38a_SZGxd1934DOhvw][addons][3]: RemoteTransportException[[Kaluu][inet[/10.8.81.67:9300]][search/phase/query]]; nested: SearchParseException[[addons][3]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"filter": {"and": [{"term": {"date": "2012-07-24T00:00:00"}}, {"term": {"count": 41}}, {"term": {"revenue": 0.0}}, {"term": {"refunds": 0}}, {"term": {"addon": 3677}}]}, "fields": ["id", "date"]}]]]; nested: NumberFormatException[For input string: "0.0"]; }

Comment 8

[Christopher Van Wiemeersch [:cvan]]

CC'ing ES folks to help take a look. We're getting hammered a lot on -dev/production with this.

Comment 9

[Christopher Van Wiemeersch [:cvan]]

Could you comment out this job in the crontab if we can't figure this out.

Comment 10

[Wil Clouser [:clouserw]]

Last time we got this was 2 days ago - maybe before the new ES cluster.  Has this happened since?

Comment 11

[Christopher Van Wiemeersch [:cvan]]

I also get like 30 failures on mkt/stats/tests/test_views.py. Kevin, can you please take a look at this? If you need help running some queries on -dev, holler at your boi.

Comment 12

[Kevin Ngo [:ngoke]]

k

Comment 13

[Kevin Ngo [:ngoke]]

https://github.com/mozilla/zamboni/commit/e554a47985c97d79b76acc0d987324255a9c2ba1

Took another stab at it. I don't really have a way of debugging this traceback.

Comment 14

[Kumar McMillan [:kumar]]

feel free to add some extra logging if you think it will help you understand the problem when it happens again in prod

Comment 15

[Kevin Ngo [:ngoke]]

closing it for now until it's reverified as still broken.

Comment 16

[Christopher Van Wiemeersch [:cvan]]

I didn't see this error last night. So I think we're good. I'll keep it open for a few more days and verify.