1131365 - Make OrangeFactor, logparser & mozautoeslib compatible with ElasticSearch 1.2

Status: RESOLVED WONTFIX

(Tree Management Graveyard :: OrangeFactor, defect)

Description

[Ed Morley [:emorley]]

Bug 1071044 / bug would like to update ElasticSearch from v0.9 to 1.2.
This has already occurred on es-dev, but not prod.

In bug 1121507 it was discovered OrangeFactor and/or the libraries it used needed changes to be compatible with v1.2.

Some things were fixed (bug 1121941, bug 1121956), however there was at least one issue left (and likely more, since we were only using ES 1.2 then for logparser submissions, and nothing else):

(In reply to Ed Morley [:edmorley] from bug 1121507 comment #9)
> (In reply to Ed Morley [:edmorley] from bug 1121507 comment #4)
> > There's also still the "ElasticSearchException: Unknown exception type"
> > (comment 2).
> 
> We're currently using pyes v0.16 (was v0.15 until bug 1121517 comment ), but
> v0.19+ gives much better output for this exception:
> https://github.com/aparo/pyes/commit/b61b4be45804abbec6fdd0f43e6127d3643e1e8e
> 
> However there are lots of breaking changes, so I've backported that fix by
> hand to
> logparser/lib/python2.6/site-packages/pyes-0.16.0-py2.6.egg/pyes/
> convert_errors.py temporarily.
> 
> This revealed:
> 
> Traceback (most recent call last):
>   File "/home/webtools/apps/logparser/src/logparser/logparser/savelogs.py",
> line 64, in parse
>     lp.parseFiles()
>   File "/home/webtools/apps/logparser/src/logparser/logparser/logparser.py",
> line 106, in parseFiles
>     self.postResultsToElasticSearch(testdata)
>   File "/home/webtools/apps/logparser/src/logparser/logparser/logparser.py",
> line 196, in postResultsToElasticSearch
>     self._post_testgroup_to_elasticsearch(data)
>   File "/home/webtools/apps/logparser/src/logparser/logparser/logparser.py",
> line 183, in _post_testgroup_to_elasticsearch
>     testgroup.submit()
>   File
> "/home/webtools/apps/logparser/src/mozautolog/mozautolog/esautolog.py", line
> 80, in submit
>     self._generate_testrun()
>   File
> "/home/webtools/apps/logparser/src/mozautolog/mozautolog/esautolog.py", line
> 58, in _generate_testrun
>     doc_type = [self.doc_type])
>   File
> "/home/webtools/apps/logparser/src/mozautoeslib/mozautoeslib/eslib.py", line
> 186, in query
>     doc_types=self.doc_type)
>   File
> "/home/webtools/apps/logparser/lib/python2.6/site-packages/pyes-0.16.0-py2.6.
> egg/pyes/es.py", line 893, in count
>     return self._query_call("_count", query, indexes, doc_types,
> **query_params)
>   File
> "/home/webtools/apps/logparser/lib/python2.6/site-packages/pyes-0.16.0-py2.6.
> egg/pyes/es.py", line 251, in _query_call
>     response = self._send_request('GET', path, body, querystring_args)
>   File
> "/home/webtools/apps/logparser/lib/python2.6/site-packages/pyes-0.16.0-py2.6.
> egg/pyes/es.py", line 223, in _send_request
>     raise_if_error(response.status, decoded)
>   File
> "/home/webtools/apps/logparser/lib/python2.6/site-packages/pyes-0.16.0-py2.6.
> egg/pyes/convert_errors.py", line 58, in raise_if_error
>     raise pyes.exceptions.ElasticSearchException(u"Unknown exception type:
> %d, %s" % (status, result), status, result)
> ElasticSearchException: Unknown exception type: 400, {u'count': 0,
> u'_shards': {u'successful': 0, u'failed': 5, u'total': 5, u'failures':
> [{u'index': u'logs', u'reason':
> u'BroadcastShardOperationFailedException[[logs][0] ]; nested:
> QueryParsingException[[logs] request does not support [bool]]; ', u'shard':
> 0}, {u'index': u'logs', u'reason':
> u'BroadcastShardOperationFailedException[[logs][2] ]; nested:
> RemoteTransportException[[elasticsearch2_dev_phx1][inet[/SNIP:9300]][count/
> s]]; nested: QueryParsingException[[logs] request does not support [bool]];
> ', u'shard': 2}, {u'index': u'logs', u'reason':
> u'BroadcastShardOperationFailedException[[logs][1] ]; nested:
> RemoteTransportException[[elasticsearch2_dev_phx1][inet[/SNIP:9300]][count/
> s]]; nested: QueryParsingException[[logs] request does not support [bool]];
> ', u'shard': 1}, {u'index': u'logs', u'reason':
> u'BroadcastShardOperationFailedException[[logs][4] ]; nested:
> QueryParsingException[[logs] request does not support [bool]]; ', u'shard':
> 4}, {u'index': u'logs', u'reason':
> u'BroadcastShardOperationFailedException[[logs][3] ]; nested:
> RemoteTransportException[[elasticsearch1_dev_phx1][inet[/SNIP:9300]][count/
> s]]; nested: QueryParsingException[[logs] request does not support [bool]];
> ', u'shard': 3}]}}