Closed Bug 1373265 Opened 9 years ago Closed 5 years ago

dxr creating invalid json for elasticsearch, fails in elasticsearch 2.x and beyond

Categories

(Webtools Graveyard :: DXR, defect)

Product:
Webtools Graveyard
Component:
DXR
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: michael.l.schneider, Unassigned)

Details

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 Steps to reproduce: updated elastisearch to 2.4 and 5.4 (starting failing in 2.4). ran dxr index, elasticsearch query failed with parser error Actual results: DXR generate the following invalid JS {"mappings": {"line": {"_all": {"enabled": false}, "properties": {"c_type_ref": {"type": "object", "properties": {"start": {"index": "no", "type": "integer"}, "qualname": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}, "end": {"index": "no", "type": "integer"}, "name": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}}}, "c_warning_opt": {"type": "object", "properties": {"start": {"index": "no", "type": "integer"}, "qualname": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}, "end": {"index": "no", "type": "integer"}, "name": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}}}, "c_namespace_ref": {"type": "object", "properties": {"start": {"index": "no", "type": "integer"}, "qualname": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}, "end": {"index": "no", "type": "integer"}, "name": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}}}, "file_name": {"index": "not_analyzed", "type": "string", "fields": {"trigrams": {"type": "string", "analyzer": "trigramalyzer"}, "trigrams_lower": {"type": "string", "analyzer": "trigramalyzer_lower"}}}, "refs": {"start": {"index": "no", "type": "integer"}, "end": {"index": "no", "type": "integer"}, "payload": {"type": "object", "properties": {"menu_data": {"index": "no", "type": "string"}, "qualname_hash": {"index": "no", "type": "long"}, "hover": {"index": "no", "type": "string"}, "id": {"index": "no", "type": "string"}, "plugin": {"index": "no", "type": "string"}}}, "type": "object"}, "c_bases": {"type": "object", "properties": {"start": {"index": "no", "type": "integer"}, "qualname": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}, "end": {"index": "no", "type": "integer"}, "name": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}}}, "number": {"type": "integer"}, "c_function_decl": {"type": "object", "properties": {"start": {"index": "no", "type": "integer"}, "qualname": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}, "end": {"index": "no", "type": "integer"}, "name": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}}}, "c_namespace": {"type": "object", "properties": {"start": {"index": "no", "type": "integer"}, "qualname": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}, "end": {"index": "no", "type": "integer"}, "name": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}}}, "content": {"ignore_above": 8191, "index": "not_analyzed", "type": "string", "fields": {"trigrams": {"type": "string", "analyzer": "trigramalyzer"}, "trigrams_lower": {"type": "string", "analyzer": "trigramalyzer_lower"}}}, "c_overridden": {"type": "object", "properties": {"start": {"index": "no", "type": "integer"}, "qualname": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}, "end": {"index": "no", "type": "integer"}, "name": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}}}, "c_type_decl": {"type": "object", "properties": {"start": {"index": "no", "type": "integer"}, "qualname": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}, "end": {"index": "no", "type": "integer"}, "name": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}}}, "regions": {"start": {"index": "no", "type": "integer"}, "end": {"index": "no", "type": "integer"}, "payload": {"index": "no", "type": "string"}, "type": "object"}, "c_call": {"type": "object", "properties": {"start": {"index": "no", "type": "integer"}, "qualname": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}, "end": {"index": "no", "type": "integer"}, "name": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}}}, "annotations": {"type": "object", "properties": {"style": {"index": "no", "type": "string"}, "class": {"index": "no", "type": "string"}, "title": {"index": "no", "type": "string"}}}, "c_namespace_alias_ref": {"type": "object", "properties": {"start": {"index": "no", "type": "integer"}, "qualname": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}, "end": {"index": "no", "type": "integer"}, "name": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}}}, "c_overrides": {"type": "object", "properties": {"start": {"index": "no", "type": "integer"}, "qualname": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}, "end": {"index": "no", "type": "integer"}, "name": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}}}, "c_namespace_alias": {"type": "object", "properties": {"start": {"index": "no", "type": "integer"}, "qualname": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}, "end": {"index": "no", "type": "integer"}, "name": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}}}, "c_member": {"type": "object", "properties": {"start": {"index": "no", "type": "integer"}, "qualname": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}, "end": {"index": "no", "type": "integer"}, "name": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}}}, "path": {"index": "not_analyzed", "type": "string", "fields": {"trigrams": {"type": "string", "analyzer": "trigramalyzer"}, "trigrams_lower": {"type": "string", "analyzer": "trigramalyzer_lower"}}}, "c_macro": {"type": "object", "properties": {"start": {"index": "no", "type": "integer"}, "qualname": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}, "end": {"index": "no", "type": "integer"}, "name": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}}}, "c_macro_ref": {"type": "object", "properties": {"start": {"index": "no", "type": "integer"}, "qualname": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}, "end": {"index": "no", "type": "integer"}, "name": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}}}, "c_var": {"type": "object", "properties": {"start": {"index": "no", "type": "integer"}, "qualname": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}, "end": {"index": "no", "type": "integer"}, "name": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}}}, "c_function_ref": {"type": "object", "properties": {"start": {"index": "no", "type": "integer"}, "qualname": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}, "end": {"index": "no", "type": "integer"}, "name": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}}}, "c_warning": {"type": "object", "properties": {"start": {"index": "no", "type": "integer"}, "qualname": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}, "end": {"index": "no", "type": "integer"}, "name": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}}}, "c_var_ref": {"type": "object", "properties": {"start": {"index": "no", "type": "integer"}, "qualname": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}, "end": {"index": "no", "type": "integer"}, "name": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}}}, "c_function": {"type": "object", "properties": {"start": {"index": "no", "type": "integer"}, "qualname": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}, "end": {"index": "no", "type": "integer"}, "name": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}}}, "ext": {"index": "not_analyzed", "type": "string"}, "c_derived": {"type": "object", "properties": {"start": {"index": "no", "type": "integer"}, "qualname": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}, "end": {"index": "no", "type": "integer"}, "name": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}}}, "c_type": {"type": "object", "properties": {"start": {"index": "no", "type": "integer"}, "qualname": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}, "end": {"index": "no", "type": "integer"}, "name": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}}}, "c_var_decl": {"type": "object", "properties": {"start": {"index": "no", "type": "integer"}, "qualname": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}, "end": {"index": "no", "type": "integer"}, "name": {"index": "not_analyzed", "type": "string", "fields": {"lower": {"type": "string", "analyzer": "lowercase"}}}}}}}, "file": {"_all": {"enabled": false}, "properties": {"description": {"index": "no", "type": "string"}, "links": {"type": "object", "properties": {"items": {"type": "object", "properties": {"href": {"index": "no", "type": "string"}, "icon": {"index": "no", "type": "string"}, "title": {"index": "no", "type": "string"}}}, "order": {"index": "no", "type": "integer"}, "heading": {"index": "no", "type": "string"}}}, "file_name": {"index": "not_analyzed", "type": "string", "fields": {"trigrams": {"type": "string", "analyzer": "trigramalyzer"}, "trigrams_lower": {"type": "string", "analyzer": "trigramalyzer_lower"}}}, "modified": {"index": "no", "type": "date"}, "name": {"index": "not_analyzed", "type": "string"}, "ext": {"index": "not_analyzed", "type": "string"}, "link": {"index": "not_analyzed", "type": "string"}, "is_folder": {"type": "boolean"}, "raw_data": {"index": "no", "type": "binary"}, "path": {"index": "not_analyzed", "type": "string", "fields": {"trigrams": {"type": "string", "analyzer": "trigramalyzer"}, "trigrams_lower": {"type": "string", "analyzer": "trigramalyzer_lower"}}}, "folder": {"index": "not_analyzed", "type": "string"}, "is_binary": {"index": "no", "type": "boolean"}, "size": {"index": "no", "type": "integer"}}}}, "settings": {"index": {"number_of_replicas": 0, "number_of_shards": 5}, "analysis": {"analyzer": {"trigramalyzer": {"type": "custom", "tokenizer": "trigram_tokenizer"}, "lowercase": {"filter": ["lowercase"], "type": "custom", "tokenizer": "keyword"}, "trigramalyzer_lower": {"filter": ["lowercase"], "type": "custom", "tokenizer": "trigram_tokenizer"}}, "tokenizer": {"trigram_tokenizer": {"type": "nGram", "max_gram": 3, "min_gram": 3}}}, "refresh_interval": "60s"}} } was sent to elasticsearch and this was returned dxr@c36c644c12f9:~/dxr/dxr/code/NX$ dxr index --verbose Starting tree 'nx1219f'. Traceback (most recent call last): File "/venv/bin/dxr", line 9, in <module> load_entry_point('dxr==2.0', 'console_scripts', 'dxr')() File "/home/dxr/dxr/dxr/cli/__init__.py", line 29, in main return dxr.invoke(ctx) File "/venv/local/lib/python2.7/site-packages/click/core.py", line 991, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/venv/local/lib/python2.7/site-packages/click/core.py", line 837, in invoke return ctx.invoke(self.callback, **ctx.params) File "/venv/local/lib/python2.7/site-packages/click/core.py", line 464, in invoke return callback(*args, **kwargs) File "/home/dxr/dxr/dxr/cli/index.py", line 26, in index index_and_deploy_tree(tree, verbose=verbose) File "/home/dxr/dxr/dxr/build.py", line 61, in index_and_deploy_tree index_name = index_tree(tree, es, verbose=verbose) File "/home/dxr/dxr/dxr/build.py", line 244, in index_tree {}) File "/home/dxr/dxr/dxr/es.py", line 110, in create_index_and_wait es.create_index(index, settings=settings) File "/venv/local/lib/python2.7/site-packages/pyelasticsearch/client.py", line 93, in decorate return func(*args, query_params=query_params, **kwargs) File "/venv/local/lib/python2.7/site-packages/pyelasticsearch/client.py", line 1033, in create_index query_params=query_params) File "/venv/local/lib/python2.7/site-packages/pyelasticsearch/client.py", line 285, in send_request self._raise_exception(status, error_message) File "/venv/local/lib/python2.7/site-packages/pyelasticsearch/client.py", line 299, in _raise_exception raise error_class(status, error_message) pyelasticsearch.exceptions.ElasticHttpError: (400, u'mapper_parsing_exception') dxr@c36c644c12f9:~/dxr/dxr/code/NX$ Expected results: The elastic search error was [2017-06-08 22:28:34,833][INFO ][node ] [Corona] starting ... [2017-06-08 22:28:35,419][INFO ][transport ] [Corona] publish_address {192.168.1.86:9300}, bound_addresses {[::]:9300} [2017-06-08 22:28:35,424][INFO ][discovery ] [Corona] elasticsearch/eJ52-aK1TMmTdUcLfmRTUw [2017-06-08 22:28:39,465][INFO ][cluster.service ] [Corona] new_master {Corona}{eJ52-aK1TMmTdUcLfmRTUw}{192.168.1.86}{192.168.1.86:9300}, reason: zen-disco-join(elected_as_master, [0] joins received) [2017-06-08 22:28:39,521][INFO ][gateway ] [Corona] recovered [0] indices into cluster_state [2017-06-08 22:28:39,716][INFO ][http ] [Corona] publish_address {192.168.1.86:9200}, bound_addresses {[::]:9200} [2017-06-08 22:28:39,716][INFO ][node ] [Corona] started [2017-06-08 22:30:02,749][DEBUG][action.admin.indices.create] [Corona] [dxr_19_nx1219f_895dfbae-4cbb-11e7-b802-0242ac110002] failed to create MapperParsingException[Failed to parse mapping [line]: Mapping definition for [regions] has unsupported parameters: [payload : {index=no, type=string}] [start : {index=no, type=integer}] [end : {index=no, type=integer}]]; nested: MapperParsingException[Mapping definition for [regions] has unsupported parameters: [payload : {index=no, type=string}] [start : {index=no, type=integer}] [end : {index=no, type=integer}]]; at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:291) at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$1.execute(MetaDataCreateIndexService.java:317) at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45) at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:468) at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772) at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231) at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: MapperParsingException[Mapping definition for [regions] has unsupported parameters: [payload : {index=no, type=string}] [start : {index=no, type=integer}] [end : {index=no, type=integer}]] at org.elasticsearch.index.mapper.DocumentMapperParser.checkNoRemainingFields(DocumentMapperParser.java:171) at org.elasticsearch.index.mapper.DocumentMapperParser.checkNoRemainingFields(DocumentMapperParser.java:165) at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:311) at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:222) at org.elasticsearch.index.mapper.object.RootObjectMapper$TypeParser.parse(RootObjectMapper.java:139) at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:118) at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:99) at org.elasticsearch.index.mapper.MapperService.parse(MapperService.java:549) at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:319) at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:289) ... 9 more [2017-06-08 22:30:40,535][DEBUG][action.admin.indices.create] [Corona] [dxr_19_nx1219f_a006d510-4cbb-11e7-95c8-0242ac110002] failed to create MapperParsingException[Failed to parse mapping [line]: Mapping definition for [regions] has unsupported parameters: [payload : {index=no, type=string}] [start : {index=no, type=integer}] [end : {index=no, type=integer}]]; nested: MapperParsingException[Mapping definition for [regions] has unsupported parameters: [payload : {index=no, type=string}] [start : {index=no, type=integer}] [end : {index=no, type=integer}]]; at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:291) at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$1.execute(MetaDataCreateIndexService.java:317) at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45) at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:468) at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772) at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231) at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: MapperParsingException[Mapping definition for [regions] has unsupported parameters: [payload : {index=no, type=string}] [start : {index=no, type=integer}] [end : {index=no, type=integer}]] at org.elasticsearch.index.mapper.DocumentMapperParser.checkNoRemainingFields(DocumentMapperParser.java:171) at org.elasticsearch.index.mapper.DocumentMapperParser.checkNoRemainingFields(DocumentMapperParser.java:165) at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:311) at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:222) at org.elasticsearch.index.mapper.object.RootObjectMapper$TypeParser.parse(RootObjectMapper.java:139) at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:118) at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:99) at org.elasticsearch.index.mapper.MapperService.parse(MapperService.java:549) at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:319) at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:289) ... 9 more [2017-06-14 15:59:04,820][DEBUG][action.admin.indices.create] [Corona] [dxr_19_nx1219f_eafd757c-513b-11e7-8125-0242ac110002] failed to create MapperParsingException[Failed to parse mapping [line]: Mapping definition for [regions] has unsupported parameters: [payload : {index=no, type=string}] [start : {index=no, type=integer}] [end : {index=no, type=integer}]]; nested: MapperParsingException[Mapping definition for [regions] has unsupported parameters: [payload : {index=no, type=string}] [start : {index=no, type=integer}] [end : {index=no, type=integer}]]; at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:291) at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$1.execute(MetaDataCreateIndexService.java:317) at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45) at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:468) at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772) at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231) at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: MapperParsingException[Mapping definition for [regions] has unsupported parameters: [payload : {index=no, type=string}] [start : {index=no, type=integer}] [end : {index=no, type=integer}]] at org.elasticsearch.index.mapper.DocumentMapperParser.checkNoRemainingFields(DocumentMapperParser.java:171) at org.elasticsearch.index.mapper.DocumentMapperParser.checkNoRemainingFields(DocumentMapperParser.java:165) at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:311) at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:222) at org.elasticsearch.index.mapper.object.RootObjectMapper$TypeParser.parse(RootObjectMapper.java:139) at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:118) at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:99) at org.elasticsearch.index.mapper.MapperService.parse(MapperService.java:549) at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:319) at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:289) ... 9 more
That spurious final } is real, or is it an artifact of copying and pasting? If the former, that is bad news indeed. But yes, we haven't tried to port DXR to modern elasticsearches yet, though we'd certainly take patches toward it.
Erik, Sorry I ran out of chars to add more info in the bug report. The } is extra. Technically. The JSON passes JSON lint. The JSON parse error is in the "syntax" of the elasticsearch DSL. I am not an elasticsearch person, but google says that elasticsearch parser became more strict in 2.x. Many commands that worked in 1.x failed in 2.x and the same falures hit in 5.x. Is the port on your radar? Our MIS folks push us to 5.4 latest for security and bugs fixes. Thanks Michael
One other note. I used TCPMON in eclipse IDE to capture and forward the JSON. That is what I used to capture the text of the message. Also, the same dxr index --verbose works fine when elasticsearch 1.7.5 use used instead or 2.4.1 SO dxr trunk + elasticsearch 1.7.5 -> indexes correctly dxr trunk + elasticsearch 2.4.1 -> fails with error logged dxr trunk + elasticsearch 5.4.1 -> fails the elasticserach used was changed my modifying dxr.config
(In reply to Michael Schneider from comment #2) > > Is the port on your radar? Our MIS folks push us to 5.4 latest for security and bugs fixes. It's definitely something we're going to need to look at later this year, as we look at moving DXR onto AWS. I don't think we'll be able to use AWS' own ES offering, which means we'll have a bit more flexibility in version choice, but CloudOps will want it one something that's actually supported. :-)
Thanks, Michael. Along with whatever strictness changes ES underwent with 2.x, ES often changes its syntax in subtle ways between versions, so this sort of breakage is expected. DXR is a back-burner project right now, but, of the things to do, supporting newer ESes is one of the top 5 or so. I'll update this bug when there's motion on this issue.
I started looking at this with elasticsearch 5.5 In 2.0 elasticsearch added the restrictions that in a mapping. If a property name appears in multiple types, then they must map to the same type. When dxr creates the initial mappings for the index. Property payload in the mapping JSON (see above) appears in 2 different types, each with a different mapping. The name of one of the properties need to change. In refs (line 130) The payload def is: "payload": { "type": "object", "properties": { "menu_data": { "index": "no", "type": "string" }, "qualname_hash": { "index": "no", "type": "long" }, "hover": { "index": "no", "type": "string" }, "id": { "index": "no", "type": "string" }, "plugin": { "index": "no", "type": "string" } } In regions (line 349) payload has a different mapping "payload": { "index": "no", "type": "string" }, What name should be changed? And what should it be changed to?
I am not an elasticsearch user. So I am climbing up the curve. Many seem to recommend one type per index. {index_name}_file {index_name}_line That would allow the underling lucene index to be less sparse. would that be appropriate? Michael
Looks like in 5.x support for payload was removed from elasticsearch https://github.com/elastic/elasticsearch-rails/issues/690
When I dropped payload from refs and modified the format of refs "refs": { "properties": { "start": { "index": "no", "type": "integer" }, "end": { "index": "no", "type": "integer" } } I got #! Deprecation: The [string] field is deprecated, please use [text] or [keyword] instead on [ext] #! Deprecation: The [string] field is deprecated, please use [text] or [keyword] instead on [file_name] #! Deprecation: The [string] field is deprecated, please use [text] or [keyword] instead on [trigrams_lower] #! Deprecation: The [string] field is deprecated, please use [text] or [keyword] instead on [trigrams] #! Deprecation: The [string] field is deprecated, please use [text] or [keyword] instead on [link] #! Deprecation: Expected a boolean [true/false] for property [index] but got [no] #! Deprecation: The [string] field is deprecated, please use [text] or [keyword] instead on [description] #! Deprecation: The [string] field is deprecated, please use [text] or [keyword] instead on [path] #! Deprecation: The [string] field is deprecated, please use [text] or [keyword] instead on [folder] #! Deprecation: The [string] field is deprecated, please use [text] or [keyword] instead on [name] #! Deprecation: The [string] field is deprecated, please use [text] or [keyword] instead on [heading] #! Deprecation: The [string] field is deprecated, please use [text] or [keyword] instead on [icon] #! Deprecation: The [string] field is deprecated, please use [text] or [keyword] instead on [href] #! Deprecation: The [string] field is deprecated, please use [text] or [keyword] instead on [title] #! Deprecation: The [string] field is deprecated, please use [text] or [keyword] instead on [qualname] #! Deprecation: The [string] field is deprecated, please use [text] or [keyword] instead on [lower] #! Deprecation: The [string] field is deprecated, please use [text] or [keyword] instead on [line_payload] #! Deprecation: The [string] field is deprecated, please use [text] or [keyword] instead on [style] #! Deprecation: The [string] field is deprecated, please use [text] or [keyword] instead on [class] #! Deprecation: The [string] field is deprecated, please use [text] or [keyword] instead on [content] { "acknowledged": true, "shards_acknowledged": true }
So a dropped payload seems to be the big difference to upgrade to 5.5 (at least from the mapping) What are your thoughts? Have any suggestions on the best way to proceed? Thanks, Michael
I think you've got a point there. Bad Things can happen if types differ within a single name. We should fix that. The one-type-per-index thing is new to me and probably postdates the version of ES I was using. It doesn't seem very feasible to me, since how would you sub out a whole group of indices atomically? Currently, we use aliases to do a single one; that's how we do updates.
Don't confuse the payload features of ES with a used-defined property that simply happens to be named "payload" in DXR. :-)
user-defined, that is. Regarding strings, it sounds like ES has deprecated those and replaced them with text fields and keywords.
(In reply to Erik Rose [:erik][:erikrose] from comment #11) > I think you've got a point there. Bad Things can happen if types differ > within a single name. We should fix that. > > The one-type-per-index thing is new to me and probably postdates the version > of ES I was using. It doesn't seem very feasible to me, since how would you > sub out a whole group of indices atomically? Currently, we use aliases to do > a single one; that's how we do updates. One type per index is optional in 5.0 so it can stay. In the future, they are going to change that. I have no problem (and elasticserach has no problem in 5.5) keeping the two types in the same index.
(In reply to Erik Rose [:erik][:erikrose] from comment #13) > user-defined, that is. > > Regarding strings, it sounds like ES has deprecated those and replaced them > with text fields and keywords. Yes, I am not familiar with all uses of each string type in the mapping. If someone in the know would change the string types to map or keyword it would be great. It should just take a couple of minutes. I could guess, but I might miss some use cases.
(In reply to Erik Rose [:erik][:erikrose] from comment #12) > Don't confuse the payload features of ES with a used-defined property that > simply happens to be named "payload" in DXR. :-) That is a good observation Erik. When I first saw the payload property, I googled payload and the elasticsearch docs said the functionality was dropped in 5.x. When I read the usage deeper, I noticed it the feature was not being leveraged in the mapping., In elasticsearch 2.0, they made a strict requirement that all properties with the same name in the mapping seciton have to have the same type (or mapping). If the two platform attributes had the same type mapping, then the property could be reused between types. Since the mappings of payload are different between the two types, elasticsearch now throws a mapping error, and flags the property as invalid. I am not sure which property is used more, maybe you could change the name of the other other. If that is too much work for you, just let me know which one to change, and what to change it to and I can take a stab at it. Michael
Like most properties, they're not part of the public API, so you can change either of them. I'd change both--maybe ref_payload and region_payload--so as not to unduly favor one. I'd entertain a PR for such a change, though it would have to update the format version (http://dxr.readthedocs.io/en/latest/glossary.html?highlight=format%20version).
Erik, Thank you, I added bug https://bugzilla.mozilla.org/show_bug.cgi?id=1383248 with your suggestion for the new names Michael

DXR is no longer available. Searchfox is now replacing it.
See meta bug 1669906 & https://groups.google.com/g/mozilla.dev.platform/c/jDRjrq3l-CY for more details.

Status: UNCONFIRMED → RESOLVED
Closed: 5 years ago
Resolution: --- → WONTFIX
Product: Webtools → Webtools Graveyard
You need to log in before you can comment on or make changes to this bug.