1342482 - Elasticsearch breaks indexing straightfoward GCC repo

Status: RESOLVED INVALID

(Webtools Graveyard :: DXR, defect)

Description

[Paulo Matos]

User Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0
Build ID: 20161213225203

Steps to reproduce:

On Linux (Fedora 25)

* Get DXR HEAD from Github;
* make shell
* make test 
Ran 538 tests in 175.057s

OK (SKIP=8)
* create dxr.config:
[DXR]
enabled_plugins=buglink clang pygmentize

[gcc]
source_folder=/code/gcc
object_folder=/code/gcc-build
build_command=
clean_command=

    [[buglink]]
    name = GCC Bugzilla
    url = https://gcc.gnu.org/bugzilla/

* go to /code and 
svn checkout svn://gcc.gnu.org/svn/gcc/trunk gcc

* go to dxr folder
* dxr index


Actual results:

dxr@4f706be47625:~/dxr$ dxr index
Starting tree 'gcc'.
/home/dxr/dxr/dxr/vcs.py:256: UserWarning: Your git remote is not supported yet. Please use a GitHub remote if you would like version control navigation links to show.
  warn("Your git remote is not supported yet. Please use a "
Running pre_build  [####################################] 100%
Running post_build [####################################] 100%
Indexing folders   [####################################]
Indexing files     [#-----------------------------------]   3%A worker failed while indexing /code/gcc/libatomic/configure:
Traceback (most recent call last):
  File "/home/dxr/dxr/dxr/build.py", line 577, in index_chunk
    index_file(tree, tree_indexers, path, es, index)
  File "/home/dxr/dxr/dxr/build.py", line 548, in index_file
    es.bulk(chunk, index=index, doc_type=LINE)
  File "/venv/local/lib/python2.7/site-packages/pyelasticsearch/client.py", line 93, in decorate
    return func(*args, query_params=query_params, **kwargs)
  File "/venv/local/lib/python2.7/site-packages/pyelasticsearch/client.py", line 460, in bulk
    raise BulkError(errors, successes)
BulkError: ([{u'create': {u'status': 429, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrJi', u'error': u'EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@37ecd690]', u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 429, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrJ_', u'error': u'EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@37ecd690]', u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 429, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrKC', u'error': u'EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@37ecd690]', u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 429, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrKk', u'error': u'EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@37ecd690]', u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 429, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrKu', u'error': u'EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@37ecd690]', u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 429, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrK9', u'error': u'EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@37ecd690]', u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 429, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrLJ', u'error': u'EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@37ecd690]', u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 429, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrLT', u'error': u'EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@37ecd690]', u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 429, u'_type': u'line', u'_id': u'AVpxExAI_UVcL34DLrLe', u'error': u'EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@37ecd690]', u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}], [{u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrJV', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrJa', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrJe', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrJm', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrJp', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrJr', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrJv', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrJz', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrJ1', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrJ5', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrJ8', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrKG', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrKK', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrKN', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrKQ', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrKT', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrKX', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrKZ', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrKd', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrKg', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrKn', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrKr', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrKx', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrK0', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrK3', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrK6', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrK_', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrLB', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrLD', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrLF', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrLH', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrLL', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrLN', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}
, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrLP', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrLR', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrLU', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrLV', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrLW', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAI_UVcL34DLrLX', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAI_UVcL34DLrLY', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAI_UVcL34DLrLZ', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAI_UVcL34DLrLa', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAI_UVcL34DLrLb', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAI_UVcL34DLrLc', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAI_UVcL34DLrLd', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAI_UVcL34DLrLf', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAI_UVcL34DLrLg', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAI_UVcL34DLrLh', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}])


Traceback (most recent call last):
  File "/venv/bin/dxr", line 9, in <module>
    load_entry_point('dxr==2.0', 'console_scripts', 'dxr')()
  File "/home/dxr/dxr/dxr/cli/__init__.py", line 29, in main
    return dxr.invoke(ctx)
  File "/venv/local/lib/python2.7/site-packages/click/core.py", line 991, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/venv/local/lib/python2.7/site-packages/click/core.py", line 837, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/venv/local/lib/python2.7/site-packages/click/core.py", line 464, in invoke
    return callback(*args, **kwargs)
  File "/home/dxr/dxr/dxr/cli/index.py", line 26, in index
    index_and_deploy_tree(tree, verbose=verbose)
  File "/home/dxr/dxr/dxr/build.py", line 61, in index_and_deploy_tree
    index_name = index_tree(tree, es, verbose=verbose)
  File "/home/dxr/dxr/dxr/build.py", line 265, in index_tree
    index_files(tree, tree_indexers, index, pool, es)
  File "/home/dxr/dxr/dxr/build.py", line 640, in index_files
    raise type, value  # exits with non-zero
pyelasticsearch.exceptions.BulkError: ([{u'create': {u'status': 429, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrJi', u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002', u'error': u'EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@37ecd690]'}}, {u'create': {u'status': 429, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrJ_', u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002', u'error': u'EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@37ecd690]'}}, {u'create': {u'status': 429, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrKC', u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002', u'error': u'EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@37ecd690]'}}, {u'create': {u'status': 429, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrKk', u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002', u'error': u'EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@37ecd690]'}}, {u'create': {u'status': 429, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrKu', u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002', u'error': u'EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@37ecd690]'}}, {u'create': {u'status': 429, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrK9', u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002', u'error': u'EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@37ecd690]'}}, {u'create': {u'status': 429, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrLJ', u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002', u'error': u'EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@37ecd690]'}}, {u'create': {u'status': 429, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrLT', u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002', u'error': u'EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@37ecd690]'}}, {u'create': {u'status': 429, u'_type': u'line', u'_id': u'AVpxExAI_UVcL34DLrLe', u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002', u'error': u'EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@37ecd690]'}}], [{u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrJV', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrJa', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrJe', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrJm', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrJp', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrJr', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrJv', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrJz', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrJ1', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrJ5', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrJ8', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrKG', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrKK', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrKN', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrKQ', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_
UVcL34DLrKT', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrKX', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrKZ', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrKd', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrKg', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrKn', u'_version': 1, u'_inex': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_tye': u'line', u'_id': u'AVpxExAH_UVcL34DLrKr', u'_version': 1, u'_index': u'dxr_19_gcc_f547952-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVxExAH_UVcL34DLrKx', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac11002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrK0', u'_vesion': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'staus': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrK3', u'_version': 1, u'_index': u'dx_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'lin', u'_id': u'AVpxExAH_UVcL34DLrK6', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVc34DLrK_', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {ucreate': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrLB', u'_version': 1,u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201,u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrLD', u'_version': 1, u'_index': u'dxr_19_gcc_f479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id' u'AVpxExAH_UVcL34DLrLF', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-024ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrLH',u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrLL', u'_version': 1, u'_index' u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type':u'line', u'_id': u'AVpxExAH_UVcL34DLrLN', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fa2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExH_UVcL34DLrLP', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrLR', u'_versio': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status' 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DLrLU', u'_version': 1, u'_index': u'dxr_19gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', '_id': u'AVpxExAH_UVcL34DLrLV', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab1-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAH_UVcL34DrLW', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'crete': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAI_UVcL34DLrLX', u'_version': 1, u'_ndex': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_ype': u'line', u'_id': u'AVpxExAI_UVcL34DLrLY', u'_version': 1, u'_index': u'dxr_19_gcc_f547924-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'VpxExAI_UVcL34DLrLZ', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac10002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAI_UVcL34DLrLa', u'_ersion': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'satus': 201, u'_type': u'line', u'_id': u'AVpxExAI_UVcL34DLrLb', u'_version': 1, u'_index': u'xr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'lne', u'_id': u'AVpxExAI_UVcL34DLrLc', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-1e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAI_UcL34DLrLd', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, u'create': {u'status': 201, u'_type': u'line', u'_id': u'AVpxExAI_UVcL34DLrLf', u'_version': , u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 20, u'_type': u'line', u'_id': u'AVpxExAI_UVcL34DLrLg', u'_version': 1, u'_index': u'dxr_19_gccf5479524-fab2-11e6-ab01-0242ac110002'}}, {u'create': {u'status': 201, u'_type': u'line', u'_i': u'AVpxExAI_UVcL34DLrLh', u'_version': 1, u'_index': u'dxr_19_gcc_f5479524-fab2-11e6-ab01-042ac110002'}}])
dxr@4f706be47625:~/dxr$ echo $?
1


Expected results:

Expected GCC to have been correctly indexed, return value 0 from dxr index.

Comment 1

[Erik Rose [:erik][:erikrose]]

Looks like ES is falling behind DXR. ES is configured with a queue of 50 something-or-others by default, and DXR outran that.

The docker setup is configured out of the box to run tests and index small things. For something big like GCC, you'll have to make tweaks and likely allocate more RAM. Have you read https://dxr.readthedocs.io/en/latest/indexing-firefox.html? You'll want to crank up the total and ES-allocated RAM as it explains.

Comment 2

[Paulo Matos]

So, I edited tooling/docker/docker-compose.yml

which now looks like:
$ cat tooling/docker/docker-compose.yml
# A container that runs elasticsearch. The upstream image has a volume defined,
# so indexes persist.
es:
  build: ./es
  environment:
    ES_HEAP_SIZE: 2g
  ports:
    - "8000:8000"

# An interactive container for experimentation and debugging
dev:
  build: ./dev
  net: container:es
  ports:
    - "8000:8000"
  volumes:
    - ../..:/home/dxr/dxr:Z
  volumes_from:
    - code
    - venv

# A Data Volume Container to persist any code you might care to index. On a
# Linux host, you can squirrel away code in the shared "dxr" source checkout,
# but, on a Mac, /code will be more attractive because it offers better IO
# performance than VirtualBox's shared folder implementation.
code:
  build: ./code

# A Data Volume Container so we don't lose our whole virtualenv and need to
# re-download all the Python packages every time the dev image is rebuilt
venv:
  build: ./venv


Then I did make docker_es and make shell. Both successful. 
Did a dxr delete, just in case and dxr index fails again on another file:

dxr@00dc997baf8b:~/dxr$ dxr index
Starting tree 'gcc'.
/home/dxr/dxr/dxr/vcs.py:256: UserWarning: Your git remote is not supported yet. Please use a GitHub remote if you would like version control navigation links to show.
  warn("Your git remote is not supported yet. Please use a "
Running pre_build  [####################################] 100%
Running post_build [####################################] 100%
Indexing folders   [####################################]
Indexing files     [#-----------------------------------]   4%A worker failed while indexing /code/gcc/libgo/go/runtime/stubs.go:
Traceback (most recent call last):
  File "/home/dxr/dxr/dxr/build.py", line 577, in index_chunk
    index_file(tree, tree_indexers, path, es, index)
  File "/home/dxr/dxr/dxr/build.py", line 548, in index_file
    es.bulk(chunk, index=index, doc_type=LINE)
  File "/venv/local/lib/python2.7/site-packages/pyelasticsearch/client.py", line 93, in decorate
    return func(*args, query_params=query_params, **kwargs)
  File "/venv/local/lib/python2.7/site-packages/pyelasticsearch/client.py", line 460, in bulk
    raise BulkError(errors, successes)
BulkError: ([{u'create': {u'status': 429, u'_type': u'line', u'_id': u'AVpxW6P-iVjRassP0Bce',


The remaining gibberish is very similar to the above. Is this a matter of bumping the ES memory to maybe 4g you think?

Comment 3

[Erik Rose [:erik][:erikrose]]

Are you still getting the "EsRejectedExecutionException[rejected execution (queue capacity 50)" among the gibberish? It could be your indexing jobs are just running faster than I'm used to for some reason. DXR is designed to self-throttle, but...wait a minute. You mentioned on IRC that you're on a 16-core machine. Does the docker container running the index job have access to all 16? DXR defaults to 1 worker job per core, which might explain why it's so handily outpacing ES.

So, here are your options:
1. Set workers=4 or something less than the number of cores. That'll slow down indexing but probably keep ES from exploding.
2. Read the ES docs until you see how to increase its queue size past 50. DXR is self-throttling, in that it waits for a response before sending another request, but ES is also keen to work asyncronously, queueing things up and returning early. The crux of it is that your ES in underperforming with respect to your Python, and you need to correct the imbalance one way or another. It's likely that you'll need to either turn more than one ES knob (not just the queue size) or start up an honest-to-goodness cluster of ES nodes.

For just getting your first build up, I recommend #1 because it's definitely attainable within the stock docker environment.

Comment 4

[Paulo Matos]

(In reply to Erik Rose [:erik][:erikrose] from comment #3)
> Are you still getting the "EsRejectedExecutionException[rejected execution
> (queue capacity 50)" among the gibberish? It could be your indexing jobs are
> just running faster than I'm used to for some reason. DXR is designed to
> self-throttle, but...wait a minute. You mentioned on IRC that you're on a
> 16-core machine. 

You're probably correct. I still see "EsRejectedExecutionException[rejected execution (queue capacity 50)" . Also, I noticed that when I do 'dxr index' I get several processes doing the indexing.

> Does the docker container running the index job have access
> to all 16? 

Don't know enough about docker but given I see several processes running the indexing the container probably has access to them all.

> DXR defaults to 1 worker job per core, which might explain why
> it's so handily outpacing ES.
> 
> So, here are your options:
> 1. Set workers=4 or something less than the number of cores. That'll slow
> down indexing but probably keep ES from exploding.

In dxr.config?

> 2. Read the ES docs until you see how to increase its queue size past 50.
> DXR is self-throttling, in that it waits for a response before sending
> another request, but ES is also keen to work asyncronously, queueing things
> up and returning early. The crux of it is that your ES in underperforming
> with respect to your Python, and you need to correct the imbalance one way
> or another. It's likely that you'll need to either turn more than one ES
> knob (not just the queue size) or start up an honest-to-goodness cluster of
> ES nodes.
> 
> For just getting your first build up, I recommend #1 because it's definitely
> attainable within the stock docker environment.

Let me try #1 first and we'll take it from there. Thanks.

Comment 5

[Paulo Matos]

This worked with:
workers=4

Great. We'll catch up in sync. Thanks.
Do you want to close this?

Comment 6

[Paulo Matos]

(In reply to Paulo Matos from comment #5)
> This worked with:
> workers=4
> 
> Great. We'll catch up in sync. Thanks.
> Do you want to close this?

s/sync/#static/

Comment 7

[Erik Rose [:erik][:erikrose]]

Great! So you were just outrunning ES, beyond DXR's ability to self-throttle; it filled up ES's queues with only one bulk call from each of your 16 workers. (Docker containers see the same number of CPUs as the host.)