1284429 - elasticsearch.helpers:BulkIndexError exception during cycle-data task

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P1)

Description

[Ed Morley [:emorley]]

Have only seen this on the Heroku prototype. However failures during cycle_data have bitten us pretty badly in the past (in the form of filling up disk without realising it), so it would good to resolve this :-)

https://rpm.newrelic.com/accounts/677903/applications/7385291/traced_errors/70b0f6-2e6c4412-427e-11e6-bd66-b82a72d22a14

Traceback (most recent call last):
...
File "/app/treeherder/model/tasks.py", line 49, in cycle_data
File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/__init__.py", line 120, in call_command
File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/base.py", line 445, in execute
File "/app/.heroku/python/lib/python2.7/site-packages/newrelic-2.66.0.49/newrelic/api/function_trace.py", line 110, in literal_wrapper
File "/app/treeherder/model/management/commands/cycle_data.py", line 68, in handle
File "/app/treeherder/model/derived/jobs.py", line 755, in cycle_data
File "/app/treeherder/model/search.py", line 115, in inner
File "/app/treeherder/model/search.py", line 153, in bulk_delete
File "/app/.heroku/python/lib/python2.7/site-packages/elasticsearch/helpers/__init__.py", line 188, in bulk
File "/app/.heroku/python/lib/python2.7/site-packages/elasticsearch/helpers/__init__.py", line 160, in streaming_bulk
File "/app/.heroku/python/lib/python2.7/site-packages/elasticsearch/helpers/__init__.py", line 132, in _process_bulk_chunk
elasticsearch.helpers:BulkIndexError: (u'61 document(s) failed to index.', [{u'delete': {u'_type': u'test_failure_line', u'_shards': {u'total': 2, u'successful': 1, u'failed': 0}, u'found': False, u'_index': u'test-failure-line', u'_version': 1, u'status': 404, u'_id': u'1693762'}}, {u'delete': {u'_type': u'test_failure_line', u'_shards': {u'total': 2, u'successful': 1, u'failed': 0}, u'found': False, u'_index': u'test-failure-line', u'_version': 1, u'status': 404, u'_id': u'1693763'}}, {u'delete': {u'_type': u'test_failure_line', u'_shards': {u'total': 2, u'successful': 1, u'failed': 0}, u'found': False, u'_index': u'test-failure-line', u'_version': 1, u'status': 404, u'_id': u'1693821'}}, {u'delete': {u'_type': u'test_failure_line', u'_shards': {u'total': 2, u'successful': 1, u'failed': 0}, u'found': False, u'_index': u'test-failure-line', u'_version': 1, u'status': 404, u'_id': u'1693822'}}, {u'delete': {u'_type': u'test_failure_line', u'_shards': {u'total': 2, u'successful': 1, u'failed': 0}, u'found': False, u'_index': u'test-failure-line', u'_version': 1, u'status': 404, u'_id': u'1693823'}}, {u'delete': {u'_type': u'test_failure_line', u'_shards': {u'total': 2, u'successful': 1, u'failed': 0}, u'found': False, u'_index': u'test-failure-line', u'_version': 1, u'status': 404, u'_id': u'1693824'}}, {u'delete': {u'_type': u'test_failure_line', u'_shards': {u'total': 2, u'successful': 1, u'failed': 0}, u'found': False, u'_index': u'test-failure-line', u'_version': 1, u'status': 404, u'_id': u'1693826'}}, {u'delete': {u'_type': u'test_failure_line', u'_shards': {u'total': 2, u'successful': 1, u'failed': 0}, u'found': False, u'_index': u'test-failure-line', u'_version': 1, u'status': 404, u'_id': u'1693828'}}, {u'delete': {u'_type': u'test_failure_line', u'_shards': {u'total': 2, u'successful': 1, u'failed': 0}, u'found': False, u'_index': u'test-failure-line', u'_version': 1, u'status': 404, u'_id': u'1693830'}}, {u'delete': {u'_type': u'test_failure_line', u'_shards': {u'total': 2, u'successful': 1, u'failed': 0}, u'found': False, u'_index': u'test-failure-line', u'_version': 1, u'status': 404, u'_id': u'1693832'}}, {u'delete': {u'_type': u'test_failure_line', u'_shards': {u'total': 2, u'successful': 1, u'failed': 0}, u'found': False, u'_index': u'test-failure-line', u'_version': 1, u'status': 404, u'_id': u'1693834'}}, {u'delete': {u'_type': u'test_failure_line', u'_shards': {u'total': 2, u'successful': 1, u'failed': 0}, u'found': False, u'_index': u'test-failure-line', u'_version': 1, u'status': 404, u'_id': u'1693836'}}, {u'delete': {u'_type': u'test_failure_line', u'_shards': {u'total': 2, u'successful': 1, u'failed': 0}, u'found': False, u'_index': u'test-failure-line', u'_version': 1, u'status': 404, u'_id': u'1693838'}}, {u'delete': {u'_type': u'test_failure_line', u'_shards': {u'total': 2, u'successful': 1, u'failed': 0}, u'found': False, u'_index': u'test-failure-line', u'_version': 1, u'status': 404, u'_id': u'1693840'}}, {u'delete': {u'_type': u'test_failure_line', u'_shards': {u'total': 2, u'successful': 1, u'failed': 0}, u'found': False, u'_index': u'test-failure-line', u'_version': 1, u'status': 404, u'_id': u'1693842'}}, {u'delete': {u'_type': u'test_failure_line', u'_shards': {u'total': 2, u'successful': 1, u'failed': 0}, u'found': False, u'_index': u'test-failure-line', u'_version': 1, u'status': 404, u'_id': u'1693844'}}, {u'delete': {u'_type': u'test_failure_line', u'_shards': {u'total': 2, u'successful': 1, u'failed': 0}, u'found': False, u'_index': u'test-failure-line', u'_version': 1, u'status': 404, u'_id': u'1693846'}}, {u'delete': {u'_type': u'test_failure_line', u'_shards': {u'total': 2, u'successful': 1, u'failed': 0}, u'found': False, u'_index': u'test-failure-line', u'_version': 1, u'status': 404, u'_id': u'1693848'}}, {u'delete': {u'_type': u'test_failure_line', u'_shards': {u'total': 2, u'successful': 1, u'failed': 0}, u'found': False, u'_index': u'test-failure-line', u'_version': 1, u'status': 404, u'_id': u'1693850'}}, {u'delete': {u'_type': u'test_failure_line', u'_shards': {u'total': 2, u'successful': 1, u'failed': 0}, u'

Comment 1

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:bulk_delete_error > mozilla:master

Comment 2

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/e5a2646e66c17d571deb21006a04160a46305e9a
Bug 1284429 - Ignore BulkIndexError when trying to bulk delete from ElasticSearch (#1681)

This is likely just deleting things that were not indexed in the first place.

Comment 3

[Ed Morley [:emorley]]

James, don't forget to close your bugs :-)