1205049 - django.db.utils:IntegrityError: (1062, "Duplicate entry 'f202d1847711c58fcb27563abeaffc8dca01b203-1528' for key 'failure_line_job_guid_7d547076_uniq'")

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Data Ingestion, defect, P1)

Description

[Ed Morley [:emorley]]

https://rpm.newrelic.com/accounts/677903/applications/4180461/traced_errors/4eda72-5d38001b-5beb-11e5-97ea-b82a72d22a14
https://rpm.newrelic.com/accounts/677903/applications/4180461/traced_errors/4f4e53-f89edd78-5be9-11e5-97ea-b82a72d22a14

Traceback (most recent call last):
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/bin/celery.py", line 769, in execute_from_commandline
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/bin/base.py", line 311, in execute_from_commandline
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/bin/celery.py", line 761, in handle_argv
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/bin/celery.py", line 693, in execute
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/bin/worker.py", line 179, in run_from_argv
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/bin/base.py", line 274, in __call__
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/bin/worker.py", line 212, in run
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/worker/__init__.py", line 206, in start
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/bootsteps.py", line 374, in start
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/worker/consumer.py", line 278, in start
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/worker/consumer.py", line 821, in start
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/worker/loops.py", line 70, in asynloop
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/kombu/async/hub.py", line 324, in create_loop
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/concurrency/asynpool.py", line 413, in _event_process_exit
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/billiard/pool.py", line 1256, in maintain_pool
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/billiard/pool.py", line 1248, in _maintain_pool
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/billiard/pool.py", line 1233, in _repopulate_pool
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/billiard/pool.py", line 1064, in _create_worker_process
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/billiard/process.py", line 137, in start
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/billiard/forking.py", line 105, in __init__
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/billiard/process.py", line 292, in _bootstrap
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/billiard/pool.py", line 291, in run
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/billiard/pool.py", line 363, in workloop
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/app/trace.py", line 349, in _fast_trace_task
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/newrelic-2.54.0.41/newrelic/hooks/application_celery.py", line 66, in wrapper
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
File "/data/www/treeherder.mozilla.org/treeherder-service/treeherder/log_parser/tasks.py", line 61, in store_error_summary
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/app/task.py", line 664, in retry
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/utils/__init__.py", line 242, in maybe_reraise
File "/data/www/treeherder.mozilla.org/treeherder-service/treeherder/log_parser/tasks.py", line 59, in store_error_summary
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 120, in call_command
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/core/management/base.py", line 444, in execute
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/newrelic-2.54.0.41/newrelic/api/function_trace.py", line 110, in literal_wrapper
File "/data/www/treeherder.mozilla.org/treeherder-service/treeherder/log_parser/management/commands/store_error_summary.py", line 49, in handle
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/models/query.py", line 392, in bulk_create
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/models/query.py", line 937, in _batched_insert
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/models/query.py", line 920, in _insert
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 974, in execute_sql
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/utils.py", line 97, in __exit__
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 124, in execute
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/newrelic-2.54.0.41/newrelic/hooks/database_dbapi2.py", line 22, in execute
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler

 django.db.utils:IntegrityError: (1062, "Duplicate entry 'f202d1847711c58fcb27563abeaffc8dca01b203-1528' for key 'failure_line_job_guid_7d547076_uniq'")

Comment 1

[Mauro Doglio [:mdoglio]]

I'll submit a patch briefly

Comment 2

[Mauro Doglio [:mdoglio]]

Attachment: PR 966

Comment 3

[Ed Morley [:emorley]]

Comment on attachment 8661725 [details] [review]
PR 966

Looks good - thank you :-)

Comment 4

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/a834fcbee3deb55d4a61e76d7594bea4f9683838
Bug 1205049 - make FailureLine bulk creation atomic

This should prevent occurences of IntegrityError happening when something goes wrong
on first insertion and the task is retried.

Comment 5

[Ed Morley [:emorley]]

Seeing similar on prod now, after the deploy:

https://rpm.newrelic.com/accounts/677903/applications/4180461/traced_errors/4eda72-52937b33-67bd-11e5-97ea-b82a72d22a14

Retry in 240s: IntegrityError(1062, "Duplicate entry '9e8caec37d22f32513c53d61587a519004876f41-9443' for key 'failure_line_job_guid_7d547076_uniq'")

Comment 6

[Ed Morley [:emorley]]

This is still occurring:
https://rpm.newrelic.com/accounts/677903/applications/5585473/traced_errors/556198-9a895ca7-6c4d-11e5-97ea-b82a72d22a14

Comment 7

[Mauro Doglio [:mdoglio]]

It looks like in some cases we have multiple `*_error_summary` logs for the same job. Because of this sometimes we have a collision between failure lines belonging to the same job on different logs with the same line number.

Comment 8

[Mauro Doglio [:mdoglio]]

Here is an example of such a job: https://treeherder.allizom.org/api/project/try/jobs/13759185/
And here you can see we have 2 error_summary logs (a11y_errorsummary.log and chrome_errorsummary.log) for that job: https://treeherder.allizom.org/api/project/try/artifact/61479122/

Comment 9

[Mauro Doglio [:mdoglio]]

I cannot work on this any time soon, feel free to take it.

Comment 10

[Ed Morley [:emorley]]

I've broken the second part of this out to bug 1233164.