1240042 - New Relic exception on prod "Duplicate entry '1849-50719' for key 'performance_alert_summary_id_438059dd79175615_uniq'"

Status: RESOLVED FIXED

(Tree Management :: Perfherder, defect, P1)

Description

[Ed Morley [:emorley]]

https://rpm.newrelic.com/accounts/677903/applications/4180461/traced_errors/52946a-625a00b3-bb82-11e5-bde2-b82a72d22a14

Traceback (most recent call last):
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/__main__.py", line 30, in main
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/bin/celery.py", line 81, in main
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/bin/celery.py", line 770, in execute_from_commandline
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/bin/base.py", line 311, in execute_from_commandline
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/bin/celery.py", line 762, in handle_argv
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/bin/celery.py", line 694, in execute
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/bin/worker.py", line 179, in run_from_argv
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/bin/base.py", line 274, in __call__
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/bin/worker.py", line 212, in run
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/worker/__init__.py", line 206, in start
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/bootsteps.py", line 374, in start
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/worker/consumer.py", line 278, in start
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/worker/consumer.py", line 821, in start
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/worker/loops.py", line 76, in asynloop
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/kombu/async/hub.py", line 340, in create_loop
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/concurrency/asynpool.py", line 413, in _event_process_exit
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/billiard/pool.py", line 1260, in maintain_pool
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/billiard/pool.py", line 1252, in _maintain_pool
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/billiard/pool.py", line 1237, in _repopulate_pool
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/billiard/pool.py", line 1068, in _create_worker_process
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/billiard/process.py", line 137, in start
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/billiard/forking.py", line 105, in __init__
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/billiard/process.py", line 292, in _bootstrap
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/billiard/pool.py", line 295, in run
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/billiard/pool.py", line 367, in workloop
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/app/trace.py", line 349, in _fast_trace_task
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/newrelic-2.60.0.46/newrelic/hooks/application_celery.py", line 66, in wrapper
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
File "/data/www/treeherder.mozilla.org/treeherder-service/treeherder/perf/tasks.py", line 10, in generate_alerts
File "/data/www/treeherder.mozilla.org/treeherder-service/treeherder/perf/alerts.py", line 88, in generate_new_alerts_in_series
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/models/query.py", line 348, in create
File "/data/www/treeherder.mozilla.org/treeherder-service/treeherder/perf/models.py", line 183, in save
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/models/base.py", line 734, in save
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/models/base.py", line 762, in save_base
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/models/base.py", line 846, in _save_table
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/models/base.py", line 885, in _do_insert
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/models/query.py", line 920, in _insert
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 974, in execute_sql
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/utils.py", line 98, in __exit__
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 124, in execute
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/newrelic-2.60.0.46/newrelic/hooks/database_dbapi2.py", line 22, in execute
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
File "/data/www/treeherder.mozilla.org/venv/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
 django.db.utils:IntegrityError: (1062, "Duplicate entry '1849-50719' for key 'performance_alert_summary_id_438059dd79175615_uniq'")

Comment 1

[GitHub Autolander Bot]

Attachment: [treeherder] wlach:1240042 > mozilla:master

Comment 2

[William Lachance (:wlach)]

Comment on attachment 8708514 [details] [review]
[treeherder] wlach:1240042 > mozilla:master

I'm pretty sure this is happening because we occasionally submit the same job information more than once. Here's a simple workaround.

Comment 3

[Ed Morley [:emorley]]

Comment on attachment 8708514 [details] [review]
[treeherder] wlach:1240042 > mozilla:master

Thank you :-)

Do we know why we submit the same job info more than once sometimes? (Is it similar to bug 1233164 where it's due to jobs that have multiple sub-parts - eg in that bug's case mochitest-chrome includes several suites)

Comment 4

[William Lachance (:wlach)]

(In reply to Ed Morley [:emorley] from comment #3)
> Comment on attachment 8708514 [details] [review]
> [treeherder] wlach:1240042 > mozilla:master
> 
> Thank you :-)
> 
> Do we know why we submit the same job info more than once sometimes? (Is it
> similar to bug 1233164 where it's due to jobs that have multiple sub-parts -
> eg in that bug's case mochitest-chrome includes several suites)

I am not entirely positive if I've found all the possible ways that this could happen, but I have observed a race condition in the log parser between a job being marked as "parsed" and the logic for queuing up a new parse job. With the current architecture of treeherder, you basically have to assume that a job's data could be submitted more than once. It's rare, but it happens.

Comment 5

[GitHub Autolander Bot]

Pull request has landed in master: https://github.com/mozilla/treeherder/commit/f25a402c8fe88c07edebca6d33627b925ceb724a

Comment 6

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/3344e68a37db731b45d833422fd2a213cc4a5de9
Bug 1240042 - Don't re-run alert generation if datum already created

If the datum was already created, it means that we submitted the same
job information twice. No need to re-run the alert generation in that
case.