Intermittent deployment "Deadlock found when trying to get lock" during fixture loading

NEW
Unassigned

Status

Tree Management
Treeherder: Infrastructure
P2
normal
13 days ago
4 days ago

People

(Reporter: emorley, Unassigned)

Tracking

Details

(Reporter)

Description

13 days ago
Twice in a row whilst trying to deploy to production (the third deploy succeeded), the deploy failed during the release phase step, in the `manage.py loaddata` management command, with:

`model.FailureClassification(pk=1): (1213, 'Deadlock found when trying to get lock; try restarting transaction')`

...whilst trying to load `fixtures/failure_classification.json`.

This is presumably due to contention with other queries running at the time.

The annoying thing about this is that the fixture file hadn't actually changed, so nothing needed updating. Perhaps we should refactor our use of fixtures to only update things that have changed?


Log excerpt:

-----> PRE-DEPLOY: Loading initial data...
...
Traceback (most recent call last):
  ...
  File "/app/treeherder/model/management/commands/load_initial_data.py", line 14, in handle
    'performance_bug_templates')
  File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/__init__.py", line 131, in call_command
    return command.execute(*args, **defaults)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/app/.heroku/python/lib/python2.7/site-packages/newrelic/api/function_trace.py", line 107, in literal_wrapper
    return wrapped(*args, **kwargs)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/commands/loaddata.py", line 69, in handle
    self.loaddata(fixture_labels)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/commands/loaddata.py", line 109, in loaddata
    self.load_label(fixture_label)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/commands/loaddata.py", line 175, in load_label
    obj.save(using=self.using)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/core/serializers/base.py", line 205, in save
    models.Model.save_base(self.object, using=using, raw=True, **kwargs)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/base.py", line 838, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/base.py", line 905, in _save_table
    forced_update)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/base.py", line 955, in _do_update
    return filtered._update(values) > 0
  File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/query.py", line 664, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 1204, in execute_sql
    cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 899, in execute_sql
    raise original_exception
django.db.utils.OperationalError: Problem installing fixture '/app/treeherder/model/fixtures/failure_classification.json': Could not load model.FailureClassification(pk=1): (1213, 'Deadlock found when trying to get lock; try restarting transaction')

Original logs:
https://dashboard.heroku.com/apps/treeherder-prod/activity/releases/63347985-98cb-4191-a383-570702583a81
https://dashboard.heroku.com/apps/treeherder-prod/activity/releases/62e50b90-d23b-4c50-bd22-f8d760669bc2
You need to log in before you can comment on or make changes to this bug.