Intermittent deployment "Deadlock found when trying to get lock" during fixture loading



Tree Management
Treeherder: Infrastructure
13 days ago
4 days ago


(Reporter: emorley, Unassigned)





13 days ago
Twice in a row whilst trying to deploy to production (the third deploy succeeded), the deploy failed during the release phase step, in the ` loaddata` management command, with:

`model.FailureClassification(pk=1): (1213, 'Deadlock found when trying to get lock; try restarting transaction')`

...whilst trying to load `fixtures/failure_classification.json`.

This is presumably due to contention with other queries running at the time.

The annoying thing about this is that the fixture file hadn't actually changed, so nothing needed updating. Perhaps we should refactor our use of fixtures to only update things that have changed?

Log excerpt:

-----> PRE-DEPLOY: Loading initial data...
Traceback (most recent call last):
  File "/app/treeherder/model/management/commands/", line 14, in handle
  File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/", line 131, in call_command
    return command.execute(*args, **defaults)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/", line 330, in execute
    output = self.handle(*args, **options)
  File "/app/.heroku/python/lib/python2.7/site-packages/newrelic/api/", line 107, in literal_wrapper
    return wrapped(*args, **kwargs)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/commands/", line 69, in handle
  File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/commands/", line 109, in loaddata
  File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/commands/", line 175, in load_label
  File "/app/.heroku/python/lib/python2.7/site-packages/django/core/serializers/", line 205, in save
    models.Model.save_base(self.object, using=using, raw=True, **kwargs)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/", line 838, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/", line 905, in _save_table
  File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/", line 955, in _do_update
    return filtered._update(values) > 0
  File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/", line 664, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/sql/", line 1204, in execute_sql
    cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/sql/", line 899, in execute_sql
    raise original_exception
django.db.utils.OperationalError: Problem installing fixture '/app/treeherder/model/fixtures/failure_classification.json': Could not load model.FailureClassification(pk=1): (1213, 'Deadlock found when trying to get lock; try restarting transaction')

Original logs:
You need to log in before you can comment on or make changes to this bug.