1924883 - Treeherder fails to deploy after latest database change: psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "django_content_type_pkey"

Status: NEW

(Tree Management :: Perfherder, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Since bug 1919588 got merged, migration stage after new commits get merge fails. It sounds as one of the unique_together rules from migration #53 is not unique.

This blocks deployment, please prioritize it.

2024-10-09 12:55:17,937 INFO     -----> PRE-DEPLOY: Running Django migration...
2024-10-09 12:55:06,188 (7/MainThread) newrelic.core.agent INFO - New Relic Python Agent (9.13.0)
2024-10-09 12:55:06,735 (7/NR-Activate-Session/treeherder-stage) newrelic.common.utilization WARNING - Invalid 'gcp' data ('machineType'): None
2024-10-09 12:55:07,518 (7/NR-Activate-Session/treeherder-stage) newrelic.core.agent_protocol INFO - Reporting to: https://rpm.newrelic.com/accounts/677903/applications/14179733
System check identified some issues:

WARNINGS:
?: (dockerflow.health.W001) Unapplied migration perf.0053_performancealertsummarytesting_and_more
?: (dockerflow.health.W001) Unapplied migration perf.0054_performancealert_confidence_and_more
Operations to perform:
  Apply all migrations: auth, changelog, contenttypes, model, perf
Running migrations:
  Applying perf.0053_performancealertsummarytesting_and_more... OK
  Applying perf.0054_performancealert_confidence_and_more... OK
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 105, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.10/site-packages/newrelic/hooks/database_psycopg2.py", line 61, in execute
    return super(CursorWrapper, self).execute(sql, parameters, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/newrelic/hooks/database_dbapi2.py", line 42, in execute
    return self.__wrapped__.execute(sql, parameters, *args, **kwargs)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "django_content_type_pkey"
DETAIL:  Key (id)=(1) already exists.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/./manage.py", line 16, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.10/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.10/site-packages/newrelic/hooks/framework_django.py", line 948, in _nr_wrapper_BaseCommand_run_from_argv_
    return BackgroundTaskWrapper(wrapped, application, subcommand, "Django")(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/newrelic/api/background_task.py", line 111, in wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 413, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 459, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.10/site-packages/newrelic/api/function_trace.py", line 166, in literal_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 107, in wrapper
    res = handle_func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/core/management/commands/migrate.py", line 384, in handle
    emit_post_migrate_signal(
  File "/usr/local/lib/python3.10/site-packages/django/core/management/sql.py", line 52, in emit_post_migrate_signal
    models.signals.post_migrate.send(
  File "/usr/local/lib/python3.10/site-packages/django/dispatch/dispatcher.py", line 189, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/usr/local/lib/python3.10/site-packages/django/contrib/auth/management/__init__.py", line 59, in create_permissions
    create_contenttypes(
  File "/usr/local/lib/python3.10/site-packages/django/contrib/contenttypes/management/__init__.py", line 142, in create_contenttypes
    ContentType.objects.using(using).bulk_create(cts)
  File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 835, in bulk_create
    returned_columns = self._batched_insert(
  File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 1875, in _batched_insert
    self._insert(
  File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 1847, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/usr/local/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1836, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 79, in execute
    return self._execute_with_wrappers(
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 92, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 100, in _execute
    with self.db.wrap_database_errors:
  File "/usr/local/lib/python3.10/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 105, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.10/site-packages/newrelic/hooks/database_psycopg2.py", line 61, in execute
    return super(CursorWrapper, self).execute(sql, parameters, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/newrelic/hooks/database_dbapi2.py", line 42, in execute
    return self.__wrapped__.execute(sql, parameters, *args, **kwargs)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "django_content_type_pkey"
DETAIL:  Key (id)=(1) already exists.

2024-10-09 12:55:08,522 (7/MainThread) newrelic.core.agent INFO - New Relic Python Agent Shutdown

2024-10-09 12:55:17,937 INFO     Deleting job migrate-stage-treeherder-app-1-migrate-1022 in namespace stage-treeherder
2024-10-09 12:55:17,937 INFO     Running: kubectl delete job -n stage-treeherder migrate-stage-treeherder-app-1-migrate-1022
2024-10-09 12:55:18,059 INFO     Running: gcloud -q config set account [e-mail]

2024-10-09 12:55:18,995 INFO     Running: gcloud -q iam service-accounts keys delete deb15fa95fa021082e61be13341ace1a2827147e --iam-account [e-mail]
2024-10-09 12:55:20,332 INFO     Running: gcloud -q auth revoke [e-mail]
Searching /tmp/tmpdrqrokhu/projects/treeherder/k8s/values
Found file prototype.yaml
Found file telegraf-stackdriver-prod.yaml
Found file prod.yaml
Found file rabbit-telegraf.yaml
Found file tc-staging.yaml
Found file stage.yaml
Searching /tmp/tmpdrqrokhu/common/k8s/values
Found file telegraf.yaml
Found file common.yaml
Searching /home/jenkins/slave/workspace/gcp-pipelines/treeherder/treeherder-production/projects/treeherder/k8s/values
Found file telegraf.yaml
Found file app.yaml
Searching /home/jenkins/slave/workspace/gcp-pipelines/treeherder/treeherder-production/common/k8s/values
Found file telegraf.yaml
Found file environments.yaml
Traceback (most recent call last):
  File "/home/jenkins/slave/workspace/gcp-pipelines/treeherder/treeherder-production/./projects/treeherder/tasks", line 56, in <module>
    cli.group()
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/jenkins/slave/workspace/gcp-pipelines/treeherder/treeherder-production/./projects/treeherder/tasks", line 41, in migrate
    kube.watch_job(change.namespace, change.name)
  File "/usr/local/lib/python3.9/site-packages/deploylib/kube.py", line 155, in watch_job
    raise KubeJobError(
deploylib.kube.KubeJobError: job.batch/migrate-stage-treeherder-app-1-migrate-1022 failed in stage-treeherder. BackoffLimitExceeded: Job has reached the specified backoff limit

Comment 1

[Greg Mierzwinski [:sparky]]

I'm looking into it.

Comment 2

[Greg Mierzwinski [:sparky]]

:aryx, can you revert the PR? I'm currently unable to reproduce this issue locally so it's going to take some time to resolve it.

Comment 3

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Okay, another migration incoming.

Comment 4

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

As mentioned in bug 1919588, the change got reverted. Please close this bug if the investigation continues over there.