1216153 - Heroku needs a |manage.py migrate| to fix "Unknown column 'performance_signature.repository_id' in 'field list'"

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P1)

Description

[Ed Morley [:emorley]]

eg:

 Oct 19 15:14:29 treeherder app/web.1: [2015-10-19 07:14:28,790] ERROR [treeherder.webapp.api.exceptions:26] (1054, "Unknown column 'performance_signature.repository_id' in 'field list'") 
Oct 19 15:14:29 treeherder app/web.1: [2015-10-19 07:14:28,791] ERROR [treeherder.webapp.api.exceptions:27] Traceback (most recent call last): 
Oct 19 15:14:29 treeherder app/web.1:   File "/app/.heroku/python/lib/python2.7/site-packages/rest_framework/views.py", line 463, in dispatch 
Oct 19 15:14:29 treeherder app/web.1:     response = handler(request, *args, **kwargs) 
Oct 19 15:14:29 treeherder app/web.1:   File "/app/treeherder/webapp/api/artifact.py", line 65, in create 
Oct 19 15:14:29 treeherder app/web.1:     artifacts_model.load_job_artifacts(artifacts, job_id_lookup) 
Oct 19 15:14:29 treeherder app/web.1:   File "/app/treeherder/model/derived/artifacts.py", line 183, in load_job_artifacts 
Oct 19 15:14:29 treeherder app/web.1:     performance_artifact_job_id_list, performance_artifact_list) 
Oct 19 15:14:29 treeherder app/web.1:   File "/app/treeherder/model/derived/artifacts.py", line 123, in store_performance_artifact 
Oct 19 15:14:29 treeherder app/web.1:     load_talos_artifacts(self.project, ref_data, job_data, perf_data) 
Oct 19 15:14:29 treeherder app/web.1:   File "/app/treeherder/etl/perf.py", line 258, in load_talos_artifacts 
Oct 19 15:14:29 treeherder app/web.1:     'last_updated': push_timestamp 
Oct 19 15:14:29 treeherder app/web.1:   File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method 
Oct 19 15:14:29 treeherder app/web.1:     return getattr(self.get_queryset(), name)(*args, **kwargs) 
Oct 19 15:14:29 treeherder app/web.1:   File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/query.py", line 405, in get_or_create 
Oct 19 15:14:29 treeherder app/web.1:     return self.get(**lookup), False 
Oct 19 15:14:29 treeherder app/web.1:   File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/query.py", line 328, in get 
Oct 19 15:14:29 treeherder app/web.1:     num = len(clone) 
Oct 19 15:14:29 treeherder app/web.1:   File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/query.py", line 144, in __len__ 
Oct 19 15:14:29 treeherder app/web.1:     self._fetch_all() 
Oct 19 15:14:29 treeherder app/web.1:   File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/query.py", line 965, in _fetch_all 
Oct 19 15:14:29 treeherder app/web.1:     self._result_cache = list(self.iterator()) 
Oct 19 15:14:29 treeherder app/web.1:   File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/query.py", line 238, in iterator 
Oct 19 15:14:29 treeherder app/web.1:     results = compiler.execute_sql() 
Oct 19 15:14:29 treeherder app/web.1:   File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 840, in execute_sql 
Oct 19 15:14:29 treeherder app/web.1:     cursor.execute(sql, params) 
Oct 19 15:14:29 treeherder app/web.1:   File "/app/.heroku/python/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute 
Oct 19 15:14:29 treeherder app/web.1:     return self.cursor.execute(sql, params) 
Oct 19 15:14:29 treeherder app/web.1:   File "/app/.heroku/python/lib/python2.7/site-packages/django/db/utils.py", line 97, in __exit__ 
Oct 19 15:14:29 treeherder app/web.1:     six.reraise(dj_exc_type, dj_exc_value, traceback) 
Oct 19 15:14:29 treeherder app/web.1:   File "/app/.heroku/python/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute 
Oct 19 15:14:29 treeherder app/web.1:     return self.cursor.execute(sql, params) 
Oct 19 15:14:29 treeherder app/web.1:   File "/app/.heroku/python/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 124, in execute 
Oct 19 15:14:29 treeherder app/web.1:     return self.cursor.execute(query, args) 
Oct 19 15:14:29 taskcluster-queue app/web.10: Mon, 19 Oct 2015 14:14:28 GMT taskcluster-client Success calling: authenticateHawk, (0 retries) 
Oct 19 15:14:29 treeherder app/web.1:   File "/app/.heroku/python/lib/python2.7/site-packages/newrelic-2.54.0.41/newrelic/hooks/database_dbapi2.py", line 22, in execute 
Oct 19 15:14:29 treeherder app/web.1:     *args, **kwargs) 
Oct 19 15:14:29 treeherder app/web.1:   File "/app/.heroku/python/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute 
Oct 19 15:14:29 treeherder app/web.1:     self.errorhandler(self, exc, value) 
Oct 19 15:14:29 treeherder app/web.1:   File "/app/.heroku/python/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler 
Oct 19 15:14:29 treeherder app/web.1:     raise errorclass, errorvalue 
Oct 19 15:14:29 treeherder app/web.1: OperationalError: (1054, "Unknown column 'performance_signature.repository_id' in 'field list'") 

https://papertrailapp.com/groups/853883/events?q=program%3Aapp%2Fweb.1&r=592767746937499660-592767773323866114

https://rpm.newrelic.com/accounts/677903/applications/7385291/traced_errors

Comment 1

[William Lachance (:wlach)]

Looking at the database, it appears that the migration didn't happen. Specifically this one:

https://github.com/mozilla/treeherder/blob/master/treeherder/perf/migrations/0002_auto_20151014_0922.py

mysql> describe  performance_signature;
+----------------------+-------------+------+-----+---------+----------------+
| Field                | Type        | Null | Key | Default | Extra          |
+----------------------+-------------+------+-----+---------+----------------+
| id                   | int(11)     | NO   | PRI | NULL    | auto_increment |
| signature_hash       | varchar(40) | NO   | UNI | NULL    |                |
| suite                | varchar(80) | NO   |     | NULL    |                |
| test                 | varchar(80) | NO   |     | NULL    |                |
| extra_properties     | longtext    | NO   |     | NULL    |                |
| framework_id         | int(11)     | NO   | MUL | NULL    |                |
| option_collection_id | int(11)     | NO   | MUL | NULL    |                |
| platform_id          | int(11)     | NO   | MUL | NULL    |                |
+----------------------+-------------+------+-----+---------+----------------+
8 rows in set (0.07 sec)

There should be repository_id and last_updated fields in that table after the migration is applied.

Comment 2

[Ed Morley [:emorley]]

manage.py migrate isn't run automatically on Heroku on the moment, that's bug 1201455.

However running manually fails with:

[~/src/treeherder]$ heroku run ./manage.py migrate
Running ./manage.py migrate on treeherder-heroku... up, run.3116
Operations to perform:
  Synchronize unmigrated apps: django_browserid, log_parser, staticfiles, autoclassify, corsheaders, workers, rest_framework_swagger, messages, etl, rest_framework_extensions, embed, rest_framework, webapp, hawkrest
  Apply all migrations: sessions, perf, admin, auth, sites, contenttypes, model, credentials
Synchronizing apps without migrations:
  Creating tables...
    Running deferred SQL...
  Installing custom SQL...
Running migrations:
  Rendering model states... DONE
  Applying credentials.0002_auto_20151014_0900... OK
  Applying model.0002_auto_20151014_0900...Traceback (most recent call last):
  File "./manage.py", line 11, in <module>
    execute_from_command_line(sys.argv)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/__init__.py", line 351, in execute_from_command_line
    utility.execute()
  File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/__init__.py", line 343, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/base.py", line 394, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/base.py", line 445, in execute
    output = self.handle(*args, **options)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 222, in handle
    executor.migrate(targets, plan, fake=fake, fake_initial=fake_initial)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/db/migrations/executor.py", line 110, in migrate
    self.apply_migration(states[migration], migration, fake=fake, fake_initial=fake_initial)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/db/migrations/executor.py", line 148, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/db/migrations/migration.py", line 115, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/db/migrations/operations/fields.py", line 201, in database_forwards
    schema_editor.alter_field(from_model, from_field, to_field)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/db/backends/base/schema.py", line 484, in alter_field
    old_db_params, new_db_params, strict)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/db/backends/base/schema.py", line 493, in _alter_field
    fk_names = self._constraint_names(model, [old_field.column], foreign_key=True)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/db/backends/base/schema.py", line 913, in _constraint_names
    constraints = self.connection.introspection.get_constraints(cursor, model._meta.db_table)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/db/backends/mysql/introspection.py", line 190, in get_constraints
    cursor.execute("SHOW INDEX FROM %s" % self.connection.ops.quote_name(table_name))
  File "/app/.heroku/python/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/db/utils.py", line 97, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/db/backends/utils.py", line 62, in execute
    return self.cursor.execute(sql)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 124, in execute
    return self.cursor.execute(query, args)
  File "/app/.heroku/python/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
    self.errorhandler(self, exc, value)
  File "/app/.heroku/python/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
django.db.utils.ProgrammingError: (1146, "Table 'treeherder.failure_match' doesn't exist")

Looks like not all of the manual changes that people have been making to stage/prod were also made to the Heroku RDS instance - guessing this needs the table adding that landed in bug 1182464.

Comment 3

[William Lachance (:wlach)]

This is a treeherder/heroku problem, not a perfherder one.

Comment 4

[Ed Morley [:emorley]]

(In reply to William Lachance (:wlach) from comment #3)
> This is a treeherder/heroku problem, not a perfherder one.

This is a combination of two problems: 
* Manual changes having not been made against the Heroku RDS instance previously (for which I've just filed bug 1217631), which has broken migrate.
* Migrate having not been run manually against Heroku since it's non-obvious that it doesn't yet happen automatically.

I'll try to get the migrate to happen as part of deploy as soon as possible (bug 1201455).

However for things for which we can't use migrate and so have to update stage/prod by hand too, it would be really useful if people could remember to update Heroku at the same time, otherwise it takes twice as long for me to figure out what's broken and which changes need to be applied (particularly when they aren't documented in the bugs :-().

Comment 5

[Ed Morley [:emorley]]

[~/src/treeherder]$ heroku run ./manage.py migrate
Running ./manage.py migrate on treeherder-heroku... up, run.6381
Operations to perform:
  Synchronize unmigrated apps: django_browserid, log_parser, rest_framework_extensions, autoclassify, rest_framework, workers, rest_framework_swagger, hawkrest, messages, embed, staticfiles, etl, webapp, corsheaders
  Apply all migrations: credentials, perf, model, auth, sites, admin, sessions, contenttypes
Synchronizing apps without migrations:
  Creating tables...
    Running deferred SQL...
  Installing custom SQL...
Running migrations:
  Rendering model states... DONE
  Applying model.0002_auto_20151014_0900... OK
  Applying perf.0002_auto_20151014_0922... OK
The following content types are stale and need to be deleted:

    model | device

Any objects related to these content types by a foreign key will also
be deleted. Are you sure you want to delete these content types?
If you're unsure, answer 'no'.

    Type 'yes' to continue, or 'no' to cancel: yes


[~/src/treeherder]$ heroku run ./manage.py migrate
Running ./manage.py migrate on treeherder-heroku... up, run.7866
Operations to perform:
  Synchronize unmigrated apps: rest_framework_swagger, embed, log_parser, django_browserid, messages, rest_framework_extensions, etl, rest_framework, hawkrest, staticfiles, workers, autoclassify, corsheaders, webapp
  Apply all migrations: auth, sites, model, contenttypes, sessions, admin, perf, credentials
Synchronizing apps without migrations:
  Creating tables...
    Running deferred SQL...
  Installing custom SQL...
Running migrations:
  No migrations to apply.