1220286 - [sync] IntegrityError on latest_translation

Status: RESOLVED WORKSFORME

(Webtools Graveyard :: Pontoon, defect, P1)

Description

[Matjaz Horvat [:mathjazz]]

I added kk in admin, ran sync and locale is still not enabled:
https://pontoon.mozilla.org/projects/firefox-accounts/

Might be related to the following error logged after Firefox Accounts Sync completes:

 Oct 30 13:06:12 mozilla-pontoon app/worker.1: [2015-10-30 20:06:12,220: INFO/Worker-2] Synced project firefox-accounts 
Oct 30 13:06:12 mozilla-pontoon app/worker.1: [2015-10-30 20:06:12,656: ERROR/MainProcess] Task pontoon.sync.tasks.sync_project[37612f27-2766-415e-a7f5-1a0a10626270] raised unexpected: IntegrityError('update or delete on table "base_translation" violates foreign key constraint "b_latest_translation_id_2e81fb9b6f7a8438_fk_base_translation_id" on table "base_project"\nDETAIL:  Key (id)=(839102) is still referenced from table "base_project".\n',) 
Oct 30 13:06:12 mozilla-pontoon app/worker.1: Traceback (most recent call last): 
Oct 30 13:06:12 mozilla-pontoon app/worker.1:   File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task 
Oct 30 13:06:12 mozilla-pontoon app/worker.1:     R = retval = fun(*args, **kwargs) 
Oct 30 13:06:12 mozilla-pontoon app/worker.1:   File "/app/.heroku/python/lib/python2.7/site-packages/newrelic-2.50.0.39/newrelic/hooks/application_celery.py", line 66, in wrapper 
Oct 30 13:06:12 mozilla-pontoon app/worker.1:     return wrapped(*args, **kwargs) 
Oct 30 13:06:12 mozilla-pontoon app/worker.1:   File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__ 
Oct 30 13:06:12 mozilla-pontoon app/worker.1:     return self.run(*args, **kwargs) 
Oct 30 13:06:12 mozilla-pontoon app/worker.1:   File "/app/pontoon/sync/tasks.py", line 22, in sync_project 
Oct 30 13:06:12 mozilla-pontoon app/worker.1:     perform_sync(project, no_pull=no_pull, no_commit=no_commit) 
Oct 30 13:06:12 mozilla-pontoon app/worker.1:   File "/app/.heroku/python/lib/python2.7/site-packages/django/utils/decorators.py", line 145, in inner 
Oct 30 13:06:12 mozilla-pontoon app/worker.1:     return func(*args, **kwargs) 
Oct 30 13:06:12 mozilla-pontoon app/worker.1:   File "/app/.heroku/python/lib/python2.7/site-packages/django/db/transaction.py", line 232, in __exit__ 
Oct 30 13:06:12 mozilla-pontoon app/worker.1:     connection.commit() 
Oct 30 13:06:12 mozilla-pontoon app/worker.1:   File "/app/.heroku/python/lib/python2.7/site-packages/django/db/backends/base/base.py", line 173, in commit 
Oct 30 13:06:12 mozilla-pontoon app/worker.1:     self._commit() 
Oct 30 13:06:12 mozilla-pontoon app/worker.1:   File "/app/.heroku/python/lib/python2.7/site-packages/django/db/backends/base/base.py", line 142, in _commit 
Oct 30 13:06:12 mozilla-pontoon app/worker.1:     return self.connection.commit() 
Oct 30 13:06:12 mozilla-pontoon app/worker.1:   File "/app/.heroku/python/lib/python2.7/site-packages/django/db/utils.py", line 97, in __exit__ 
Oct 30 13:06:12 mozilla-pontoon app/worker.1:     six.reraise(dj_exc_type, dj_exc_value, traceback) 
Oct 30 13:06:12 mozilla-pontoon app/worker.1:   File "/app/.heroku/python/lib/python2.7/site-packages/django/db/backends/base/base.py", line 142, in _commit 
Oct 30 13:06:12 mozilla-pontoon app/worker.1:     return self.connection.commit() 
Oct 30 13:06:12 mozilla-pontoon app/worker.1:   File "/app/.heroku/python/lib/python2.7/site-packages/newrelic-2.50.0.39/newrelic/hooks/database_dbapi2.py", line 71, in commit 
Oct 30 13:06:12 mozilla-pontoon app/worker.1:     return self.__wrapped__.commit() 
Oct 30 13:06:12 mozilla-pontoon app/worker.1: IntegrityError: update or delete on table "base_translation" violates foreign key constraint "b_latest_translation_id_2e81fb9b6f7a8438_fk_base_translation_id" on table "base_project" 
Oct 30 13:06:13 mozilla-pontoon app/worker.1: DETAIL:  Key (id)=(839102) is still referenced from table "base_project".

Comment 1

[Matjaz Horvat [:mathjazz]]

I'm unable to repeat that error locally.

Comment 2

[Osmose [:osmose, :mkelly]]

I also cannot replicate this locally. Are we still hitting this on prod?

Comment 3

[Matjaz Horvat [:mathjazz]]

Yup, search for "IntegrityError".

Comment 4

[Osmose [:osmose, :mkelly]]

I wonder if the index mentioned in the traceback exists on staging or our local copies. Perhaps prod has an index that the rest of us don't?

Comment 5

[Matjaz Horvat [:mathjazz]]

It's back. This time with AMO. Tried to add a new locale (ar) and I get the error after sync. Locale is not added.

https://papertrailapp.com/groups/1435104/events?centered_on_id=599395357616812062&q=program%3Aapp%2Fworker.2

Comment 6

[Osmose [:osmose, :mkelly]]

The only time during sync that a translation can get deleted is when we delete resources, but testing locally I can't get deleting a resource to trigger this kind of IntegrityError; latest_translation always gets set to null correctly.

https://github.com/mozilla/pontoon/pull/251 will add error reporting to the celery tasks, I suggest we wait until that lands and then see the errors as they get reported for more information.

Comment 7

[Matjaz Horvat [:mathjazz]]

The problem disappeared after:
https://github.com/mozilla/pontoon/commit/a8082009b6605c9a25ed416d09c6331c3a073062