822202 - [Mozillians] Drop celery tables on dev

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P3)

Description

[Giorgos Logiotatidis [:giorgos]]

Hello IT,

We are upgrading celery libraries on mozillians from an ancient version and we need your help to do that:

 1. Please run the following command on mozillians-dev

./manage.py dbshell < ./scripts/drop_celery_tables.sql

which will alter the database and drop the celery tables.


 2. Manually execute the update script to update mozillians-dev.


The HEAD is on 9cb0a324f91

Thanks!

Comment 1

[Brandon Burton [:solarce]]

Tables dropped, migrations ran successfully, push complete.

Comment 2

[Ben (:bensternthal)]

Re-opening.

Giorgos we are getting reports of search not working on dev, see

#822202

I thought maybe i had to kick off the indexer, however my admin login no longer works on dev.

This might be a symptom of the migrations not be run correctly/successfully.

Comment 3

[Giorgos Logiotatidis [:giorgos]]

Ben,

I re-enabled your admin login on dev (stage is OK).

This bug should not be related to search, given that the push was completed successfully as :solarce says in comment #1. I suspect that maybe bug 810960 is related (depending bug 818508 had some activity yesterday).

:solarce, maybe you check dev ElasticSearch or Celery logs for something unusual?

Thanks!

Comment 4

[Giorgos Logiotatidis [:giorgos]]

BTW I triggered an re-index which didn't change much.

Comment 5

[Ben (:bensternthal)]

Doh just realized I meant to reference  bug 822342 above

Ok let me request feedback from solarce to see if anything funky is in the logs.

Comment 6

[Ben (:bensternthal)]

Update? We still have 0 search results appearing on dev.

Comment 7

[Brandon Burton [:solarce]]

Sorry, I missed this, been a lot of bugmail the last day, I can take a look in the morning

Comment 8

[Giorgos Logiotatidis [:giorgos]]

:solarce

1. Please make sure that celery is running (maybe also restart to be on the safe side?)
2. Run

./manage.py cron index_all_profiles

This should sent celery a few jobs, maybe these are failing?

3. ElasticSearch server get an index 'mozillians' with a couple thousand rows.

Thanks!

Comment 9

[Brandon Burton [:solarce]]

Looking into this now

Comment 10

[Brandon Burton [:solarce]]

Yes, there is an error in the celeryd log

[[e-mail]1 log]# grep '2012-12-21 10\:11*' celeryd-mozillians-dev.log

[2012-12-21 10:11:10,008: WARNING/MainProcess] Traceback (most recent call last):
[2012-12-21 10:11:10,013: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/manage.py", line 24, in <module>
[2012-12-21 10:11:10,014: WARNING/MainProcess] manage.main()
[2012-12-21 10:11:10,016: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/src/funfactory/funfactory/manage.py", line 121, in main
[2012-12-21 10:11:10,018: WARNING/MainProcess] execute_manager(current_settings)
[2012-12-21 10:11:10,028: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/src/django/django/core/management/__init__.py", line 459, in execute_manager
[2012-12-21 10:11:10,035: WARNING/MainProcess] utility.execute()
[2012-12-21 10:11:10,036: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/src/django/django/core/management/__init__.py", line 382, in execute
[2012-12-21 10:11:10,038: WARNING/MainProcess] self.fetch_command(subcommand).run_from_argv(self.argv)
[2012-12-21 10:11:10,039: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/djcelery/management/base.py", line 74, in run_from_argv
[2012-12-21 10:11:10,040: WARNING/MainProcess] return super(CeleryCommand, self).run_from_argv(argv)
[2012-12-21 10:11:10,042: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/src/django/django/core/management/base.py", line 196, in run_from_argv
[2012-12-21 10:11:10,043: WARNING/MainProcess] self.execute(*args, **options.__dict__)
[2012-12-21 10:11:10,044: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/djcelery/management/base.py", line 67, in execute
[2012-12-21 10:11:10,048: WARNING/MainProcess] super(CeleryCommand, self).execute(*args, **options)
[2012-12-21 10:11:10,051: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/src/django/django/core/management/base.py", line 232, in execute
[2012-12-21 10:11:10,052: WARNING/MainProcess] output = self.handle(*args, **options)
[2012-12-21 10:11:10,053: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/djcelery/management/commands/celeryd.py", line 25, in handle
[2012-12-21 10:11:10,055: WARNING/MainProcess] worker.run(*args, **options)
[2012-12-21 10:11:10,056: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/celery/bin/celeryd.py", line 153, in run
[2012-12-21 10:11:10,058: WARNING/MainProcess] return self.app.Worker(**kwargs).run()
[2012-12-21 10:11:10,059: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/celery/apps/worker.py", line 183, in run
[2012-12-21 10:11:10,060: WARNING/MainProcess] self.run_worker()
[2012-12-21 10:11:10,062: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/celery/apps/worker.py", line 268, in run_worker
[2012-12-21 10:11:10,063: WARNING/MainProcess] **self.confopts_as_dict())
[2012-12-21 10:11:10,064: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/celery/worker/__init__.py", line 335, in __init__
[2012-12-21 10:11:10,066: WARNING/MainProcess] self.namespace = Namespace(app=self.app).apply(self, **kwargs)
[2012-12-21 10:11:10,067: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/celery/worker/bootsteps.py", line 64, in apply
[2012-12-21 10:11:10,069: WARNING/MainProcess] self.load_modules()
[2012-12-21 10:11:10,070: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/celery/worker/bootsteps.py", line 51, in load_modules
[2012-12-21 10:11:10,071: WARNING/MainProcess] self.import_module(m)
[2012-12-21 10:11:10,073: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/celery/worker/bootsteps.py", line 84, in import_module
[2012-12-21 10:11:10,076: WARNING/MainProcess] return import_module(module)
[2012-12-21 10:11:10,079: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/importlib/__init__.py", line 37, in import_module
[2012-12-21 10:11:10,080: WARNING/MainProcess] __import__(name)
[2012-12-21 10:11:10,081: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/celery/worker/consumer.py", line 99, in <module>
[2012-12-21 10:11:10,084: WARNING/MainProcess] from .control import Panel
[2012-12-21 10:11:10,085: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/celery/worker/control/__init__.py", line 3, in <module>
[2012-12-21 10:11:10,086: WARNING/MainProcess] from celery.messaging import ControlReplyPublisher, with_connection
[2012-12-21 10:11:10,087: WARNING/MainProcess] ImportError
[2012-12-21 10:11:10,088: WARNING/MainProcess] :
[2012-12-21 10:11:10,089: WARNING/MainProcess] cannot import name ControlReplyPublisher

Comment 11

[Brandon Burton [:solarce]]

Just to be extra sure I did some tests to make sure that the celery box is getting the latest mozillians code and I can confirm this is the case,

Please let me know on the error above

Comment 12

[Brandon Burton [:solarce]]

I am on PTO from this afternoon until 01/02/2013, so I am putting this back in the queue if you need further webops help

Have a merry christmas, see you next year!

Comment 13

[Daniel Maher [:phrawzty]]

06:00:29 < giorgos> phrawzty, we need to run the commands for comment #1 again
06:00:44 < giorgos> i.e. delete the celery tables and pull the new code

Comment 14

[Daniel Maher [:phrawzty]]

07:11:54 < giorgos> phrawzty, ok. I believe the problem is that south believes that the tables are there
07:12:05 < giorgos> so it doesn't re-create them on "migrate"
07:12:41 < giorgos> we need to drop them again, to be sure
07:12:49 < giorgos> ./manage.py dbshell < ./scripts/drop_celery_tables.sql
07:13:14 < giorgos> then alter the migration history 
07:13:59 < giorgos> ./manage.py dbshell 
07:16:12 < giorgos> and run http://pastebin.mozilla.org/2016495


[[e-mail]1 mozillians]$ sudo ./manage.py dbshell < ./scripts/drop_celery_tables.sql
/data/genericrhel6-dev/src/mozillians-dev.allizom.org/mozillians/vendor/src/django/django/conf/__init__.py:75: DeprecationWarning: The ADMIN_MEDIA_PREFIX setting has been removed; use STATIC_URL instead.
  "use STATIC_URL instead.", DeprecationWarning)
ERROR 1051 (42S02) at line 1: Unknown table 'djcelery_taskstate'
[[e-mail]1 mozillians]$ sudo ./manage.py dbshell

mysql> drop table celery_taskmeta;
Query OK, 0 rows affected (0.29 sec)

mysql> drop table celery_tasksetmeta;
Query OK, 0 rows affected (0.14 sec)

mysql> delete from south_migrationhistory where app_name='djcelery'; 
Query OK, 2 rows affected (0.01 sec)

Comment 15

[Daniel Maher [:phrawzty]]

After much investigation and peering into log files here and there, the problem - ultimately - was that the "new" celery for mozillians-dev wasn't actually starting properly (despite the service restart exiting 0).  :Giorgos made a code edit, I pushed, and now everything appears to be working as expected.

Comment 16

[Matt Brandt [:mbrandt]]

Bumping to verified - I suspect the passage of time has overtaken this bug.