Hello IT, We are upgrading celery libraries on mozillians from an ancient version and we need your help to do that: 1. Please run the following command on mozillians-dev ./manage.py dbshell < ./scripts/drop_celery_tables.sql which will alter the database and drop the celery tables. 2. Manually execute the update script to update mozillians-dev. The HEAD is on 9cb0a324f91 Thanks!
Assignee: server-ops-webops → bburton
Priority: -- → P3
Whiteboard: [push interrupt][triaged 20121217]
Tables dropped, migrations ran successfully, push complete.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Re-opening. Giorgos we are getting reports of search not working on dev, see #822202 I thought maybe i had to kick off the indexer, however my admin login no longer works on dev. This might be a symptom of the migrations not be run correctly/successfully.
Ben, I re-enabled your admin login on dev (stage is OK). This bug should not be related to search, given that the push was completed successfully as :solarce says in comment #1. I suspect that maybe bug 810960 is related (depending bug 818508 had some activity yesterday). :solarce, maybe you check dev ElasticSearch or Celery logs for something unusual? Thanks!
BTW I triggered an re-index which didn't change much.
Doh just realized I meant to reference bug 822342 above Ok let me request feedback from solarce to see if anything funky is in the logs.
Update? We still have 0 search results appearing on dev.
Sorry, I missed this, been a lot of bugmail the last day, I can take a look in the morning
:solarce 1. Please make sure that celery is running (maybe also restart to be on the safe side?) 2. Run ./manage.py cron index_all_profiles This should sent celery a few jobs, maybe these are failing? 3. ElasticSearch server get an index 'mozillians' with a couple thousand rows. Thanks!
Looking into this now
Status: REOPENED → ASSIGNED
Yes, there is an error in the celeryd log [email@example.com log]# grep '2012-12-21 10\:11*' celeryd-mozillians-dev.log [2012-12-21 10:11:10,008: WARNING/MainProcess] Traceback (most recent call last): [2012-12-21 10:11:10,013: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/manage.py", line 24, in <module> [2012-12-21 10:11:10,014: WARNING/MainProcess] manage.main() [2012-12-21 10:11:10,016: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/src/funfactory/funfactory/manage.py", line 121, in main [2012-12-21 10:11:10,018: WARNING/MainProcess] execute_manager(current_settings) [2012-12-21 10:11:10,028: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/src/django/django/core/management/__init__.py", line 459, in execute_manager [2012-12-21 10:11:10,035: WARNING/MainProcess] utility.execute() [2012-12-21 10:11:10,036: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/src/django/django/core/management/__init__.py", line 382, in execute [2012-12-21 10:11:10,038: WARNING/MainProcess] self.fetch_command(subcommand).run_from_argv(self.argv) [2012-12-21 10:11:10,039: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/djcelery/management/base.py", line 74, in run_from_argv [2012-12-21 10:11:10,040: WARNING/MainProcess] return super(CeleryCommand, self).run_from_argv(argv) [2012-12-21 10:11:10,042: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/src/django/django/core/management/base.py", line 196, in run_from_argv [2012-12-21 10:11:10,043: WARNING/MainProcess] self.execute(*args, **options.__dict__) [2012-12-21 10:11:10,044: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/djcelery/management/base.py", line 67, in execute [2012-12-21 10:11:10,048: WARNING/MainProcess] super(CeleryCommand, self).execute(*args, **options) [2012-12-21 10:11:10,051: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/src/django/django/core/management/base.py", line 232, in execute [2012-12-21 10:11:10,052: WARNING/MainProcess] output = self.handle(*args, **options) [2012-12-21 10:11:10,053: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/djcelery/management/commands/celeryd.py", line 25, in handle [2012-12-21 10:11:10,055: WARNING/MainProcess] worker.run(*args, **options) [2012-12-21 10:11:10,056: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/celery/bin/celeryd.py", line 153, in run [2012-12-21 10:11:10,058: WARNING/MainProcess] return self.app.Worker(**kwargs).run() [2012-12-21 10:11:10,059: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/celery/apps/worker.py", line 183, in run [2012-12-21 10:11:10,060: WARNING/MainProcess] self.run_worker() [2012-12-21 10:11:10,062: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/celery/apps/worker.py", line 268, in run_worker [2012-12-21 10:11:10,063: WARNING/MainProcess] **self.confopts_as_dict()) [2012-12-21 10:11:10,064: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/celery/worker/__init__.py", line 335, in __init__ [2012-12-21 10:11:10,066: WARNING/MainProcess] self.namespace = Namespace(app=self.app).apply(self, **kwargs) [2012-12-21 10:11:10,067: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/celery/worker/bootsteps.py", line 64, in apply [2012-12-21 10:11:10,069: WARNING/MainProcess] self.load_modules() [2012-12-21 10:11:10,070: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/celery/worker/bootsteps.py", line 51, in load_modules [2012-12-21 10:11:10,071: WARNING/MainProcess] self.import_module(m) [2012-12-21 10:11:10,073: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/celery/worker/bootsteps.py", line 84, in import_module [2012-12-21 10:11:10,076: WARNING/MainProcess] return import_module(module) [2012-12-21 10:11:10,079: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/importlib/__init__.py", line 37, in import_module [2012-12-21 10:11:10,080: WARNING/MainProcess] __import__(name) [2012-12-21 10:11:10,081: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/celery/worker/consumer.py", line 99, in <module> [2012-12-21 10:11:10,084: WARNING/MainProcess] from .control import Panel [2012-12-21 10:11:10,085: WARNING/MainProcess] File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/celery/worker/control/__init__.py", line 3, in <module> [2012-12-21 10:11:10,086: WARNING/MainProcess] from celery.messaging import ControlReplyPublisher, with_connection [2012-12-21 10:11:10,087: WARNING/MainProcess] ImportError [2012-12-21 10:11:10,088: WARNING/MainProcess] : [2012-12-21 10:11:10,089: WARNING/MainProcess] cannot import name ControlReplyPublisher
Just to be extra sure I did some tests to make sure that the celery box is getting the latest mozillians code and I can confirm this is the case, Please let me know on the error above
I am on PTO from this afternoon until 01/02/2013, so I am putting this back in the queue if you need further webops help Have a merry christmas, see you next year!
Assignee: bburton → server-ops-webops
06:00:29 < giorgos> phrawzty, we need to run the commands for comment #1 again 06:00:44 < giorgos> i.e. delete the celery tables and pull the new code
Assignee: server-ops-webops → dmaher
07:11:54 < giorgos> phrawzty, ok. I believe the problem is that south believes that the tables are there 07:12:05 < giorgos> so it doesn't re-create them on "migrate" 07:12:41 < giorgos> we need to drop them again, to be sure 07:12:49 < giorgos> ./manage.py dbshell < ./scripts/drop_celery_tables.sql 07:13:14 < giorgos> then alter the migration history 07:13:59 < giorgos> ./manage.py dbshell 07:16:12 < giorgos> and run http://pastebin.mozilla.org/2016495 [firstname.lastname@example.org mozillians]$ sudo ./manage.py dbshell < ./scripts/drop_celery_tables.sql /data/genericrhel6-dev/src/mozillians-dev.allizom.org/mozillians/vendor/src/django/django/conf/__init__.py:75: DeprecationWarning: The ADMIN_MEDIA_PREFIX setting has been removed; use STATIC_URL instead. "use STATIC_URL instead.", DeprecationWarning) ERROR 1051 (42S02) at line 1: Unknown table 'djcelery_taskstate' [email@example.com mozillians]$ sudo ./manage.py dbshell mysql> drop table celery_taskmeta; Query OK, 0 rows affected (0.29 sec) mysql> drop table celery_tasksetmeta; Query OK, 0 rows affected (0.14 sec) mysql> delete from south_migrationhistory where app_name='djcelery'; Query OK, 2 rows affected (0.01 sec)
After much investigation and peering into log files here and there, the problem - ultimately - was that the "new" celery for mozillians-dev wasn't actually starting properly (despite the service restart exiting 0). :Giorgos made a code edit, I pushed, and now everything appears to be working as expected.
Status: ASSIGNED → RESOLVED
Last Resolved: 6 years ago → 6 years ago
Resolution: --- → FIXED
Bumping to verified - I suspect the passage of time has overtaken this bug.
Status: RESOLVED → VERIFIED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.