Closed
Bug 1172904
Opened 9 years ago
Closed 9 years ago
reindex command fails with kombu.exceptions.EncodeError: <class 'mkt.webapps.indexers.WebappIndexer'> is not JSON serializable
Categories
(Marketplace Graveyard :: General, defect, P1)
Tracking
(Not tracked)
VERIFIED
FIXED
2015-06-16
People
(Reporter: jason, Assigned: robhudson)
Details
I tried to reindex altdev today and received the following: ../venv/bin/python manage.py reindex' [alias:apps_altdev] Indexing 20405 items into 41 chunks of size 500Traceback (most recent call last): File "manage.py", line 73, in <module> execute_from_command_line(sys.argv) File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line utility.execute() File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 377, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/django/core/management/base.py", line 288, in run_from_argv self.execute(*args, **options.__dict__) File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/django/core/management/base.py", line 338, in execute output = self.handle(*args, **options) File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/zamboni/lib/es/management/commands/reindex.py", line 272, in handle body=post_task)).apply_async() File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/celery/canvas.py", line 251, in apply_async return _apply(args, kwargs, **options) File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/celery/app/builtins.py", line 293, in apply_async tasks[0].apply_async(**options) File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/celery/canvas.py", line 251, in apply_async return _apply(args, kwargs, **options) File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/celery/app/task.py", line 559, in apply_async **dict(self._get_exec_options(), **options) File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/celery/app/base.py", line 353, in send_task reply_to=reply_to or self.oid, **options File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/celery/app/amqp.py", line 305, in publish_task **kwargs File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/kombu/messaging.py", line 165, in publish compression, headers) File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/kombu/messaging.py", line 241, in _prepare body) = dumps(body, serializer=serializer) File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/kombu/serialization.py", line 164, in dumps payload = encoder(data) File "/opt/rh/python27/root/usr/lib64/python2.7/contextlib.py", line 35, in __exit__ self.gen.throw(type, value, traceback) File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/kombu/serialization.py", line 59, in _reraise_errors reraise(wrapper, wrapper(exc), sys.exc_info()[2]) File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/kombu/serialization.py", line 55, in _reraise_errors yield File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/kombu/serialization.py", line 164, in dumps payload = encoder(data) File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/anyjson/__init__.py", line 141, in dumps return implementation.dumps(value) File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/anyjson/__init__.py", line 87, in dumps return self._encode(data) File "/opt/rh/python27/root/usr/lib64/python2.7/json/__init__.py", line 243, in dumps return _default_encoder.encode(obj) File "/opt/rh/python27/root/usr/lib64/python2.7/json/encoder.py", line 207, in encode chunks = self.iterencode(o, _one_shot=True) File "/opt/rh/python27/root/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode return _iterencode(o, 0) File "/opt/rh/python27/root/usr/lib64/python2.7/json/encoder.py", line 184, in default raise TypeError(repr(o) + " is not JSON serializable") kombu.exceptions.EncodeError: <class 'mkt.webapps.indexers.WebappIndexer'> is not JSON serializable
Comment 1•9 years ago
|
||
This is following bug 1136820 and those 2 changes: https://github.com/mozilla/zamboni/commit/57d310c49ba1e49319f51026273d44ffca97b166 and https://github.com/mozilla/zamboni/commit/d3eac680fa39b6a78e8cfe13d3625e1914b727eb
Assignee: nobody → robhudson.mozbugs
Priority: -- → P1
Assignee | ||
Comment 2•9 years ago
|
||
I see what we're doing. We're passing the whole indexer class in with the pre and post tasks (the INDEXER variable). We can't do that with json (and shouldn't have been doing that with pickle preferably). I can fix by making a dict mapping from a string name to the class and passing that instead. It'll make the queue messages smaller and json can encode/decode it.
Target Milestone: --- → 2015-06-16
Assignee | ||
Comment 3•9 years ago
|
||
https://github.com/mozilla/zamboni/pull/3126
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Reporter | ||
Comment 4•9 years ago
|
||
reindex ran successfully on -dev and altdev.
Status: RESOLVED → VERIFIED
You need to log in
before you can comment on or make changes to this bug.
Description
•