reindex command fails with kombu.exceptions.EncodeError: <class 'mkt.webapps.indexers.WebappIndexer'> is not JSON serializable

VERIFIED FIXED in 2015-06-16

Status

P1
normal
VERIFIED FIXED
3 years ago
3 years ago

People

(Reporter: jason, Assigned: robhudson)

Tracking

Avenir
2015-06-16
Points:
---

Details

(Reporter)

Description

3 years ago
I tried to reindex altdev today and received the following:

../venv/bin/python manage.py reindex'

[alias:apps_altdev] Indexing 20405 items into 41 chunks of size 500Traceback (most recent call last):
  File "manage.py", line 73, in <module>
    execute_from_command_line(sys.argv)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 377, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/django/core/management/base.py", line 338, in execute
    output = self.handle(*args, **options)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/zamboni/lib/es/management/commands/reindex.py", line 272, in handle
    body=post_task)).apply_async()
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/celery/canvas.py", line 251, in apply_async
    return _apply(args, kwargs, **options)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/celery/app/builtins.py", line 293, in apply_async
    tasks[0].apply_async(**options)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/celery/canvas.py", line 251, in apply_async
    return _apply(args, kwargs, **options)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/celery/app/task.py", line 559, in apply_async
    **dict(self._get_exec_options(), **options)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/celery/app/base.py", line 353, in send_task
    reply_to=reply_to or self.oid, **options
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/celery/app/amqp.py", line 305, in publish_task
    **kwargs
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/kombu/messaging.py", line 165, in publish
    compression, headers)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/kombu/messaging.py", line 241, in _prepare
    body) = dumps(body, serializer=serializer)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/kombu/serialization.py", line 164, in dumps
    payload = encoder(data)
  File "/opt/rh/python27/root/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/kombu/serialization.py", line 59, in _reraise_errors
    reraise(wrapper, wrapper(exc), sys.exc_info()[2])
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/kombu/serialization.py", line 55, in _reraise_errors
    yield
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/kombu/serialization.py", line 164, in dumps
    payload = encoder(data)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/anyjson/__init__.py", line 141, in dumps
    return implementation.dumps(value)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/anyjson/__init__.py", line 87, in dumps
    return self._encode(data)
  File "/opt/rh/python27/root/usr/lib64/python2.7/json/__init__.py", line 243, in dumps
    return _default_encoder.encode(obj)
  File "/opt/rh/python27/root/usr/lib64/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/opt/rh/python27/root/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
  File "/opt/rh/python27/root/usr/lib64/python2.7/json/encoder.py", line 184, in default
    raise TypeError(repr(o) + " is not JSON serializable")
kombu.exceptions.EncodeError: <class 'mkt.webapps.indexers.WebappIndexer'> is not JSON serializable
(Assignee)

Comment 2

3 years ago
I see what we're doing. We're passing the whole indexer class in with the pre and post tasks (the INDEXER variable). We can't do that with json (and shouldn't have been doing that with pickle preferably).

I can fix by making a dict mapping from a string name to the class and passing that instead. It'll make the queue messages smaller and json can encode/decode it.
Target Milestone: --- → 2015-06-16
(Assignee)

Comment 3

3 years ago
https://github.com/mozilla/zamboni/pull/3126
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
(Reporter)

Comment 4

3 years ago
reindex ran successfully on -dev and altdev.
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.