1172904 - reindex command fails with kombu.exceptions.EncodeError: <class 'mkt.webapps.indexers.WebappIndexer'> is not JSON serializable

Status: VERIFIED FIXED

(Marketplace Graveyard :: General, defect, P1)

Description

[Jason Thomas [:jason]]

I tried to reindex altdev today and received the following:

../venv/bin/python manage.py reindex'

[alias:apps_altdev] Indexing 20405 items into 41 chunks of size 500Traceback (most recent call last):
  File "manage.py", line 73, in <module>
    execute_from_command_line(sys.argv)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 377, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/django/core/management/base.py", line 338, in execute
    output = self.handle(*args, **options)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/zamboni/lib/es/management/commands/reindex.py", line 272, in handle
    body=post_task)).apply_async()
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/celery/canvas.py", line 251, in apply_async
    return _apply(args, kwargs, **options)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/celery/app/builtins.py", line 293, in apply_async
    tasks[0].apply_async(**options)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/celery/canvas.py", line 251, in apply_async
    return _apply(args, kwargs, **options)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/celery/app/task.py", line 559, in apply_async
    **dict(self._get_exec_options(), **options)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/celery/app/base.py", line 353, in send_task
    reply_to=reply_to or self.oid, **options
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/celery/app/amqp.py", line 305, in publish_task
    **kwargs
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/kombu/messaging.py", line 165, in publish
    compression, headers)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/kombu/messaging.py", line 241, in _prepare
    body) = dumps(body, serializer=serializer)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/kombu/serialization.py", line 164, in dumps
    payload = encoder(data)
  File "/opt/rh/python27/root/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/kombu/serialization.py", line 59, in _reraise_errors
    reraise(wrapper, wrapper(exc), sys.exc_info()[2])
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/kombu/serialization.py", line 55, in _reraise_errors
    yield
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/kombu/serialization.py", line 164, in dumps
    payload = encoder(data)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/anyjson/__init__.py", line 141, in dumps
    return implementation.dumps(value)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/anyjson/__init__.py", line 87, in dumps
    return self._encode(data)
  File "/opt/rh/python27/root/usr/lib64/python2.7/json/__init__.py", line 243, in dumps
    return _default_encoder.encode(obj)
  File "/opt/rh/python27/root/usr/lib64/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/opt/rh/python27/root/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
  File "/opt/rh/python27/root/usr/lib64/python2.7/json/encoder.py", line 184, in default
    raise TypeError(repr(o) + " is not JSON serializable")
kombu.exceptions.EncodeError: <class 'mkt.webapps.indexers.WebappIndexer'> is not JSON serializable

Comment 1

[Mathieu Pillard [:mat]]

This is following bug 1136820 and those 2 changes: https://github.com/mozilla/zamboni/commit/57d310c49ba1e49319f51026273d44ffca97b166 and https://github.com/mozilla/zamboni/commit/d3eac680fa39b6a78e8cfe13d3625e1914b727eb

Comment 2

[Rob Hudson [:robhudson]]

I see what we're doing. We're passing the whole indexer class in with the pre and post tasks (the INDEXER variable). We can't do that with json (and shouldn't have been doing that with pickle preferably).

I can fix by making a dict mapping from a string name to the class and passing that instead. It'll make the queue messages smaller and json can encode/decode it.

Comment 3

[Rob Hudson [:robhudson]]

https://github.com/mozilla/zamboni/pull/3126

Comment 4

[Jason Thomas [:jason]]

reindex ran successfully on -dev and altdev.