Closed Bug 1172904 Opened 9 years ago Closed 9 years ago

reindex command fails with kombu.exceptions.EncodeError: <class 'mkt.webapps.indexers.WebappIndexer'> is not JSON serializable

Categories

(Marketplace Graveyard :: General, defect, P1)

Avenir
defect

Tracking

(Not tracked)

VERIFIED FIXED
2015-06-16

People

(Reporter: jason, Assigned: robhudson)

Details

I tried to reindex altdev today and received the following:

../venv/bin/python manage.py reindex'

[alias:apps_altdev] Indexing 20405 items into 41 chunks of size 500Traceback (most recent call last):
  File "manage.py", line 73, in <module>
    execute_from_command_line(sys.argv)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 377, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/django/core/management/base.py", line 338, in execute
    output = self.handle(*args, **options)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/zamboni/lib/es/management/commands/reindex.py", line 272, in handle
    body=post_task)).apply_async()
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/celery/canvas.py", line 251, in apply_async
    return _apply(args, kwargs, **options)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/celery/app/builtins.py", line 293, in apply_async
    tasks[0].apply_async(**options)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/celery/canvas.py", line 251, in apply_async
    return _apply(args, kwargs, **options)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/celery/app/task.py", line 559, in apply_async
    **dict(self._get_exec_options(), **options)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/celery/app/base.py", line 353, in send_task
    reply_to=reply_to or self.oid, **options
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/celery/app/amqp.py", line 305, in publish_task
    **kwargs
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/kombu/messaging.py", line 165, in publish
    compression, headers)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/kombu/messaging.py", line 241, in _prepare
    body) = dumps(body, serializer=serializer)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/kombu/serialization.py", line 164, in dumps
    payload = encoder(data)
  File "/opt/rh/python27/root/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/kombu/serialization.py", line 59, in _reraise_errors
    reraise(wrapper, wrapper(exc), sys.exc_info()[2])
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/kombu/serialization.py", line 55, in _reraise_errors
    yield
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/kombu/serialization.py", line 164, in dumps
    payload = encoder(data)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/anyjson/__init__.py", line 141, in dumps
    return implementation.dumps(value)
  File "/data/addons-dev/www/marketplace-altdev.allizom.org/deploy-zamboni-altdev-20150609113659-da91659706/venv/lib/python2.7/site-packages/anyjson/__init__.py", line 87, in dumps
    return self._encode(data)
  File "/opt/rh/python27/root/usr/lib64/python2.7/json/__init__.py", line 243, in dumps
    return _default_encoder.encode(obj)
  File "/opt/rh/python27/root/usr/lib64/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/opt/rh/python27/root/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
  File "/opt/rh/python27/root/usr/lib64/python2.7/json/encoder.py", line 184, in default
    raise TypeError(repr(o) + " is not JSON serializable")
kombu.exceptions.EncodeError: <class 'mkt.webapps.indexers.WebappIndexer'> is not JSON serializable
I see what we're doing. We're passing the whole indexer class in with the pre and post tasks (the INDEXER variable). We can't do that with json (and shouldn't have been doing that with pickle preferably).

I can fix by making a dict mapping from a string name to the class and passing that instead. It'll make the queue messages smaller and json can encode/decode it.
Target Milestone: --- → 2015-06-16
https://github.com/mozilla/zamboni/pull/3126
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
reindex ran successfully on -dev and altdev.
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.