617940 - Occasional stack trace from _rebuild_kb_chunk

Status: VERIFIED FIXED

(support.mozilla.org :: Knowledge Base Software, task, P2)

Description

[James Socol [:jsocol, :james]]

Since we're pushing celery logs to syslog, I noticed this stack trace show up occasionally. I'm not sure what the ValidationError is from yet:

Task wiki.tasks._rebuild_kb_chunk[5d6e917d-c90b-4f62-8645-5c8984813b0f] raised exception: 
UnpickleableExceptionWrapper('django.core.exceptions', 'ValidationError', ())
Traceback (most recent call last):
   File "/data/www/master.support.mozilla.com/kitsune/vendor/packages/celery/celery/execute/trace.py", line 29, in trace
     return cls(states.SUCCESS, retval=fun(*args, **kwargs))
   File "/data/www/master.support.mozilla.com/kitsune/vendor/packages/celery/celery/task/base.py", line 222, in __call__
     return self.run(*args, **kwargs)
   File "/data/www/master.support.mozilla.com/kitsune/vendor/packages/celery/celery/decorators.py", line 52, in run
     return fun(*args, **kwargs)
   File "/data/www/master.support.mozilla.com/kitsune/apps/wiki/tasks.py", line 133, in _rebuild_kb_chunk
     document.save()
   File "/data/www/master.support.mozilla.com/kitsune/apps/wiki/models.py", line 279, in save
     self.

Unfortunately the stack trace cuts off there. :(

I've never seen this running a kb_rebuild task, so it might be from more up-to-date data, or just something on master.support. Once 2.3.2 goes out we'll be able to look at celery logs from production, too, to see if this is happening.

If anyone can reproduce this, please try to investigate.

Comment 1

[Ricky Rosario [:rrosario, :r1cky]]

I don't get any errors locally either. I'll try it out fresh database dump if I can get one downloaded by today.

Comment 2

[James Socol [:jsocol, :james]]

Full stack trace:

Traceback (most recent call last):
  File "/data/www/master.support.mozilla.com/kitsune/vendor/packages/celery/celery/execute/trace.py", line 34, in trace
    return cls(states.SUCCESS, retval=fun(*args, **kwargs))
  File "/data/www/master.support.mozilla.com/kitsune/vendor/packages/celery/celery/task/base.py", line 248, in __call__
    return self.run(*args, **kwargs)
  File "/data/www/master.support.mozilla.com/kitsune/vendor/packages/celery/celery/decorators.py", line 52, in run
    return fun(*args, **kwargs)
  File "/data/www/master.support.mozilla.com/kitsune/apps/wiki/tasks.py", line 133, in _rebuild_kb_chunk
    document.save()
  File "/data/www/master.support.mozilla.com/kitsune/apps/wiki/models.py", line 279, in save
    self._clean_is_localizable()
  File "/data/www/master.support.mozilla.com/kitsune/apps/wiki/models.py", line 216, in _clean_is_localizable
    unicode(self), unicode(self.parent)))
ValidationError: ('django.core.exceptions', 'ValidationError', ())


My first thought is: we need to add messages to exceptions we raise, when we don't have a special exception defined (e.g. "FileTooLargeException" is explicit, "ValidationError" is not).

My second is: hmm? Why doesn't this happen locally for anyone?

Comment 3

[James Socol [:jsocol, :james]]

(In reply to comment #2)
> 216, in _clean_is_localizable
>     unicode(self), unicode(self.parent)))
> ValidationError: ('django.core.exceptions', 'ValidationError', ())

> My first thought is: we need to add messages to exceptions we raise, when we
> don't have a special exception defined (e.g. "FileTooLargeException" is
> explicit, "ValidationError" is not).

https://github.com/jsocol/kitsune/blob/master/apps/wiki/models.py#L214

OK, we are trying to set a message and failing, or Celery just chooses not to include the message, which wouldn't surprise me.

It looks like this arises from a document with children marked is_localizable = False.

1) How did this happen?
2) Can we clean it up in pure SQL? Something like...

> UPDATE wiki_document d1 SET is_localizable = 1
> WHERE (SELECT COUNT(*) FROM wiki_document d2
> WHERE d2.parent_id = d1.id) > 0;

If we can clean it up, let's do that ASAP, even if we can't find the cause.

Comment 4

[Paul Craciunoiu [:paulc]]

This was cleaned up on Thursday by Ricky and I, and bug 619836 better handles future exceptions of this kind.

Comment 5

[Rebecca Billings [:rbillings]]

Verified no longer able to documents with children marked as not localizable