rebuild_kb doesn't kick off indexing tasks

NEW
Unassigned

Status

support.mozilla.org
Code Quality
P3
normal
6 years ago
3 years ago

People

(Reporter: willkg, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: u=dev c=search p=2)

The rebuild_kb task doesn't kick of indexing tasks because we have this goofy de-duping code that creates the indexing tasks and clears the queue when a request is completed. Since celery tasks aren't running in an HTTP request context, it probably means the queue grows to a certain point and hangs out.

Ricky, James and I talked on IRC about this and it's not clear we want rebuild_kb to be reindexing things every 6 hours. That seems excessive since we're live indexing and rebuild the kb index once a night.

Ricky mentioned looking at register_for_indexing and not registering in the case of a celery task. That seems like a good idea.

Also, we might want to look at the de-duping code Andy just wrote for AMO. It's different and might solve the problem better.


Making this a 2 pointer. It involves verifying what we're assuming is true is true, then seeing what AMO did, then figuring out whether to rework the de-duping code or to fix register_for_indexing code.
Here's the commit Andy showed me yesterday:

https://github.com/andymckay/zamboni/commit/6edbf40f8f26250b842315ba500ee8794452ebdc

It uses the request_finished signal, so that's pretty much what we're doing, too.
I have no idea if this is still an issue or not. If it is, then we should probably look into it.
You need to log in before you can comment on or make changes to this bug.