749696 - Elasticsearch churn

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: Code Quality, defect)

Description

[Rob Hudson [:robhudson]]

While looking at logs there seems to be a lot of ES indexing calls for the same add-on within seconds of each other. These would then get sent to ES for indexing repeatedly. The cause is the indexing task is spawned when the add-on is saved, which can happen multiple times in a single request/response cycle. We really only need the final save()'s index to happen, but this is easy to say and hard to do.

All of this may be fine, but it may cause unnecessary churn in Elasticsearch. This is especially evident in our tests.

We could remove the post_save signal, but putting an index call where we need it would be difficult.

Another option I've found is to use the Celery batches:
http://ask.github.com/celery/reference/celery.contrib.batches.html

Set it up to flush every few seconds (?) and the task method would get the batch of tasks. The task method itself would need to collapse multiple same tasks into a single task that then gets executed.

Comment 1

[Rob Hudson [:robhudson]]

See https://github.com/mozilla/kitsune/commit/85936b11e91c375e3fec9b86f12e0ad069f92524 for a potential solution for AMO.

Comment 2

[Andy McKay]

https://github.com/mozilla/zamboni/commit/980a44

Only enabled for tests at the moment, I don't know if its as big a deal for requests.

Comment 3

[Andy McKay]

Didn't seem to help the time it took tests to run. But hopefully it helped the ES cluster jenkins uses.

Comment 4

[Rob Hudson [:robhudson]]

(In reply to Andy McKay [:andym] from comment #2)
> Only enabled for tests at the moment, I don't know if its as big a deal for
> requests.

Agreed. Closing...