867289 - Reindex management command doesn't chain and parallelize tasks properly

Status: RESOLVED FIXED

(Marketplace Graveyard :: Search, defect, P3)

Description

[Rob Hudson [:robhudson]]

I've seen the side effect a couple times but never fully understood why until I wrote a separate "reindex_mkt" management command recently for the upcoming work to separate AMO and MKT search indexes.

The problem is we're chaining tasks with a TaskSet in the middle like:
task1 -> task2 -> task3(n) -> task4 -> task5.

Where task3(n) is a TaskSet of many indexing tasks and tasks 4 and 5 should only be run once all the task3(n)s are done. But when adding debug logging I see tasks 4 and 5 happening before the task3(n)s.

The result is that the new index is aliased before anything is indexed into it causing search results to return empty. The reason we moved to aliasing was to avoid breaking search while a reindexing job was happening.

I've solved it by not using a taskset in the middle and just use a straight line for indexing. This isn't ideal b/c we could parallelize the indexing jobs but our current version of celery doesn't support chords, which solves this problem.

Once bug 867288 lands we should fix this and also remove marketplace specific indexing from the AMO index.

Comment 1

[Rob Hudson [:robhudson]]

Just to be clear, the `reindex_mkt` command works fine. It could be optimized a bit but I'd consider that low priority.

The `reindex` command, however, does need to be fixed.

Comment 2

[Allen Short [:ashort]]

https://github.com/mozilla/zamboni/commit/164d41e4b

Comment 3

[Iulian Timis]

Please add STR here or mark it with [qa-] if no QA is needed.