724849 - implement writeindex for elastic search

Status: VERIFIED FIXED

(support.mozilla.org :: Search, defect, P1)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

(Cribbing from Ricky's email:)

Basically, add a version to each index and separate out the configuration setting for the read and write indexes. The way the code is setup now, this was pretty easy to do [1] (unless I missed something!).

If we have to do a reindex, we would:

1- Push a new settings.py with `ES_WRITE_INDEXES['default'] = 'sumo_v{next}'`
2- Press the reindex button and wait until it is done
3- Push a new settings.py with `ES_INDEXES['default'] = 'sumo_v{next}'`

If we have to do a remapping, we would:
1- Same as #1 from above in addition to updated `Model.get_get_mapping()`
2- Same as #2 above
3- Same as #3 above in addition to the new code that requires the new mapping(s)

I did some quick and dirty test runs locally fiddling with my settings and it seems to work.

Pros:
* Very little code changes now
* No downtime other than deploying code

Cons:
* Annoying two commit process to reindex (if this is very infrequent, then who cares?)
* During the time between steps 1 and 3, the live read index won't be updated.
** We could potentially change the `SearchMixin.index()` method to write to both indexes if they are different.

https://github.com/rlr/kitsune/compare/es-write-index

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

I'm putting this in the 2012.3 sprint to get fleshed out and implemented.

Making it a 2 pter since some work has been done on it already.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Whoops--blocked on the wrong bug.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

I'm almost done working on this. I need to:

1. add the ability to delete non-read/write indexes to the search admin

2. go through and double-check to make sure everything is doing things to the correct index

Almost done, though.

Things I'm doing above and beyond Ricky's changes:

1. Fixing code to work with dual indexes better.

2. Updating eswhazzup and the search admin so these utilities give you a high-level view of exactly what's going on making it easier to discover mistakes and less likely we incur manual errors.

3. Updating documentation.

Comment 4

[Ricky Rosario [:rrosario, :r1cky]]

I am pretty sure willkg is working on this.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

I'm adding code that allows us to delete indexes from the search admin interface. I was thinking of putting the following precautions on it:

1. you can't delete the index used for reading
2. you can't delete indexes when we're reindexing

Does that make sense? I don't want to overdo it, but I want to minimize the chance of getting into a downtime/hosed state in production.

Also, how do we want to deal with dev/staging? Do we want to deal with having to push twice for reindexing?

Comment 6

[Ricky Rosario [:rrosario, :r1cky]]

(In reply to Will Kahn-Greene [:willkg] from comment #5)
> Does that make sense?
Yes.

> Also, how do we want to deal with dev/staging? Do we want to deal with
> having to push twice for reindexing?
Other than testing the approach out, we don't really care about reindexing downtime on dev and stage. We would have to remember to reindex every time the index name changes though.

Comment 7

[Will Kahn-Greene [:willkg] ET needinfo? me]

Pull request: https://github.com/mozilla/kitsune/pull/493

Comment 8

[Will Kahn-Greene [:willkg] ET needinfo? me]

Landed in master in https://github.com/mozilla/kitsune/commit/2faae1501d5cc6bcf17ebaded0d5960460543222

Yay!

Marking this as [qa-] since it's all push-to-production kinds of stuff.

Comment 9

[Rebecca Billings [:rbillings]]

Closed as [qa-]