746609 - rewrite elasticsearch related code to use one doctype

Status: RESOLVED FIXED

(support.mozilla.org :: Search, defect, P1)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

This covers two tasks because they need to be done together:

1. rewrite the ES indexing to use one doctype 
2. rewrite the elasticsearch search view to simulate the search bucket results with the new index structure

There are a lot of unknowns here, so it's likely we'll unearth problems that require thinking. Also, it's possible we'll have to make changes to elasticutils.

Setting the whiteboard and putting this in the 2012.8 sprint. I'm making this a 3, but that's kind of a lie since I'm estimating that this is a 2 week project or so.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

The more I mull over this the more I realize there are a lot of unknowns here that might make this more complex.

So I'm turning this into a research bug that'll be a 3-pter and the end result will generate a bunch of actual task bugs.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Here's my update:

elasticsearch supports searching multiple doctypes at the same time (http://www.elasticsearch.org/guide/reference/api/search/indices-types.html) however, elasticutils (the API we're using to do searches in elasticsearch) doesn't support that at all and I'm pretty sure it'll be non-trivial to implement since there's an implied 1:1 connection between doctypes and Django models. So... while it's nice elasticsearch supports that, it doesn't really help, so I'm nixing that option.

I spent some time writing code that takes a bunch of mappings and merges it into a single mapping. I figured this was easier to do than doing it by hand, plus it'll tell us in the future if one of our mappings doesn't merge well (it throws an exception).

That unearthed some problems which I fixed--no biggie.

Now I have the problem where elasticutils derives the doctype for a model from the model._meta.db_table. That sucks because we're moving to a unified doctype. I'm now trying to figure out what to do about this. I had similar issues with index names a few weeks ago.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

Pull request for this and some other stuff: https://github.com/mozilla/kitsune/pull/582

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

Landed in master in fdafc92bc9f57c1cef1b387d1a157a72a2342995

Pushed to production just now.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

Nixing the [research] part. I ended up doing the work in the bug itself.