784742 - implement indexing infrastructure

Status: RESOLVED FIXED

(Input :: General, defect, P2)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

This bug covers implementing the infrastructure for indexing and searching.

The plan is to use ES. I propose we use ElasticUtils because it's in-house and thus easy to modify, and Input is probably going to have some significant search needs.

For the indexing side, I propose we look at what's in ElasticUtils in the django contrib module and use as much of that as we can and upstream fixes. We should also take inspiration from the Kitsune search infrastructure.

This covers:

1. adding elasticutils to vendor-local
2. writing indexing infrastructure
3. writing command-line interface to indexing infrastructure
4. upstreaming any fixes to elasticutils project
5. writing tests for indexing/search infrastructure
6. writing docs for installing ES and maintaining index/search stuff


There's a bunch here, so I'm making it a 3 pointer for 2012.16.

Making this block the dashboard bug, though we could implement part of that without this.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

This is partially waiting on:

* https://github.com/mozilla/elasticutils/pull/58
* https://github.com/mozilla/elasticutils/pull/61

I'm working through the rest of it now, though.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Making all of these P2--they need to get done asap.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

This is a huge bug.

I rewrote types in ElasticUtils so it has a new MappingType system that separates index document types from django db model types so the two are different. This fixes a bunch of sharp edges with ElasticUtils and makes it easier for Input to use. That took a while to do, but I landed that work this morning.

I've got esreindex, esstatus and esdelete command lines working. It currently indexes fjord.feedback.models.Simple data.

I still have a few things to do:

* documentation for setting up ES, maintaining indexes, and using the command line utilities
* admin pages for maintaining the index
* live indexing infrastructure (I will probably push this off to another bug)
* tests to verify everything is working and test infrastructure for testing search/index-related things

Probably another day or two of work.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

Nixing s= data from whiteboard--scrumbugz manages that itself now.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

Oy. I'm breaking this into a few more bugs.

This one covers the search infrastructure and the cli commands and that's it.

1. bug #788524 covers documenting the indexing infrastructure
2. bug #788527 covers implementing the admin pages
3. bug #788529 covers implementing the live indexing infrastructure

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

In pull request: https://github.com/mozilla/fjord/pull/19

Comment 7

[Will Kahn-Greene [:willkg] ET needinfo? me]

Landed in master in https://github.com/mozilla/fjord/commit/855702468fdd32d6f34d4be852f7c6a8e41d81e9