implement indexing infrastructure

RESOLVED FIXED

Status

P2
normal
RESOLVED FIXED
6 years ago
6 years ago

People

(Reporter: willkg, Assigned: willkg)

Tracking

Details

(Whiteboard: u=dev c=infrastructure p=3)

This bug covers implementing the infrastructure for indexing and searching.

The plan is to use ES. I propose we use ElasticUtils because it's in-house and thus easy to modify, and Input is probably going to have some significant search needs.

For the indexing side, I propose we look at what's in ElasticUtils in the django contrib module and use as much of that as we can and upstream fixes. We should also take inspiration from the Kitsune search infrastructure.

This covers:

1. adding elasticutils to vendor-local
2. writing indexing infrastructure
3. writing command-line interface to indexing infrastructure
4. upstreaming any fixes to elasticutils project
5. writing tests for indexing/search infrastructure
6. writing docs for installing ES and maintaining index/search stuff


There's a bunch here, so I'm making it a 3 pointer for 2012.16.

Making this block the dashboard bug, though we could implement part of that without this.
This is partially waiting on:

* https://github.com/mozilla/elasticutils/pull/58
* https://github.com/mozilla/elasticutils/pull/61

I'm working through the rest of it now, though.
Making all of these P2--they need to get done asap.
Priority: -- → P2
Depends on: 788286
This is a huge bug.

I rewrote types in ElasticUtils so it has a new MappingType system that separates index document types from django db model types so the two are different. This fixes a bunch of sharp edges with ElasticUtils and makes it easier for Input to use. That took a while to do, but I landed that work this morning.

I've got esreindex, esstatus and esdelete command lines working. It currently indexes fjord.feedback.models.Simple data.

I still have a few things to do:

* documentation for setting up ES, maintaining indexes, and using the command line utilities
* admin pages for maintaining the index
* live indexing infrastructure (I will probably push this off to another bug)
* tests to verify everything is working and test infrastructure for testing search/index-related things

Probably another day or two of work.
Nixing s= data from whiteboard--scrumbugz manages that itself now.
Whiteboard: u=dev c=infrastructure p=3 s=2012.16 → u=dev c=infrastructure p=3
Oy. I'm breaking this into a few more bugs.

This one covers the search infrastructure and the cli commands and that's it.

1. bug #788524 covers documenting the indexing infrastructure
2. bug #788527 covers implementing the admin pages
3. bug #788529 covers implementing the live indexing infrastructure
Summary: implement search infrastructure → implement indexing infrastructure
Landed in master in https://github.com/mozilla/fjord/commit/855702468fdd32d6f34d4be852f7c6a8e41d81e9
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Duplicate of this bug: 781717
Blocks: 780626
You need to log in before you can comment on or make changes to this bug.