793010 - Research: Elastic search

Status: RESOLVED FIXED

(developer.mozilla.org Graveyard :: Dashboards, defect)

Description

[John Karahalis [:openjck]]

We talked about using Elastic Search to power some of the items on the MDN dashboard. We should look into this.

We decided to timebox this at about 1-point worth of work. If you find yourself spending more time than that, let me know and we can set aside more time for research next Sprint.

Please capture your findings at the following page. Again, this is sort of a experiment in using the Wiki to help with planning. Let me know if this helps, or if it's just a distraction.

https://wiki.mozilla.org/MDN/Development/Components/Dashboards

Comment 1

[John Karahalis [:openjck]]

The first (and maybe only) action on this is to start an email thread with SUMO to get advice on what to do next.

Comment 2

[Luke Crouch [:groovecoder]]

Let's get the advice here so it's more public and accessible.

James & Will, how do you suggest we start with ElasticSearch on MDN?

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

First, you'll need a library to talk to ElasticSearch. It's got a REST api and everything is in JSON which is nice, but ... you'll want an abstraction layer to make it easier to use.

There are (IMO) two good alternatives for ElasticSearch libraries:

1. django-haystack: http://haystacksearch.org/
2. ElasticUtils: http://elasticutils.readthedocs.org/en/latest/

The former works across a bunch of different search systems and acts like the Django ORM and tries to make it really easy to add search to your Django app. I'm using it with richard, but that's it.

The latter is maintained by Mozilla. We've been fleshing out issues with the project and slowly restructuring it to be easy to use and convenient, but still allow you "bare-metal" access to ElasticSearch. We've succeeded in that in some ways and sucked in others. It's undergoing a lot of rapid improvement, still. That might slow down soon. SUMO, Input, Mozillians and AMO all use ElasticUtils. Probably some other projects, too.

My vote is that the first thing you do is figure out which library you want to use.

After doing that, you need to figure out what you're indexing and how you want to access it and then it's the IR equivalent to DB stuff.

Comment 4

[Luke Crouch [:groovecoder]]

I had a so-so experience with haystack & solr about a year ago - it's nice for doing basic search in a django-like way, but I quickly reached for pysolr to do stuff like facets and boosting.

So, I vote for elasticutils.

We will index docs first for sure - Document objects/records with HTML content and metadata including tags. I'd say we do:

* https://github.com/mozilla/kitsune/blob/master/apps/wiki/models.py#L558
* https://github.com/mozilla/kitsune/blob/master/apps/wiki/models.py#L583

-product, -is_archived, -recent_helpful_votes
+tags, +review_tags, +creator

decide: 

* use categories on docs and add them to index?
* restore keywords and add them to index?

Comment 5

[David Walsh :davidwalsh]

Does this have a rest interface we can hook up an autocompleter to?

Comment 6

[Luke Crouch [:groovecoder]]

Sure does. http://www.elasticsearch.org/guide/reference/api/

Closing this research bug now that I've created https://bugzilla.mozilla.org/showdependencytree.cgi?id=794525&hide_resolved=1