793762 - fix indexing so we're using language-specific analyzers

Status: RESOLVED DUPLICATE of bug 889890

(support.mozilla.org :: Search, defect)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

Currently, we index all content using an English-based analyzer. The problem with this is that the analyzer is using English stems and stop words.

We need to fix this to use the correct analyzer.

AMO does this currently. They have a field per language per model field. e.g. In Kitsune, we'd have document_title_en-US, document_title_fr, document_title-it, ... Each field uses the appropriate language analyzer and the content goes into the field for the language it's in.

This is a major change to indexing as it changes the mappings and also the indexing code itself. It will require a two-phase push to production.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

I'm pretty sure this is a 3-point bug. It requires some research to make sure we know how to do it, work to implement it, and then some work to test the results. Additionally, we need to reorder the commits so that we can do a two-phase push to production.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

As a side note, I think Rob is revisiting how AMO does locale-specific analysis now. Just mentioning that here since it's prudent we look at implementation options instead of just doing what AMO is doing.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

Marking this as a duplicate of bug #889890. We're not doing the solution proposed here, but we are solving the same problem.