710469 - Remove wikimarkup from search excerpts and indexes for elastic search

Status: RESOLVED FIXED

(support.mozilla.org :: Search, defect)

Description

[Erik Rose [:erik][:erikrose]]

We can feed the index HTML rather than wikimarkup for Questions and Forums, then use the lovely HTML Strip charfilter (http://www.elasticsearch.org/guide/reference/index-modules/analysis/htmlstrip-charfilter.html). That way, we won't have any more embarrassing equal signs in our search results:

Firefox Crashes at http://de.rs-online.com...com == Crash ID(s) == Crash at http://de.rs-online.com ...To look up....

Comment 1

[Ricky Rosario [:rrosario, :r1cky]]

This is:
* Index html for questions and answers
* Improve display of question results (if necessary)

Comment 2

[Erik Rose [:erik][:erikrose]]

Nah, forums are full of [links] (https://support.mozilla.org/en-US/search?a=1&updated=0&e=sph&created=0&sortby=0&w=4&page=3). For the life of me, I can't get a test question to show up in search.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

Erik: I think you need to give the test question an answer, then mark that answer as helpful. That'll put it in the search results. Otherwise it gets filtered out.

Comment 4

[Erik Rose [:erik][:erikrose]]

Yeah, that wasn't cutting it for some reason. :-(

Comment 5

[Erik Rose [:erik][:erikrose]]

QA: Make sure items with wiki links in them, like [[SomePage|some link text]] aren't found by a search for "SomePage". Likewise, searching for "br" shouldn't find all the wiki pages with <br> tags in them. Exerpts on the search results page should likewise lack both wiki markup ("== Header ==") and HTML.

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

Clarifying that this is specific to Elastic Search searches--not Sphinx searches.

Comment 7

[Will Kahn-Greene [:willkg] ET needinfo? me]

This seems like a duplicate of bug #557451, though that's really old and probably covers whatever search system we had back then.

Should we mark that one as a duplicate of this one and move forward? Or is there something about that one that isn't covered in this one?

Comment 8

[Ricky Rosario [:rrosario, :r1cky]]

Moving to next sprint due to the added dependency on Bug 720935.

Comment 9

[Will Kahn-Greene [:willkg] ET needinfo? me]

We should probably bump this from the 2012.3 sprint. We could look at landing it after the read/write index split lands, but it's not as high priority as the gazillion other things on our plates right now, so maybe it's better to push it off a bit?

Comment 10

[Ricky Rosario [:rrosario, :r1cky]]

(In reply to Will Kahn-Greene [:willkg] from comment #9)
> so maybe it's better to push it off a bit?

Agreed. Let's come back to this once we are done with blockers and higher priority ES bugs.

Comment 11

[Will Kahn-Greene [:willkg] ET needinfo? me]

I'm going to make this block bug #729688 since it'll be a lot easier to get this into production before we ditch Sphinx rather than afterwards.

Having said that, this shouldn't be at the tippy top of the priority list.

Comment 12

[Ricky Rosario [:rrosario, :r1cky]]

Erik isnt around anymore, Q1 is way over, not sure if this still needs to be done

Comment 13

[Will Kahn-Greene [:willkg] ET needinfo? me]

My vote is we keep it around. It's definitely something lame and it probably affects the results scoring, but I think there's still a decent amount of low-hanging fruit that'd affect things more.

Comment 14

[Joni Chan]

This looks like it's fixed? I don't see HTML in the results anymore. Please re-open if it's still an issue.