528607 - Ensure search v3 accounts for {REDIRECT} articles

Status: VERIFIED FIXED

(support.mozilla.org :: Search, defect, P2)

Description

[Paul Craciunoiu [:paulc]]

We have some articles that are simply redirects of others. These shouldn't show up in the search results. We previously did this in the XML indexer, but we won't be using that anymore.

Comment 1

[Paul Craciunoiu [:paulc]]

I was thinking, worst case, we can do a filtering by page id for existing redirects. That wouldn't add much to indexing time.

Chris: why do we need article redirects? could you explain more about that? could we do them in htaccess instead?

Comment 2

[Chris Ilias [:cilias]]

Two reasons:
* To preserve old URLs of renamed articles.
* To provide alternative names to articles, where we think it is needed. The articles that have text before the redirect are supposed to show up in search results. See bug 489046.

Comment 3

[James Socol [:jsocol, :james]]

If we just add

AND content NOT LIKE '{REDIRECT(%'

to the WHERE clause for wiki_pages in sphinx.conf, we should be good, right?

Comment 4

[Paul Craciunoiu [:paulc]]

That sounds good to me. I hope that won't triple indexing time :)

Comment 5

[David Tenser [:djst]]

To clarify, the two reasons Ilias listed are also two very distinct use cases:

* To preserve old URLs of renamed articles -- these redirects shouldn't be included in search results

* To provide alternative names to articles where we think it is needed -- these redirects SHOULD be included in search results

Typically they're distinguished by being _just_ the {REDIRECT} code in the formst, and also having an actual article summary/description paragraph in the latter.

Chris: can you please clarify how we distinguish the two so we don't end up with a situation where our alternate article names are ignored by the search engine, as that would defeat the purpose of having them?

Comment 6

[James Socol [:jsocol, :james]]

I talked to Chris and he gave me a couple example articles. It shouldn't be a problem.

Comment 7

[James Socol [:jsocol, :james]]

Attachment: tell the indexer to ignore pure {REDIRECT()} articles

This just tells the indexer that if the page content starts with "{REDIRECT" to ignore the whole page. I tested this by searching for "Shockwave".

Comment 8

[Paul Craciunoiu [:paulc]]

Comment on attachment 413229 [details] [diff] [review]
tell the indexer to ignore pure {REDIRECT()} articles

Looks good, no more "Shockwave" in results for me either.

Comment 9

[James Socol [:jsocol, :james]]

r56386. Should be affecting the results on stage by now.

Comment 10

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED on https://sumo-forumux.stage.mozilla.com/search.php?locale=en-US&q=shockwave&sa=.