607318 - [tracker] Improve quality of search results

Status: RESOLVED WORKSFORME

(support.mozilla.org :: Search, defect)

Description

[Verdi [:verdi] Best to slack me]

This one is hard to explain. Here's a video illustrating the problem (2min):
http://sumo.graymattergravy.com/videos/nonsense.ogv
The issue is that the user searched for "can't find downloaded files" and we don't have an article that addresses that specific problem. The search results though, instead of turning up forum posts like https://support.mozilla.com/rm/questions/754176 or KB articles like https://support.mozilla.com/en-US/kb/Downloads+window turn up results that are irrelevant with our least trafficked article listed as the #1 result: https://support.mozilla.com/en-US/kb/AUS+Update+XML+File+Malformed+200

Comment 1

[David Tenser [:djst]]

The exact search query on kitsune: https://master.support.mozilla.com/k/en-US/search?q=can%27t+find+downloaded+files&x=0&y=0

Comment 2

[James Socol [:jsocol, :james]]

The Question you linked isn't marked as solved, so is excluded from the default search. Maybe it's worth reconsidering that? We could search "questions with answers marked helpful" by default, instead, for example.

The article, "Downloads window" doesn't have the search term "find", so is unlikely to match, since I think we're in "ALL" mode now (see bug 607306). (If "can't" is treated as a unit, and I'd have to check that, it doesn't appear in the article, either.)

This situation is the reason for the keywords field: try sticking "can't find downloaded files" into the keywords of "Downloads window".

Comment 3

[David Tenser [:djst]]

(In reply to comment #2)
> The Question you linked isn't marked as solved, so is excluded from the default
> search. Maybe it's worth reconsidering that? We could search "questions with
> answers marked helpful" by default, instead, for example.

Oh yeah, we should definitely change that. A question with answers marked helpful is as close to "solved" as you can get without hearing back from the original poster.

Cheng, any additional thoughts?

Comment 4

[[:Cww]]

At some point we had the feature of emailing the user back if they haven't voted in X days.  What happened to that?

Also, having a helpful answer should be included in search results.  In fact, I think every thread should be included in search results since we have the whole me-too system.

Comment 5

[James Socol [:jsocol, :james]]

(In reply to comment #4)
> In fact, I
> think every thread should be included in search results since we have the whole
> me-too system.

Just clarifying: all threads are indexed and searchable via advanced search, we're just talking about the default, basic search behavior here.

Comment 6

[Kadir Topal [:atopal]]

It's probably a matter of prioritization. People will already see those unanswered threads when they want to ask a question. But in the search results they would need to be the lowest priority.

Comment 7

[David Tenser [:djst]]

In reply to comment #4)
> At some point we had the feature of emailing the user back if they haven't
> voted in X days.  What happened to that?

Not related to this bug, but yeah, we want that. It was a P2 I think, so it wasn't included in the release? (I'm guessing)

> Also, having a helpful answer should be included in search results.  In fact, I
> think every thread should be included in search results since we have the whole
> me-too system.

Agreed -- showing all results actually makes more sense and now that I recall, that's what we discussed when designing the new forum. I guess this change slipped the radar. Of course, unsolved/unanswered-helpful posts should have lower priority if all other weights in the search match are the same.

As James said elsewhere, this is likely an iterative process to get it exactly right.

Comment 8

[Verdi [:verdi] Best to slack me]

I guess I'm hoping that we can find some way to make the search smarter.

Unfortunately, for the search "can't find downloaded files" the article "AUS Update XML File Malformed 200" has a term in it's title and versions of all of them in the actual article. The problem is this article has absolutely nothing to do with the question. In fact all of the results are useless. But if I type that same search into Google, it knows what I'm talking about. Can we make our search smarter without being Google? 

From the other example in the video:
When you do a search for "find recently downloaded" on kitsune https://master.support.mozilla.com/en-US/search?q=find+recently+downloaded&x=12&y=18&page=1 you get "Quick find bar opens when typing in text fields" as the top article. It has one term - "find" in the tile and part of another - "download" in the article itself. It has no keywords. The "Downloads window" article doesn't show up at all in the results yet it has part of one term in it's title - "Downloads"and the actual term "downloaded" in the article itself plus it has the keyword "Downloads". It doesn't make sense that Downloads Window doesn't show up at all. 

I'll try adding new keywords to the Downloads Window article but with a different search I ran into the problem in bug 607306:
I added the keywords "Flash flashplayer player plugin crash adobe" to "The Adobe Flash plugin has crashed" but a search for "What to do when flashplayer crash" still gets zero results. Limiting the search to just "flashplayer crash" does give you "The Adobe Flash plugin has crashed" listed number one.

Comment 9

[Verdi [:verdi] Best to slack me]

(In reply to comment #2)
> 
> This situation is the reason for the keywords field: try sticking "can't find
> downloaded files" into the keywords of "Downloads window".

I did this and the result is that Downloads Window is now #1 for "can't find downloaded files" but AUS Update XML File Malformed 200 is #2 and the rest of the results are meaningless. Also Downloads Window again doesn't show up at all in a search for "find recently downloaded" - the users first attempt at this search.

So I think keywords certainly have their place and will help, we still need some way to make the search smarter.

Comment 10

[James Socol [:jsocol, :james]]

I think we can do a lot better just by fixing two things:

* We're using the default match mode, ALL.
* We have almost no stop words.

That means (A) if you search for "what to do when flashplayer crash", the terms used are "what" "to" "do" "when" "flashplay" "crash", and (B) if the article is written using declarative language, "what" is probably not in it. Combining A and B excludes the article from the results.

We're fixing the stop word issue in bug 586034. Since this is clearly causing you headaches, I'm willing to up the priority (it's not currently a 2.3 blocker).

As I said in bug 607306, I think we should be using the EXTENDED match mode, not ALL. Unfortunately that switch isn't trivial, since it means we need to add some input handling to stop users from triggering Server Errors. So unless we get 2.3 buttoned up sooner than I expect, I don't think we'll be able to get to this.

Those will both make things better.

But before, and even with these changes, remember the first rule of SEO: "Content is king." The single best way to affect the results for a query is to change the content. If you don't want the AUS article to show up in searches for "download," reword it to remove the word "download." If you want more articles to come up, add content about finding downloaded files to them.

(Note that I'm talking about what ends up in the result set. When we talk about ordering the result set there are different things to consider.)

Comment 11

[David Tenser [:djst]]

(In reply to comment #10)
> I think we can do a lot better just by fixing two things:
> 
> * We're using the default match mode, ALL.
> * We have almost no stop words.

Yeah, things will especially improve once bug 607306 is fixed, as we will still show matches even if not all words are in the results.

To put things into perspective though, even without these two fixes, the Kitsune search demonstrably works much better than the current one according to tests Michael already did, so 2.3 will be an improvement regardless. Yay!

Comment 12

[Kadir Topal [:atopal]]

I think we can close this, since we have fixed numerous things in individual bugs, and there is no way to actually call this bug done.