755843 - Enable proper search weights in production

Status: RESOLVED FIXED

(support.mozilla.org :: Search, defect, P1)

Description

[Matt Grimes [:Matt_G]]

Currently search is treating all matches with equal weight. This is impacting our search results. 

We'd like to do a quick sanity check with waffling so I can get a general sense of search accuracy. Once the sanity check is done, please waffle 50% of users on the new search weights and 50% on the current equal weights. We will monitor the CTR for a few days before making the change for all users. 

We will need the urls for the waffling to pull the CTR. Let me know if you have any questions.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

So... couple of questions:

1. Next sprint I'll finish up the work for the unified search results view. I was under the impression we were going to leave the existing bucketed search results view as is and only fiddle with weights and other similar things with the unified search results view. Is this supposed to apply to the unified search results view, the bucketed search results view, or both?

2. This doesn't specify the weights you want to use. Do we have weights we want to use? We could use the weights we were using with Sphinx as a start.

Comment 2

[Matt Grimes [:Matt_G]]

Hey Will. I spoke to Cheng and it looks like CTR reporting is broken right now anyway. We can discuss in more detail when we meet on Tues as I'll need to fully understand the unified search question. I was under the impression that these values were the "Out of the box" values that we were going to use:

http://kitsune.readthedocs.org/en/latest/searchchapter.html#search-scoring

We can tweak them as needed after we see how well they work.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

Those are the values that Sphinx search uses and that I thought our Elasticsearch code was using, but discovered last week that wasn't the case because the weights weren't being applied.

I'm game for using those weights. Other than "that's what we used for Sphinx", I don't know what they're based on.

It's less "unified search" and more "unified search results". Right now the site shows a "bucketed results view"--it shows 10 kb articles first, then shows support forum results. It's bucketed in that the results come from one group, then the next regardless of the scoring.

We're moving to a "unified search results" view where the results are interweaved and ordered only by score. i.e. One big result set rather than kb results first, then support forum results, ...

The rest of that work will hopefully get done in the next sprint. Then you'll have a unified search results view to work with.

However, the plan is that we're going to tinker with the scoring to get it to "good enough" first before letting regular users see the unified search results view. Given that, I'm not sure the plan you've outlined in the description of this bug will work.

We can discuss more on Tuesday, but that's the gist of where things are at.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

Putting this in the 2012.12 sprint and making it a 1 pointer. All we're going to do for this bug is add query-time weights to the ES unified search results view using the numbers we use for Sphinx search.

Any additional work will be shunted to other bugs.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

Also, making this a P1. This needs to get done for the next sprint.

Comment 6

[Matt Grimes [:Matt_G]]

This sounds great. Can we waffle a couple of days in Prod to see current vs weighted search? I'd also like to grab Verdi and few others to do some manual testing before we push to prod 100%. Sound good?

Comment 7

[Will Kahn-Greene [:willkg] ET needinfo? me]

Let's try not to use shorthand since it gets confusing because there are a lot of things going on here.

The current plan is that I'm pushing out the code changes for the unified search results and that they will be behind the esunified waffle flag. Users won't see it until the search results are "good enough".

This bug is one of the things we'll be doing to get the search results "good enough".

I'm hesitant to add a waffle flag covering weighted and unweighted ES unified search results. Why is weight vs. unweighted an interesting contrast? Depending on the outcome of that, what's the next step?

Comment 8

[Matt Grimes [:Matt_G]]

If we won't be unveiling any changes until the results are "good enough" then I am fine with your plan. My concern is that we assume adding the weights as implemented in Sphinx will be an improvement. In the off chance that it is not an improvement, we would want to know so that could either back the weights out or tweak them as needed. It sounds like all of that can still be done before any users see the change. Is that correct?

Comment 9

[Will Kahn-Greene [:willkg] ET needinfo? me]

Matt: Seems like there's some confusion here, but I don't understand what's going on. Let me know when you're around and we can chat on skype, then summarize the conclusion here.

Comment 10

[Will Kahn-Greene [:willkg] ET needinfo? me]

Talked to Matt via the phone.

All we're going to do here is add the weights that we used in Sphinx to the unified search results view.

After that, any further weight tweaking will be spun out from work done on the analysis bug #765768.

Comment 11

[Will Kahn-Greene [:willkg] ET needinfo? me]

In a pull request now: https://github.com/mozilla/kitsune/pull/660

Comment 12

[Will Kahn-Greene [:willkg] ET needinfo? me]

Landed in master in https://github.com/mozilla/kitsune/commit/d7ad2f8fb9bf6e2cabbb3a0824a26a10a12cb263

Comment 13

[Will Kahn-Greene [:willkg] ET needinfo? me]

Pushed to production just now.