797321 - re-add text_phrase

Status: RESOLVED FIXED

(support.mozilla.org :: Search, defect, P2)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

This bug is for undoing the change in bug #796979.

We want enough time to measure CTR, so we shouldn't do this until October 10th. Also, talk to Kadir before making the changes.

Making this a 0 pointer since it's easy peasy to do. Also, making it block the excerpts bug.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

We're changing the order of events and planning to do this after we make changes to excerpts. Given that, this should get moved to the 2012.20 sprint.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Making this block on bug #799445 since we want to know what effect adjusting excerpts have before we re-add text_phrase.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

Anyone can do this--it's pretty easy. It's changing line 324 (or somewhere around there) in apps/search/views.py:

           for field in query_fields:
                # Note: Commenting this out for a week to watch CTR.
                # for query_type in ['text', 'text_phrase']:
                for query_type in ['text']:
                    query['%s__%s' % (field, query_type)] = cleaned_q

to:

           for field in query_fields:
                for query_type in ['text', 'text_phrase']:
                    query['%s__%s' % (field, query_type)] = cleaned_q

That's it.

This is just waiting for Kadir to finish his CTR analysis work and giving us the go-ahead.

Unassigning myself.

Comment 4

[Kadir Topal [:atopal]]

On a hunch I looked at the CTR for forum and KB results separately and graphed them over time :http://www.evernote.com/shard/s4/sh/90b36703-9ed3-4b3b-b0af-c7803349868f/90e12dae1ea2216a2238866f533bef06

As you can see the CTR for KB articles goes down by 10 percentage points on the 25th of September, exactly when text_phrase was pushed to production, and it recovers again on the 3rd of October, exactly when it was turned off again. At the same time we see a jump in forum CTR with text_phrase added. 

I suspect that text_phrase adds more forum results into the top spots for search results. Unfortunately we can't see if people who go on to the forum results are actually happy. However we know that they were happy to go to KB articles before, because the drop in traffic by 10 percentage point did not lead to spike in helpful votes. If those were unhappy with the article results it would follow that sending them to the forum would increase the helpful votes from search in the KB. That is not the case.

I'm not sure yet, what to do given the data. There are several options. I'll lay them out tomorrow. And if you have any idea, let me know.

Comment 5

[Ricky Rosario [:rrosario, :r1cky]]

I think the text_phrase might be better matching question content (since there is so much of it, there is a higher chance there is a match for any given query). This might be causing questions to come up higher than articles. Maybe we need to adjust boosts so that kb articles stay mostly higher when that makes sense.

Not sure if any of that makes sense... tl;dr we might need to adjust boosts as well.

Comment 6

[Matt Grimes [:Matt_G]]

That sounds like a pretty sound approach. I'd be happy to give that a try and see how we do.

Comment 7

[Ricky Rosario [:rrosario, :r1cky]]

We are holding off on this and doing bug 800896 first.

Comment 8

[Ricky Rosario [:rrosario, :r1cky]]

Since we pushed search changes for limiting the age of support forum results, I am pushing this out to the next week.

Comment 9

[Ricky Rosario [:rrosario, :r1cky]]

Moving to next sprint.

Comment 10

[Ricky Rosario [:rrosario, :r1cky]]

Landed on master:

https://github.com/mozilla/kitsune/commit/e9ff134448bc0c9030949d5eafee782213592d2d

Comment 11

[Ricky Rosario [:rrosario, :r1cky]]

Just deployed.