768242 - Unified Search: Limit age of forum posts in results

Status: RESOLVED FIXED

(support.mozilla.org :: Search, defect, P3)

Description

[Matt Grimes [:Matt_G]]

There should be an upward threshold for the age of forum posts returned in unified search. We assume that newer content will be more accurate and cut down on users "reviving" ancient threads. This threshold should apply to unified search only and should not impact advanced search.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

How do you determine the age of a forum post?

What does "upward threshold" mean?

Comment 2

[Matt Grimes [:Matt_G]]

Some forums posts are more than a year old and unsolved. They are showing up fairly high in the search results in some instances. We should set a limit on how old a post can be if we are going to display it. Kadir, Michael, and I are working out what that limit should be. I'll add that information in the next day or so.

Comment 3

[Kadir Topal [:atopal]]

Baseline: Don't show any question older than 180 days (creation date). We should probably be more aggressive though. We'll work that out later.

Comment 4

[Ricky Rosario [:rrosario, :r1cky]]

See bug 688463 for a bunch of implementation ideas.

Comment 5

[Ricky Rosario [:rrosario, :r1cky]]

Making this a 3pter as there is some research/prototyping that needs to happen to figure out how to implement this.

Comment 6

[Ricky Rosario [:rrosario, :r1cky]]

I talked with willkg again and this is easier than all the comments in bug 688463. We need to add something like `.filter(created__gt=convert_to_timestamp(now()-timedelta(days=180)))`

Also add tests.

I don't think that takes over half day => 1pt

Comment 7

[Ricky Rosario [:rrosario, :r1cky]]

working on this today.

Comment 8

[Ricky Rosario [:rrosario, :r1cky]]

in a pull request:
https://github.com/mozilla/kitsune/pull/938

Comment 9

[Will Kahn-Greene [:willkg] ET needinfo? me]

It's actually: https://github.com/mozilla/kitsune/pull/939

Comment 10

[Ricky Rosario [:rrosario, :r1cky]]

Landed and deployed:

https://github.com/mozilla/kitsune/commit/c170683211985896a7b30cee0c88c1e634b53729

I'm not sure what the CTR will do, but helpfulness in the Support Forum will hopefully go up.

Comment 11

[Kadir Topal [:atopal]]

First day impact is in: http://www.evernote.com/shard/s4/sh/830733d7-6a82-4227-8be3-d3a6e1d958fd/c601a8d599fa47a5b6ac7f11532262be

CTR for KB is up by 7 percentage points, but CTR for forum is down 13 percentage points. Overall CTR is down 6 percentage points. 

The helpfulness in the KB from search is flat, while the helpfulness in the forums from search is still not measurable and the overall helpfulness in the forums is flat. We only have one day of CTR data, so let's wait a little longer before calling anything.

Comment 12

[Kadir Topal [:atopal]]

The first full day is in, and the CTR is down 10 percentage points. At this point one thing is clear to me: We have to find out better ways to judge the impact of this. For once, we'd need to know, are we serving people better or worse now that we eliminate threads that are meant for way older releases?

As far as I can tell, we have currently no automated way to measure that. Here are some ideas on what do do, and please jump in if you have any other ideas:

1. Add helpful vote meta data for forum votes (might help, but will take several weeks to produce the confidence level we need)

2. Add SurveyGizmo survey to those threads that are visited from our internal search (we'd need to figure out how and when to show the survey so it makes sense)

3. Measure how many people click through to a search result and go back to the search results again (I don't think we can get this from the logs, but maybe GA can do that?)

4. Compare results manually. Look at queries that would have produced an old forum thread at the first position before and produces a KB thread now.

5. Experiment with more weight to forum threads now that we have fewer of them.

Any thoughts on this, or other ideas?