Closed Bug 688463 Opened 15 years ago Closed 14 years ago

Filter or de-emphasize old questions in elastic search results

Categories

(support.mozilla.org :: Search, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WORKSFORME
Future

People

(Reporter: jsocol, Unassigned)

References

Details

(Whiteboard: u=user c=search s= p=1)

(In reply to Ibai from bug 670428 comment #10) > No trying to stop the migration because I'm 100% agree that the fine tunning > should happen after we move to the new Search Engine. > > That said, Kadir's last comment is based on a discussion we had yesterday. > My big concern is that showing not-helpful/not-solved questions that are old > will not be that helpful after all. Here is the usecase: > > - User searches for "tabs randomly closed". > - No KB article is found. > - One thread found in the Support Forum. One year old and without answers. > - User visits the threads and adds a "Me too". > - User will expect to be contacted once the thread gets an answer. > - The thread doesn't get surfaced to Contributors so never gets answered. > - User never gets an answer to his issue. > > The only positive usecase I see is to avoid thread duplication, but this is > only positive when the thread is going to have an answer. > > I'm trying to think about a positive usecase for older threads and I can't > find any (i.e. why is better to show not useful content than no content at > all?) Three solutions, from easiest to hardest: * "Easy". By default, filter out questions older than X (where X is something like 6 months or 1 year). * "Middle". (I don't know if oedipus/EU supports something like this.) Set the Sort Mode for questions to SPH_SORT_TIME_SEGMENTS. This buckets the results into last hour/day/week/month/3-months/everything before ordering by relevance, so the newest results are at the top even if they're not as good of a match. * "Hard". Make oedipus support Sphinx's SPH_SORT_EXPR[1] mode, and define an expression that involves age. Thoughts? [1] http://sphinxsearch.com/docs/manual-0.9.9.html#sorting-modes
I think we should do "Easy in any case". It just doesn't make much sense to surface stale questions. The used Firefox version will be outdated, extensions and websites will have evolved and the original poster will probably not come back to mark any solution as solving. So, anything without a reply and older than 6 month should be discarded anyway. The question there is how far to the present should we prune. I'd propose looking at the time it took 80% of the questions that are solved to be marked as solved. If that's 30 days, then we won't show anything older then 30 days. We should probably start with 6 month, measure that value over a few weeks and adjust based on the findings. As to the middle, I'm not sure if I understand correctly, but I'd say recency should not trump relevancy, especially when we limit recency to 6 month or less anyway. About hard: Not sure how long it would take to configure this and especially to test and refine it based on real results, so I can't say anything about it.
I sort of intended easy/middle/hard as mutually exclusive. We really don't want to spend a ton of time on Sphinx improvements right now. Let's do the easy one. Kadir, can you do whatever research you need to give us a cutoff time? Then this is pretty simple to move forward.
Assignee: nobody → a.topal
Blocks: 670428
James, like I said, let's start with 6 month and take it from there.
(In reply to Kadir Topal [:atopal] from comment #1) > So, anything without > a reply and older than 6 month should be discarded anyway. I'm totally on board with actually throwing out questions older than 6 months, and if that's what you meant you should go ahead and file a bug for it ;) > The question there is how far to the present should we prune. I'd propose > looking at the time it took 80% of the questions that are solved to be > marked as solved. So you don't want to look into this first, just go with 6 months? > If that's 30 days, then we won't show anything older then > 30 days. We should probably start with 6 month, measure that value over a > few weeks and adjust based on the findings. What metrics are we going to use to evaluate this?
(In reply to James Socol [:jsocol, :james] from comment #4) > (In reply to Kadir Topal [:atopal] from comment #1) > > So, anything without > > a reply and older than 6 month should be discarded anyway. > > I'm totally on board with actually throwing out questions older than 6 > months, and if that's what you meant you should go ahead and file a bug for > it ;) Questions older than 6 month can still be valuable if they have an answer. For questions without a reply that's probably not true, but I guess throwing them out would have other implications. I haven't thought about that enough, but in any case, what I mean is discarding them in search results. > > The question there is how far to the present should we prune. I'd propose > > looking at the time it took 80% of the questions that are solved to be > > marked as solved. > > So you don't want to look into this first, just go with 6 months? > > > If that's 30 days, then we won't show anything older then > > 30 days. We should probably start with 6 month, measure that value over a > > few weeks and adjust based on the findings. > > What metrics are we going to use to evaluate this? We need to set this up, but it would probably be a simple DB query, that we can run once a week for 4 weeks to determine a value based on real data. Is there a way to compute how long it took for a question to be marked as solved? I'm sure there is a time stamp for question_created, but is there also a time stamp for question_solved?
And sorry, jumped the second question. Yes, let's start with 6 month before we do any measuring, because resurfacing older questions without replies will probably change the results anyway.
(In reply to Kadir Topal [:atopal] from comment #5) > Questions older than 6 month can still be valuable if they have an answer. > For questions without a reply that's probably not true, but I guess throwing > them out would have other implications. I haven't thought about that enough, > but in any case, what I mean is discarding them in search results. Sphinx can't really do ORs like this, unfortunately ("has_answer OR created > now-6months"). Another reason to move quickly to ES. > > What metrics are we going to use to evaluate this? > > We need to set this up, but it would probably be a simple DB query, that we > can run once a week for 4 weeks to determine a value based on real data. Is > there a way to compute how long it took for a question to be marked as > solved? I'm sure there is a time stamp for question_created, but is there > also a time stamp for question_solved? You'll need to look at the created timestamp of the answer identified as the solution. We don't have a timestamp right now for the "solved" event. (In reply to Kadir Topal [:atopal] from comment #6) > And sorry, jumped the second question. Yes, let's start with 6 month before > we do any measuring, because resurfacing older questions without replies > will probably change the results anyway. We can go ahead and make this change but what metrics will we look at for search result quality?
(In reply to James Socol [:jsocol, :james] from comment #7) > (In reply to Kadir Topal [:atopal] from comment #5) > > Questions older than 6 month can still be valuable if they have an answer. > > For questions without a reply that's probably not true, but I guess throwing > > them out would have other implications. I haven't thought about that enough, > > but in any case, what I mean is discarding them in search results. > > Sphinx can't really do ORs like this, unfortunately ("has_answer OR created > > now-6months"). Another reason to move quickly to ES. Hmm, without the age restriction this will probably mess up the results pretty bad. Is there no workaround for this? Like first filter for >has_answer and in a second step for now-6month? You'd think that filtering is the core of any search engine. Well, if it's not doable let's put this on hold, until we have ES. But I'll at least rant about Sphinx on Twitter. > > > What metrics are we going to use to evaluate this? > > > > We need to set this up, but it would probably be a simple DB query, that we > > can run once a week for 4 weeks to determine a value based on real data. Is > > there a way to compute how long it took for a question to be marked as > > solved? I'm sure there is a time stamp for question_created, but is there > > also a time stamp for question_solved? > > You'll need to look at the created timestamp of the answer identified as the > solution. We don't have a timestamp right now for the "solved" event. I didn't write any DB queries on my own so far. I take it that it's possible to filter for has_solution, and then identify answers marked as solution related to those filtered questions? > (In reply to Kadir Topal [:atopal] from comment #6) > > And sorry, jumped the second question. Yes, let's start with 6 month before > > we do any measuring, because resurfacing older questions without replies > > will probably change the results anyway. > > We can go ahead and make this change but what metrics will we look at for > search result quality? This would not be about search result quality, just about how long it takes 80% of the solved questions to be solved. I assume that when we add questions without replies that time will go up. Because questions without replies are basically dead after a day for now, unless someone finds them via Google. With this change those questions can get votes and will trend up on the "most requested" list. The search result quality on the other hand is tracked via the indicators here: http://etherpad.mozilla.com:9000/kitsune-next-gen-search
Depends on: 694979
Target Milestone: 2011Q4 → ---
Whiteboard: u=user c=search s=2012.1 p=
Target Milestone: --- → 2012Q1
Only for ES.
Assignee: a.topal → nobody
Summary: Filter or de-emphasize old questions in search results → Filter or de-emphasize old questions in elastic search results
For 2012.1, research how to do this.
Whiteboard: u=user c=search s=2012.1 p= → u=user c=search s=2012.1 p=1
We'll probably have to figure out how to get ahold of age in ES, since, unlike in Sphinx, we're not reindexing every 20 minutes.
In order to do an "age", we'd have to do what we're doing now: update all of the questions in the index every day so that the age increments appropriately. Otherwise the age field gets stale. However, it's possible we don't need an "age", but rather just a "last updated date". If we had the latter, then in ES we could do filters like (in ridiculous pseudo-code): today_minus_six_months = datetime.now() - timedelta(months=6) question_s.filter(last_updated_date__lt=today_minus_six_months) Is that good enough?
I had a stupid-great idea: we add the last-updated time to documents when we index/update them. Then we write a cronjob that uses an ES query for all the ones that haven't been updated in 24 hours and updates them. And by "update them" we could either: 1. do a real update where we pull all the data from SQL and then rebuild the document and reindex it 2. we can just pull out the age field, add one, and push that back in--no db stuff If we did the former once a day, then we possibly catch all the failed incremental indexing that happened that day. It'd be our fail-safe cleanup pass that fixes the state of things. Any thoughts on that?
#1 is basically equivalent to a complete daily rebuild, modulo the docs that have changed that day. To make it much better, we'd have to store a "Hey, doc ID such and such should have been updated at such and such time" off to the side someplace, in something more reliable, and then bang those against your ES query result to do an "audit". And if we're going to do that, we might as well just put a "Hey maybe try Doc X again" in a queue someplace when our exception handler catches a failure. (All these failures happen detectably, don't they?) #2 is still going to involve having ES reindex, isn't it? I didn't think ES docs were mutable on a per-field basis, though it would be cool if they were. Personally, I'm still holding out hope that there's a way of doing some math in ES. I've been meaning to get around to looking.
Had you seen http://www.elasticsearch.org/guide/reference/query-dsl/custom-score-query.html? I think we can do exactly what we want, making score dynamically proportional to age. The unblemished score can be got out of the _score var, and then we can say doc['created'] to get the numeric age out (which we might have to store in second since epoch or something). I already see (roughly) how we can add this to elasticutils without making a mess.
Trading this one off with bug 715665, which is oh-so-much-more-blocking.
Whiteboard: u=user c=search s=2012.1 p=1 → u=user c=search s=2012.2 p=1
Whiteboard: u=user c=search s=2012.2 p=1 → u=user c=search s= p=1
Moving all 2012Q1 bugs to Future.
Target Milestone: 2012Q1 → Future
We can close this off, since we are not showing any threads older than 180 days at all anymore.
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.