799445 - [october 21st] check CTR for A/B testing on search results excerpts

Status: RESOLVED FIXED

(support.mozilla.org :: Search, defect, P2)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

This bug is to make sure we check the CTR results for the A/B test set up in bug #790425 on or around October 16th.

I added a querystring parameter "esab" which has values "a" and "b".

"a" - User sees the first 500 characters of question_content.
"b" - User sees at most 3 fragments of question_content as determined by ES.

Then we can decide whether we need more CTR data, whether it mattered to users at all, or which one was better and make the appropriate code changes.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Assigning this to Kadir who will either do the CTR analysis or pass it off to whoever should.

Comment 2

[Kadir Topal [:atopal]]

This went into effect on 2012-10-09 at 14:20:40 PST

Comment 3

[Ricky Rosario [:rrosario, :r1cky]]

Kadir wants to have 10 days of data.

Comment 4

[Kadir Topal [:atopal]]

The data is in. I had to remove the 12th, because it was dominated by 'spam'


date	 CT a	CT b	mobile	total CT  searches CTR a  CTR b	CTR total
10.10.12 11675	11660	406	23741	  31901	   0,37	  0,37	0,74
11.10.12 11523	11196	439	23158	  31439	   0,37	  0,36	0,74
13.10.12 9424	9553	420	19397	  26074	   0,36	  0,37	0,74
14.10.12 9578	9682	472	19732	  26422	   0,36	  0,37	0,75
15.10.12 10872	11093	338	22303	  29557	   0,37	  0,38	0,75
16.10.12 10827	10516	414	21757	  28715	   0,38	  0,37	0,76
17.10.12 10458	10570	441	21469	  28396	   0,37	  0,37	0,76
18.10.12 10310	10175	430	20915	  27641	   0,37	  0,37	0,76
19.10.12 9618	9710	427	19755	  26038	   0,37	  0,37	0,76
total	 94285	94155	3787	192227	  256183   0,37	  0,37	0,75

As you can see the mean CTR for both is the same. I also looked at whether one of them was giving us a better CTR for forum posts specifically, or for KB articles specifically. The result was negative. There is, for all practical purposes, no difference in CTR between the two options.

However, both options are a huge improvement over the past. I looked at the CTR for forum and KB results separately and graphed them over time :http://www.evernote.com/shard/s4/sh/90b36703-9ed3-4b3b-b0af-c7803349868f/90e12dae1ea2216a2238866f533bef06

There was huge jump by about 9 percentage points in forum CTR which did not affect the KB CTR negatively. Awesome news! We just need to decide whether we want to keep option a or b, and we can rely on our intuition here, since the data is inconclusive.

Comment 5

[Ricky Rosario [:rrosario, :r1cky]]

(In reply to Kadir Topal [:atopal] from comment #4)
> We just need to decide whether
> we want to keep option a or b, and we can rely on our intuition here, since
> the data is inconclusive.

I vote for option a because it plays nicer with Bug 800896 (highlighting kb search summaries). We don't really want those to be fragmented and we can't apply different logic to kb vs questions.

Comment 6

[Kadir Topal [:atopal]]

I heard from Michael that he prefers option a too. Personally I find option a better for consistency with KB results. So, Matt, if you are also okay with it, let's go with option a.

Comment 7

[Ricky Rosario [:rrosario, :r1cky]]

We removed option b from the code:
https://github.com/mozilla/kitsune/commit/6ce9e179bfe47c2f804e20b3d23dc9b97c678a0f

Comment 8

[Matt Grimes [:Matt_G]]

Sorry for the delay. Option A is perfect!