Closed Bug 484283 Opened 15 years ago Closed 15 years ago

Conduct A/B test on SUMO's inproduct start page

Categories

(support.mozilla.org :: Knowledge Base Software, task)

Product:
support.mozilla.org
Component:
Knowledge Base Software
Type:
task
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: djst, Assigned: oremj)

References

(
URL
)

Details

(Whiteboard: sumo_only) 

Let's conduct a first A/B test on the most trafficked page on SUMO to gain knowledge about users' expectations of the inproduct start page.

The current URL of that page is http://support.mozilla.com/en-US/kb/Firefox+Help. This page is linked to from Firefox itself via .htaccess rewrite rules. The target URL is for the en-US locale is: http://support.mozilla.com/en-US/kb/Firefox+Help?style_mode=inproduct

During a test period of 7 days, rather that directing people to the normal inproduct start page as specified above, distribute the traffic evenly between the two pages:

http://support.mozilla.com/en-US/kb/Firefox+Help+1?style_mode=inproduct
http://support.mozilla.com/en-US/kb/Firefox+Help+2?style_mode=inproduct

For details, see https://wiki.mozilla.org/Support/StartPageOptimization/FirstTest

As for when the test should start, anytime works.

mrz: I assigned this to you to get your attention since we talked about the logistics of this test in a conf call recently. Please help me assign this to the right person. Thanks!
Ken Kovash approves this test plan.
Assignee: mrz → server-ops
Component: General → Server Operations
Product: support.mozilla.com → mozilla.org
QA Contact: general → mrz
Version: unspecified → other
Assignee: server-ops → oremj
When is this or when should it be scheduled to go live?
Someone will need to implement comment #0
Assignee: oremj → nobody
Component: Server Operations → Knowledge Base Software
Product: mozilla.org → support.mozilla.com
QA Contact: mrz → kb-software
Version: other → unspecified
(In reply to comment #2)
> When is this or when should it be scheduled to go live?

Whenever we're ready and you have cycles to spare. The two different start pages are already up and ready to go.

(In reply to comment #3)
> Someone will need to implement comment #0

What part of comment 0 needs to be implemented?
Ping?
Severity: normal → major
Back to server-ops. Jeremy, could you clarify what you mean in comment 3? Thanks!
Assignee: nobody → server-ops
Component: Knowledge Base Software → Server Operations
Product: support.mozilla.com → mozilla.org
QA Contact: kb-software → mrz
Version: unspecified → other
Someone needs to implement at 50/50 redirection (htaccess, php, ...) at the test URL.
Assignee: server-ops → nobody
Component: Server Operations → Knowledge Base Software
Product: mozilla.org → support.mozilla.com
QA Contact: mrz → kb-software
Version: other → unspecified
Who did the one for www.mozilla.com?  Was that webdev?
(In reply to comment #8)
> Who did the one for www.mozilla.com?  Was that webdev?

Yes; Fred did in https://bugzilla.mozilla.org/show_bug.cgi?id=478952#c45
I did one of the mozilla.com ones and webdev did one.  I'm not really as comfortable altering sumo as I am mozilla.com though, so the sumo team should handle this and coordinate a push with IT.
When do you want to go live with this?
Asap please. Friday - next Friday would be good I think. We should test on stage to make sure it works first and then go live.

Thanks!
Is this still going ahead tomorrow?
No one commented after Laura asked and this never got pushed.  

Tenser, can you draw a line in the sand and pick a date we can all commit to?  Thursday night?

Is there anything blocking pushing this out then?
We're pushing 1.0.2 next Tuesday. Laura, would there be any problems from your point of view to push this out Thursday night and have it run for 7 days even if that means we'll push 1.0.2 while this test is still running?
If everything's ready we can push tonight as well.  Not sure what "everything" is.
Just talked to Laura and she's fine with these dates. We need to test on stage first though. 

So, testing on stage - asap
Push it out on prod - Thursday night April 16
Stop test (another push) - Thursday April 23

Is that OK?
Assignee: nobody → oremj
As long as we've verified that the A/B mechanism works on support-stage.mozilla.org, pushing it out tonight would work for me as well, no problem. The content is ready, I just haven't seen the actual behind-the-scene magic to make the A/B test happen.
This doesn't feel like it has to be an announced push (let me know if you disagree) so you basically have all day today to confirm it works before a 7pm push.
Just confirming... there isn't anything we need to do in advance from an Omniture/tracking perspective, so we should be good to go in terms of metrics.
(In reply to comment #19)
> This doesn't feel like it has to be an announced push (let me know if you
> disagree) so you basically have all day today to confirm it works before a 7pm
> push.

Laura can confirm that it works as soon as the patch is applied on stage, which I assume oremj is working on. Some help from QA would be good too -- Stephen?

oremj, let us know when it's ready. Thanks!
Added the following to stage:
  RewriteCond %{TIME_SEC} .[02468]
  RewriteCond %{THE_REQUEST} "^GET /en-US/kb/Firefox\+Help\?style_mode=inproduct"
  RewriteRule . /en-US/kb/Firefox+Help+1?style_mode=inproduct [R]

  RewriteCond %{TIME_SEC} .[13579]
  RewriteCond %{THE_REQUEST} "^GET /en-US/kb/Firefox\+Help\?style_mode=inproduct"
  RewriteRule . /en-US/kb/Firefox+Help+2?style_mode=inproduct [R]

Ken, since we are doing redirects you will see "Firefox+Help+2" and "Firefox+Help+1" in the page name reports.
Is this supposed to be up and running on stage?

Testing the inproduct URL http://support-stage.mozilla.org/1/firefox/3.6a1pre/Darwin/en-US/firefox-help still takes me to http://support-stage.mozilla.org/en-US/kb/Firefox+Help?style_mode=inproduct.
Via Laura: If you edited htaccess.dist then you'll need to run sudo ./htaccess.sh. If you edited .htaccess it should just work.
Doesn't seem to be working - I get the Firefox+Help+1 version every time
Bah, actually /en-US/kb/Firefox+Help?style_mode=inproduct every time.  Forgive me.
Wfm now too.
I didn't push this tonight, because I wasn't 100% sure everyone involved felt it was complete.  I am fine with pushing this out tomorrow or Thursday, if we want to wait for a maintenance window.
This seems to be working perfectly. Push this out anytime, please. Thanks!
Pushed to production.
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
There are a few things in our Omniture reporting that don't make much sense:

(1) Looking at the top pages for *today*, there are about 85K page views for +1 and 85K page views +2... and about 45K page views for the regular "en-US/kb/Firefox+Help".

(2) For the "en-US/kb/Firefox+Help" traffic, almost all of it comes from people directly entering the site (which makes sense since it's the in-product page).  For the +1 and +2 pages, only about half of their traffic comes from people directly entering the site.  The other half of their traffic comes from someone hitting +1, then hitting +2 (or vise versa).

Any insights?
How are people still hitting the Firefox+Help page when we're supposed to distribute those visits to Firefox+Help+1 or Firefox+Help+2?

More importantly for our goal, though, it looks like the bounce rate on +1 and +2 are identical. If that's true, it appears that the main reason for the high bounce rate has virtually nothing to do with the articles we're listing below the search bar.
(In reply to comment #34)

> More importantly for our goal, though, it looks like the bounce rate on +1 and
> +2 are identical. If that's true, it appears that the main reason for the high
> bounce rate has virtually nothing to do with the articles we're listing below
> the search bar.

That actually seems like a logical conclusion when you think about the user flow when using the page.  It seems to me the big fail point is going to be whether or not people understand how to search for the article they're looking for. 

Is there a newsgroup or forum where we're actually discussing the results and what to test next? I have further thoughts, but don't want to clutter the bug.
Feel free to post in this bug.
Also, you'll want to read this: https://wiki.mozilla.org/Support/StartPageOptimization
Ok, well in theory the current flow is focused around the search box, so users

1. Get to the page
2. Search on something
then possibly
3. Not sure what to search so read down further
4. Find what you're looking for in a link 
or
4. Don't find what you're looking for, now what?

Unless there are some specifically popular pages from inproduct odds are very low that changing the links will cover most cases where people need help.  So, if people aren't getting the help they need, we should probably assume it's because they don't know how to effectively search, or that they don't realize they need to search to access the rest of the articles. 

One of two things I think will be the answer, either doing a better job of helping users effectively search or taking the focus off of search and using a better mechanism to get users where they want to go. (A tag cloud, for example, could be helpful in both cases, either supplementing search, or replacing it entirely).

So for the next A/B test I would suggest trying to convince more users to search by playing with the instructions that are currently in black text above the search box: "Have a question about Firefox? Search the Knowledge Base or check out the tutorials and reference articles below."

- Try putting a background behind this information, or even move it inside the search box
- Try different wording, something a bit more like instructions?


The other assumption we can make is that it's because they don't realize there are more articles than what's on the front page. I'm not sure what I think of this one because users reading the list *should* see the "Browse all KB articles" link and be bouncing from there instead, but it's still worth exploring. In this case we'd want to play with the text around the suggested articles, make it clear that these are only *some* of the articles and tell users what to do to get to the rest.

- Mention search here again
- Make the "Browse all articles" text a bit more informative
- Make the text stand out better, again maybe with a background or just a bigger font with a color that stands out from the list

I do, however, believe that the ultimate problem is that users don't know how to describe the feature or problem they need more information on. I'd be most interested to see the results of attempts to fix that problem, maybe with a tag cloud of terms, or perhaps some interactive map of the browser where users can point to the area they want to know more about.
Looking at https://wiki.mozilla.org/images/e/ec/SUMO_mockup_with_description.png I have a few comments as well, though it'd be great to move it to proper forum to actually discuss this stuff.

The smaller text principle doesn't actually work on the text under the heading, the rest of the page has headings so people will skip that smaller text in lieu of skimming the headings, the search bar text and the headings of "popular support articles" etc.

I really like the Step idea, I would put that outside the search box under the header

Step 1 - Find out if your answer is already in our knowledge base

Search 
- Enter a few words to search for an answer

or

Browse
- View a directory of our articles based on title


the other thing i wanted to say is that I don't view a tag cloud as browsing. For me it's very clearly searching, we're just suggesting search terms instead of making the user think of them. Tiki's tag system also suggests tags for refining your search.

We could definitely create some categorical tags, and closely control which articles they're used on, and then show a list of those tags. In that case I would consider that browsing by section, eg Tutorials would be a good category to browse by. Maybe show a few examples of the type of article in that category under each heading. However I would definitely link the tag *cloud* with search, not browse.
David and Lucy -- before we get too far into this discussion, I'd suggest that we first spend some time looking at the data, analysis, and findings.  Omniture has a bunch of valuable info in their pathing and heatmap reports.  David -- I can spend some time with you today going through everything.
Is this info going to be shared somewhere publicly? Can we pick up this discussion somewhere more appropriate?
The short answer is "absolutely".  We hit a bit of a snag last week, when some info from Omniture's reporting didn't make a whole lot of sense.  And we've been slowed down by this week's all hands.  David can better speak to the timing/location of the public sharing of the results.
Time to stop this test. Filed bug 492507.

We should blog about the outcome of this test, and our plans for the next test. If anyone has other ideas and thoughts to share right now, the contributor forum of SUMO would be a good place, as that would also get other SUMO contributors involved in the discussion. http://support.mozilla.com/tiki-view_forum.php?forumId=3
Whiteboard: sumo_only
You need to log in before you can comment on or make changes to this bug.