Closed
Bug 484283
Opened 15 years ago
Closed 15 years ago
Conduct A/B test on SUMO's inproduct start page
Categories
(support.mozilla.org :: Knowledge Base Software, task)
support.mozilla.org
Knowledge Base Software
Tracking
(Not tracked)
VERIFIED
FIXED
People
(Reporter: djst, Assigned: oremj)
References
()
Details
(Whiteboard: sumo_only)
Let's conduct a first A/B test on the most trafficked page on SUMO to gain knowledge about users' expectations of the inproduct start page. The current URL of that page is http://support.mozilla.com/en-US/kb/Firefox+Help. This page is linked to from Firefox itself via .htaccess rewrite rules. The target URL is for the en-US locale is: http://support.mozilla.com/en-US/kb/Firefox+Help?style_mode=inproduct During a test period of 7 days, rather that directing people to the normal inproduct start page as specified above, distribute the traffic evenly between the two pages: http://support.mozilla.com/en-US/kb/Firefox+Help+1?style_mode=inproduct http://support.mozilla.com/en-US/kb/Firefox+Help+2?style_mode=inproduct For details, see https://wiki.mozilla.org/Support/StartPageOptimization/FirstTest As for when the test should start, anytime works. mrz: I assigned this to you to get your attention since we talked about the logistics of this test in a conf call recently. Please help me assign this to the right person. Thanks!
Comment 1•15 years ago
|
||
Ken Kovash approves this test plan.
Updated•15 years ago
|
Assignee: mrz → server-ops
Component: General → Server Operations
Product: support.mozilla.com → mozilla.org
QA Contact: general → mrz
Version: unspecified → other
Updated•15 years ago
|
Assignee: server-ops → oremj
Comment 2•15 years ago
|
||
When is this or when should it be scheduled to go live?
Assignee | ||
Comment 3•15 years ago
|
||
Someone will need to implement comment #0
Assignee: oremj → nobody
Component: Server Operations → Knowledge Base Software
Product: mozilla.org → support.mozilla.com
QA Contact: mrz → kb-software
Version: other → unspecified
Reporter | ||
Comment 4•15 years ago
|
||
(In reply to comment #2) > When is this or when should it be scheduled to go live? Whenever we're ready and you have cycles to spare. The two different start pages are already up and ready to go. (In reply to comment #3) > Someone will need to implement comment #0 What part of comment 0 needs to be implemented?
Reporter | ||
Comment 6•15 years ago
|
||
Back to server-ops. Jeremy, could you clarify what you mean in comment 3? Thanks!
Assignee: nobody → server-ops
Component: Knowledge Base Software → Server Operations
Product: support.mozilla.com → mozilla.org
QA Contact: kb-software → mrz
Version: unspecified → other
Assignee | ||
Comment 7•15 years ago
|
||
Someone needs to implement at 50/50 redirection (htaccess, php, ...) at the test URL.
Updated•15 years ago
|
Assignee: server-ops → nobody
Component: Server Operations → Knowledge Base Software
Product: mozilla.org → support.mozilla.com
QA Contact: mrz → kb-software
Version: other → unspecified
Comment 8•15 years ago
|
||
Who did the one for www.mozilla.com? Was that webdev?
Comment 9•15 years ago
|
||
(In reply to comment #8) > Who did the one for www.mozilla.com? Was that webdev? Yes; Fred did in https://bugzilla.mozilla.org/show_bug.cgi?id=478952#c45
Assignee | ||
Comment 10•15 years ago
|
||
I did one of the mozilla.com ones and webdev did one. I'm not really as comfortable altering sumo as I am mozilla.com though, so the sumo team should handle this and coordinate a push with IT.
Assignee | ||
Comment 11•15 years ago
|
||
When do you want to go live with this?
Reporter | ||
Comment 12•15 years ago
|
||
Asap please. Friday - next Friday would be good I think. We should test on stage to make sure it works first and then go live. Thanks!
Comment 13•15 years ago
|
||
Is this still going ahead tomorrow?
Comment 14•15 years ago
|
||
No one commented after Laura asked and this never got pushed. Tenser, can you draw a line in the sand and pick a date we can all commit to? Thursday night? Is there anything blocking pushing this out then?
Reporter | ||
Comment 15•15 years ago
|
||
We're pushing 1.0.2 next Tuesday. Laura, would there be any problems from your point of view to push this out Thursday night and have it run for 7 days even if that means we'll push 1.0.2 while this test is still running?
Comment 16•15 years ago
|
||
If everything's ready we can push tonight as well. Not sure what "everything" is.
Reporter | ||
Comment 17•15 years ago
|
||
Just talked to Laura and she's fine with these dates. We need to test on stage first though. So, testing on stage - asap Push it out on prod - Thursday night April 16 Stop test (another push) - Thursday April 23 Is that OK?
Assignee | ||
Updated•15 years ago
|
Assignee: nobody → oremj
Reporter | ||
Comment 18•15 years ago
|
||
As long as we've verified that the A/B mechanism works on support-stage.mozilla.org, pushing it out tonight would work for me as well, no problem. The content is ready, I just haven't seen the actual behind-the-scene magic to make the A/B test happen.
Comment 19•15 years ago
|
||
This doesn't feel like it has to be an announced push (let me know if you disagree) so you basically have all day today to confirm it works before a 7pm push.
Comment 20•15 years ago
|
||
Just confirming... there isn't anything we need to do in advance from an Omniture/tracking perspective, so we should be good to go in terms of metrics.
Reporter | ||
Comment 21•15 years ago
|
||
(In reply to comment #19) > This doesn't feel like it has to be an announced push (let me know if you > disagree) so you basically have all day today to confirm it works before a 7pm > push. Laura can confirm that it works as soon as the patch is applied on stage, which I assume oremj is working on. Some help from QA would be good too -- Stephen? oremj, let us know when it's ready. Thanks!
Assignee | ||
Comment 22•15 years ago
|
||
Added the following to stage: RewriteCond %{TIME_SEC} .[02468] RewriteCond %{THE_REQUEST} "^GET /en-US/kb/Firefox\+Help\?style_mode=inproduct" RewriteRule . /en-US/kb/Firefox+Help+1?style_mode=inproduct [R] RewriteCond %{TIME_SEC} .[13579] RewriteCond %{THE_REQUEST} "^GET /en-US/kb/Firefox\+Help\?style_mode=inproduct" RewriteRule . /en-US/kb/Firefox+Help+2?style_mode=inproduct [R] Ken, since we are doing redirects you will see "Firefox+Help+2" and "Firefox+Help+1" in the page name reports.
Reporter | ||
Comment 23•15 years ago
|
||
Is this supposed to be up and running on stage? Testing the inproduct URL http://support-stage.mozilla.org/1/firefox/3.6a1pre/Darwin/en-US/firefox-help still takes me to http://support-stage.mozilla.org/en-US/kb/Firefox+Help?style_mode=inproduct.
Reporter | ||
Comment 24•15 years ago
|
||
Via Laura: If you edited htaccess.dist then you'll need to run sudo ./htaccess.sh. If you edited .htaccess it should just work.
Comment 25•15 years ago
|
||
Doesn't seem to be working - I get the Firefox+Help+1 version every time
Comment 26•15 years ago
|
||
Bah, actually /en-US/kb/Firefox+Help?style_mode=inproduct every time. Forgive me.
I didn't see any mention of a login requirement on https://wiki.mozilla.org/Support/StartPageOptimization/FirstTest (that is, clicking "Log in" doesn't return the user to the page from which they clicked it, but rather returns them to https://support-stage.mozilla.org/en-US/kb/Contributor+Home+Page), so testing this as-is, I see what I'd expect: http://support-stage.mozilla.org/en-US/kb/Firefox+Help?style_mode=inproduct yields either: http://support-stage.mozilla.org/en-US/kb/Firefox+Help+2?style_mode=inproduct -or- http://support-stage.mozilla.org/en-US/kb/Firefox+Help+1?style_mode=inproduct, 50% of the time.
Comment 28•15 years ago
|
||
Wfm now too.
Assignee | ||
Comment 29•15 years ago
|
||
I didn't push this tonight, because I wasn't 100% sure everyone involved felt it was complete. I am fine with pushing this out tomorrow or Thursday, if we want to wait for a maintenance window.
Reporter | ||
Comment 30•15 years ago
|
||
This seems to be working perfectly. Push this out anytime, please. Thanks!
Assignee | ||
Comment 31•15 years ago
|
||
Pushed to production.
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
Verified FIXED; http://support.mozilla.com/en-US/kb/Firefox+Help?style_mode=inproduct gives me http://support.mozilla.com/en-US/kb/Firefox+Help+1?style_mode=inproduct and http://support.mozilla.com/en-US/kb/Firefox+Help+2?style_mode=inproduct about 50% of the time each, respectively.
Status: RESOLVED → VERIFIED
Comment 33•15 years ago
|
||
There are a few things in our Omniture reporting that don't make much sense: (1) Looking at the top pages for *today*, there are about 85K page views for +1 and 85K page views +2... and about 45K page views for the regular "en-US/kb/Firefox+Help". (2) For the "en-US/kb/Firefox+Help" traffic, almost all of it comes from people directly entering the site (which makes sense since it's the in-product page). For the +1 and +2 pages, only about half of their traffic comes from people directly entering the site. The other half of their traffic comes from someone hitting +1, then hitting +2 (or vise versa). Any insights?
Reporter | ||
Comment 34•15 years ago
|
||
How are people still hitting the Firefox+Help page when we're supposed to distribute those visits to Firefox+Help+1 or Firefox+Help+2? More importantly for our goal, though, it looks like the bounce rate on +1 and +2 are identical. If that's true, it appears that the main reason for the high bounce rate has virtually nothing to do with the articles we're listing below the search bar.
Comment 35•15 years ago
|
||
(In reply to comment #34) > More importantly for our goal, though, it looks like the bounce rate on +1 and > +2 are identical. If that's true, it appears that the main reason for the high > bounce rate has virtually nothing to do with the articles we're listing below > the search bar. That actually seems like a logical conclusion when you think about the user flow when using the page. It seems to me the big fail point is going to be whether or not people understand how to search for the article they're looking for. Is there a newsgroup or forum where we're actually discussing the results and what to test next? I have further thoughts, but don't want to clutter the bug.
Reporter | ||
Comment 36•15 years ago
|
||
Feel free to post in this bug.
Reporter | ||
Comment 37•15 years ago
|
||
Also, you'll want to read this: https://wiki.mozilla.org/Support/StartPageOptimization
Comment 38•15 years ago
|
||
Ok, well in theory the current flow is focused around the search box, so users 1. Get to the page 2. Search on something then possibly 3. Not sure what to search so read down further 4. Find what you're looking for in a link or 4. Don't find what you're looking for, now what? Unless there are some specifically popular pages from inproduct odds are very low that changing the links will cover most cases where people need help. So, if people aren't getting the help they need, we should probably assume it's because they don't know how to effectively search, or that they don't realize they need to search to access the rest of the articles. One of two things I think will be the answer, either doing a better job of helping users effectively search or taking the focus off of search and using a better mechanism to get users where they want to go. (A tag cloud, for example, could be helpful in both cases, either supplementing search, or replacing it entirely). So for the next A/B test I would suggest trying to convince more users to search by playing with the instructions that are currently in black text above the search box: "Have a question about Firefox? Search the Knowledge Base or check out the tutorials and reference articles below." - Try putting a background behind this information, or even move it inside the search box - Try different wording, something a bit more like instructions? The other assumption we can make is that it's because they don't realize there are more articles than what's on the front page. I'm not sure what I think of this one because users reading the list *should* see the "Browse all KB articles" link and be bouncing from there instead, but it's still worth exploring. In this case we'd want to play with the text around the suggested articles, make it clear that these are only *some* of the articles and tell users what to do to get to the rest. - Mention search here again - Make the "Browse all articles" text a bit more informative - Make the text stand out better, again maybe with a background or just a bigger font with a color that stands out from the list I do, however, believe that the ultimate problem is that users don't know how to describe the feature or problem they need more information on. I'd be most interested to see the results of attempts to fix that problem, maybe with a tag cloud of terms, or perhaps some interactive map of the browser where users can point to the area they want to know more about.
Comment 39•15 years ago
|
||
Looking at https://wiki.mozilla.org/images/e/ec/SUMO_mockup_with_description.png I have a few comments as well, though it'd be great to move it to proper forum to actually discuss this stuff. The smaller text principle doesn't actually work on the text under the heading, the rest of the page has headings so people will skip that smaller text in lieu of skimming the headings, the search bar text and the headings of "popular support articles" etc. I really like the Step idea, I would put that outside the search box under the header Step 1 - Find out if your answer is already in our knowledge base Search - Enter a few words to search for an answer or Browse - View a directory of our articles based on title the other thing i wanted to say is that I don't view a tag cloud as browsing. For me it's very clearly searching, we're just suggesting search terms instead of making the user think of them. Tiki's tag system also suggests tags for refining your search. We could definitely create some categorical tags, and closely control which articles they're used on, and then show a list of those tags. In that case I would consider that browsing by section, eg Tutorials would be a good category to browse by. Maybe show a few examples of the type of article in that category under each heading. However I would definitely link the tag *cloud* with search, not browse.
Comment 40•15 years ago
|
||
David and Lucy -- before we get too far into this discussion, I'd suggest that we first spend some time looking at the data, analysis, and findings. Omniture has a bunch of valuable info in their pathing and heatmap reports. David -- I can spend some time with you today going through everything.
Comment 41•15 years ago
|
||
Is this info going to be shared somewhere publicly? Can we pick up this discussion somewhere more appropriate?
Comment 42•15 years ago
|
||
The short answer is "absolutely". We hit a bit of a snag last week, when some info from Omniture's reporting didn't make a whole lot of sense. And we've been slowed down by this week's all hands. David can better speak to the timing/location of the public sharing of the results.
Reporter | ||
Comment 43•15 years ago
|
||
Time to stop this test. Filed bug 492507. We should blog about the outcome of this test, and our plans for the next test. If anyone has other ideas and thoughts to share right now, the contributor forum of SUMO would be a good place, as that would also get other SUMO contributors involved in the discussion. http://support.mozilla.com/tiki-view_forum.php?forumId=3
Reporter | ||
Updated•15 years ago
|
Whiteboard: sumo_only
You need to log in
before you can comment on or make changes to this bug.
Description
•