Closed
Bug 971027
Opened 12 years ago
Closed 9 years ago
Enable users to opt-out of Optimizely tests on webmaker.org and tools
Categories
(Webmaker Graveyard :: Metrics, defect, P2)
Tracking
(Not tracked)
RESOLVED
INCOMPLETE
People
(Reporter: humph, Assigned: adam)
References
Details
(Whiteboard: [metrics] [icebox])
Last week we were building some infrastructure and processes around AB testing, and one thing we did was play with "Remix" vs. "Fork". It's not a useful test, beyond allowing us to learn how to get this stuff right; however, it caused confusion for users who were taking a course that references the "Remix" button in the UI.
We're creating an engagement ladder for our users, and trying to help encourage people to move from being users of the web to makers. One step on such a ladder is helping to improve Webmaker, whether through testing new features, reporting bugs, or participating in AB tests like this.
I'd like to suggest that we consider making such tests "opt in" such that users can check some box indicating that they'd like to be placed in a pool for AB experiments, would receive emails about new features and things to test, etc. It would essentially create a tier of users who are more involved in the development of Webmaker, even if they aren't developers or working in github/bugzilla, etc.
Comment 1•12 years ago
|
||
Interesting idea - my understanding of the tests from the work week indicated that you needed a sufficient level of traffic to determine the results of the tests however. Having them be opt-in by definition would mean most users wouldn't see them, causing the tests to take far longer to run or potential be inconclusive.
Adam?
Flags: needinfo?(adam)
| Reporter | ||
Comment 2•12 years ago
|
||
For reference, a similar experiment and its fallout with Facebook:
http://stackoverflow.com/questions/21692646/how-does-facebook-disable-developer-tools/21693931#21693931
And how they allow users to turn it on/off:
https://www.facebook.com/selfxss
WRT the point about needing larger numbers for tests, I think we just need to try hard to get many users checking the "yes please" box.
Comment 3•12 years ago
|
||
Yeah, it is my understanding too that we need large numbers to make it worth while.
I agree there is a problem running tests on prod though, and opt-in is just a proposed solution.
I'm really interested in what Adam has to say because maybe he has hit this before.
Another option is maybe we need to be more careful as to what we test.
Example, my idea to ab test save/publish, the events tab name, and position of the save button are probably better limited to user testing, and might not be good candidates for ab testing.
It looks to be very powerful in a limited space. A clear call to action, or to improve bounce rate on the homepage from new users, maybe, but inside an editor/finished project? Probably not.
Example, ab testing clicks to download firefox is a clear action, and something that is only done once. Another example is getting a donation.
Moving the save button or something else that a person is going to hit multiple times feels like a different thing.
I'm going to propose we turn off the remix/fork test too, but it's cool if we decide to keep it on.
Comment 4•12 years ago
|
||
I think it's more important that a class or group has consistency than it is to be able to opt in or out.
So: why not make it so that within a given IP block (e.g. a /24 or even a /16), they always get A, or always get B?
Gerv
Comment 5•12 years ago
|
||
(In reply to Gervase Markham [:gerv] from comment #4)
> I think it's more important that a class or group has consistency than it is
> to be able to opt in or out.
>
> So: why not make it so that within a given IP block (e.g. a /24 or even a
> /16), they always get A, or always get B?
>
> Gerv
I disagree with this approach. I teach Webmaker at various locations, and I usually prepare my classes at home. If the tools look/act different based on a random selection process, and I hit it, it can create confusion on my part and on the students' parts too. They are tinkering with the tools at home, school, Starbucks, so it can get really confusing to why this is happening, since they didn't ask for it.
Production environment should not be a guinea pig for users who don't sign up for it. An opt-in/opt-out solution for advanced users seems more suitable.
| Assignee | ||
Comment 6•12 years ago
|
||
There are lots of interesting points in this thread, so I'm going to reply with a few comments to try and keep the conversations clearer. Comments to follow now.
Flags: needinfo?(adam)
| Assignee | ||
Comment 7•12 years ago
|
||
(In reply to David Humphrey (:humph) from comment #0)
> I'd like to suggest that we consider making such tests "opt in" such that
> users can check some box indicating that they'd like to be placed in a pool
> for AB experiments
Unfortunately, this would effectively kill the testing capacity of Webmaker:
* Based on % of people who actively opt-in to GA tracking we'd lose at least 90% of traffic, meaning tests would take 10x longer to complete. We don't currently have enough traffic overall for that to be practical. ref: http://chinwag.com/blogs/sam-michel/cookiepocalypse-implementing-new-law-drops-use-90
* This kind of optimization is most relevant to brand new visitors to the site. It's mostly about how our messaging and design choices influence a cold audience to sign up. To limit this to existing users who actively opt-in, we'd be testing against a segment of people already 'on board'. And in this example of the remix button test which has caused confusion, we're testing on published content which has a much wider audience than people who know about webmaker. We couldn't expect people who view their friends published popcorn video to first opt-in to an A/B test. But this is a broad audience of people, more of whom we'd love to get involved in Webmaker - and testing of the content bar at the top of published makes is a potential way to bring more people on board.
| Assignee | ||
Comment 8•12 years ago
|
||
(In reply to David Humphrey (:humph) from comment #0)
> [...] would receive emails about new features and things to
> test, etc. It would essentially create a tier of users who are more
> involved in the development of Webmaker, even if they aren't developers or
> working in github/bugzilla, etc.
I think communicating our tests (not just what is being tested, but the results of what we learn from running these tests) is a great way to involve people in Webmaker, and webmaking in general. I'm hesitant to start a new list or group for this, so will start by posting to the existing webmaker list and we can gauge how useful that is. If it becomes too noisy, we could filter this out to somewhere else.
| Assignee | ||
Comment 9•12 years ago
|
||
(In reply to Scott [:thecount] Downe from comment #3)
> Another option is maybe we need to be more careful as to what we test.
This initial test complicated things, which is unfortunate, but it's also useful to hit this problem early so we can plan for it. In future, tests will be talked about with more people before they start so potential confusion points like this are more likely to be flagged up. Also the launch of a test will also be shared, so we won't have the problem of surprises.
> It looks to be very powerful in a limited space. A clear call to action, or
> to improve bounce rate on the homepage from new users, maybe, but inside an
> editor/finished project? Probably not.
Don't be put off here! The wording of this button was problematic, but the changing color and icon still have the potential increase the number of people clicking remix. Cumulatively, out published content with the make-valet bar gets more traffic than our homepage. This is a place where we can be 'recruiting' more people to use Webmaker and our tools, and testing is the way to optimize this.
> I'm going to propose we turn off the remix/fork test too, but it's cool if
> we decide to keep it on.
Just to note here that we've switched off the word "Fork" in the test, so new people won't see this now. But the color and icon variations are still running.
| Assignee | ||
Comment 10•12 years ago
|
||
(In reply to Gervase Markham [:gerv] from comment #4)
> I think it's more important that a class or group has consistency than it is
> to be able to opt in or out.
>
> So: why not make it so that within a given IP block (e.g. a /24 or even a
> /16), they always get A, or always get B?
This is a nice idea for managing group consistency but we're using a 3rd party tool to distribute the traffic so I'd expect limited interest on their part to implement this. We could add a layer of logic to our systems to distribute people among variations, but the setup and maintenance feels like it would outweigh the benefit.
FWIW I'd still encourage being open about testing as I like talking to people about it, but I know this won't apply to everyone and in all teaching situations. Embrace the variations! ;-P
I'll be adding an article to SUMO today that might help with a top-level explanation for people want to talk about testing when they are teaching.
| Assignee | ||
Comment 11•12 years ago
|
||
(In reply to Dumitru Gherman [:dumitru] from comment #5)
> Production environment should not be a guinea pig for users who don't sign
> up for it.
I'd like to challenge this and point out that this kind of testing happens all around us every day in 'production' environments. We see slight variations of TV ads across different channels with varying phone numbers to measure response. Supermarkets getting different versions of product display boards to distribute among their stores. Etc, etc. And pretty much any major website you visit today will be split testing something. On the plus side, the numbers are only useful in aggregate so it's not a process of tracking an individual's behavior - which is a good thing. Our production environment is the only place we can find out how a real audience behaves at scale, and the goal of our testing is to make our site and tools better for our real audience so we can get more people teaching and making the web.
> An opt-in/opt-out solution for advanced users seems more suitable.
I think after working through this genuinely interesting set of questions, this is what it comes down to:
* Opt-in would kill the testing process
* Opt-out could solve the complications in teaching scenarios if we make it simple
| Assignee | ||
Comment 12•12 years ago
|
||
I'd like feedback from all, and if this is idea is positive I'll setup a new bug and close this one.
I propose:
Adding a new link to the footer of Webmaker.org: "A/B Testing"
* This would go to a new content page that explains the fact that we run tests and how we hope to make the site better by doing so. Including links to more detail about this on SUMO, and how to get involved with testing on the Wiki.
* This would also include a big one-click OPT-OUT button (https://www.optimizely.com/docs/api#opt-out)
That way if a test is causing any confusion during a teaching session, students/teachers can go to this page and quickly click the button to get the default version of the site.
We'll also have better communications around tests running, and be more careful about what we test to reduce the times when you might want to use the opt-out button. But in general does this solution work for everyone?
Flags: needinfo?(thereallove)
Flags: needinfo?(scott)
Flags: needinfo?(gerv)
Flags: needinfo?(david.humphrey)
Flags: needinfo?(brett)
Comment 13•12 years ago
|
||
I think the link needs a better name than "A/B Testing", which no-one will understand, but apart from that, I can live with this solution :-)
Gerv
Flags: needinfo?(gerv)
Comment 14•12 years ago
|
||
I agree that the term "A/B testing" is probably not the best term for the front facing warning on the homepage, but inside the SUMO article and documentation is where users could be intoduced to the technical term. However, we could A/B test the wording :P (half kidding)
Other than that though, I think it's a fair solution.
My main takeaway lesson from this is:
Be very very careful about changing things like text or position of something that has a lot of repeat users. It has a huge cost, and has the potential to lose more users than you gain.
Flags: needinfo?(scott)
Comment 15•12 years ago
|
||
Thanks for these detailed responses, Adam - it's great this conversation is documented.
I agree with the opt-out approach, as well as the need to have the link be named something more familiar to users - "seeing differences in Webmaker?" or something to that effect.
I'd propose just going ahead with making those pages and then closing this issue.
Flags: needinfo?(brett)
| Reporter | ||
Updated•12 years ago
|
Assignee: nobody → adam
Status: NEW → ASSIGNED
| Reporter | ||
Updated•12 years ago
|
Flags: needinfo?(david.humphrey)
| Assignee | ||
Comment 16•12 years ago
|
||
I've renamed the bug to reflect the modified task:
* Adding a new page to webmaker.org: "Seeing changes in Webmaker?" which:
** explains the fact that we run tests and how we hope to make the site better by doing so. Including links to more detail about this on SUMO (sumo in turn links to the wiki if people want to contribute). bug 970843
** includes a one-click opt-out button, explaining how this could be useful in a class room situation: https://www.optimizely.com/docs/api#opt-out
* Add this a footer link on webmaker.org
* Communicate this feature with webmaker mentors
Flags: needinfo?(thereallove)
Summary: Consider ways to make testing experiments "opt in" → Enable users to opt-out of Optimizely tests on webmaker.org and tools
| Assignee | ||
Comment 17•12 years ago
|
||
This ticket fell off the radar, but we're planning to run A/B test as part of the Maker Party strategy (on maker party landing pages rather than the tools), so I'll revive this work before then.
Priority: -- → P2
Comment 18•12 years ago
|
||
* Adam: is this work still required? Given that we're going to be using Unbounce instead?
* Maybe we should punt this to [june13], once we've run our first Unbounce test?
Whiteboard: [June13]
Updated•12 years ago
|
Flags: needinfo?(adam)
Whiteboard: [June13] → [metrics] [June27]
| Assignee | ||
Comment 19•12 years ago
|
||
This is less urgent because the things we're going to test are on the comms/marketing landing pages rather than in the tools where this can cause more confusion.
But in principal, it's still a good thing for us to do.
Flags: needinfo?(adam)
Updated•12 years ago
|
Whiteboard: [metrics] [June27] → [metrics] [icebox]
Comment 20•9 years ago
|
||
Closing this bug as part of the deprecation of the Webmaker product on Bugzilla.
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → INCOMPLETE
You need to log in
before you can comment on or make changes to this bug.
Description
•