Closed Bug 1168359 Opened 9 years ago Closed 5 years ago

A/B testing for several tasks under /contribute/signup/

Categories

(www.mozilla.org :: Bedrock, defect)

Production
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: nikos, Unassigned)

References

Details

(Whiteboard: [kb=1776839] )

This a tracker for bug for the A/B testing around various tasks to live under the /contribute/signup/ page.
Depends on: 1169186
Depends on: 1169188
Depends on: 1169189
Depends on: 1169190
Whiteboard: [kb=1776837]
Whiteboard: [kb=1776837] → [kb=1776839]
Wireframes of all variations implemented here (although some things have changed in the UI since then):
http://cl.ly/image/0A3C1q1N3B2h/o
http://cl.ly/image/221b3x0s2u0a/o
http://cl.ly/image/2C2R0g0a3h2m/o
http://cl.ly/image/2O360r1N071N/o
No longer depends on: 1169186, 1169188, 1169189, 1169190
Summary: [tracker] A/B testing for several tasks under /contribute/signup/ → A/B testing for several tasks under /contribute/signup/
We many need some extra review for Optimizely.
Flags: needinfo?(jbertsch)
(In reply to Nikos Roussos [:nikos] from comment #2)
> We many need some extra review for Optimizely.

Hi Cory-

Could you please review the Optimizely set up?  The test is "/bedrock contribute" and is here https://app.optimizely.com/projects/246059135/experiments/3011470152

Here's the PR https://github.com/mozilla/bedrock/pull/3059
Flags: needinfo?(jbertsch) → needinfo?(cprice)
Thanks Jen.

My recommendation is:

1. Get the PR r+'d
2. Merge into production
3. Update the Optimizely experiment to create goals on the new variations (currently 404'ing since the variations aren't merged)
4. Start experiment
Flags: needinfo?(cprice)
Once the PR is reviewed and merged, I'll take another look at the experiment.
Depends on: 1183706
Note: The signup page has an Optimizely snippet wrapped in a waffle switch named "contribute-signup-optimizely". We'll need to enable that switch when this goes to production and disable it when the test period ends. (CCing jgmize and pmac)
Waiting for nikos to add the snippet to a couple more places

https://github.com/mozilla/bedrock/pull/3059#issuecomment-121716854
Flags: needinfo?(cprice)
Just did on the PR
Need a timeline for when this will be ready. Whose work are we waiting on?
Flags: needinfo?(nikos)
I created a demo5 experiment in Optimizely and added the 4 variations. Experiment is: Bedrock /contribute (demo5)

There hasn't been much other configuration done in Optimizely yet with respect to goals and conversions (rightfully so since there was no live page to configure against until now).

Nikos - please update 'Bedrock /contribute (demo5)' with your desired element tracking and goal setting.

If you would like to collaborate together to create your experiment, I can be available to assist.

Once your experiment has been set up to everyones' liking, we can push the bedrock changes to production, update the experiment to point to the production URL's, and start the experiment.

Thanks
This is kind of a nitpick and something I should have spotted sooner... the intro text on the survey page says "Keep an eye on your inbox for an important message from us" even though the user hasn't submitted an email yet. Since this experiment is English only and we don't need to worry about breaking strings, I suggest rewriting that blurb on the survey page.

Also, if I click the "not ready" button and get to the survey page without a survey, the title says "Thanks!" when I haven't done anything to be thanked for, and the form title says "Interested in trying something else?" when I haven't tried anything. We could use the same conditional logic we're already using to tailor the copy for the non-task case as well.

Thoughts? I'm happy to take a crack at writing some different copy.
Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/f36e60b081602fffd3356f344ac1a6a1014a1eb5
[Fix bug 1168359] A/B testing for several tasks under /contribute/signup/

https://github.com/mozilla/bedrock/commit/342876fb7fda52f8631f6d3b82623b3b4c05ad27
Bug 1168359: Add tests and fix revealed issues.

https://github.com/mozilla/bedrock/commit/bf33ba24a1b016220cc399432ce6462d4fc72093
Merge pull request #3135 from pmclanahan/comzeradd-signup-abtesting

[Fix bug 1168359] A/B testing for several tasks under /contribute/signup/
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
It seems it's already merged. I changed Optimizely urls for production to be ready. What we want to see in this is whether users will follow the tasks. And since all of them open in a new tab I think Engagement is enough as goal on the Optimizely experiment.
Flags: needinfo?(nikos)
Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/ac5d2b9e2569b46fd09a6e3ab953758ced3b5459
Bug 1168359 - contribute A/B test copy tweaks

With some conditional messaging on the survey page.

https://github.com/mozilla/bedrock/commit/5d05dd4e0093bf3ebbe7dfbff6b4a5da5bf421b3
Merge pull request #3137 from mozilla/bug-1168359-participation-signup-abtesting

Bug 1168359 - contribute A/B test copy tweaks
(In reply to Nikos Roussos [:nikos] from comment #15)
> It seems it's already merged. I changed Optimizely urls for production to be
> ready. What we want to see in this is whether users will follow the tasks.
> And since all of them open in a new tab I think Engagement is enough as goal
> on the Optimizely experiment.

Thanks Nikos.

I have turned the experiment on. This page received 37k en-US visits the last 30 days, so it could take a few days to see significant traffic.
@ckprice @nikos awesome! When will it be showing up here (https://www.mozilla.org/en-US/contribute/signup/) I've visited the page several times and haven't been able to see any of the variations.
Flags: needinfo?(cprice)
Hi Lucy,

Missed one last config thing.

Should be good to go - http://cl.ly/image/0R170E1V290k!

May need a hard refresh to start seeing it.
Flags: needinfo?(cprice)
I can confirm that it's working. I watch Optimizely results page and visitors are evenly distributed among all variations.
(In reply to Nikos Roussos [:nikos] from comment #15)
> It seems it's already merged. I changed Optimizely urls for production to be
> ready. What we want to see in this is whether users will follow the tasks.
> And since all of them open in a new tab I think Engagement is enough as goal
> on the Optimizely experiment.

Hi-

Sorry I missed this earlier.  Just adding a record of the test goals for when we turn the tests back on:
test goals
1.  Learn if visitors to the Get involved page more interested in:

    a.  providing an email address to learn more about participating in the Mozilla community?

    b.  clicking to view more information about easy tasks?

    c.  clicking to view more information about hard tasks?

    d.  clicking to view more information about tasks that require a shorter time commitment?

    e.  clicking to view more information about tasks that require a longer time commitment?


2.  Learn whether or not offering the following two different types of content on the sign up page is more likely to drive conversion:

    a.  a list of specific tasks (conversion = clicking on a task or providing email address) on the Get Involved sign up page vs

    b.  a list of "pathways" with an email CTA is more likely to drive conversion (conversion = picking a contribution pathway and providing an email address)


Thx,
Jen
> I have turned the experiment on. This page received 37k en-US visits the last 30 days, so it could take a few days to see significant traffic.

Can someone explain this statement please and the term "experiment" in this context?

Are there Firefox users that are being sent to this page automatically? What exactly is the experiment?
Hi all:

I've found a bug in Safari

https://www.mozilla.org/en-US/contribute/tasks/?variation=1

Clicking on the circles switches the current tab to the survey, but it does not open the new tab.

TODO: assign someone to fix this.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(In reply to Cory Price [:ckprice] from comment #23)
> Hi all:
> 
> I've found a bug in Safari
> 
> https://www.mozilla.org/en-US/contribute/tasks/?variation=1
> 
> Clicking on the circles switches the current tab to the survey, but it does
> not open the new tab.
> 
> TODO: assign someone to fix this.

Let's turn off the test for Safari.  http://cl.ly/image/2I1I2B0x2P1c
Cory: I would create a new audience that is non-Safari + English any instead of the "Everyone" it is set to now to eliminate the bug above and to ensure the person's language is English.
With cmore's help, we got the test up and running. We'll report on results in a week or so.

Thanks.
Can someone please answer my question?

> Can someone explain this statement please and the term "experiment" in this context?

> Are there Firefox users that are being sent to this page automatically? What exactly is the experiment?
(In reply to Mike Kaply [:mkaply] from comment #22)
> Are there Firefox users that are being sent to this page automatically? What
> exactly is the experiment?

The experiment is to show different variants of the page (see comment 1) to different en-US visitors who visit that URL, nothing is sent to Firefox users.
(In reply to Cory Price [:ckprice] from comment #26)
> With cmore's help, we got the test up and running. We'll report on results
> in a week or so.
> 
> Thanks.

Thanks for the help, Cmore and Cory!
Depends on: 1197945

This work was completed a long time ago (the aforementioned page no longer exists, but I'm closing it as FIXED).

Status: REOPENED → RESOLVED
Closed: 9 years ago5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.