Optimizely test: linkedin.com unsupported browser referrals

NEW
Unassigned

Status

a year ago
a year ago

People

(Reporter: frios, Unassigned)

Tracking

Production

Details

(Reporter)

Description

a year ago
Linkedin is referring traffic to our /new page via an unsupported browser notification on their site. This traffic has a lower than expected download rate on Firefox. We'd like to test a variation of this landing page to improve the download rate.
(Reporter)

Comment 1

a year ago
Hey Justin,

Are you able to help me set up this test in Optimizely? Check out the Mana write up. I started drafting the Optimizely test, but I need some help making sure it's targeting the download properly.

https://mana.mozilla.org/wiki/display/FIREFOX/%5BBub+1396105%5D+linked.com+unsupported+browser+referral+traffic+test

https://app.optimizely.com/edit?experiment_id=8547171181
Flags: needinfo?(hoosteeno)
(In reply to Fabio Rios [:frios] from comment #1)
> I need some help
> making sure it's targeting the download properly.

Do you mean the goal tracking? It looks good to me. Everything else looks good, too. Great job setting this up.

:jpetto, could you do code review on this? Looks like snippet is already inline on both pages, so no push needed.
Flags: needinfo?(hoosteeno) → needinfo?(jon)
Making sure that:

- this should be en-US only
- this should run on ?scene=2 or any other /new variation if somehow the user got there directly from LinkedIn
- we are okay losing the referrer after Optimizely does the redirect (we are still working w/Gareth to resolve this issue)

If all the above are okay, I think we're good to run this.

Note that we are about to start running a survey on /new as well (see https://github.com/mozilla/bedrock/pull/5082).
Flags: needinfo?(jon)
(Reporter)

Comment 4

a year ago
(In reply to Jon Petto [:jpetto] from comment #3)
> Making sure that:
> 
> - this should be en-US only
Yes, en-US only
> - this should run on ?scene=2 or any other /new variation if somehow the
> user got there directly from LinkedIn
I think the answer is yes, but not sure I understand the question? The visitors that are referred to us from linkedin for this test are coming exclusively from this page: https://www.linkedin.com/error_pages/unsupported-browser.html
> - we are okay losing the referrer after Optimizely does the redirect (we are
> still working w/Gareth to resolve this issue)
That's fine for this test. As long as we can track downloads in optimizely, we'll use that.
> If all the above are okay, I think we're good to run this.
Thank you
> Note that we are about to start running a survey on /new as well (see
> https://github.com/mozilla/bedrock/pull/5082).
Thanks for the heads up. I don't expect that to disrupt the results. We'll take an undetectable amount of traffic away from /new and redirect them to our xv=unsupported-browser page during this test
Flags: needinfo?(jon)
(In reply to Fabio Rios [:frios] from comment #4)
> (In reply to Jon Petto [:jpetto] from comment #3)
> > - this should run on ?scene=2 or any other /new variation if somehow the
> > user got there directly from LinkedIn
> I think the answer is yes, but not sure I understand the question? The
> visitors that are referred to us from linkedin for this test are coming
> exclusively from this page:
> https://www.linkedin.com/error_pages/unsupported-browser.html

The current URL targeting in the Optimizely experiment does not exclude query params, meaning that if the user came from LinkedIn and landed on https://www.mozilla.org/en-US/firefox/new/?scene=2 or https://www.mozilla.org/en-US/firefox/new/?xv=fast, they would be entered into the experiment and redirected. We could tighten up the URL targeting in Optimizely to ensure we only target folks coming to https://www.mozilla.org/en-US/firefox/new/ exactly.

I recommend we do the above to be safe.
Flags: needinfo?(jon)
I modified the regex for URL targeting, pinning it to the end as well:

^.+www\.mozilla\.org\/en\-US\/firefox\/new\/$

Now this experiment is targeted at anyone coming from the unsupported browser page on LinkedIn to the /new/ page exactly.

:jpetto, if you're good with that, I'll launch today.
Flags: needinfo?(jon)
New targeting looks good. Go for launch.

How long do we expect this test to run?
Flags: needinfo?(jon)
Current conversion rate is appx. 30%. If we want to be confident in a 20% improvement (i.e. 36%+) we need 580 visitors per variation, or 1.2k total. We get more than that on a regular weekday.

https://www.optimizely.com/sample-size-calculator/?conversion=30&effect=20&significance=95

For extra confidence we could run through next week.

Launching now.
Cool. Please comment here when the experiment is turned off, and when we have some results on the data.
(Reporter)

Comment 10

a year ago
Hey Justin,

I've made a few changes to the test using a copy of the previous one. Can we re-run? I've added the /firefox/ page and a variation of the xv=unsupported browser to the test.

https://app.optimizely.com/projects/246059135/experiments/8733782124
Flags: needinfo?(hoosteeno)
:jpetto, the original experiment linked above is paused and archived. We have a new version of it that sends visitors to 2 additional variations -- /firefox/ home page and /firefox/new/ with some text changes. URL target, audience definition, and goals definition are unchanged.

Can you review and/or waive review? 

https://app.optimizely.com/edit?experiment_id=8733782124
Flags: needinfo?(hoosteeno) → needinfo?(jon)
Experiment looks good to me.

Noting that Optimizely is not currently enabled on /firefox.
Flags: needinfo?(jon)
(In reply to Jon Petto [:jpetto] from comment #12)
> Experiment looks good to me.

OK, I will launch now. :frios please keep an eye on it?

> Noting that Optimizely is not currently enabled on /firefox.

Since we're not measuring or changing elements on that page, I think this is OK.
Experiment is launched.

:frios, can you update the experiment page with new variations? I already updated start date.

https://mana.mozilla.org/wiki/display/FIREFOX/%5BBug+1396105%5D+linked.com+unsupported+browser+referral+traffic+test
Flags: needinfo?(frios)
(In reply to Justin Crawford [:hoosteeno] [:jcrawford] from comment #13)
> 
> > Noting that Optimizely is not currently enabled on /firefox.
> 
> Since we're not measuring or changing elements on that page, I think this is
> OK.

I thought Optimizely had to be enabled on redirects to measure engagement? Happy to be wrong though - means a lighter weight/faster page. :)
> 
> I thought Optimizely had to be enabled on redirects to measure engagement?
> Happy to be wrong though - means a lighter weight/faster page. :)

I've launched dozens of these experiments and remember only one or two where "clicked any element on the page" was a metric we cared about. We're almost always trying to move something more specific than that. In this case, with downloads, we're using a goal that counts sessions landing on scene=2. Since optimizely is enabled on scene=2, this'll work. I went ahead and removed "engagement" goal from this experiment because it wouldn't be useful.
(Reporter)

Updated

a year ago
Flags: needinfo?(frios)
(Reporter)

Comment 17

a year ago
Justin, lets pause the test.

Given the big improvement using the /firefox/ page, we'll want to make this permanent. I've written LinkedIn and asked if they can help us make the change to the page: https://www.linkedin.com/error_pages/unsupported-browser.html

If we can't find a contact there to help us, perhaps we can run a redirect on our side.
Flags: needinfo?(hoosteeno)
It is paused. 

This experiment was a great success -- we learned a lot from it. It may also help us achieve a 50% increase in conversions on this channel. Good work :frios.
Flags: needinfo?(hoosteeno)
You need to log in before you can comment on or make changes to this bug.