1396105 - Optimizely test: linkedin.com unsupported browser referrals

Status: NEW

(Websites :: Web Analytics, task)

Description

[Fabio Rios [:frios]]

Linkedin is referring traffic to our /new page via an unsupported browser notification on their site. This traffic has a lower than expected download rate on Firefox. We'd like to test a variation of this landing page to improve the download rate.

Comment 1

[Fabio Rios [:frios]]

Hey Justin,

Are you able to help me set up this test in Optimizely? Check out the Mana write up. I started drafting the Optimizely test, but I need some help making sure it's targeting the download properly.

https://mana.mozilla.org/wiki/display/FIREFOX/%5BBub+1396105%5D+linked.com+unsupported+browser+referral+traffic+test

https://app.optimizely.com/edit?experiment_id=8547171181

Comment 2

[Justin Crawford [:hoosteeno] [:jcrawford]]

(In reply to Fabio Rios [:frios] from comment #1)
> I need some help
> making sure it's targeting the download properly.

Do you mean the goal tracking? It looks good to me. Everything else looks good, too. Great job setting this up.

:jpetto, could you do code review on this? Looks like snippet is already inline on both pages, so no push needed.

Comment 3

[Jonathan Petto [:jpetto]]

Making sure that:

- this should be en-US only
- this should run on ?scene=2 or any other /new variation if somehow the user got there directly from LinkedIn
- we are okay losing the referrer after Optimizely does the redirect (we are still working w/Gareth to resolve this issue)

If all the above are okay, I think we're good to run this.

Note that we are about to start running a survey on /new as well (see https://github.com/mozilla/bedrock/pull/5082).

Comment 4

[Fabio Rios [:frios]]

(In reply to Jon Petto [:jpetto] from comment #3)
> Making sure that:
> 
> - this should be en-US only
Yes, en-US only
> - this should run on ?scene=2 or any other /new variation if somehow the
> user got there directly from LinkedIn
I think the answer is yes, but not sure I understand the question? The visitors that are referred to us from linkedin for this test are coming exclusively from this page: https://www.linkedin.com/error_pages/unsupported-browser.html
> - we are okay losing the referrer after Optimizely does the redirect (we are
> still working w/Gareth to resolve this issue)
That's fine for this test. As long as we can track downloads in optimizely, we'll use that.
> If all the above are okay, I think we're good to run this.
Thank you
> Note that we are about to start running a survey on /new as well (see
> https://github.com/mozilla/bedrock/pull/5082).
Thanks for the heads up. I don't expect that to disrupt the results. We'll take an undetectable amount of traffic away from /new and redirect them to our xv=unsupported-browser page during this test

Comment 5

[Jonathan Petto [:jpetto]]

(In reply to Fabio Rios [:frios] from comment #4)
> (In reply to Jon Petto [:jpetto] from comment #3)
> > - this should run on ?scene=2 or any other /new variation if somehow the
> > user got there directly from LinkedIn
> I think the answer is yes, but not sure I understand the question? The
> visitors that are referred to us from linkedin for this test are coming
> exclusively from this page:
> https://www.linkedin.com/error_pages/unsupported-browser.html

The current URL targeting in the Optimizely experiment does not exclude query params, meaning that if the user came from LinkedIn and landed on https://www.mozilla.org/en-US/firefox/new/?scene=2 or https://www.mozilla.org/en-US/firefox/new/?xv=fast, they would be entered into the experiment and redirected. We could tighten up the URL targeting in Optimizely to ensure we only target folks coming to https://www.mozilla.org/en-US/firefox/new/ exactly.

I recommend we do the above to be safe.

Comment 6

[Justin Crawford [:hoosteeno] [:jcrawford]]

I modified the regex for URL targeting, pinning it to the end as well:

^.+www\.mozilla\.org\/en\-US\/firefox\/new\/$

Now this experiment is targeted at anyone coming from the unsupported browser page on LinkedIn to the /new/ page exactly.

:jpetto, if you're good with that, I'll launch today.

Comment 7

[Jonathan Petto [:jpetto]]

New targeting looks good. Go for launch.

How long do we expect this test to run?

Comment 8

[Justin Crawford [:hoosteeno] [:jcrawford]]

Current conversion rate is appx. 30%. If we want to be confident in a 20% improvement (i.e. 36%+) we need 580 visitors per variation, or 1.2k total. We get more than that on a regular weekday.

https://www.optimizely.com/sample-size-calculator/?conversion=30&effect=20&significance=95

For extra confidence we could run through next week.

Launching now.

Comment 9

[Jonathan Petto [:jpetto]]

Cool. Please comment here when the experiment is turned off, and when we have some results on the data.

Comment 10

[Fabio Rios [:frios]]

Hey Justin,

I've made a few changes to the test using a copy of the previous one. Can we re-run? I've added the /firefox/ page and a variation of the xv=unsupported browser to the test.

https://app.optimizely.com/projects/246059135/experiments/8733782124

Comment 11

[Justin Crawford [:hoosteeno] [:jcrawford]]

:jpetto, the original experiment linked above is paused and archived. We have a new version of it that sends visitors to 2 additional variations -- /firefox/ home page and /firefox/new/ with some text changes. URL target, audience definition, and goals definition are unchanged.

Can you review and/or waive review? 

https://app.optimizely.com/edit?experiment_id=8733782124

Comment 12

[Jonathan Petto [:jpetto]]

Experiment looks good to me.

Noting that Optimizely is not currently enabled on /firefox.

Comment 13

[Justin Crawford [:hoosteeno] [:jcrawford]]

(In reply to Jon Petto [:jpetto] from comment #12)
> Experiment looks good to me.

OK, I will launch now. :frios please keep an eye on it?

> Noting that Optimizely is not currently enabled on /firefox.

Since we're not measuring or changing elements on that page, I think this is OK.

Comment 14

[Justin Crawford [:hoosteeno] [:jcrawford]]

Experiment is launched.

:frios, can you update the experiment page with new variations? I already updated start date.

https://mana.mozilla.org/wiki/display/FIREFOX/%5BBug+1396105%5D+linked.com+unsupported+browser+referral+traffic+test

Comment 15

[Jonathan Petto [:jpetto]]

(In reply to Justin Crawford [:hoosteeno] [:jcrawford] from comment #13)
> 
> > Noting that Optimizely is not currently enabled on /firefox.
> 
> Since we're not measuring or changing elements on that page, I think this is
> OK.

I thought Optimizely had to be enabled on redirects to measure engagement? Happy to be wrong though - means a lighter weight/faster page. :)

Comment 16

[Justin Crawford [:hoosteeno] [:jcrawford]]

> 
> I thought Optimizely had to be enabled on redirects to measure engagement?
> Happy to be wrong though - means a lighter weight/faster page. :)

I've launched dozens of these experiments and remember only one or two where "clicked any element on the page" was a metric we cared about. We're almost always trying to move something more specific than that. In this case, with downloads, we're using a goal that counts sessions landing on scene=2. Since optimizely is enabled on scene=2, this'll work. I went ahead and removed "engagement" goal from this experiment because it wouldn't be useful.

Comment 17

[Fabio Rios [:frios]]

Justin, lets pause the test.

Given the big improvement using the /firefox/ page, we'll want to make this permanent. I've written LinkedIn and asked if they can help us make the change to the page: https://www.linkedin.com/error_pages/unsupported-browser.html

If we can't find a contact there to help us, perhaps we can run a redirect on our side.

Comment 18

[Justin Crawford [:hoosteeno] [:jcrawford]]

It is paused. 

This experiment was a great success -- we learned a lot from it. It may also help us achieve a 50% increase in conversions on this channel. Good work :frios.