Open
Bug 1396105
Opened 8 years ago
Updated 8 years ago
Optimizely test: linkedin.com unsupported browser referrals
Categories
(Websites :: Web Analytics, task)
Tracking
(Not tracked)
NEW
People
(Reporter: frios, Unassigned)
Details
Linkedin is referring traffic to our /new page via an unsupported browser notification on their site. This traffic has a lower than expected download rate on Firefox. We'd like to test a variation of this landing page to improve the download rate.
Reporter | ||
Comment 1•8 years ago
|
||
Hey Justin,
Are you able to help me set up this test in Optimizely? Check out the Mana write up. I started drafting the Optimizely test, but I need some help making sure it's targeting the download properly.
https://mana.mozilla.org/wiki/display/FIREFOX/%5BBub+1396105%5D+linked.com+unsupported+browser+referral+traffic+test
https://app.optimizely.com/edit?experiment_id=8547171181
Flags: needinfo?(hoosteeno)
Comment 2•8 years ago
|
||
(In reply to Fabio Rios [:frios] from comment #1)
> I need some help
> making sure it's targeting the download properly.
Do you mean the goal tracking? It looks good to me. Everything else looks good, too. Great job setting this up.
:jpetto, could you do code review on this? Looks like snippet is already inline on both pages, so no push needed.
Flags: needinfo?(hoosteeno) → needinfo?(jon)
Comment 3•8 years ago
|
||
Making sure that:
- this should be en-US only
- this should run on ?scene=2 or any other /new variation if somehow the user got there directly from LinkedIn
- we are okay losing the referrer after Optimizely does the redirect (we are still working w/Gareth to resolve this issue)
If all the above are okay, I think we're good to run this.
Note that we are about to start running a survey on /new as well (see https://github.com/mozilla/bedrock/pull/5082).
Flags: needinfo?(jon)
Reporter | ||
Comment 4•8 years ago
|
||
(In reply to Jon Petto [:jpetto] from comment #3)
> Making sure that:
>
> - this should be en-US only
Yes, en-US only
> - this should run on ?scene=2 or any other /new variation if somehow the
> user got there directly from LinkedIn
I think the answer is yes, but not sure I understand the question? The visitors that are referred to us from linkedin for this test are coming exclusively from this page: https://www.linkedin.com/error_pages/unsupported-browser.html
> - we are okay losing the referrer after Optimizely does the redirect (we are
> still working w/Gareth to resolve this issue)
That's fine for this test. As long as we can track downloads in optimizely, we'll use that.
> If all the above are okay, I think we're good to run this.
Thank you
> Note that we are about to start running a survey on /new as well (see
> https://github.com/mozilla/bedrock/pull/5082).
Thanks for the heads up. I don't expect that to disrupt the results. We'll take an undetectable amount of traffic away from /new and redirect them to our xv=unsupported-browser page during this test
Flags: needinfo?(jon)
Comment 5•8 years ago
|
||
(In reply to Fabio Rios [:frios] from comment #4)
> (In reply to Jon Petto [:jpetto] from comment #3)
> > - this should run on ?scene=2 or any other /new variation if somehow the
> > user got there directly from LinkedIn
> I think the answer is yes, but not sure I understand the question? The
> visitors that are referred to us from linkedin for this test are coming
> exclusively from this page:
> https://www.linkedin.com/error_pages/unsupported-browser.html
The current URL targeting in the Optimizely experiment does not exclude query params, meaning that if the user came from LinkedIn and landed on https://www.mozilla.org/en-US/firefox/new/?scene=2 or https://www.mozilla.org/en-US/firefox/new/?xv=fast, they would be entered into the experiment and redirected. We could tighten up the URL targeting in Optimizely to ensure we only target folks coming to https://www.mozilla.org/en-US/firefox/new/ exactly.
I recommend we do the above to be safe.
Flags: needinfo?(jon)
Comment 6•8 years ago
|
||
I modified the regex for URL targeting, pinning it to the end as well:
^.+www\.mozilla\.org\/en\-US\/firefox\/new\/$
Now this experiment is targeted at anyone coming from the unsupported browser page on LinkedIn to the /new/ page exactly.
:jpetto, if you're good with that, I'll launch today.
Flags: needinfo?(jon)
Comment 7•8 years ago
|
||
New targeting looks good. Go for launch.
How long do we expect this test to run?
Flags: needinfo?(jon)
Comment 8•8 years ago
|
||
Current conversion rate is appx. 30%. If we want to be confident in a 20% improvement (i.e. 36%+) we need 580 visitors per variation, or 1.2k total. We get more than that on a regular weekday.
https://www.optimizely.com/sample-size-calculator/?conversion=30&effect=20&significance=95
For extra confidence we could run through next week.
Launching now.
Comment 9•8 years ago
|
||
Cool. Please comment here when the experiment is turned off, and when we have some results on the data.
Reporter | ||
Comment 10•8 years ago
|
||
Hey Justin,
I've made a few changes to the test using a copy of the previous one. Can we re-run? I've added the /firefox/ page and a variation of the xv=unsupported browser to the test.
https://app.optimizely.com/projects/246059135/experiments/8733782124
Flags: needinfo?(hoosteeno)
Comment 11•8 years ago
|
||
:jpetto, the original experiment linked above is paused and archived. We have a new version of it that sends visitors to 2 additional variations -- /firefox/ home page and /firefox/new/ with some text changes. URL target, audience definition, and goals definition are unchanged.
Can you review and/or waive review?
https://app.optimizely.com/edit?experiment_id=8733782124
Flags: needinfo?(hoosteeno) → needinfo?(jon)
Comment 12•8 years ago
|
||
Experiment looks good to me.
Noting that Optimizely is not currently enabled on /firefox.
Flags: needinfo?(jon)
Comment 13•8 years ago
|
||
(In reply to Jon Petto [:jpetto] from comment #12)
> Experiment looks good to me.
OK, I will launch now. :frios please keep an eye on it?
> Noting that Optimizely is not currently enabled on /firefox.
Since we're not measuring or changing elements on that page, I think this is OK.
Comment 14•8 years ago
|
||
Experiment is launched.
:frios, can you update the experiment page with new variations? I already updated start date.
https://mana.mozilla.org/wiki/display/FIREFOX/%5BBug+1396105%5D+linked.com+unsupported+browser+referral+traffic+test
Flags: needinfo?(frios)
Comment 15•8 years ago
|
||
(In reply to Justin Crawford [:hoosteeno] [:jcrawford] from comment #13)
>
> > Noting that Optimizely is not currently enabled on /firefox.
>
> Since we're not measuring or changing elements on that page, I think this is
> OK.
I thought Optimizely had to be enabled on redirects to measure engagement? Happy to be wrong though - means a lighter weight/faster page. :)
Comment 16•8 years ago
|
||
>
> I thought Optimizely had to be enabled on redirects to measure engagement?
> Happy to be wrong though - means a lighter weight/faster page. :)
I've launched dozens of these experiments and remember only one or two where "clicked any element on the page" was a metric we cared about. We're almost always trying to move something more specific than that. In this case, with downloads, we're using a goal that counts sessions landing on scene=2. Since optimizely is enabled on scene=2, this'll work. I went ahead and removed "engagement" goal from this experiment because it wouldn't be useful.
Reporter | ||
Updated•8 years ago
|
Flags: needinfo?(frios)
Reporter | ||
Comment 17•8 years ago
|
||
Justin, lets pause the test.
Given the big improvement using the /firefox/ page, we'll want to make this permanent. I've written LinkedIn and asked if they can help us make the change to the page: https://www.linkedin.com/error_pages/unsupported-browser.html
If we can't find a contact there to help us, perhaps we can run a redirect on our side.
Flags: needinfo?(hoosteeno)
Comment 18•8 years ago
|
||
It is paused.
This experiment was a great success -- we learned a lot from it. It may also help us achieve a 50% increase in conversions on this channel. Good work :frios.
Flags: needinfo?(hoosteeno)
You need to log in
before you can comment on or make changes to this bug.
Description
•