1195391 - Remove spring campaign video from Fx 38+ /firstrun/ pages

Status: RESOLVED FIXED

(www.mozilla.org :: Pages & Content, defect)

Description

[Chris More [:cmore]]

For the spring campaign, we added a brand video /firstrun/ page. We did a 50/50 split with a video overlay and video below to understand which one impacts the key conversions on the page.

We kept the video for the Fx40 release so that we could compare the new embedded FxA form to the previous version that was just a button. We will have enough data by 8/19/2015.

Let's get a PR done and have it staged after we get enough data to compare Fx38/39 to Fx40.

Comment 1

[Jonathan Petto [:jpetto]]

Attachment: GitHub PR

PR will also address displaying FxA error messages in the iframe. See https://github.com/mozilla/fxa-content-server/issues/2934

Comment 2

[Jonathan Petto [:jpetto]]

PR above has been pushed to demo4 for testing:

https://www-demo4.allizom.org/firefox/40.0/firstrun/
https://www-demo4.allizom.org/firefox/38.0.5/firstrun/

Comment 3

[Chris More [:cmore]]

Thanks :jpetto. We are ready to push this once we get this QA'd and code reviewed.

Comment 4

[[github robot]]

Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/e31e7366850432b99f4021139e3210c6b1b84f46
[fix bug 1195391] Remove video from Fx38+ firstrun.

- Enable display of FxA frame error.

https://github.com/mozilla/bedrock/commit/e0724bd88d0739fa975c33ff26ada7389e03724a
Merge pull request #3215 from mozilla/bug-1195391-remove-video-fx38+-firstrun

[fix bug 1195391] Remove video from Fx38+ firstrun.

Comment 5

[Chris More [:cmore]]

Video was removed yesterday and we will monitor the conversion rates. From previous analysis, the video wasn't helping with conversion rates and only 5% of users were even watching the video.

Comment 6

[Chris More [:cmore]]

Here's the stats from the video analysis:

Video plays: 2-4% of all /firstrun/ users
Video complete: 20% of the video play users completed or about 0.06% of all /firstrun/ users.
Video impact to FxA conversion: The video lowered the conversion rate to engaging with FxA by 15-20%
Video impact to bounce rate: The presence of the video increased page bounce rate by 20%
Video impact to ADIs: The presence of the video decreased retention by 20% over 2.5 months

From these results, there is a new hypothesis:

Given that the FxA rate increased when no video was present and bounce rate decreased, users that signed up for an account and setup sync were less likely to churn over time. The video wasn't making them churn, but the video negatively impacted sign up rates and getting more people to sign up retained the no-video cohort longer.