1151528 - Add optimizely to hello.firefox.com for research and optimization purposes

Status: RESOLVED FIXED

(Hello (Loop) :: Client, defect, P2)

Description

[Chris More [:cmore]]

The Firefox Growth Team is going to be doing research on Firefox Hello usage and running some small A/B tests. We use optimizely on www.mozilla.org and other websites at Mozilla to do A/B tests and we would add it to hello.firefox.com right before the Google Analytics tag and within the DNT conditional block. 

Tag:

<script src="//cdn.optimizely.com/js/2768540301.js"></script>

Comment 1

[Mark Banner (:standard8)]

Hi Chris,

What should the url be for non-production sites? or can it be the same?

Should we be updating the privacy policy before doing this?

Comment 2

[Chris More [:cmore]]

(In reply to Mark Banner (:standard8) from comment #1)
> Hi Chris,
> 
> What should the url be for non-production sites? or can it be the same?
> 
> Should we be updating the privacy policy before doing this?

Hi Mark.

Good questions.

> Should we be updating the privacy policy before doing this?

We should have the privacy/legal team review the Hello privacy statements:

https://www.mozilla.org/en-US/privacy/firefox-hello/

and websites privacy statement that mentions Optimizely:

https://www.mozilla.org/en-US/privacy/websites/

and determine if the Hello privacy policy + websites privacy policy covers us here given that hello.firefox.com is a really a website that has abilities to interact with the product. Slightly a grey area.

Stacy: Can you provide us some guidance here?

Comment 3

[Chris More [:cmore]]

(In reply to Mark Banner (:standard8) from comment #1)
> Hi Chris,
> 
> What should the url be for non-production sites? or can it be the same?

The JS in comment 0 would be the same for prod/dev/stage websites. In Optimizely, you match tests to the fully qualified domain name, thus we can do sample tests on stage by adding rules to Optimizely to only match non-prod domains.

Comment 4

[Alexis Metaireau (:alexis)]

hello.firefox.com is handled by the client side component, updating bugzilla to reflect this.

Comment 5

[Romain Testard [:RT]]

(In reply to Chris More [:cmore] from comment #2)
> (In reply to Mark Banner (:standard8) from comment #1)
> > Hi Chris,
> > 
> > What should the url be for non-production sites? or can it be the same?
> > 
> > Should we be updating the privacy policy before doing this?
> 
> Hi Mark.
> 
> Good questions.
> 
> > Should we be updating the privacy policy before doing this?
> 
> We should have the privacy/legal team review the Hello privacy statements:
> 
> https://www.mozilla.org/en-US/privacy/firefox-hello/
> 
> and websites privacy statement that mentions Optimizely:
> 
> https://www.mozilla.org/en-US/privacy/websites/
> 
> and determine if the Hello privacy policy + websites privacy policy covers
> us here given that hello.firefox.com is a really a website that has
> abilities to interact with the product. Slightly a grey area.
> 
> Stacy: Can you provide us some guidance here?

Mika, I loop you in there since you're looking at legal issues with Hello.

Comment 6

[Romain Testard [:RT]]

Mark/Shell, could we make this part of 0.17 since it seems like a fairly small change that will help a lot for the growth model and A/B testing.

Comment 7

[:shell escalante]

Hi Mark,  Per Alexis this looks like it's on the link-clicker side.  Is this truly a one-liner patch that could go up for the next link-clicker roll-out?  If there is work on server side, i'd not try for this 0.17 (since we need to get 0.17.4 released without holding it up).

Comment 8

[Mark Banner (:standard8)]

(In reply to :shell escalante from comment #7)
> Hi Mark,  Per Alexis this looks like it's on the link-clicker side.  Is this
> truly a one-liner patch that could go up for the next link-clicker roll-out?

It is link-clicker only. I'm waiting for the response from Mika or Stacy as requested in comment 2/comment 5 before rolling this out.

Mika: note, we're also using Google Analytics at the moment as well.

Comment 9

[Mika]

Hi, just saw this, sorry for delay.  FYI, any questions involving changes to analytics or data should go to Marshall Erwin (not Stacy). 

(1) is the addition of Optimizely temporary (for a defined period of testing) or indefinite?
(2) I understand that the script running on hello.firefox.com (which is covered by the Websites Privacy Notice).  I'm unclear on how it runs for linkclickers, can you explain?
(3) what's the data that is collected? 
(4) what are the A/B tests?

Comment 10

[Chris More [:cmore]]

(In reply to Mika from comment #9)
> Hi, just saw this, sorry for delay.  FYI, any questions involving changes to
> analytics or data should go to Marshall Erwin (not Stacy). 
> 
> (1) is the addition of Optimizely temporary (for a defined period of
> testing) or indefinite?

We will be testing on and off over periods of time while we are learning. It is not trivial to add/remove the tag, but when we have no tests running, the tag is just sitting there.

> (2) I understand that the script running on hello.firefox.com (which is
> covered by the Websites Privacy Notice).  I'm unclear on how it runs for
> linkclickers, can you explain?

The script will run on the hello.firefox.com website domain. The only thing we can monitor and optimize with Optimizely are actions that are web-based. If it is something on hello.firefox.com, we should be able to see it. We also have Google Analytics on the site now.

> (3) what's the data that is collected? 

The same data that we use on every other site that uses optimizely. See also the implementation on www.mozilla.org, developer.mozilla.org, careers.mozilla.org, marketplace.firefox.com, support.mozilla.org, and webmaker.org. It is anonymous aggregate data about how successful a change being tests impacts a specific goal, like a download button. Bug 840805 is the original privacy review of Optimizely for various mozilla websites.

> (4) what are the A/B tests?

TBD. Some will be real A/B tests where we will make small adjustments to the page to see how it impacts the conversion rate of whatever we are testing. Other tests will be to simply deploy a survey to returning users on see what they like about Firefox Hello.

Comment 11

[Merwin]

Just talked to Mika about this...We will have to update the firefox hello privacy notice to include Optimizely, since it wouldn't be covered under our websites notice. 

What is the timeline for adding Optimizely to Hello?

Comment 12

[Chris More [:cmore]]

(In reply to merwin from comment #11)
> Just talked to Mika about this...We will have to update the firefox hello
> privacy notice to include Optimizely, since it wouldn't be covered under our
> websites notice. 
> 
> What is the timeline for adding Optimizely to Hello?

We have it as an important Q2 goal to understand, test, and improve the Firefox Hello funnel and work on our growth model. Without optimizely on the site, it will be difficult to deploy a small sampled survey to returning users or see if non-Firefox users are willing to try Firefox. With that, the timeline is soon, like we'll need to get something on the site in the next few weeks.

Comment 13

[Merwin]

Mika, I'm putting you back on the needinfo line so we can discuss changes to the hello privacy policy.

Comment 14

[Romain Testard [:RT]]

Mika confirmed by e-mail that we can move forward with implementation. The privacy notice will be updated but we don't need to wait for that to be done.

Comment 15

[Mark Banner (:standard8)]

Attachment: Add optimizely to Loop standalone for research and optimization purposes. Also tidy up some of the in-line javascript in the index page.

This adds loading of optimizely to the standalone page within the do not track check.

I've also taken the opportunity to fix some of the eslint warnings in the rest of the file.

Comment 16

[Mike de Boer [:mikedeboer]]

Comment on attachment 8612199 [details] [diff] [review]
Add optimizely to Loop standalone for research and optimization purposes. Also tidy up some of the in-line javascript in the index page.

Review of attachment 8612199 [details] [diff] [review]:
-----------------------------------------------------------------

Ship it!

::: browser/components/loop/standalone/content/index.html
@@ +36,5 @@
> +            window.navigator.msDoNotTrack !== "1") {
> +
> +          insertScript("//cdn.optimizely.com/js/2768540301.js");
> +
> +          window.GoogleAnalyticsObject = "ga";

Maybe add a comment here that this is an unfolded version of the official GA script that is functionally exactly the same, only readable?

Comment 17

[Pulsebot]

https://hg.mozilla.org/integration/fx-team/rev/e8806ac05aeb

Comment 18

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/e8806ac05aeb