1202004 - Setup Optimizely for AVIM Funnelcake

Status: RESOLVED FIXED

(www.mozilla.org :: Analytics, defect)

Description

[Alex Davis [:adavis]]

Test Details:
https://mana.mozilla.org/wiki/display/FIREFOX/Vietnam+Funnelcake+with+AVIM+Pre-Installed

Test Hypothesis:
If we pre-install the AVIM add-on for Vietnamese user, then we will remove friction from the onboarding process and increase user retention in Vietnam because users will be able to more easily input text in their native language without having to go look for this add-on.

Targetting:
OS: Win
Lang: Vi
Country: Vi

Optimizely Test:
https://app.optimizely.com/edit?experiment_id=3418011432

Comment 1

[Alex Davis [:adavis]]

Chris, please review the Optimizely setup. I've inspired myself from your previously funnelcake tests.

Note the installer is not yet on the CDN.

They are here for your reference:
http://stage.mozilla.org/pub/mozilla.org/firefox/candidates/40.0.3-candidates/build1/partner-repacks-bug1195843/funnelcake45/win32/vi/Firefox%20Setup%2040.0.3.exe
http://stage.mozilla.org/pub/mozilla.org/firefox/candidates/40.0.3-candidates/build1/partner-repacks-bug1195843/funnelcake46/win32/vi/Firefox%20Setup%2040.0.3.exe

Comment 2

[Chris More [:cmore]]

r+ from an Optimizely perspective.

Reviewed:

* Audience targeting
* variation code
* URL targeting
* Goal setting
* Traffic allocation


The only change I made is the traffic allocation. I normally pause the original and make the test the two funnelcakes and then lower the sample down to 20%. This is also more understandable because you can say we are running a 20% sample for two variations split at 50/50. Instead of saying a 100% where the original is getting 80% and the two variations are getting the other %20.

See new traffic allocations:

http://cl.ly/image/2k1X0T063y3M

Comment 3

[Chris More [:cmore]]

Once the fc 45/46 funnelcakes are up on download.mozilla.org, update the download link in the variation and mana page code.

Comment 4

[Chris More [:cmore]]

Another change! I added 9/21 at noon at the end date for the experiment since Firefox 41.0 will be released the next day. That's a fail safe so we don't run it into 41.0's window.

Comment 5

[Alex Davis [:adavis]]

(In reply to Chris More [:cmore] from comment #4)
> Another change! I added 9/21 at noon at the end date for the experiment
> since Firefox 41.0 will be released the next day. That's a fail safe so we
> don't run it into 41.0's window.

Great suggestion!

Comment 6

[Chris More [:cmore]]

:jeptto: can you review this experiment knowing that the URLs 404? They are not on the production download.mozilla.org yet, but wanted to get the rest reviewed for now and then we can cycle back after the Fx URLs are live.

Comment 7

[Chris More [:cmore]]

Anthony: can you do some QA here?

We have two funnelcakes build. 

Funnelcake 45, which has the AVIM add-on installed.

Funnelcake 46, has no add-on installed.

These are win32 vi language full installers.

We can do 4 verifies, 2 for each funnelcake, and 2 for with and without profiles.

==First verify==

1) Ensure there is no Firefox profile on the Win32 machine.

2) Download this: http://stage.mozilla.org/pub/mozilla.org/firefox/candidates/40.0.3-candidates/build1/partner-repacks-bug1195843/funnelcake45/win32/vi/Firefox%20Setup%2040.0.3.exe

3) Install Firefox

4) Run Firefox

Verify: Pre-installed the AVIM Add-on as seen in left of the hamburger menu:

https://mana.mozilla.org/wiki/download/attachments/49451007/win10-fx40-vi-2.png?version=1&modificationDate=1439852064046&api=v2

==Second verify==

1) Ensure is an existing Firefox install on the machine (but not version from above)

2) Download this: http://stage.mozilla.org/pub/mozilla.org/firefox/candidates/40.0.3-candidates/build1/partner-repacks-bug1195843/funnelcake45/win32/vi/Firefox%20Setup%2040.0.3.exe

3) Install Firefox

4) Run Firefox

Verify: Pre-installed the AVIM Add-on as seen in left of the hamburger menu:

https://mana.mozilla.org/wiki/download/attachments/49451007/win10-fx40-vi-2.png?version=1&modificationDate=1439852064046&api=v2

==Third verify==

1) Ensure there is no Firefox profile on the Win32 machine.

2) Download this: http://stage.mozilla.org/pub/mozilla.org/firefox/candidates/40.0.3-candidates/build1/partner-repacks-bug1195843/funnelcake46/win32/vi/Firefox%20Setup%2040.0.3.exe

3) Install Firefox

4) Run Firefox

Verify: No ADVIM add-on installed

==Forth verify==

1) Ensure is an existing Firefox install on the machine (but not version from above)

2) Download this: http://stage.mozilla.org/pub/mozilla.org/firefox/candidates/40.0.3-candidates/build1/partner-repacks-bug1195843/funnelcake46/win32/vi/Firefox%20Setup%2040.0.3.exe

3) Install Firefox

4) Run Firefox

Verify: No ADVIM add-on installed

Comment 8

[u279076]

Everything looks like it's working as expected on Windows 10 and Windows 7.

Comment 9

[Alex Davis [:adavis]]

(In reply to Anthony Hughes, QA Mentor (:ashughes) from comment #8)
> Everything looks like it's working as expected on Windows 10 and Windows 7.

Thanks so much for doing that so quickly! :)

Comment 10

[Jonathan Petto [:jpetto]]

Optimizely code looks good to me. r+ on that front.

Comment 11

[Chris More [:cmore]]

Woo hoo. Funnelcake builds are on production and I have also verified the URLs :adavis has in Optimizely are the correct ones and I will also r+ the experiment.

:adavis: enable the experiment and note the date in the tracker: https://docs.google.com/a/mozilla.com/spreadsheets/d/1gZ-GFyubAAlIxh-u-5SR52rqKKfBRLRuwrjSq97QkWA/edit?usp=sharing

Thanks jpetto

Comment 12

[Jennifer Bertsch [:jbertsch]]

Is this bug in the right product/component?

Comment 13

[Alex Davis [:adavis]]

(In reply to Jennifer Bertsch [:jbertsch] from comment #12)
> Is this bug in the right product/component?

This is what Cmore has been doing with his Optimizely bugs so I've been doing the same. Let me know if you think it would be better elsewhere.

Comment 14

[Chris More [:cmore]]

(In reply to Jennifer Bertsch [:jbertsch] from comment #12)
> Is this bug in the right product/component?

It could be in firefox:general, but since the optimizely test for the distribution of the test is on www.mozilla.org, it seems to make sense to put a/b tests on www.mozilla.org (even if it is not a website a/b test) in www.mozilla.org:analytics product:component since it is where the code lives.

Comment 15

[Alex Davis [:adavis]]

I hate to re-open this bug but due to the FHR incident, we need to re-run this test.

Thanks in advance for your help.

Comment 16

[Alex Davis [:adavis]]

I'm re-closing this bug.

FHR data has been coming back in and despite missing a month of data, if we look at the usage of each funnelcake 1 month later, there is a really clear winner.