Closed Bug 1202004 Opened 4 years ago Closed 4 years ago

Setup Optimizely for AVIM Funnelcake

Categories

(www.mozilla.org :: Analytics, defect)

Development/Staging
defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: adavis, Assigned: adavis)

References

(Blocks 1 open bug)

Details

(Whiteboard: [fxgrowth])

Test Details:
https://mana.mozilla.org/wiki/display/FIREFOX/Vietnam+Funnelcake+with+AVIM+Pre-Installed

Test Hypothesis:
If we pre-install the AVIM add-on for Vietnamese user, then we will remove friction from the onboarding process and increase user retention in Vietnam because users will be able to more easily input text in their native language without having to go look for this add-on.

Targetting:
OS: Win
Lang: Vi
Country: Vi

Optimizely Test:
https://app.optimizely.com/edit?experiment_id=3418011432
r+ from an Optimizely perspective.

Reviewed:

* Audience targeting
* variation code
* URL targeting
* Goal setting
* Traffic allocation


The only change I made is the traffic allocation. I normally pause the original and make the test the two funnelcakes and then lower the sample down to 20%. This is also more understandable because you can say we are running a 20% sample for two variations split at 50/50. Instead of saying a 100% where the original is getting 80% and the two variations are getting the other %20.

See new traffic allocations:

http://cl.ly/image/2k1X0T063y3M
Flags: needinfo?(chrismore.bugzilla)
Once the fc 45/46 funnelcakes are up on download.mozilla.org, update the download link in the variation and mana page code.
Another change! I added 9/21 at noon at the end date for the experiment since Firefox 41.0 will be released the next day. That's a fail safe so we don't run it into 41.0's window.
(In reply to Chris More [:cmore] from comment #4)
> Another change! I added 9/21 at noon at the end date for the experiment
> since Firefox 41.0 will be released the next day. That's a fail safe so we
> don't run it into 41.0's window.

Great suggestion!
:jeptto: can you review this experiment knowing that the URLs 404? They are not on the production download.mozilla.org yet, but wanted to get the rest reviewed for now and then we can cycle back after the Fx URLs are live.
Flags: needinfo?(jon)
Anthony: can you do some QA here?

We have two funnelcakes build. 

Funnelcake 45, which has the AVIM add-on installed.

Funnelcake 46, has no add-on installed.

These are win32 vi language full installers.

We can do 4 verifies, 2 for each funnelcake, and 2 for with and without profiles.

==First verify==

1) Ensure there is no Firefox profile on the Win32 machine.

2) Download this: http://stage.mozilla.org/pub/mozilla.org/firefox/candidates/40.0.3-candidates/build1/partner-repacks-bug1195843/funnelcake45/win32/vi/Firefox%20Setup%2040.0.3.exe

3) Install Firefox

4) Run Firefox

Verify: Pre-installed the AVIM Add-on as seen in left of the hamburger menu:

https://mana.mozilla.org/wiki/download/attachments/49451007/win10-fx40-vi-2.png?version=1&modificationDate=1439852064046&api=v2

==Second verify==

1) Ensure is an existing Firefox install on the machine (but not version from above)

2) Download this: http://stage.mozilla.org/pub/mozilla.org/firefox/candidates/40.0.3-candidates/build1/partner-repacks-bug1195843/funnelcake45/win32/vi/Firefox%20Setup%2040.0.3.exe

3) Install Firefox

4) Run Firefox

Verify: Pre-installed the AVIM Add-on as seen in left of the hamburger menu:

https://mana.mozilla.org/wiki/download/attachments/49451007/win10-fx40-vi-2.png?version=1&modificationDate=1439852064046&api=v2

==Third verify==

1) Ensure there is no Firefox profile on the Win32 machine.

2) Download this: http://stage.mozilla.org/pub/mozilla.org/firefox/candidates/40.0.3-candidates/build1/partner-repacks-bug1195843/funnelcake46/win32/vi/Firefox%20Setup%2040.0.3.exe

3) Install Firefox

4) Run Firefox

Verify: No ADVIM add-on installed

==Forth verify==

1) Ensure is an existing Firefox install on the machine (but not version from above)

2) Download this: http://stage.mozilla.org/pub/mozilla.org/firefox/candidates/40.0.3-candidates/build1/partner-repacks-bug1195843/funnelcake46/win32/vi/Firefox%20Setup%2040.0.3.exe

3) Install Firefox

4) Run Firefox

Verify: No ADVIM add-on installed
Flags: needinfo?(anthony.s.hughes)
Everything looks like it's working as expected on Windows 10 and Windows 7.
Flags: needinfo?(anthony.s.hughes)
(In reply to Anthony Hughes, QA Mentor (:ashughes) from comment #8)
> Everything looks like it's working as expected on Windows 10 and Windows 7.

Thanks so much for doing that so quickly! :)
Optimizely code looks good to me. r+ on that front.
Flags: needinfo?(jon)
Woo hoo. Funnelcake builds are on production and I have also verified the URLs :adavis has in Optimizely are the correct ones and I will also r+ the experiment.

:adavis: enable the experiment and note the date in the tracker: https://docs.google.com/a/mozilla.com/spreadsheets/d/1gZ-GFyubAAlIxh-u-5SR52rqKKfBRLRuwrjSq97QkWA/edit?usp=sharing

Thanks jpetto
Is this bug in the right product/component?
(In reply to Jennifer Bertsch [:jbertsch] from comment #12)
> Is this bug in the right product/component?

This is what Cmore has been doing with his Optimizely bugs so I've been doing the same. Let me know if you think it would be better elsewhere.
Blocks: 1198541
(In reply to Jennifer Bertsch [:jbertsch] from comment #12)
> Is this bug in the right product/component?

It could be in firefox:general, but since the optimizely test for the distribution of the test is on www.mozilla.org, it seems to make sense to put a/b tests on www.mozilla.org (even if it is not a website a/b test) in www.mozilla.org:analytics product:component since it is where the code lives.
Whiteboard: [fxgrowth]
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
I hate to re-open this bug but due to the FHR incident, we need to re-run this test.

Thanks in advance for your help.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
I'm re-closing this bug.

FHR data has been coming back in and despite missing a month of data, if we look at the usage of each funnelcake 1 month later, there is a really clear winner.
Status: REOPENED → RESOLVED
Closed: 4 years ago4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.