1620021 - Implement ExperimentManager and API that can integrate with MS and Normandy

Status: VERIFIED FIXED

(Firefox :: Messaging System, enhancement, P1)

Description

[Kate Hudson :k88hudson]

In order to support Messaging System Experiments, I'd like to implementing an ExperimentManager module that handles branch generation, caching, telemtry, and other elements of experiment lifecycle management. Right now these are handled individually for Prefs and Addons so we'd have to implement them separately for Messaging System / AboutWelcome anyway.

It should:

• Expose lifecycle methods for handling new recipes, ready states
• Work at startup and first run
• Provide an API that allows consumer code (including the MessagingSystemRouter and about:welcome) to retrieve experiment state and handle error or update events, in both the parent and content processes.

Comment 1

[Kate Hudson :k88hudson]

Attachment: Bug 1620021 - Add ExperimentsManager and ExperimentAPI

Comment 2

[Kate Hudson :k88hudson]

Attachment: Bug 1620021 - Integrate ExperimentManager with Normandy

Depends on D65365

Comment 3

[Kate Hudson :k88hudson]

Attachment: Bug 1620021 - Integrate aboutwelcome with ExperimentManager

Depends on D65366

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/50bc2329abae
Add ExperimentsManager and ExperimentAPI r=mythmon
https://hg.mozilla.org/integration/autoland/rev/600778a596aa
Integrate ExperimentManager with Normandy r=mythmon

Comment 5

[Noemi Erli[:noemi_erli]]

Backed out 2 changesets (Bug 1620021) for failures in test_ExperimentManager_lifecycle.js

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedJob=297183604&resultStatus=testfailed%2Cbusted%2Cexception&revision=600778a596aac5395ea31b238268784fe243f4d5

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=297183604&repo=autoland&lineNumber=1382

Backout: https://hg.mozilla.org/integration/autoland/rev/fc403bd161cc6d683ca485c24956793c9261bc5d

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/392200a6f8ea
Add ExperimentsManager and ExperimentAPI r=mythmon
https://hg.mozilla.org/integration/autoland/rev/c4d219534696
Integrate ExperimentManager with Normandy r=mythmon

Comment 7

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/392200a6f8ea
https://hg.mozilla.org/mozilla-central/rev/c4d219534696

Comment 8

[Tim Spurway [:tspurway]]

[Tracking Requested - why for this release]: this is intended for uplift into Beta and will provide a new way to control feature flag experiments without relying on prefs

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/835b24a1d43e
Integrate aboutwelcome with ExperimentManager r=andreio

Comment 10

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/835b24a1d43e

Comment 11

[Kate Hudson :k88hudson]

Comment on attachment 9130838 [details]
Bug 1620021 - Add ExperimentsManager and ExperimentAPI

Beta/Release Uplift Approval Request

• User impact if declined: We will not be able to launch the onboarding Pull Factor Experiment in Firefox 76. This is critical for us to be able to learn about how to make onboarding better and do more experimentation in Q2.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: [Punam to add]
• List of other uplifts needed: Bug 1624309
• Risk to taking this patch: Medium
• Why is the change risky/not risky? (and alternatives if risky): Note that we are still in the process of verifying in Nightly today since this requires real content to be published to Remote Settings to test.

Risk is medium because although we could run into unexpected bugs:
(a) we can turn it off via a pref / unpublish any remote data from remote settings
(b) the failure state it does not have any user-facing effects, since it only controls experimentation enrollment

• String changes made/needed: None

Comment 12

[Kate Hudson :k88hudson]

Can you help add steps to test launching the pull-factor experiment via RS staging (feel free to link to a document)?

Comment 13

[Punam Dahiya [:pdahiya]]

Started documenting steps to verify pull-factor experiment below:

https://docs.google.com/document/d/1mlajaljqPZYmPXo1nm9wQy-FrSTFuDTV2CtSQ7a0Ibc/edit#heading=h.lxd5be9sbxuf

Comment 14

[Marius Coman [:mcoman], Ecosystem QA]

I have verified that the "variant_1" version of the "about:welcome" page is successfully displayed after following the steps from here using the latest Firefox Nightly (77.0a1 Build ID - 20200421094220) installed on Windows 10 x64.
However, we have not tested this enhancement on macOS and Linux, because as far as we know, from the past experiments, a "first-run" experiment type depends on the full installer which is available only on Windows.

Punam, could you please tell us if the experiment will target other platforms besides Windows?
I'm asking this because we observed that there is a discrepancy between the "Experimentation Platform Design for about:welcome in 76" doc and the experimenter ticked, more exactly the documentation says that the experiment will target the Windows platform only (due to FirstStartup first-run limitations), meanwhile, the experimenter ticket states that the experiment will target all the platforms.

Comment 15

[Punam Dahiya [:pdahiya]]

(In reply to Marius Coman [:mcoman], Ecosystem QA from comment #14)

I have verified that the "variant_1" version of the "about:welcome" page is successfully displayed after following the steps from here using the latest Firefox Nightly (77.0a1 Build ID - 20200421094220) installed on Windows 10 x64.
However, we have not tested this enhancement on macOS and Linux, because as far as we know, from the past experiments, a "first-run" experiment type depends on the full installer which is available only on Windows.

Punam, could you please tell us if the experiment will target other platforms besides Windows?
I'm asking this because we observed that there is a discrepancy between the "Experimentation Platform Design for about:welcome in 76" doc and the experimenter ticked, more exactly the documentation says that the experiment will target the Windows platform only (due to FirstStartup first-run limitations), meanwhile, the experimenter ticket states that the experiment will target all the platforms.

Hi Marius,
I have updated ticket to reflect the experiment will be Windows only . Thanks!

Comment 16

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9130838 [details]
Bug 1620021 - Add ExperimentsManager and ExperimentAPI

Needed for experiments targeting 76. Approved for 76.0b7.

Comment 17

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/d58dff3b3032
https://hg.mozilla.org/releases/mozilla-beta/rev/11e0faa7bc6d
https://hg.mozilla.org/releases/mozilla-beta/rev/86753f535c44

Comment 18

[Arthur Iakab [arthur_iakab]]

Backed out 4 changesets (bug 1620021, bug 1624309) for causing browser-chrome faliures on browser_aboutwelcome_simplified.js and browser_ext_themes_ntp_colors.js a=Backout

Backout revision https://hg.mozilla.org/releases/mozilla-beta/rev/19a84d3cda6d15e0bf9f52fa67ed0fd2877b1704

Failed push https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&resultStatus=testfailed%2Cbusted%2Cexception&revision=22851affc3b941cff06a869f2b322ccc60c789c0&searchStr=bc

Failure logs https://treeherder.mozilla.org/logviewer.html#?job_id=298703999&repo=mozilla-beta for browser_ext_themes_ntp_colors.js
https://treeherder.mozilla.org/logviewer.html#?job_id=298704015&repo=mozilla-beta for browser_aboutwelcome_simplified.js

Kate can you please take alook?

Comment 19

[Kate Hudson :k88hudson]

Attachment: Bug 1620021 - Dont block aboutwelcome on loading experiments from disk

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1877c5d6fa3e
Dont block aboutwelcome on loading experiments from disk r=andreio

Comment 21

[Kate Hudson :k88hudson]

The additional patch I landed https://phabricator.services.mozilla.com/D71932 should resolve the intermittent test failures and should apply cleanly to beta as well, do I need to add a separate request for that?

Comment 22

[Kate Hudson :k88hudson]

Comment on attachment 9142354 [details]
Bug 1620021 - Dont block aboutwelcome on loading experiments from disk

Beta/Release Uplift Approval Request

• User impact if declined: This is a fix to address the test failures in the other dependent patches.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: Yes
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky):
• String changes made/needed:

Comment 23

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9142354 [details]
Bug 1620021 - Dont block aboutwelcome on loading experiments from disk

I did a sanity check Try push and things look good. Thanks for the quick fix. Let's retry this for 76.0b8.

Comment 24

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/308fff043c34
https://hg.mozilla.org/releases/mozilla-beta/rev/2d124a1f6a41
https://hg.mozilla.org/releases/mozilla-beta/rev/c72ef3c31d5b
https://hg.mozilla.org/releases/mozilla-beta/rev/30493cca8413

Comment 25

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/1877c5d6fa3e

Comment 26

[Marius Coman [:mcoman], Ecosystem QA]

This enhancement was part of the experiment work from PI-580 that was signed off as GREEN. Considering this I am marking this enhancement as VERIFIED.