Implement ExperimentManager and API that can integrate with MS and Normandy
Categories
(Firefox :: Messaging System, enhancement, P1)
Tracking
()
People
(Reporter: k88hudson, Assigned: k88hudson)
References
Details
Attachments
(4 files)
47 bytes,
text/x-phabricator-request
|
RyanVM
:
approval-mozilla-beta+
|
Details | Review |
47 bytes,
text/x-phabricator-request
|
RyanVM
:
approval-mozilla-beta+
|
Details | Review |
47 bytes,
text/x-phabricator-request
|
RyanVM
:
approval-mozilla-beta+
|
Details | Review |
47 bytes,
text/x-phabricator-request
|
RyanVM
:
approval-mozilla-beta+
|
Details | Review |
In order to support Messaging System Experiments, I'd like to implementing an ExperimentManager
module that handles branch generation, caching, telemtry, and other elements of experiment lifecycle management. Right now these are handled individually for Prefs and Addons so we'd have to implement them separately for Messaging System / AboutWelcome anyway.
It should:
- Expose lifecycle methods for handling new recipes, ready states
- Work at startup and first run
- Provide an API that allows consumer code (including the MessagingSystemRouter and about:welcome) to retrieve experiment state and handle error or update events, in both the parent and content processes.
Assignee | ||
Comment 1•5 years ago
|
||
Assignee | ||
Updated•5 years ago
|
Assignee | ||
Updated•5 years ago
|
Updated•5 years ago
|
Updated•5 years ago
|
Updated•5 years ago
|
Pushed by khudson@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/50bc2329abae Add ExperimentsManager and ExperimentAPI r=mythmon https://hg.mozilla.org/integration/autoland/rev/600778a596aa Integrate ExperimentManager with Normandy r=mythmon
Comment 5•5 years ago
|
||
Backed out 2 changesets (Bug 1620021) for failures in test_ExperimentManager_lifecycle.js
Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedJob=297183604&resultStatus=testfailed%2Cbusted%2Cexception&revision=600778a596aac5395ea31b238268784fe243f4d5
Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=297183604&repo=autoland&lineNumber=1382
Backout: https://hg.mozilla.org/integration/autoland/rev/fc403bd161cc6d683ca485c24956793c9261bc5d
Pushed by khudson@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/392200a6f8ea Add ExperimentsManager and ExperimentAPI r=mythmon https://hg.mozilla.org/integration/autoland/rev/c4d219534696 Integrate ExperimentManager with Normandy r=mythmon
Comment 7•5 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/392200a6f8ea
https://hg.mozilla.org/mozilla-central/rev/c4d219534696
Comment 8•5 years ago
|
||
[Tracking Requested - why for this release]: this is intended for uplift into Beta and will provide a new way to control feature flag experiments without relying on prefs
Pushed by khudson@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/835b24a1d43e Integrate aboutwelcome with ExperimentManager r=andreio
Comment 10•4 years ago
|
||
bugherder |
Assignee | ||
Comment 11•4 years ago
|
||
Comment on attachment 9130838 [details]
Bug 1620021 - Add ExperimentsManager and ExperimentAPI
Beta/Release Uplift Approval Request
- User impact if declined: We will not be able to launch the onboarding Pull Factor Experiment in Firefox 76. This is critical for us to be able to learn about how to make onboarding better and do more experimentation in Q2.
- Is this code covered by automated tests?: Yes
- Has the fix been verified in Nightly?: No
- Needs manual test from QE?: Yes
- If yes, steps to reproduce: [Punam to add]
- List of other uplifts needed: Bug 1624309
- Risk to taking this patch: Medium
- Why is the change risky/not risky? (and alternatives if risky): Note that we are still in the process of verifying in Nightly today since this requires real content to be published to Remote Settings to test.
Risk is medium because although we could run into unexpected bugs:
(a) we can turn it off via a pref / unpublish any remote data from remote settings
(b) the failure state it does not have any user-facing effects, since it only controls experimentation enrollment
- String changes made/needed: None
Assignee | ||
Updated•4 years ago
|
Assignee | ||
Updated•4 years ago
|
Assignee | ||
Updated•4 years ago
|
Assignee | ||
Comment 12•4 years ago
|
||
Can you help add steps to test launching the pull-factor experiment via RS staging (feel free to link to a document)?
Comment 13•4 years ago
|
||
Started documenting steps to verify pull-factor experiment below:
Updated•4 years ago
|
Comment 14•4 years ago
|
||
I have verified that the "variant_1" version of the "about:welcome" page is successfully displayed after following the steps from here using the latest Firefox Nightly (77.0a1 Build ID - 20200421094220) installed on Windows 10 x64.
However, we have not tested this enhancement on macOS and Linux, because as far as we know, from the past experiments, a "first-run" experiment type depends on the full installer which is available only on Windows.
Punam, could you please tell us if the experiment will target other platforms besides Windows?
I'm asking this because we observed that there is a discrepancy between the "Experimentation Platform Design for about:welcome in 76" doc and the experimenter ticked, more exactly the documentation says that the experiment will target the Windows platform only (due to FirstStartup first-run limitations), meanwhile, the experimenter ticket states that the experiment will target all the platforms.
Comment 15•4 years ago
|
||
(In reply to Marius Coman [:mcoman], Ecosystem QA from comment #14)
I have verified that the "variant_1" version of the "about:welcome" page is successfully displayed after following the steps from here using the latest Firefox Nightly (77.0a1 Build ID - 20200421094220) installed on Windows 10 x64.
However, we have not tested this enhancement on macOS and Linux, because as far as we know, from the past experiments, a "first-run" experiment type depends on the full installer which is available only on Windows.Punam, could you please tell us if the experiment will target other platforms besides Windows?
I'm asking this because we observed that there is a discrepancy between the "Experimentation Platform Design for about:welcome in 76" doc and the experimenter ticked, more exactly the documentation says that the experiment will target the Windows platform only (due to FirstStartup first-run limitations), meanwhile, the experimenter ticket states that the experiment will target all the platforms.
Hi Marius,
I have updated ticket to reflect the experiment will be Windows only . Thanks!
Comment 16•4 years ago
|
||
Comment on attachment 9130838 [details]
Bug 1620021 - Add ExperimentsManager and ExperimentAPI
Needed for experiments targeting 76. Approved for 76.0b7.
Updated•4 years ago
|
Updated•4 years ago
|
Comment 17•4 years ago
|
||
bugherder uplift |
https://hg.mozilla.org/releases/mozilla-beta/rev/d58dff3b3032
https://hg.mozilla.org/releases/mozilla-beta/rev/11e0faa7bc6d
https://hg.mozilla.org/releases/mozilla-beta/rev/86753f535c44
Comment 18•4 years ago
|
||
Backed out 4 changesets (bug 1620021, bug 1624309) for causing browser-chrome faliures on browser_aboutwelcome_simplified.js and browser_ext_themes_ntp_colors.js a=Backout
Backout revision https://hg.mozilla.org/releases/mozilla-beta/rev/19a84d3cda6d15e0bf9f52fa67ed0fd2877b1704
Failure logs https://treeherder.mozilla.org/logviewer.html#?job_id=298703999&repo=mozilla-beta for browser_ext_themes_ntp_colors.js
https://treeherder.mozilla.org/logviewer.html#?job_id=298704015&repo=mozilla-beta for browser_aboutwelcome_simplified.js
Kate can you please take alook?
Assignee | ||
Comment 19•4 years ago
|
||
Comment 20•4 years ago
|
||
Pushed by khudson@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/1877c5d6fa3e Dont block aboutwelcome on loading experiments from disk r=andreio
Assignee | ||
Comment 21•4 years ago
|
||
The additional patch I landed https://phabricator.services.mozilla.com/D71932 should resolve the intermittent test failures and should apply cleanly to beta as well, do I need to add a separate request for that?
Assignee | ||
Comment 22•4 years ago
|
||
Comment on attachment 9142354 [details]
Bug 1620021 - Dont block aboutwelcome on loading experiments from disk
Beta/Release Uplift Approval Request
- User impact if declined: This is a fix to address the test failures in the other dependent patches.
- Is this code covered by automated tests?: Yes
- Has the fix been verified in Nightly?: Yes
- Needs manual test from QE?: Yes
- If yes, steps to reproduce:
- List of other uplifts needed: None
- Risk to taking this patch: Low
- Why is the change risky/not risky? (and alternatives if risky):
- String changes made/needed:
Comment 23•4 years ago
|
||
Comment on attachment 9142354 [details]
Bug 1620021 - Dont block aboutwelcome on loading experiments from disk
I did a sanity check Try push and things look good. Thanks for the quick fix. Let's retry this for 76.0b8.
Comment 24•4 years ago
|
||
bugherder uplift |
https://hg.mozilla.org/releases/mozilla-beta/rev/308fff043c34
https://hg.mozilla.org/releases/mozilla-beta/rev/2d124a1f6a41
https://hg.mozilla.org/releases/mozilla-beta/rev/c72ef3c31d5b
https://hg.mozilla.org/releases/mozilla-beta/rev/30493cca8413
Comment 25•4 years ago
|
||
bugherder |
Comment 26•4 years ago
|
||
This enhancement was part of the experiment work from PI-580 that was signed off as GREEN. Considering this I am marking this enhancement as VERIFIED.
Description
•