[Experiment] The "WNP_MOMENTS_15" page is wrongly triggered if the "Recommend extensions/feature as you browse" options are unchecked in the same session where the "MOMENTS_PAGE_SET" telemetry ping was logged
Categories
(Firefox :: Messaging System, defect, P3)
Tracking
()
Tracking | Status | |
---|---|---|
firefox115 | --- | affected |
People
(Reporter: mcoman, Unassigned)
References
Details
Attachments
(1 file)
1.88 MB,
image/gif
|
Details |
[Affected versions]:
- Firefox Release 115.0 - Build ID: 20230629134642
- Firefox Release 114.0.2 - Build ID: 20230619081400
[Affected Platforms]:
- Windows 10 x64
- Windows 11 x64
[Prerequisites]:
- Have a Firefox profile older than 28 days.
- You are not connected to Firefox Sync with a Firefox account linked to a VPN subscription.
- Have any from the following Firefox locales installed: 'de', 'en-CA', 'en-US', 'fr', 'it'.
- Have the "browser.search.region" pref set to any of the following regions: 'AT', 'BE', 'CA', 'CH', 'DE', 'ES', 'FI', 'FR', 'GB', 'IE', 'IT', 'NL', 'SE', 'US'.
- Have the "browser.ping-centre.log" pref set to "true" in the "about:config" page.
- Have the "Browser Console" opened.
- Have the "user.js" filed saved to your PC.
[Steps to reproduce]:
- Open the browser with the profile from the prerequisites.
- Navigate to the profile folder and paste the "user.js" file from the prerequisites.
- Restart the browser until the "MOMENTS_PAGE_SET" telemetry ping related to the "WNP_MOMENTS_15" page is displayed in the "Browser Console".
- Navigate to the "about:preferences" page and scroll to the "Browsing" section.
- Uncheck the "Recommend extensions as you browse" and the "Recommend feature as you browse" options.
- Restart the browser and observe the behavior.
[Expected result]:
- The "WNP_MOMENTS_15" page is not triggered.
[Actual result]:
- The "WNP_MOMENTS_15" page is triggered even if the "Recommend extensions/feature as you browse" options are unchecked.
[Additional Notes]:
- Attached is a screen recording of the issue:
Updated•2 years ago
|
Comment 1•2 years ago
|
||
This is how the moments page feature works, not an issue in the experiment recipe. It has the same targeting for these prefs as previous moments pages. The reason it happens is because the targeting matches when the moments page action is invoked. But the moments page action doesn't immediately open a moments page, it just sets a pref that overrides the user's homepage for one session. This is because moments pages are only supposed to open at app startup. So although the moments page action was invoked in session 1, the page will not show until session 2. If, in between sessions 1 and 2, you change something like prefs or locale or region so that the targeting no longer matches, you will still see the moments page, because there is no opportunity for the targeting to re-evaluate.
It would require an on-train patch, but it wouldn't really be a simple fix. How would firefox know when to re-evaluate the targeting and clear the override pref? 1) Targeting includes an arbitrary number of things to listen to, so we can't just listen to any changes to the targeting environment, and 2) other components may set the override pref too, it's not solely owned by the messaging system, so we can't necessarily assume that the value is safe to clear.
I suspect the reason this issue didn't show up for previous moments pages is because this is the first time a moments page has been attempted via a Nimbus experiment. I suspect Nimbus experiments take longer to enroll and load at startup, so they miss the initial moments page trigger, and have to retrigger on the next session. Whereas Remote Settings is faster, so they can be loaded in time for the initial moments page trigger. So there is not enough time between trigger and actually loading the homepage for the targeting environment to change much.
I think the fix, for now, would be to just do this experiment via Remote Settings. There isn't really any need for a proper Nimbus experiment, because there's nothing to compare in the control branch. The moments page URL is completely unique to this moments page. The only other way to access it is by typing it into the URL. So there's no reason for a no-message control, we know its performance will be 0.
Comment 2•2 years ago
|
||
This is a known issue with moments pages. Will be good to check if a user gets enrolled on reevaluation of Nimbus targeting (~6 hrs) and the flow on restart of browser. This limitation should be noted in moments docs on source doc.
Will be good to check if message group 'moments-page' honors cfr.features pref from settings and help unenroll user on re-evaluation of targeting.
Updated•2 years ago
|
Comment 4•2 years ago
|
||
Created a PR in experimenter for advanced targeting: Exisisting users(Profile age >=28), No VPN or enterprise history, and haven't disabled "recommend features as you browse" or "recommend extensions as you browse". I have verified this targeting on my local experimenter instance and the output targeting expression is as follows:
(!hasActiveEnterprisePolicies && (currentDate|date - profileAgeCreated|date) / 86400000 >= 28 && (!os.isWindows || os.windowsBuildNumber >= 18362) && userMonthlyActivity|length >= 1 && !('e6eb0d1e856335fc' in attachedFxAOAuthClients|mapToProperty('id')) && 'browser.newtabpage.activity-stream.asrouter.userprefs.cfr.features'|preferenceValue&&'browser.newtabpage.activity-stream.asrouter.userprefs.cfr.addons'|preferenceValue)
Comment 5•2 years ago
•
|
||
I've updated the recipe to use the new advanced targeting. This helps NOT enroll a user if they have disabled "recommend features as you browse" or "recommend extensions as you browse". But, as Shane mentioned above, if the user gets enrolled in the experiment and in the same session disables these settings(or changes locale or region), in the next session they will still see the moments page although they get unenrolled from the experiment.
Updated•2 years ago
|
Updated•1 year ago
|
Description
•