1645370 - The "Set Firefox as Default" snippet is wrongly displayed on the first opened tab after the user opt-out from the experiment

Status: RESOLVED WONTFIX

(Firefox :: Messaging System, defect, P2)

Description

[Marius Coman [:mcoman], Ecosystem QA]

Attachment: rec of the issue.gif

[Affected versions]:

• Firefox Release 72.0 - Build ID: 20200103162918
• Firefox Release 73.0.1 - Build ID: 20200217142647
• Firefox Release 74.0 - Build ID: 20200309095159
• Firefox Release 75.0 - Build ID: 20200403170909
• Firefox Release 76.0 - Build ID: 20200429185419
• Firefox Release 77.0.1 - Build ID: 20200602222727

[Affected Platforms]:

• Windows 10 x64
• macOS 10.15
• Ubuntu 18.04 x64

[Prerequisites]:

• Have a Firefox en-US locale build installed.
• Have a Firefox profile older than 17 weeks. In order to do that you can change the creation date from the "times.json" file that can be found in the current profile folder.
• Have an active multi pref recipe.
• Have the following pref in the "about:config" page:
  • "security.content.signature.root_hash" pref value set to "DB:74:CE:58:E4:F9:D0:9E:E0:42:36:BE:6C:C5:C4:F6:6A:E7:74:7D:C0:21:42:7A:03:BC:2F:57:0C:8B:9B:90".
  • "app.normandy.api_url" pref set to "https://stage.normandy.nonprod.cloudops.mozgcp.net/api/v1".
  • "app.normandy.dev_mode" set to “true”.
  • "app.normandy.logging.level" set to “0”.
  • "services.settings.server" pref set to "https://settings.stage.mozaws.net/v1".
  • Create a new pref named "mcoman-snip" and set it to true.
• Have the "Set Firefox as Default" snippet displayed.

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites.
2. Navigate to the "about:studies" page and remove the "special name" experiment.
3. Open a new tab and observe the displayed sections.

[Expected result]:

• The following sections are displayed:
  • "In-content Search Bar"
  • "Top Sites"
  • "Recommended by Pocket" (if available)
  • "Highlights"

[Actual result]:

• The "Set Firefox to Default" snippet is still displayed under the in-content "Search Bar".

[Notes]:

• The impression ping is also displayed in the "Browser Console" when the snippet is displayed for the second time.
• This might be a preloading issue considering the fact that the snippet is displayed only on the first opened tab and the fact that the "Set Firefox to Default" snippets are blocked in the "about:newtab#devtools" page right after the "Set Firefox as Default" button is clicked.
• Attached a screen recording of the issue.

Comment 1

[:shell escalante]

Pinging Message Router team for insight. What is being seen is the pref value that should be controlling what is shown is being reset to default upon opting out of the study - but the snippet treatment is still displayed.

Discussion:
Asked Michael Cooper and "Opting-out in about:studies ends the study immediately. I suspect what is going on is that although the preference changes immediately, Snippets does not update it's cache of messages to show in reaction to that, and so it waits until the next normal update period. I don't know what this is for Snippets though."

Marius Coman tested that the value in browser.newtabpage.activity-stream.asrouter.providers.snippets for users after going to about:studies and dismissing this study is reset to the default value ({"id":"snippets","enabled":true,"type":"remote","url":"https://snippets.cdn.mozilla.net/%STARTPAGE_VERSION%/%NAME%/%VERSION%/%APPBUILDID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/","updateCycleInMs":14400000}) after the study is removed from the "about:studies" page.

Comment 2

[Tim Spurway [:tspurway]]

This behaviour is intentional. When the user opens a new tab, their next newtab page will be rendered and cached. That's why creating a new tab in Firefox is so snappy - it's always cached. I am unsure we would want to accept a perf regression just to save the user an additional snippet impression.

Comment 3

[Tim Spurway [:tspurway]]

This is experiment specific and the experiment has concluded