1565558 - What's New panel holdback study for Firefox 70

Status: RESOLVED FIXED

(Firefox :: Messaging System, task, P1)

Description

[Ricky Rosario [:rrosario, :r1cky]]

Bug 1565555 will enable the What's New panel by default for everybody. We want to set up a holdback study so some users dont get the feature.

Comment 1

[Andrei Oprea [:andreio]]

:mythmon, just to confirm how the hold back study will happen (details in the user story doc): the feature is shown at browser startup (with a 5 mins delay) and is dependent on users updating from Firefox 69 to 70 (new profiles on Firefox 70 will not see it).

• the feature does not exist in 69
• the pref that needs to be flipped does not exist in 69
  We want to show the notification within the first 5 minutes of startup.

1. Does it make the study more difficult if we want to set the pref in 69 and track users after they upgraded
2. If we instead flip the pref in 70 what kind response time should we expect (this might influence the 5 minute delay we have implemented)

What do you think is more reasonable?

Comment 2

[Michael Cooper [:mythmon]]

This is an ideal situation for a hold back study. Since the preference and the feature don't exist on 69, it is safe to start enrolling users in 69. Since the feature only takes affect 5 minutes after startup, even stragglers should get the correct behavior and we don't have to worry about race conditions.

The strategy I'd recommend is this: Create a recipe that targets 69 and 70 with the preference change, both the control and treatment branches. This recipe should launch on the Tuesday before the release (so probably about a week before the release). Only about a day is needed, but launching a Tuesday fits our schedule well.

• Users that use Firefox in that week before will get primed for the holdback, being assigned to their respective groups. Of course nothing will change in Firefox 69 since the feature doesn't exist. When these users upgrade to 70, they'll already have the preference set appropriately.

• Users that don't use Firefox in the week between the experiment starts and start directly with 70 (new users, for example) should receive the recipe and the change before the 5 minutes, and will get assigned to the group.

To directly answer your questions:

1. No, this actually makes things easier.
2. In this case we could see up to 6 hours delay in the worst case. I don't recommend this course.

Comment 3

[Ed Lee :Mardak]

It looks like the pref to hold back exists in 69 landed with bug 1561307.

From a quick test and code search, it looks like 69 beta does indeed not do anything if that pref is set to true. The export bug 1565293 included bug 1563319 which landed in 70. So I believe we're still good for holdback even though the pref technically exists in 69.

Comment 4

[Liz Henry (:lizzard) (relman/hg->git project)]

Is there anything here which needs to land for 70 release?

Comment 5

[Tim Spurway [:tspurway]]

:lizzard, everything has landed for this feature and it's study

Comment 6

[Liz Henry (:lizzard) (relman/hg->git project)]

Great, thanks!

Comment 7

[Marius Coman [:mcoman], Ecosystem QA]

Pref-Flip Experiment: hold-back experiment for What's New Panel

Targeted: Firefox Release 70

We have finished testing the Pref-Flip Experiment: hold-back experiment for What's New Panel experiment.

QA’s recommendation: YELLOW - SHIP IT CONDITIONALLY

Reasoning:

• During testing we have found and logged a new issue (Bug 1590721), where the “WNP” impression ping is wrongly sent even if the “What’s New” toolbar button is not displayed. This issue could affect the telemetry metrics in the event that the “WNP” is disabled through Normandy.

Testing Summary:

• Verified that the “What’s New” toolbar button is not displayed if the “browser.messaging-system.whatsNewPanel.enabled” pref is set to “false”.

• Verified that the “browser.messaging-system.whatsNewPanel.enabled” pref’s value is not changed to “true” after a browser update.

• Verified that “What’s New” option is not displayed in the hamburger menu after setting the “browser.messaging-system.whatsNewPanel.enabled” pref to “false”.

• Verified that the “Firefox Account” toolbar button is successfully badged if the “What’s New” button is not displayed.

• Verified that the “What’s New” telemetry pings are not displayed in the “Browser Console”, if the “browser.messaging-system.whatsNewPanel.enabled” pref is set to “false” (see Bug 1590721).

• Verified that the “browser.messaging-system.whatsNewPanel.enabled” pref’s value is successfully changed to “false” using Normandy pref rollouts or multi-preference actions.

Tested Platforms:

• Windows 10 x64

• Mac OS 10.15

• Arch Linux 5.3.6

Tested Firefox versions:

• Firefox Release 69.0.3

• Firefox Release 70.0

Comment 8

[Liz Henry (:lizzard) (relman/hg->git project)]

This experiment launched as planned. I don't think we need this bug now since we have it tracked in Experimenter.