1595390 - [Experiment]: Pref-Flip Experiment: activity-stream-Newtab-FxA-Messaging-for-pre-Trailhead-Users

Status: RESOLVED WONTFIX

(Shield :: Shield Study, task, P3)

Description

[Experimenter Bot]

activity-stream-Newtab-FxA-Messaging-for-pre-Trailhead-Users

Due to missed KPIs in Fx70, we are introducing a feature to show Onboarding Newtab Triplets to existing users who had not previously seen them.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

More information: https://experimenter.services.mozilla.com/experiments/newtab-fxa-messaging-for-pre-trailhead-users/

Comment 1

[Marius Coman [:mcoman], Ecosystem QA]

Experiment for NewTab FxA Messaging for Pre-Trailhead Profiles
Firefox Release 70

We have finished testing the Experiment for NewTab FxA Messaging for Pre-Trailhead Profiles experiment.

QA’s recommendation: YELLOW - SHIP IT CONDITIONALLY

Reasoning:

• We were unable to verify the two branches on the targeted build (Firefox Release 70.0.1) due to the fact that the actual build will only be available on the 15th of Nov, but the enrollment has to start one week before the actual release.
• During testing, we have found and logged two new issues.
• While bug 1595058 is reproducible only on the latest Firefox Nightly and will not affect the experiment end-users in a harmful way, it will affect data collection, and bug 1595074 will be reproducible on all Firefox channels after bug 1594125 is uplifted and will be easily encountered by the end-users causing the “Pre-Trailhead Triplets” cards to be displayed on a new tab after being dismissed.

Testing Summary:

• Full Functional test suite: TestRail

Tested Platforms:

• Windows 10 x64
• Mac OS 10.15.1
• Arch Linux 5.3.6

Tested Firefox versions:

• Firefox Nightly 72.0a1

Comment 2

[Liz Henry (:lizzard)]

Is this OK to close now?