1571817 - Hold back rollout test for Extended Triplets in Firefox 69

Status: VERIFIED FIXED

(Firefox :: Messaging System, enhancement, P1)

Description

[Tim Spurway [:tspurway]]

Let's create a test to hold back 1% of the extended first run triplets treatment we are doing for Firefox 69. Let's leverage our code for first-run experiments from Trailhead to do cohort selection.

Goal: to measure the interaction/engagement on triplets tiles and compare that to folks who haven't been given the 'extended' version of the triplets.

Control branch: because this is a hold back, the control group will be the 99% of folks that get the new "extended triplets" treatment

Experiment branch: the hold back branch will target the same selection criteria as the extended triplets, which is profiles that are less than seven days old. In this case, the user should see the same onboarding and newtab behaviour that users saw in Firefox 68

Comment 1

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/activity-stream/pull/5233

Comment 2

[Nan Jiang [:nanj]]

Notes for QA testing:

This patch implements a holdback experiment for the extended triplets added in bug 1568909. There are two branchs in this experiment:

• The control branch (99%), in which one should see the extended triplets if the profile age is younger than 7 days;
• The holdback branch (1%), in which one should NOT see the extended triplets regardless of the profile age.

To test the control branch is straightforward, simply use a new profile, it should show the triplets on the about:home or about:newtab. It's worth checking the shield_id field in the onboarding IMPRESSION telemetry ping, it should include the string activity-stream-extended-triplets:control. You can find how to check the telemetry in this doc.

It's hard to verify the holdback branch since it only accounts for 1% of the targeting audience. However, we can manually enroll ourselves into this branch by overwriting trailhead.extendedTriplets.experiment in about:config, simply set it as holdback then restart the browser. Users in this branch will not see the extended triplets, hence it won't send the IMPRESSION ping. Still, shield_id should include activity-stream-extended-triplets:holdback in other pings of Activity Stream.

Comment 3

[Ed Lee :Mardak]

https://hg.mozilla.org/mozilla-central/rev/0c5028732786

Comment 4

[Ciprian Muresan [:cmuresan], Ecosystem QA]

I have verified that the control branch related pings have the "shield_id":"activity-stream-extended-triplets:control;" argument for Extended Triplets IMPRESSION pings as well as on other pings generated by actions on the New Tab page. I have also verified that the "shield_id":"activity-stream-extended-triplets:holdback;" argument appears for other pings generated by actions on the New Tab page while on the holdback branch.
The above verification was done on Windows 10, macOS 10.14, and Arch Linux 4.14.3.

@Nan, thanks for the detailed steps and expected results!

Comment 5

[Nan Jiang [:nanj]]

:cmuresan - Thanks for verifying this! Glad to hear it worked as expected. \o/

Comment 6

[Ed Lee :Mardak]

https://hg.mozilla.org/releases/mozilla-beta/rev/c2aa1c31c07e4427bea5ec58432f8806dbe4d973