1581912 - Wrapping up the extended triplets rollout in 69

Status: VERIFIED FIXED

(Firefox :: Messaging System, enhancement, P1)

Comment 1

[Ed Lee :Mardak]

[Tracking Requested - why for this release]: If we do nothing, the extended triplet rollout will continue from 69 release to 70 with the same 5% holdback. This could be fine if we need to continue the experiment. Otherwise we should probably change the behavior to be the default behavior for 100% of users.

Comment 2

[Ed Lee :Mardak]

[Tracking Requested - why for this release]: If we do nothing, the extended triplet rollout will continue from 69 release to 70 with the same 5% holdback. This could be fine if we need to continue the experiment. Otherwise we should probably change the behavior to be the default behavior for 100% of users.

Comment 3

[Tim Spurway [:tspurway]]

Step 1: Let's do an analysis to see if we can optimize frequency capping (currently 20) and profile age targeting (currently 7 days) and be a bit more conservative

Step 2: Land patch to extended triplets messaging with new values. Let's continue to run the hold-back (95-5%) for 70, but change the slug so we can differentiate the study run in 69

NOTE: this will be a JSON configuration change and will not change functionality

Comment 4

[Tim Spurway [:tspurway]]

Note that the 5% holdback group for this experiment should see the single Triplet newtab, exactly like the holdback group in the Extended Triplets experiment for 69 in bug #1574003

Comment 5

[Nan Jiang [:nanj]]

Based on the telemetry in Firefox 69, on average, each user generates 5.3 impressions in the extended triplets group. I'd recommend us to decrease the impression cap from 20 to 5 in order for us to optimize the dismissal rate without affecting the overall click-through rate. In terms of the profile age targeting critiron, I believe we can leave it as it is (i.e. 7 days and younger) to make this tweak easier to understand. We can leave it as a followup option for future optimizations.

Comment 6

[Nan Jiang [:nanj]]

Attachment: Bug 1581912 - Follow up tuning for the extended triplets rollout

• Updates the frequency cap for the extended triplets feature
• Updates the experiment ID

Comment 7

[Nan Jiang [:nanj]]

What's changed

This patch makes following changes to the extended triplets holdback experiment (bug 1571817) rolled out in Firefox 69.

• Decreased impression cap for the extended triplets from 20 to 5
• Updated the experiment ID from activity-stream-extended-triplets to activity-stream-extended-triplets-v2-1581912

Notes for QA testing:

There are still two branches in this experiment:

• The control branch (95%), in which one should see the extended triplets if the profile age is younger than 7 days;
• The holdback branch (5%), in which one should NOT see the extended triplets regardless of the profile age.

To test the control branch is straightforward, simply use a new profile, it should show the triplets on the about:home or about:newtab.

• Verify it only gets shown for 5 times in total, i.e. the first about:home and the subsequent 4 newtab pages.
• Verify the shield_id field in the onboarding IMPRESSION telemetry ping, it should include the string activity-stream-extended-triplets-v2-1581912:control. You can find how to check the telemetry in this doc.

It's harder to verify the holdback branch since it only accounts for 5% of the targeting audience. However, we can manually enroll ourselves into this branch by overwriting trailhead.extendedTriplets.experiment in about:config, simply set it as holdback then restart the browser. Users in this branch will not see the extended triplets, hence it won't send the IMPRESSION ping. Still, shield_id should include activity-stream-extended-triplets-v2-1581912:holdback in other pings of Activity Stream.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f45cc699e7a1
Follow up tuning for the extended triplets rollout r=k88hudson

Comment 9

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/f45cc699e7a1

Comment 10

[Nan Jiang [:nanj]]

Comment on attachment 9097445 [details]
Bug 1581912 - Follow up tuning for the extended triplets rollout

Beta/Release Uplift Approval Request

• User impact if declined: This is to improve the user experience for certain users, as they might find the extended triplets getting overly shown.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: Please see comment 7 for QA steps.
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): It merely updated the experiment ID and a minor frequency cap tweak for an existing rollout.
• String changes made/needed: None

Comment 11

[Liz Henry (:lizzard) (relman/hg->git project)]

Comment on attachment 9097445 [details]
Bug 1581912 - Follow up tuning for the extended triplets rollout

Version update and decrease in # of times shown to users, looks simple enough.
Let's uplift now for beta 12 and verify this in beta 70 rather than in nightly.

Comment 12

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/releases/mozilla-beta/rev/6de5423ca135

Comment 13

[Andreea Cupsa [:acupsa], Experiments QA]

I have verified this issue on Latest Firefox Nightly 71.0a1 (Build ID: 20191021095628) and Release Candidate 70 (Build ID: 20191016161957) on Windows 10 x64, Mac 10.14.6 and Arch Linux 5.3.6 x64.
I have verified that:

• On the control branch, the Extended Triplets are shown for 5 times in total, the first about:home and the subsequent 4 new tab pages and only on profiles that are younger than 7 days. The shield_id field in the onboarding IMPRESSION telemetry ping includes the value activity-stream-extended-triplets-v2-1581912:control.
• On the holdback branch (trailhead.extendedTriplets.experiment preference set to holdback) the Extended Triplets are not displayed at all and no IMPRESSION ping is sent. Actions taken on the about:newtab page send a telemetry ping which include the shield id with the activity-stream-extended-triplets-v2-1581912:holdback value.