Closed Bug 1581912 Opened 6 months ago Closed 6 months ago

Wrapping up the extended triplets rollout in 69

Categories

(Firefox :: Messaging System, enhancement, P1)

enhancement

Tracking

()

VERIFIED FIXED
Firefox 71
Iteration:
71.2 - Sept 16 - 29
Tracking Status
firefox70 + verified
firefox71 --- verified

People

(Reporter: nanj, Assigned: nanj)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

No description provided.

[Tracking Requested - why for this release]: If we do nothing, the extended triplet rollout will continue from 69 release to 70 with the same 5% holdback. This could be fine if we need to continue the experiment. Otherwise we should probably change the behavior to be the default behavior for 100% of users.

[Tracking Requested - why for this release]: If we do nothing, the extended triplet rollout will continue from 69 release to 70 with the same 5% holdback. This could be fine if we need to continue the experiment. Otherwise we should probably change the behavior to be the default behavior for 100% of users.

Step 1: Let's do an analysis to see if we can optimize frequency capping (currently 20) and profile age targeting (currently 7 days) and be a bit more conservative

Step 2: Land patch to extended triplets messaging with new values. Let's continue to run the hold-back (95-5%) for 70, but change the slug so we can differentiate the study run in 69

NOTE: this will be a JSON configuration change and will not change functionality

Assignee: nobody → najiang
Iteration: --- → 71.2 - Sept 16 - 29
Priority: -- → P1

Note that the 5% holdback group for this experiment should see the single Triplet newtab, exactly like the holdback group in the Extended Triplets experiment for 69 in bug #1574003

Blocks: 1584219
Iteration: 71.2 - Sept 16 - 29 → 71.3 - Sept 30 - Oct 13

Based on the telemetry in Firefox 69, on average, each user generates 5.3 impressions in the extended triplets group. I'd recommend us to decrease the impression cap from 20 to 5 in order for us to optimize the dismissal rate without affecting the overall click-through rate. In terms of the profile age targeting critiron, I believe we can leave it as it is (i.e. 7 days and younger) to make this tweak easier to understand. We can leave it as a followup option for future optimizations.

Iteration: 71.3 - Sept 30 - Oct 13 → 71.2 - Sept 16 - 29
  • Updates the frequency cap for the extended triplets feature
  • Updates the experiment ID

What's changed

This patch makes following changes to the extended triplets holdback experiment (bug 1571817) rolled out in Firefox 69.

  • Decreased impression cap for the extended triplets from 20 to 5
  • Updated the experiment ID from activity-stream-extended-triplets to activity-stream-extended-triplets-v2-1581912

Notes for QA testing:

There are still two branches in this experiment:

  • The control branch (95%), in which one should see the extended triplets if the profile age is younger than 7 days;
  • The holdback branch (5%), in which one should NOT see the extended triplets regardless of the profile age.

To test the control branch is straightforward, simply use a new profile, it should show the triplets on the about:home or about:newtab.

  • Verify it only gets shown for 5 times in total, i.e. the first about:home and the subsequent 4 newtab pages.
  • Verify the shield_id field in the onboarding IMPRESSION telemetry ping, it should include the string activity-stream-extended-triplets-v2-1581912:control. You can find how to check the telemetry in this doc.

It's harder to verify the holdback branch since it only accounts for 5% of the targeting audience. However, we can manually enroll ourselves into this branch by overwriting trailhead.extendedTriplets.experiment in about:config, simply set it as holdback then restart the browser. Users in this branch will not see the extended triplets, hence it won't send the IMPRESSION ping. Still, shield_id should include activity-stream-extended-triplets-v2-1581912:holdback in other pings of Activity Stream.

Pushed by najiang@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/f45cc699e7a1
Follow up tuning for the extended triplets rollout r=k88hudson
Status: NEW → RESOLVED
Closed: 6 months ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 71

Comment on attachment 9097445 [details]
Bug 1581912 - Follow up tuning for the extended triplets rollout

Beta/Release Uplift Approval Request

  • User impact if declined: This is to improve the user experience for certain users, as they might find the extended triplets getting overly shown.
  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Nightly?: Yes
  • Needs manual test from QE?: Yes
  • If yes, steps to reproduce: Please see comment 7 for QA steps.
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): It merely updated the experiment ID and a minor frequency cap tweak for an existing rollout.
  • String changes made/needed: None
Attachment #9097445 - Flags: approval-mozilla-beta?
Flags: qe-verify+
QA Whiteboard: [qa-triaged]

Comment on attachment 9097445 [details]
Bug 1581912 - Follow up tuning for the extended triplets rollout

Version update and decrease in # of times shown to users, looks simple enough.
Let's uplift now for beta 12 and verify this in beta 70 rather than in nightly.

Attachment #9097445 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Blocks: 1589146

I have verified this issue on Latest Firefox Nightly 71.0a1 (Build ID: 20191021095628) and Release Candidate 70 (Build ID: 20191016161957) on Windows 10 x64, Mac 10.14.6 and Arch Linux 5.3.6 x64.
I have verified that:

  • On the control branch, the Extended Triplets are shown for 5 times in total, the first about:home and the subsequent 4 new tab pages and only on profiles that are younger than 7 days. The shield_id field in the onboarding IMPRESSION telemetry ping includes the value activity-stream-extended-triplets-v2-1581912:control.
  • On the holdback branch (trailhead.extendedTriplets.experiment preference set to holdback) the Extended Triplets are not displayed at all and no IMPRESSION ping is sent. Actions taken on the about:newtab page send a telemetry ping which include the shield id with the activity-stream-extended-triplets-v2-1581912:holdback value.
Status: RESOLVED → VERIFIED
Flags: qe-verify+
You need to log in before you can comment on or make changes to this bug.