1577008 - Fx 69 - Pref Rollout: Pocket New Tab - en-US/US geo

Status: RESOLVED FIXED

(Shield :: Rollouts, task)

Description

[Jessilyn Davis]

Pending the completion of enrollment of a Holdback study group (Bug 1573312), we'd like to rollout the new, Pocket New Tab experience to the en-US/US geo for users that have the default Firefox experience with New Tab enabled.

Timeline:
Sept 3 - Holdback study enrollment start
Sept 10 - Holdback study enrollment ended
Sept 10 - Rollout Pocket New Tab experience to the en-US/US geo at 25%*
Sept 11 - Rollout Pocket New Tab experience to the en-US/US geo at 50%*
Sept 12 - Rollout Pocket New Tab experience to the en-US/US geo at 100%*

*Pending your recommendation. This is a well tested feature that the only reason we're rolling out slowly is to ensure our content backend can handle the full en-US/US geo population. Risks & Mitigations for this feature rollout can be found in this Tech Documentation

Comment 1

[Michael Cooper [:mythmon]]

we'd like to rollout the new, Pocket New Tab experience

Can you list the preference that needs to change, and to what value, for this to happen?

for users that have the default Firefox experience with New Tab enabled

Can you specify how we can detect this? Is it a preference or group of preferences we can read?

Pending the completion of enrollment of a Holdback study group

This isn't a technical requirement. We can start enrollment in the rollout whenever we want, because we can guarantee there will be no overlap between the rollout and experiment populations. The schedule you've proposed sounds fine to me though, pending relman approval.

We can't actually take this rollout to 100%, because the holdback experiment will use up 1% of the population. That's not a big deal though, we'll just stop at 99%.

Given the above, do you want to change anything about the plan?

Comment 2

[Scott [:thecount] Downe]

we'd like to rollout the new, Pocket New Tab experience

Can you list the preference that needs to change, and to what value, for this to happen?

Use browser.newtabpage.activity-stream.discoverystream.enabled and set it to true for people that are enrolled.

for users that have the default Firefox experience with New Tab enabled

Can you specify how we can detect this? Is it a preference or group of preferences we can read?

Do we need to filter them out? This feature is handled in the code, so if you set this feature to on, and either newtab or pocket is off, the feature is ignored.
If you still need to filter the rollout to certain people, you want to look at these prefs: browser.newtabpage.activity-stream.feeds.section.topstories if that is false, our feature in browser.newtabpage.activity-stream.discoverystream.enabled is ignored.
and if browser.newtabpage.enabled is false our feature is ignored in new tabs but still enabled for home.
For home, you want to look at browser.startup.homepage if that value is not about:home our feature is also ignored for home.
I'm also sure if any of those prefs are simply user set, it's the same as their off values.

Comment 3

[Michael Cooper [:mythmon]]

Do we need to filter them out? This feature is handled in the code, so if you set this feature to on, and either newtab or pocket is off, the feature is ignored.

In this case, we don't need to filter the users out for the rollout. This information is useful for the experiment though.

I've updated the user story to reflect the above comments. Additionally, we are currently planning to run the experiment with an 8% sample, so I've lowered the maximum sample achieved t 92% to reflect that.

Comment 4

[Jessilyn Davis]

Update on experiment sample - we'll be targeting 6% for the holdback. This can be updated to 94% for the rollout.

Details here: https://experimenter.services.mozilla.com/experiments/activity-stream-pkt-new-tab-release69-layout-holdback-1573312/

Thanks mythmon! :D

Comment 5

[Jessilyn Davis]

Heyo! Per slack convos, we're now all rolled out! Thanks y'all!