1495577 - [Shield] Pref Rollout: Roll out Activity Stream Search Shortcuts to 100% US region in Firefox 63&64&65

Status: RESOLVED FIXED

(Shield :: Rollouts, task)

Description

[Tim Spurway [:tspurway]]

This pref rollout is the response to a successful 1% experiment in the US region of Search Shortcuts.

The idea is to roll out the feature in Firefox to 100% of users in US region after 63 is released.  We would like to use Normandy rollout feature to roll out to different regions at different rates and to be able to dial back the percentages in any region if undesirable outcomes are reached.

Like the blocked Shield Study (#1483396), the rollout will set the pref:

browser.newtabpage.activity-stream.improvesearch.noDefaultSearchTile=true

for all locales in the US region

Comment 1

[Michael Cooper [:mythmon]]

A note for anyone looking at this later: Activity Stream preferences are not normally compatible with Preference Rollout. This particular preference has been given special support to be compatible (see bug 1495517). For future Activity Stream rollouts, the individual preferences would also need a similar migration in order to be compatible.

Tim: Is the goal to do a rollout for en-US 100%, and separately do further experiments for other locales? Or is the goal to do rollouts for all the locales you have in mind?

I ask because rollbacks (if something goes wrong) is a blunt instrument. We can't rollout to 100% of en-US and 10% of another locale, and then roll back the other locale without also rolling back en-US. To work around this, we can deploy this as a group of many rollout recipes, each of which can be controlled separately. This is more complex, but doable.

Additionally, we can't really go up to some percentage, and then down to a different percentage. Rollbacks are generally all-or-nothing. If you expect that you'd want to go up to 10% and then down to 5%, for example, we'll also have to take special precautions.

If this rollout is only for the en-US population, than the previous two paragraphs are moot, and this is a simple rollout.

Comment 2

[Tim Spurway [:tspurway]]

Let's follow this roll-out schedule (where Day 1 is shortly after 63 release on Oct 23, 2018):

Day  1:  10%
Day  4:  25%
Day  7:  50%
Day 10: 100%

Comment 3

[Nick Nguyen [:osunick]]

Approved.

Comment 4

[Pascal Chevrel:pascalc (PTO until April 26)]

(In reply to Tim Spurway [:tspurway] from comment #2)
> Let's follow this roll-out schedule (where Day 1 is shortly after 63 release
> on Oct 23, 2018):
> 
> Day  1:  10%
> Day  4:  25%
> Day  7:  50%
> Day 10: 100%

This has relman approval. The last patch for uplift was approved today and will be part of 63 beta 13.

Comment 5

[Michael Cooper [:mythmon]]

What metrics will we using to determine if the rollout is going or not going as planned? Do we want to run analysis between each rate increase, or are we going to increase the rate unless a problem comes up (such as crashes)?

Comment 6

[Tim Spurway [:tspurway]]

We will be keeping a close eye on AS dashboards to keep track of any anomalous engagement with newtab features (especially the search), as well as overall search and address bar engagement:

https://sql.telemetry.mozilla.org/dashboard/activity-stream-system-addon-metrics-summary_1
https://sql.telemetry.mozilla.org/dashboard/activity-stream-search

Comment 7

[Tim Spurway [:tspurway]]

r? :pascalc

we are ready to roll-out this feature to:

10% population of
US region
en-US locale
release 63
for users who are NOT first-run

we'd like you to give us the relman go-ahead when you feel the release has stabilized and is ready for the feature

Comment 8

[Tim Spurway [:tspurway]]

https://delivery-console.prod.mozaws.net/recipe/617/

Comment 9

[Pascal Chevrel:pascalc (PTO until April 26)]

(In reply to Tim Spurway [:tspurway] from comment #7)
> r? :pascalc
> 
> we are ready to roll-out this feature to:
> 
> 10% population of
> US region
> en-US locale
> release 63
> for users who are NOT first-run
> 
> we'd like you to give us the relman go-ahead when you feel the release has
> stabilized and is ready for the feature

Feedback and stability from our initial rollout are positive so we are now rolling Fx63 out to 100% of our release users, you can proceed with the shield study rollout.

Comment 10

[Nan Jiang [:nanj]]

hey :mythmon, we've updated the recipe to roll it further up to 25% of the target audience. Could you review it, plz?

The first phase rollout looks good, and we are tracking the telemetry for this rollout in https://sql.telemetry.mozilla.org/queries/59769#154505.

Comment 11

[Michael Cooper [:mythmon]]

I approved and enabled the 25% yesterday.

Comment 12

[Nan Jiang [:nanj]]

Hey :mythmon, the last rollout (25%) looks good, We're looking to skip the 50% one and bump to 100% directly, since BD would like to see this go to 100% USA asap. Can you take a look at it, please?

Comment 13

[Michael Cooper [:mythmon]]

We'll need relman to sign off on the change of schedule.

Pascal, are you ok with jumping from 25% to 100%, skipping the 50% stage and accelerating the rollout schedule by a few days?

Comment 14

[Pascal Chevrel:pascalc (PTO until April 26)]

(In reply to Michael Cooper [:mythmon] from comment #13)
> We'll need relman to sign off on the change of schedule.
> 
> Pascal, are you ok with jumping from 25% to 100%, skipping the 50% stage and
> accelerating the rollout schedule by a few days?

a+ from relman

Comment 15

[Michael Cooper [:mythmon]]

The 100% bump was shipped on October 31st.

Comment 16

[Nan Jiang [:nanj]]

Recipe updated at https://delivery-console.prod.mozaws.net/recipe/617/

:mythmon - This recipe uses the raw JEXL filters, which have a normandy.version >= '63', assuming that would include 65. So I only updated its name. Do that make sense?

Comment 17

[Michael Cooper [:mythmon]]

Nan, that makes sense and I think is the right thing. I've approved the name change in Delivery Console.

Comment 18

[Ryan VanderMeulen [:RyanVM]]

IIUC, we're handling this in-product for 66+, but please set the status to affected and nominate for tracking if I'm misunderstanding.

Comment 19

[Ciprian Muresan [:cmuresan], Ecosystem QA]

We have finished testing the Search Shortcuts feature.

QA’s recommendation: GREEN - SHIP IT

Reasoning:

• We haven’t found any issues during testing.

Testing Summary:

• Verified that the Search Shortcuts appear as expected.
• Verified that searches can be performed using them.
• Verified that additional Search Shortcuts can be added or removed.
• Verified that the Search Shortcuts are dismissible.
• Verified all telemetry interactions tied to the Search Shortcuts.

Tested Platforms:

• Windows 10 x64
• MacOS 10.14.2
• Arch Linux 4.14.3 x64

Tested Firefox versions:

• Firefox 65 RC1 (en-US, en-GB, de, it, fr, ja)
• Firefox 65 RC2 (en-US, en-GB, de, it, fr, ja)

Comment 20

[Ryan VanderMeulen [:RyanVM]]

Recipe #617 was already set for version >=63, so we should be good to go here.

Comment 21

[Michael Cooper [:mythmon]]

I disabled recipe 617 today.