Closed Bug 1495577 Opened 1 year ago Closed 5 months ago

[Shield] Pref Rollout: Roll out Activity Stream Search Shortcuts to 100% US region in Firefox 63&64&65

Categories

(Shield :: Rollouts, task)

task
Not set

Tracking

(firefox63+ fixed, firefox64+ fixed, firefox65+ fixed, firefox66 unaffected)

RESOLVED FIXED
Tracking Status
firefox63 + fixed
firefox64 + fixed
firefox65 + fixed
firefox66 --- unaffected

People

(Reporter: tspurway, Unassigned)

References

()

Details

This pref rollout is the response to a successful 1% experiment in the US region of Search Shortcuts.

The idea is to roll out the feature in Firefox to 100% of users in US region after 63 is released.  We would like to use Normandy rollout feature to roll out to different regions at different rates and to be able to dial back the percentages in any region if undesirable outcomes are reached.

Like the blocked Shield Study (#1483396), the rollout will set the pref:

browser.newtabpage.activity-stream.improvesearch.noDefaultSearchTile=true

for all locales in the US region
Flags: needinfo?(mgrimes)
A note for anyone looking at this later: Activity Stream preferences are not normally compatible with Preference Rollout. This particular preference has been given special support to be compatible (see bug 1495517). For future Activity Stream rollouts, the individual preferences would also need a similar migration in order to be compatible.

Tim: Is the goal to do a rollout for en-US 100%, and separately do further experiments for other locales? Or is the goal to do rollouts for all the locales you have in mind?

I ask because rollbacks (if something goes wrong) is a blunt instrument. We can't rollout to 100% of en-US and 10% of another locale, and then roll back the other locale without also rolling back en-US. To work around this, we can deploy this as a group of many rollout recipes, each of which can be controlled separately. This is more complex, but doable.

Additionally, we can't really go up to some percentage, and then down to a different percentage. Rollbacks are generally all-or-nothing. If you expect that you'd want to go up to 10% and then down to 5%, for example, we'll also have to take special precautions.

If this rollout is only for the en-US population, than the previous two paragraphs are moot, and this is a simple rollout.
Depends on: 1495517
Let's follow this roll-out schedule (where Day 1 is shortly after 63 release on Oct 23, 2018):

Day  1:  10%
Day  4:  25%
Day  7:  50%
Day 10: 100%
Approved.
(In reply to Tim Spurway [:tspurway] from comment #2)
> Let's follow this roll-out schedule (where Day 1 is shortly after 63 release
> on Oct 23, 2018):
> 
> Day  1:  10%
> Day  4:  25%
> Day  7:  50%
> Day 10: 100%

This has relman approval. The last patch for uplift was approved today and will be part of 63 beta 13.
What metrics will we using to determine if the rollout is going or not going as planned? Do we want to run analysis between each rate increase, or are we going to increase the rate unless a problem comes up (such as crashes)?
Depends on: 1497908
We will be keeping a close eye on AS dashboards to keep track of any anomalous engagement with newtab features (especially the search), as well as overall search and address bar engagement:

https://sql.telemetry.mozilla.org/dashboard/activity-stream-system-addon-metrics-summary_1
https://sql.telemetry.mozilla.org/dashboard/activity-stream-search
r? :pascalc

we are ready to roll-out this feature to:

10% population of
US region
en-US locale
release 63
for users who are NOT first-run

we'd like you to give us the relman go-ahead when you feel the release has stabilized and is ready for the feature
Flags: needinfo?(pascalc)
(In reply to Tim Spurway [:tspurway] from comment #7)
> r? :pascalc
> 
> we are ready to roll-out this feature to:
> 
> 10% population of
> US region
> en-US locale
> release 63
> for users who are NOT first-run
> 
> we'd like you to give us the relman go-ahead when you feel the release has
> stabilized and is ready for the feature

Feedback and stability from our initial rollout are positive so we are now rolling Fx63 out to 100% of our release users, you can proceed with the shield study rollout.
Flags: needinfo?(pascalc)
hey :mythmon, we've updated the recipe to roll it further up to 25% of the target audience. Could you review it, plz?

The first phase rollout looks good, and we are tracking the telemetry for this rollout in https://sql.telemetry.mozilla.org/queries/59769#154505.
Flags: needinfo?(mcooper)
I approved and enabled the 25% yesterday.
Flags: needinfo?(mcooper)
Hey :mythmon, the last rollout (25%) looks good, We're looking to skip the 50% one and bump to 100% directly, since BD would like to see this go to 100% USA asap. Can you take a look at it, please?
Flags: needinfo?(mcooper)
We'll need relman to sign off on the change of schedule.

Pascal, are you ok with jumping from 25% to 100%, skipping the 50% stage and accelerating the rollout schedule by a few days?
Flags: needinfo?(pascalc)
(In reply to Michael Cooper [:mythmon] from comment #13)
> We'll need relman to sign off on the change of schedule.
> 
> Pascal, are you ok with jumping from 25% to 100%, skipping the 50% stage and
> accelerating the rollout schedule by a few days?

a+ from relman
Flags: needinfo?(pascalc)
The 100% bump was shipped on October 31st.
Flags: needinfo?(mcooper)
Flags: needinfo?(mgrimes)
Summary: [Shield] Pref Rollout: Roll out Activity Stream Search Shortcuts to 100% US region in Firefox 63 → [Shield] Pref Rollout: Roll out Activity Stream Search Shortcuts to 100% US region in Firefox 63&64
Summary: [Shield] Pref Rollout: Roll out Activity Stream Search Shortcuts to 100% US region in Firefox 63&64 → [Shield] Pref Rollout: Roll out Activity Stream Search Shortcuts to 100% US region in Firefox 63&64&65

Recipe updated at https://delivery-console.prod.mozaws.net/recipe/617/

:mythmon - This recipe uses the raw JEXL filters, which have a normandy.version >= '63', assuming that would include 65. So I only updated its name. Do that make sense?

Flags: needinfo?(mcooper)

Nan, that makes sense and I think is the right thing. I've approved the name change in Delivery Console.

Flags: needinfo?(mcooper)

IIUC, we're handling this in-product for 66+, but please set the status to affected and nominate for tracking if I'm misunderstanding.

We have finished testing the Search Shortcuts feature.

QA’s recommendation: GREEN - SHIP IT

Reasoning:

  • We haven’t found any issues during testing.

Testing Summary:

  • Verified that the Search Shortcuts appear as expected.
  • Verified that searches can be performed using them.
  • Verified that additional Search Shortcuts can be added or removed.
  • Verified that the Search Shortcuts are dismissible.
  • Verified all telemetry interactions tied to the Search Shortcuts.

Tested Platforms:

  • Windows 10 x64
  • MacOS 10.14.2
  • Arch Linux 4.14.3 x64

Tested Firefox versions:

  • Firefox 65 RC1 (en-US, en-GB, de, it, fr, ja)
  • Firefox 65 RC2 (en-US, en-GB, de, it, fr, ja)

Recipe #617 was already set for version >=63, so we should be good to go here.

I disabled recipe 617 today.

Status: NEW → RESOLVED
Closed: 5 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.