Closed Bug 1495577 Opened 1 year ago Closed 5 months ago
[Shield] Pref Rollout: Roll out Activity Stream Search Shortcuts to 100% US region in Firefox 63&64&65
This pref rollout is the response to a successful 1% experiment in the US region of Search Shortcuts. The idea is to roll out the feature in Firefox to 100% of users in US region after 63 is released. We would like to use Normandy rollout feature to roll out to different regions at different rates and to be able to dial back the percentages in any region if undesirable outcomes are reached. Like the blocked Shield Study (#1483396), the rollout will set the pref: browser.newtabpage.activity-stream.improvesearch.noDefaultSearchTile=true for all locales in the US region
A note for anyone looking at this later: Activity Stream preferences are not normally compatible with Preference Rollout. This particular preference has been given special support to be compatible (see bug 1495517). For future Activity Stream rollouts, the individual preferences would also need a similar migration in order to be compatible. Tim: Is the goal to do a rollout for en-US 100%, and separately do further experiments for other locales? Or is the goal to do rollouts for all the locales you have in mind? I ask because rollbacks (if something goes wrong) is a blunt instrument. We can't rollout to 100% of en-US and 10% of another locale, and then roll back the other locale without also rolling back en-US. To work around this, we can deploy this as a group of many rollout recipes, each of which can be controlled separately. This is more complex, but doable. Additionally, we can't really go up to some percentage, and then down to a different percentage. Rollbacks are generally all-or-nothing. If you expect that you'd want to go up to 10% and then down to 5%, for example, we'll also have to take special precautions. If this rollout is only for the en-US population, than the previous two paragraphs are moot, and this is a simple rollout.
Let's follow this roll-out schedule (where Day 1 is shortly after 63 release on Oct 23, 2018): Day 1: 10% Day 4: 25% Day 7: 50% Day 10: 100%
(In reply to Tim Spurway [:tspurway] from comment #2) > Let's follow this roll-out schedule (where Day 1 is shortly after 63 release > on Oct 23, 2018): > > Day 1: 10% > Day 4: 25% > Day 7: 50% > Day 10: 100% This has relman approval. The last patch for uplift was approved today and will be part of 63 beta 13.
What metrics will we using to determine if the rollout is going or not going as planned? Do we want to run analysis between each rate increase, or are we going to increase the rate unless a problem comes up (such as crashes)?
We will be keeping a close eye on AS dashboards to keep track of any anomalous engagement with newtab features (especially the search), as well as overall search and address bar engagement: https://sql.telemetry.mozilla.org/dashboard/activity-stream-system-addon-metrics-summary_1 https://sql.telemetry.mozilla.org/dashboard/activity-stream-search
r? :pascalc we are ready to roll-out this feature to: 10% population of US region en-US locale release 63 for users who are NOT first-run we'd like you to give us the relman go-ahead when you feel the release has stabilized and is ready for the feature
(In reply to Tim Spurway [:tspurway] from comment #7) > r? :pascalc > > we are ready to roll-out this feature to: > > 10% population of > US region > en-US locale > release 63 > for users who are NOT first-run > > we'd like you to give us the relman go-ahead when you feel the release has > stabilized and is ready for the feature Feedback and stability from our initial rollout are positive so we are now rolling Fx63 out to 100% of our release users, you can proceed with the shield study rollout.
hey :mythmon, we've updated the recipe to roll it further up to 25% of the target audience. Could you review it, plz? The first phase rollout looks good, and we are tracking the telemetry for this rollout in https://sql.telemetry.mozilla.org/queries/59769#154505.
I approved and enabled the 25% yesterday.
Hey :mythmon, the last rollout (25%) looks good, We're looking to skip the 50% one and bump to 100% directly, since BD would like to see this go to 100% USA asap. Can you take a look at it, please?
We'll need relman to sign off on the change of schedule. Pascal, are you ok with jumping from 25% to 100%, skipping the 50% stage and accelerating the rollout schedule by a few days?
(In reply to Michael Cooper [:mythmon] from comment #13) > We'll need relman to sign off on the change of schedule. > > Pascal, are you ok with jumping from 25% to 100%, skipping the 50% stage and > accelerating the rollout schedule by a few days? a+ from relman
The 100% bump was shipped on October 31st.
Summary: [Shield] Pref Rollout: Roll out Activity Stream Search Shortcuts to 100% US region in Firefox 63 → [Shield] Pref Rollout: Roll out Activity Stream Search Shortcuts to 100% US region in Firefox 63&64
Type: defect → task
Status: NEW → RESOLVED
Closed: 5 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.