1507237 - [Shield] Pref Rollout: Roll out Activity Stream Search Shortcuts to 100% DE region in Firefox 63&64&65

Status: RESOLVED FIXED

(Shield :: Rollouts, task)

Description

[Nan Jiang [:nanj]]

Overview
========

As a followup rollout of bug 1495577. We'd like to roll out the feature in Firefox to 100% of users in DE (Germany) region in the 63 release channel.

Like bug 1495577, the rollout will set the pref:

browser.newtabpage.activity-stream.improvesearch.noDefaultSearchTile=true

for all locales in the DE region.

Schedule
========

The rollout is scheduled as follows:

Day  1:  10%
Day  4:  25%
Day  7:  50%
Day 10: 100%

Rollout Metrics
===============

The rollout related metrics will be tracked at https://sql.telemetry.mozilla.org/dashboard/activity-stream-search-shortcuts

Comment 1

[Nan Jiang [:nanj]]

Normandy recipe:

https://delivery-console.prod.mozaws.net/recipe/636/

Comment 2

[Nan Jiang [:nanj]]

hey :pascalc, I've requested for the tracking for 63, could you take a look at this please?

Comment 3

[Pascal Chevrel (relman team) -> :pascalc]

Tracking for 63 as this is the target.

Comment 4

[Marissa (Reese) Wood]

I approve this change on behalf of Nick.  Thanks!

Comment 5

[Pascal Chevrel (relman team) -> :pascalc]

A couple notes on the recipe:

Comment #0 says that the pref set to true is:
browser.newtabpage.activity-stream.improvesearch.noDefaultSearchTile

But https://delivery-console.prod.mozaws.net/recipe/636/ modifies this pref:
browser.newtabpage.activity-stream.improvesearch.topSiteSearchShortcuts

From the purpose of this bug, I think comment #0 was wrong but I'd like a confirmation that the pref to update is the right one.

Also, there is a typo in the slug:
pref-rollout-activity-stream-search-shortucts-1507237
shortucts -> shortcuts

Finally, I notice that there is no indication in the recipe about the targeted Firefox version.

Comment 6

[Nan Jiang [:nanj]]

(In reply to Pascal Chevrel:pascalc from comment #5)
> A couple notes on the recipe:
> 
> Comment #0 says that the pref set to true is:
> browser.newtabpage.activity-stream.improvesearch.noDefaultSearchTile
> 
> But https://delivery-console.prod.mozaws.net/recipe/636/ modifies this pref:
> browser.newtabpage.activity-stream.improvesearch.topSiteSearchShortcuts
> 
> From the purpose of this bug, I think comment #0 was wrong but I'd like a
> confirmation that the pref to update is the right one.

Yes, browser.newtabpage.activity-stream.improvesearch.topSiteSearchShortcuts is correct one.

> 
> Also, there is a typo in the slug:
> pref-rollout-activity-stream-search-shortucts-1507237
> shortucts -> shortcuts

Fixed.

> Finally, I notice that there is no indication in the recipe about the
> targeted Firefox version.

Hmm, I've set the version to 63 in the recipe, somehow it didn't pick up. Perhaps I should do the raw JEXL filters.

Comment 7

[Nick Nguyen [:osunick]]

Approved

Comment 8

[Nan Jiang [:nanj]]

:mythmon, could you r? this recipe on Normandy, please?

Comment 9

[Michael Cooper [:mythmon]]

The recipe looks fine by me. I'd like to see explicit sign off from relman about this. Pascal, are your concerns from comment 5 addressed?

Comment 10

[Pascal Chevrel (relman team) -> :pascalc]

(In reply to Michael Cooper [:mythmon] from comment #9)
> The recipe looks fine by me. I'd like to see explicit sign off from relman
> about this. Pascal, are your concerns from comment 5 addressed?

Yes this looks fine, a+

Comment 11

[Michael Cooper [:mythmon]]

I have approved and enabled the recipe in Delivery Console. It is now shipping to users.

Comment 12

[Nan Jiang [:nanj]]

Hi :pascal and :mythmon,

The first phase (10%) of rollout in DE worked pretty well, see metrics at https://sql.telemetry.mozilla.org/dashboard/activity-stream-search-shortcuts.

We'd like to bump this rollout directly to 100% now, as this feature has already been proven to work in bug 1495577. What do you think?

PS: I've updated the recipe and requested for the approval on Normandy.

Comment 13

[Pascal Chevrel (relman team) -> :pascalc]

a+ for relman

Comment 14

[Michael Cooper [:mythmon]]

Approved and shipped.

Comment 15

[Nan Jiang [:nanj]]

We'd like to include users from 64 release in Germany into this rollout.

:jcristau, could you take a look at this, please?

Comment 16

[Julien Cristau [:jcristau]]

updated recipe (revision 8) approved in delivery console, this should now be live for 64 as well as 63.

Comment 17

[Julien Cristau [:jcristau]]

I guess we can call this fixed?

Comment 18

[Ryan VanderMeulen [:RyanVM]]

IIUC, we're still needing to use Normandy for 65 but intend to do something in-product for 66 and beyond.

Comment 19

[Nan Jiang [:nanj]]

Correct, we still want to use Normandy for 65, and the recipe has been updated and pending to review at https://delivery-console.prod.mozaws.net/recipe/636/

The game plan for 66 is tracked by bug 1520853.

:RyanVM - do you want us to ask for QA verification for 65?

Comment 20

[Ryan VanderMeulen [:RyanVM]]

Yes please, thanks.

Comment 21

[Ciprian Muresan [:cmuresan], Formerly of Ecosystem QA - please needinfo @mcoman for things regarding the team]

We have finished testing the Search Shortcuts feature.

QA’s recommendation: GREEN - SHIP IT

Reasoning:

• We haven’t found any issues during testing.

Testing Summary:

• Verified that the Search Shortcuts appear as expected.
• Verified that searches can be performed using them.
• Verified that additional Search Shortcuts can be added or removed.
• Verified that the Search Shortcuts are dismissible.
• Verified all telemetry interactions tied to the Search Shortcuts.

Tested Platforms:

• Windows 10 x64
• MacOS 10.14.2
• Arch Linux 4.14.3 x64

Tested Firefox versions:

• Firefox 65 RC1 (en-US, en-GB, de, it, fr, ja)
• Firefox 65 RC2 (en-US, en-GB, de, it, fr, ja)

Comment 22

[Nan Jiang [:nanj]]

:cmuresan - Thanks for verifying this!

:RyanVM - could you review the following recipe updates?

For "DE" - https://delivery-console.prod.mozaws.net/recipe/636/
For "FR/IT/GB/JP" (bug 1511423) - https://delivery-console.prod.mozaws.net/recipe/637/

Note: we don't need to update the recipe for "US", since that rollout already has version>=63 in its recipe.

Comment 23

[Ryan VanderMeulen [:RyanVM]]

I've signed off on both. It might have been a bit clearer if we could have used a >=63 rule for those like we did for en-US, but this works too :)

Comment 24

[Nan Jiang [:nanj]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #23)

I've signed off on both. It might have been a bit clearer if we could have used a >=63 rule for those like we did for en-US, but this works too :)

Thanks!

Yep, v >= 63 is indeed clearer. When we built the other two on Normandy, we also wanted to try its new GUI-based filter builders :)

Comment 25

[Liz Henry (:lizzard)]

Is this still going to be active for 66 release?

Comment 26

[Nan Jiang [:nanj]]

(In reply to Liz Henry (:lizzard) (use needinfo) from comment #25)

Is this still going to be active for 66 release?

Hi Liz, this feature has been turned on by default in 66, so we don't need the rollout(s) any more for 66. Will work with mythmon to shut it down.

Comment 27

[Michael Cooper [:mythmon]]

This recipe is only targeted at 65 and below, so it won't affect users of 66 and above. However, we usually keep rollout recipes running for quite a while after the release, to pick up users that don't update right away.

Comment 28

[Michael Cooper [:mythmon]]

I've disabled this recipe today.