1512212 - Roll Out CFR addon recommendations in Firefox Desktop 64

Status: RESOLVED INVALID

(Firefox :: Messaging System, defect, P1)

Description

[Tim Spurway [:tspurway]]

Let's use the Normandy roll-out mechanism to pref on CFR in Firefox 64.  Let's start the roll-out when we have reached 100% deployment of 64 after the Dec12 release.

Let's pref the feature on for all locales, all regions along the following schedule:

Day 0 - 1% rollout
Day 3 - 50% rollout
Day 4 - 100% rollout

We will monitor on: https://sql.telemetry.mozilla.org/dashboard/activity-stream-system-addon-metrics-summary_1 

Julien - when is a good day to launch our rollout for this?

Comment 1

[Julien Cristau [:jcristau] (back April 22)]

I'll let you know when we go 100%.  If all goes well that would be Dec 13 or 14, if we need to spin 64.0.1 it'll be the following week.  Keeping the needinfo until then.

Comment 2

[Julien Cristau [:jcristau] (back April 22)]

Hey Tim, 64 is rolled out at 100% as of yesterday.  One thing I'm wondering, did we get QA coverage for CFR on 64, and if so can you point me at test results?

Comment 3

[Tim Spurway [:tspurway]]

hey Julien,  we have QA for 64 on:

https://bugzilla.mozilla.org/show_bug.cgi?id=1499110
https://bugzilla.mozilla.org/show_bug.cgi?id=1510678

is that what you were looking for?

Comment 4

[Ryan VanderMeulen [:RyanVM]]

List compiled by Kate of all CFR changes between 63 and 64:
https://docs.google.com/document/d/1pZgRbr4j5W6cgT4eTE1sbln7IBIDJxktmrfF3wyNtLc/edit

Comment 5

[Ryan VanderMeulen [:RyanVM]]

I had a chance to look over the list in comment 4 and the main concern I have at this point is that the opt-out in about:preferences doesn't seem to have gone through much QA at this point. Given that it's already Friday evening and the urgency isn't high, let's defer on rolling out until next week. In the mean time, it would be great if QA could go through the list and give things another pass for any other gaps in coverage they see.

Comment 6

[Julien Cristau [:jcristau] (back April 22)]

clearing my needinfo pending QA results.

Comment 7

[Marius Coman [:mcoman], Ecosystem QA]

Roll Out CFR addon recommendations in Firefox Desktop 64
Targeted: Firefox Release 64

We have finished testing the Roll Out CFR addon recommendations in Firefox Desktop 64.

QA’s recommendation: GREEN - SHIP IT

Reasoning: We haven't found any issues during testing.

Testing Summary:
- Full Functional test suite: https://goo.gl/jM45bW;

Tested Platforms:
- Windows 10 x64;
- Ubuntu 16.04 x64;
- Mac 10.13.3;

Tested Firefox versions:
- Firefox Release 64.0

Comment 8

[Nan Jiang [:nanj]]

Recipe created on Normandy: https://delivery-console.prod.mozaws.net/recipe/660/

:mythmon r?, please?

Comment 9

[Michael Cooper [:mythmon]]

The recipe looks good to me, and the preference is compatible with rollout.

Julien, Ryan: this still needs relman approval.

Comment 10

[Nan Jiang [:nanj]]

Hi :jscritau, could you confirm?

Comment 11

[Julien Cristau [:jcristau] (back April 22)]

1% rollout is fine with me.

Is the intent still to go 50% on day 3 and 100% on day 4 (per comment 0)?  That runs right into the weekend before christmas, so might need adjustment?

Comment 12

[Tim Spurway [:tspurway]]

This is a pretty good point, Julien.  We would really like to get to 100% before Jan 1 in order to get a 'full' dataset for January and 2019.

We have really good (fast) telemetry and monitoring on this, so why don't we do the following schedule:

1% today (dec19)
50% tomorrow 1pm EST (dec20)
100% dec27 

this will give us lots of time to measure and roll back if required and will still get us rolled out so that 2019 will have a full cohort of data.

Comment 13

[Michael Cooper [:mythmon]]

Nan and I launched this at 1% about an hour ago.

Comment 14

[Julien Cristau [:jcristau] (back April 22)]

Assuming we double check that things look as expected today from the 1% rollout, I'm OK with the plan in comment 12.

Comment 15

[Nan Jiang [:nanj]]

Hi :julien and :RyanVM, we've found an issue (bug 1515742) in 64 that prevents CFR from making any recommendations for the user. To fix that, it would need a code change.

While we're looking at options for the fix, we'd like to rollback this rollout now. :mythmon already made a recipe to undo this rollout, could anyone of you review this, please?

https://delivery-console.prod.mozaws.net/recipe/661/

Comment 16

[Julien Cristau [:jcristau] (back April 22)]

Done, thanks.

Comment 17

[Nan Jiang [:nanj]]

Close it as we've decided not to rollout CFR in 64 due to bug 1515742.