1467514 - Gradual rollout of retained-dl for Fx61 via Normandy after release

Status: VERIFIED FIXED

(Core :: Web Painting, defect, P1)

Description

[Jet Villegas (inactive)]

+++ This bug was initially created as a clone of Bug #1404181 +++

The plan is to ship FF61 with RDL preffed off, then use the Normandy system to slowly ramp up the FF61 population with RDL enabled. This bug will track the disabling and deployment.

Comment 1

[Jet Villegas (inactive)]

* what preference is changing 
layout.display-list.retain

* what the new value will be
true

* what versions and channels should be included
61 release

* the percentage you want to start with, and how you want it to change over time
61 goes live with RDL turned off (0%)
2 days after go-live barring any dot release drivers, push RDL to 25% users.
Wait for a week, then push RDL to 50%
Wait for a week, then push RDL to 100%

* the users this should roll out to
All *desktop* Firefox users

Comment 2

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

I'd like to add here that any dot releases that are pushed after Normandy pref rollout is at 100% for RDL, we should turn the pref on by default in the dot release build. 

This will avoid an unnecessary delay of shipping the dot release build with RDL off, wait ~6 hrs for the recipe to be downloaded on the client which turns RDL on.

Comment 3

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Hi Jet, Matt, once the retailed-dl pref is turned on, do we need a browser restart to make the change effective? I wasn't sure about the behavior.

Comment 4

[Michael Cooper [:mythmon]]

(In reply to Ritu Kothari (:ritu) from comment #2)
> I'd like to add here that any dot releases that are pushed after Normandy
> pref rollout is at 100% for RDL, we should turn the pref on by default in
> the dot release build. 
> 
> This will avoid an unnecessary delay of shipping the dot release build with
> RDL off, wait ~6 hrs for the recipe to be downloaded on the client which
> turns RDL on.

I agree that this is a good idea, but I think the impact of not doing it would be low. I see three cases:

1. Users that upgraded from a previous version that the pref rollout applied to would keep the value in the new version (no impact).

2. Users that start a new profile on the new version would get the new value very quickly during startup. (very low impact)

3. Users that upgrade from an old profile not covered by the rollout (57, for example) would have to wait between 0 and 6 hours to get the rollout on 61.0.z. If this profile hasn't been used in the last 6 hours, the update would happen quickly, but a user that just was running 57 on that profile and then upgraded would have to wait on average 3 hours. I'd call this a low impact, but maybe enough to worry about.

Again, updating the default in a future dot release is totally fine. It also is something I don't think we should stress much about (if timing is really tight, for example).

Comment 5

[Michael Cooper [:mythmon]]

I've created a recipe on Normandy in a draft state based on the details in comment 1: https://normandy-admin.prod.mozaws.net/recipe/483/

You can also see the details on the public API, if you don't have access to the Normandy UI: https://normandy.cdn.mozilla.net/api/v1/recipe/483/

This recipe will stay in a draft state until we are ready to launch. Sometime before we launch, Jet (or someone else) will need to approve the recipe in Normandy.

Additionally, I've talked with Matt Woodrow, and did some manual testing with Normandy staging and 61.0b12, and between the two I'm confident that we don't need a browser restart for the feature to work. Clearing the NI for this information.

Comment 6

[Ryan VanderMeulen [:RyanVM]]

Bug 1467530 will be disabling RDL by default prior to Fx61 shipping. Since this bug covers the gradual roll-out side of things after we ship, I'm updating the summary to be a bit less confusing about what's being tracked here now.

Comment 7

[Maire Reavy [:mreavy]]

Hey David -- assigning this to you so it's clear you are the engineering manager tracking the rollout. Obviously you and Matt are a team on this one.

Comment 8

[Ryan VanderMeulen [:RyanVM]]

Hey Adrian, can you please test the recipe on stage now that we have final 61 builds with the pref set to the correct default-off setting (or redirect as appropriate)? Would be good to have a more formal QA signoff in a bug rather than just a preliminary test from an email thread. Thanks!

Comment 9

[Matt Woodrow (:mattwoodrow)]

Recipe is now approved within Normandy.

I'm planning on publishing the 25% recipe 2 days after the initial release, which would be the 28th. Around 3pm PST (10am NZST) would probably be the best time for me.

Comment 10

[Adrian Florinescu [:aflorinescu]]

Based on request from comment 8, I've:

1. Set up a staging recipe almost identical (changing just staging to sample 0.5, then updating to 1) to the one approved on production.
2. Verified that the default is for RDL to be false on build RC3 61.0 20180621125625 .
3. Started the rollout for RDL using staging environment.
4. Using the user_pref("layout.display-list.show-rebuild-area",true) I've confirmed that RDL gets turned on 'on the fly' once the recipe is executed.
5. Upgraded the Staging recipe from 0.5 sample to 1 sample and verified that after the change, the clients that did not enroll with 0.5 sample are enrolled with 1 sample.

All in all, I'd say we are ready to roll (out).


I've also set up a very fast to run test that anybody can run and which can be reused in case we want to validate further changes to the recipe, if case requires. 

A. Download the attached prefs.js . (it sets everything you need)
B. Use profile manager to create a new profile (do not start firefox yet with it) - copy the above prefs.js in it.
C. Start Firefox.

Expected Result:
-given that is a first run profile, you will notice that the rebuild-area for first run page (any page) is solid (entire page) and in a few seconds as the recipe is executed and RDL turned on, the rebuild-area is now made of small blue rectangles.
-hitting CTRL+J and bringing out browser console will state:
app.normandy.recipe-runner	INFO	Executing recipe "Pref Rollout: Retained Display Lists, release 61 [Bug 1467514]" (action=preference-rollout)

Comment 11

[Adrian Florinescu [:aflorinescu]]

Attachment: prefs.js

Configuration to use with test in comment 10.

Comment 12

[Ryan VanderMeulen [:RyanVM]]

(In reply to Matt Woodrow (:mattwoodrow) from comment #9)
> Recipe is now approved within Normandy.
> 
> I'm planning on publishing the 25% recipe 2 days after the initial release,
> which would be the 28th. Around 3pm PST (10am NZST) would probably be the
> best time for me.

Signing off for RelMan. Deploy when ready.

Comment 13

[Ryan VanderMeulen [:RyanVM]]

From an email thread: "I've just deployed the Normandy recipe, so this should be rolling out to 25% of 61 users right now."

Comment 14

[Michael Cooper [:mythmon]]

As of July 1st, 5pm Pacific, I see about 7.9 million enrollments in this recipe. Given that the current recipe is targeting 25% of Release 61, which is itself at 25% rollout, I think this is about the number we expect to see. From an enrollment point of view, this is looking good. Do we have any signs from Telemetry about how this rollout is performing in terms of RDL's metrics?

Comment 15

[Ryan VanderMeulen [:RyanVM]]

Not seeing any reason to block the increase to 50% rollout. None of the reported RDL bugs we've seen so far are high severity. Let's proceed.

Comment 16

[Michael Cooper [:mythmon]]

The Normandy recipe has been edited to go to 50%.

Comment 17

[Ryan VanderMeulen [:RyanVM]]

No new issues since we bumped to 50% last week. I think this can go to 100% whenever Matt says so.

Comment 18

[Matt Woodrow (:mattwoodrow)]

The Normandy recipe has been edited to go to 100%, so that should be in effect now.

Comment 19

[Ryan VanderMeulen [:RyanVM]]

Given that we're now at 100%, I think we can close this bug out. That said, we still want to land an in-tree patch on m-r to backout bug 1467530 for any subsequent dot releases I believe? Did we want to do that here or in another bug?

Comment 20

[Matt Woodrow (:mattwoodrow)]

I'll do that in bug 1477404.

Comment 21

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Hi Mythmon, this is also a recipe that is still enabled. I think we shipped 62 onwards with retained-dl pref ON by default. Do you think it makes sense to disable this recipe now?

Comment 22

[Michael Cooper [:mythmon]]

I think we should keep probably keep this active for a while longer. We've had 1 million enrollments on this recipe in the last 24 hours. I think in general we should keep recipes active for at least one version after their graduating version is released, since users update fairly slowly, historically.

Comment 23

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Mythmon and I review Normandy feature rollout recipes once a month to see which ones can be disabled. This recipe now has a very low enrollment (weekly and 4-week number) that we have disabled it. Please let me know if there are any concerns.