Closed Bug 1467514 Opened 2 years ago Closed 2 years ago

Gradual rollout of retained-dl for Fx61 via Normandy after release

Categories

(Core :: Web Painting, defect, P1, blocker)

61 Branch
defect

Tracking

()

VERIFIED FIXED
Tracking Status
firefox61 blocking verified

People

(Reporter: bugs, Assigned: davidb)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

+++ This bug was initially created as a clone of Bug #1404181 +++

The plan is to ship FF61 with RDL preffed off, then use the Normandy system to slowly ramp up the FF61 population with RDL enabled. This bug will track the disabling and deployment.
Depends on: 1467530
* what preference is changing 
layout.display-list.retain

* what the new value will be
true

* what versions and channels should be included
61 release

* the percentage you want to start with, and how you want it to change over time
61 goes live with RDL turned off (0%)
2 days after go-live barring any dot release drivers, push RDL to 25% users.
Wait for a week, then push RDL to 50%
Wait for a week, then push RDL to 100%

* the users this should roll out to
All *desktop* Firefox users
I'd like to add here that any dot releases that are pushed after Normandy pref rollout is at 100% for RDL, we should turn the pref on by default in the dot release build. 

This will avoid an unnecessary delay of shipping the dot release build with RDL off, wait ~6 hrs for the recipe to be downloaded on the client which turns RDL on.
Hi Jet, Matt, once the retailed-dl pref is turned on, do we need a browser restart to make the change effective? I wasn't sure about the behavior.
Flags: needinfo?(matt.woodrow)
Flags: needinfo?(bugs)
(In reply to Ritu Kothari (:ritu) from comment #2)
> I'd like to add here that any dot releases that are pushed after Normandy
> pref rollout is at 100% for RDL, we should turn the pref on by default in
> the dot release build. 
> 
> This will avoid an unnecessary delay of shipping the dot release build with
> RDL off, wait ~6 hrs for the recipe to be downloaded on the client which
> turns RDL on.

I agree that this is a good idea, but I think the impact of not doing it would be low. I see three cases:

1. Users that upgraded from a previous version that the pref rollout applied to would keep the value in the new version (no impact).

2. Users that start a new profile on the new version would get the new value very quickly during startup. (very low impact)

3. Users that upgrade from an old profile not covered by the rollout (57, for example) would have to wait between 0 and 6 hours to get the rollout on 61.0.z. If this profile hasn't been used in the last 6 hours, the update would happen quickly, but a user that just was running 57 on that profile and then upgraded would have to wait on average 3 hours. I'd call this a low impact, but maybe enough to worry about.

Again, updating the default in a future dot release is totally fine. It also is something I don't think we should stress much about (if timing is really tight, for example).
I've created a recipe on Normandy in a draft state based on the details in comment 1: https://normandy-admin.prod.mozaws.net/recipe/483/

You can also see the details on the public API, if you don't have access to the Normandy UI: https://normandy.cdn.mozilla.net/api/v1/recipe/483/

This recipe will stay in a draft state until we are ready to launch. Sometime before we launch, Jet (or someone else) will need to approve the recipe in Normandy.

Additionally, I've talked with Matt Woodrow, and did some manual testing with Normandy staging and 61.0b12, and between the two I'm confident that we don't need a browser restart for the feature to work. Clearing the NI for this information.
Flags: needinfo?(matt.woodrow)
Flags: needinfo?(bugs)
Bug 1467530 will be disabling RDL by default prior to Fx61 shipping. Since this bug covers the gradual roll-out side of things after we ship, I'm updating the summary to be a bit less confusing about what's being tracked here now.
Summary: Pref off retained-dl for slow rollout in FF61 → Gradual rollout of retained-dl for Fx61 via Normandy after release
Priority: -- → P1
Hey David -- assigning this to you so it's clear you are the engineering manager tracking the rollout. Obviously you and Matt are a team on this one.
Assignee: nobody → dbolter
Hey Adrian, can you please test the recipe on stage now that we have final 61 builds with the pref set to the correct default-off setting (or redirect as appropriate)? Would be good to have a more formal QA signoff in a bug rather than just a preliminary test from an email thread. Thanks!
Flags: needinfo?(adrian.florinescu)
Depends on: 1471046
Recipe is now approved within Normandy.

I'm planning on publishing the 25% recipe 2 days after the initial release, which would be the 28th. Around 3pm PST (10am NZST) would probably be the best time for me.
Based on request from comment 8, I've:

1. Set up a staging recipe almost identical (changing just staging to sample 0.5, then updating to 1) to the one approved on production.
2. Verified that the default is for RDL to be false on build RC3 61.0 20180621125625 .
3. Started the rollout for RDL using staging environment.
4. Using the user_pref("layout.display-list.show-rebuild-area",true) I've confirmed that RDL gets turned on 'on the fly' once the recipe is executed.
5. Upgraded the Staging recipe from 0.5 sample to 1 sample and verified that after the change, the clients that did not enroll with 0.5 sample are enrolled with 1 sample.

All in all, I'd say we are ready to roll (out).


I've also set up a very fast to run test that anybody can run and which can be reused in case we want to validate further changes to the recipe, if case requires. 

A. Download the attached prefs.js . (it sets everything you need)
B. Use profile manager to create a new profile (do not start firefox yet with it) - copy the above prefs.js in it.
C. Start Firefox.

Expected Result:
-given that is a first run profile, you will notice that the rebuild-area for first run page (any page) is solid (entire page) and in a few seconds as the recipe is executed and RDL turned on, the rebuild-area is now made of small blue rectangles.
-hitting CTRL+J and bringing out browser console will state:
app.normandy.recipe-runner	INFO	Executing recipe "Pref Rollout: Retained Display Lists, release 61 [Bug 1467514]" (action=preference-rollout)
Flags: needinfo?(adrian.florinescu)
Attached file prefs.js
Configuration to use with test in comment 10.
(In reply to Matt Woodrow (:mattwoodrow) from comment #9)
> Recipe is now approved within Normandy.
> 
> I'm planning on publishing the 25% recipe 2 days after the initial release,
> which would be the 28th. Around 3pm PST (10am NZST) would probably be the
> best time for me.

Signing off for RelMan. Deploy when ready.
Flags: needinfo?(matt.woodrow)
From an email thread: "I've just deployed the Normandy recipe, so this should be rolling out to 25% of 61 users right now."
Flags: needinfo?(matt.woodrow)
As of July 1st, 5pm Pacific, I see about 7.9 million enrollments in this recipe. Given that the current recipe is targeting 25% of Release 61, which is itself at 25% rollout, I think this is about the number we expect to see. From an enrollment point of view, this is looking good. Do we have any signs from Telemetry about how this rollout is performing in terms of RDL's metrics?
Not seeing any reason to block the increase to 50% rollout. None of the reported RDL bugs we've seen so far are high severity. Let's proceed.
The Normandy recipe has been edited to go to 50%.
No new issues since we bumped to 50% last week. I think this can go to 100% whenever Matt says so.
The Normandy recipe has been edited to go to 100%, so that should be in effect now.
Given that we're now at 100%, I think we can close this bug out. That said, we still want to land an in-tree patch on m-r to backout bug 1467530 for any subsequent dot releases I believe? Did we want to do that here or in another bug?
Flags: needinfo?(matt.woodrow)
I'll do that in bug 1477404.
Status: NEW → RESOLVED
Closed: 2 years ago
Flags: needinfo?(matt.woodrow)
Resolution: --- → FIXED
Status: RESOLVED → VERIFIED
Hi Mythmon, this is also a recipe that is still enabled. I think we shipped 62 onwards with retained-dl pref ON by default. Do you think it makes sense to disable this recipe now?
Flags: needinfo?(mcooper)
I think we should keep probably keep this active for a while longer. We've had 1 million enrollments on this recipe in the last 24 hours. I think in general we should keep recipes active for at least one version after their graduating version is released, since users update fairly slowly, historically.
Flags: needinfo?(mcooper)

Mythmon and I review Normandy feature rollout recipes once a month to see which ones can be disabled. This recipe now has a very low enrollment (weekly and 4-week number) that we have disabled it. Please let me know if there are any concerns.

You need to log in before you can comment on or make changes to this bug.