1541488 - Gradual rollout of WebRender in release 67

Status: RESOLVED FIXED

(Core :: Graphics: WebRender, task, P1)

Description

[Jeff Muizelaar [:jrmuizel]]

Aiming for full rollout in 1 week with this plan in mind:
Feature is off by default at mainline release go-live/launch.
Turn on feature at 25% update Monday post go-live.
Push to 50% 3 days later, Thursday
Push to 100% a week later IF there are no blocking issues.

Comment 1

[Jeff Muizelaar [:jrmuizel]]

Pascal, does this seem like a reasonable plan?

Comment 2

[Jeff Muizelaar [:jrmuizel]]

The pref to change from false to true to do the rollout is gfx.webrender.all.qualified.

Comment 3

[Pascal Chevrel:pascalc]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #1)

Pascal, does this seem like a reasonable plan?

sgtm, thanks.

Comment 4

[Michael Cooper [:mythmon]]

I've prepared a Normandy recipe for this rollout. It is not automatically scheduled, so we will need to enable it during release week, and manually update it to increase the population size. I chose a manual schedule because of the impact of the feature: I think it's likely we'll want to tweak the schedule by hand anyways. If not, I can implement an automatic schedule as well.

I have not requested review for the recipe yet.

Normandy API: https://normandy.cdn.mozilla.net/api/v3/recipe/755/history/
Delivery Console: https://delivery-console.prod.mozaws.net/recipe/755/

Comment 5

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Heads-up that we want to run a concurrent experiment in 67 to validate performance and stability in bug 1526094, with WR-enabled and WR-disabled branches for qualified users. That experiment will target the same pref. The experiment and the rollout might need to be aware of each other.

Comment 6

[Michael Cooper [:mythmon]]

In light of comment 5, I propose that we modify the schedule of this rollout slightly. The experiment would occupy the range from 95% to 100%, making a total of a 5% sample. The rollout would start as 0%-25%, then 3 days later it would be 0%-50%, and finally a week later it would be 0%-95%. When we end the experiment in mid june, we can then expand the rollout to 100%.

Comment 7

[Michael Cooper [:mythmon]]

The rollout is live at 25%. I'll update the recipe on Thursday to bump to 50%.

Comment 8

[Darkspirit]

List of known Windows issues.

Comment 9

[Michael Cooper [:mythmon]]

I have updated the recipe from 25% to 50%, and requested approval. Ritu, can you review? https://delivery-console.prod.mozaws.net/recipe/755/

Comment 10

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

I confirmed with Mythmon that we are seeing the expected level of activity with the 25% rollout so far, before going to 50%. He confirmed ~13.8M enrollments, and no significant amount of other events.

With that I have approved the recipe, lgtm.

Comment 11

[Jessie [:jbonisteel] pls NI]

I would like to hold off on bumping up the remainder of the rollout until Monday the 10th, just to make sure nothing pops up over the weekend.

Comment 12

[Michael Cooper [:mythmon]]

The rollout continues to look healthy. We've seen ~52.5M enrollments, with a negligible error rate. I've queued a bump to a 95% sample rate for this rollout (as discussed in comment 6).

Ritu, can you review the change? https://delivery-console.prod.mozaws.net/recipe/755/

For reference, the experiment recipe that is taking up the 95%-100% range is the below. It is scheduled to end tomorrow.

• Delivery Console: https://delivery-console.prod.mozaws.net/recipe/783/
• Experimenter: https://experimenter.services.mozilla.com/experiments/webrender-performance-67/

Comment 13

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Done. Approved. Thanks (as always) for sharing the recipe enrollment numbers.

Comment 14

[Kartikaya Gupta (email:[e-mail])]

67 happened.