Closed Bug 1416975 Opened 8 years ago Closed 6 years ago

[Shield] Pref Flip Study: Pref Flip Retained Display List

Categories

(Shield :: Shield Study, enhancement)

Product:
Shield
Component:
Shield Study
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: Harald, Unassigned)

References

(
URL
)

Details

What is your basic hypothesis? By enabling retained display lists, the browser will handle updates on sites with complex layouts better, making top sites like Facebook, LinkedIn or Twitter more responsive and therefore improving performance at large What high level metric are you attempting to influence? We hope to improve performance (input latency telemetry), and as a result product engagement like page loads, usage hours, etc. Details Section for Bugs and Rel-Drivers Email Basic description of experiment: We want to test enabling the retained display list feature to get feedback on its effectiveness while still minimizing risk and controlling the number of users we ship it to. What is the preference we will be changing? layout.display-list.retain What are the branches of the study and what values should each branch be set to? We just want a single branch with the pref set to true, and the rest as a control with it false. What percentage of users do you want in each branch? I think 50/50 makes the most sense, but I’m happy to take suggestions. What Channels and locales do you intend to ship to? Nightly 58 is preffed on since Nov 14. This experiment till ship on Beta 58, and pending the outcome, shipping enabled to Release 58. All locales. What is your intended go live date and how long will the study run? 2 weeks, probably starting 1 week after the feature got enabled on Nightly. Are there specific criteria for participants? Beta 58 population. What is the main effect you are looking for and what data will you use to make these decisions? The treatment group will see a reduction in CONTENT_PAINT_TIME, which is the amount of time spent painting. We also expect the ‘dl’ portion of CONTENT_LARGE_PAINT_PHASE_WEIGHT to reduce (and the others increase, since they are percentages of total time). Who is the owner of the data analysis for this study? Experiments team with assistance from Product (:digitarald) and layout team (:mwoodrow)? Will this experiment require uplift? No. QA Status of your code: Unsure. We haven’t had any QA done yet, but working with the QA team to get that started this week. Do you plan on surveying users at the end of the study? No. Link to any relevant google docs / Drive files that describe the project. Links to prior art if it exists: - https://groups.google.com/forum/#!topic/mozilla.dev.platform/jSuLiuXqs8w - https://docs.google.com/document/d/1RxUl0z5ViwyAbcQgYvUoorjIQ38M5ZGwwjljBujgE7o
Flags: needinfo?(mgrimes)
Got a +1 from :Matt_G in person. Will prep the Intent to Experiment
Flags: needinfo?(mgrimes)
Matt, can you confirm the QA status?
Flags: needinfo?(matt.woodrow)
Gerry, following up on your comment on the intent email; could you give your preliminary approval here, pending QA sign off; so we are good to go when we have the green light from QA?
Flags: needinfo?(gchang)
I can! QA is complete, and they consider the feature to be stable. Happy to forward to the report email to anyone interested. However, I'm a bit worried about bug 1418722, which is a relatively high volume crash that has shown up. I've got a fix that I'll push as soon as the tree opens, but I think we should try uplift to beta before starting the experiment.
Flags: needinfo?(matt.woodrow)
:Harald, sure. Once we got the QA signoff. We are good to go. Like Matt mentioned, we should monitor bug 1418722.
Flags: needinfo?(gchang)
I've requested beta uplift for bug 1418722, and a few other small rendering regressions that have been fixed on m-c. I think once those are uplifted we will be good to go!
This study went live yesterday. We plan on ending the study 12/18
Is this definitely live? On the dashboard I'm using (https://www.jeffersonscher.com/sumo/shield.php) it shows as not yet live. On the other hand I'm not sure how that data is being maintained.
This study is no longer live. It ended 12/18 as indicated above. @jgaunt is the analysis owner and will coordinate with Harald.
@jgaunt now that the study has ended can you recap the results here and close the bug?
Flags: needinfo?(jgaunt)
Jupyter notebook with results gisted here on Github: https://gist.github.com/jtg567/8819794fdb8239f59d97cc2dd638c9cd
Flags: needinfo?(jgaunt)

closing all experiment bugs - for experiments that have ended.

Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.