[Experiment]: Add-On Experiment: User sensitivity to page load regressions
Categories
(Shield :: Shield Study, task, P3)
Tracking
(Not tracked)
People
(Reporter: experimenter, Assigned: ccd)
References
()
Details
User sensitivity to page load regressions
This experiment is composed of three phases, each of 4-week duration, that artificially regresses Firefox page load speeds. The experiment will test the impact of engagement and retention on known page load regressions. In addition, it will determine how quickly users acclimate to these regressions. Finally, a secondary objective will be to review telemetry page load probes with these known regressions, to help tune regression detection algorithms.
Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.
Data Science Issue: https://jira.mozilla.com/browse/DS-466
More information: https://experimenter.services.mozilla.com/experiments/user-sensitivity-to-page-load-regressions/
|
Reporter | |
Updated•4 years ago
|
This experiment garnered some interest on social media and there were concerns about what we might have been planning to do. We're sorry for the confusion, but we don't plan to run this experiment. I'm happy to talk about why we considered it, though, and why we decided not to go ahead.
Our "Fission" project - a major undertaking for Firefox engineering - involves splitting the proverbial process atom to mitigate SPECTRE-style attacks aimed at Firefox users. We're getting close, but that project isn't ready; last December our performance team's testing showed that enabling Fission regressed some important performance benchmarks by as much as 30%. We assumed that users would really feel that large of a performance hit after upgrading.
The difficulty facing us was that we weren't sure how much we could improve that situation before we needed to release if Fission. Specifically, if a major uptick of in-the-wild SPECTRE-style attacks made an early release necessary to protect our users. We felt the need to understand how our users might react to changes in performance more precisely than “faster is better,” to ensure that we made the right tradeoffs and focused our limited resources where they make the greatest difference. We couldn't make a decision like that blindly.
Once planning was completed for this experiment, the next steps would have been to get buy-in from the Fission and Desktop teams, and then get leadership approval before committing to the experiment and making an announcement. We pride ourselves on working in the open, though, and I guess that means that what we’re working on may generate questions before we have all the answers.
Fortunately it now looks like this experiment is unnecessary; this week's good news is that test results from the morning of May 4 show that we’re much closer to performance parity than we thought that we’d be, so much so that we feel comfortable putting this experiment aside. I should add that we’re hugely thankful for the millions of users using the Firefox Nightly and Beta releases that let us to run experiments and tune Firefox before changes hit our official release channel; we wouldn't be able to do the work we do without their help.
Again, sorry for the confusion; it's nice that we're a lot further along than we thought we'd be, but we'll try to do a better job of communicating this sort of thing in the future.
Description
•