Closed Bug 1373241 Opened 7 years ago Closed 7 years ago

Pref flip study: JavaScript Start-up Bytecode Cache

Categories

(Shield :: Shield Study, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: nbp, Unassigned)

References

(Depends on 2 open bugs)

Details

> What is the preference we’ll be changing?

 - dom.script_loader.bytecode_cache.enabled

> What are the values of each branch of the study? You can have multiple branches, but every study must include a control.

false - (default)
 true - Enable the JSBC.

> What percentage of users do you want in each branch? We can do branches of uneven proportions. Example: 20% overall with 5% in control and 15% in treatment 1.

I have no idea what population size might be enough. Probably something small to start getting bug reports, and larger later to tune the heuristics.

> What Channels/locales do you intend to ship to?

Nightly, only.

> What is your intended go live date and how long will the study run? All pref experiments have expiration dates. If you are looking to flip a pref permanently we have different tools for that.

I expect switch this pref-flag to "true" around mid-June. (before Firefox 57 moves to beta)

> Are there specific criteria for participants? We can include/exclude based on anything available in telemetry.

Having both Desktop and Mobile users.

If we can associate telemetry data exclusively for this experiment, then I will also request that we clear the cache of the users when the experiment is installed. (Bug 1362114 comment 11)

> What is the main effect you are looking for?

1. crash-stat reports for the alternate-data implementation of necko, as well as reports for SpiderMonkey (XDR encoding and decoding) functions, as well as the integration of the 2 (Script*Load* from /dom/script/ directory).

2. Collecting telemetry reports for the probes added in Bug 1362114, to tune the heuristics as described in Bug 1362114 comment 3.

> Who is the owner of the data analysis for this study?

Me?

> Do you plan on surveying users at the end of the study? If so, please link to the survey.

No, telemetry data should provide enough information to see if there is any improvement, such as the TIME_TO_* probes.


> Prior art

Chrome added a similar feature, but there is no detail about how the selected their heuristics.  Also, we do not have the same information as part of the current interface of necko, so we cannot even use there approach.
Flags: needinfo?(mgrimes)
We can probably get this out today or tomorrow. Have you sent an intent to ship email to release drivers? For the Nightly channel this just needs to be a heads up and does not require explicit sign off.
Flags: needinfo?(mgrimes)
Please fill out the template below and send it to release-drivers@mozilla.org

************************

Subject:
Intent to ship: [CHANNEL] Shield Pref Study [NAME OF YOUR PREF]

Description:
We intend to run a shield pref study on CHANNEL to evaluate effect of YOUR PREF

https://bugzilla.mozilla.org/show_bug.cgi?id=1373241

Population Affected:
X% of CHANNEL channel, all locales and configuration OR YOUR CONDITIONS.
 - variation 1: pref value true: 50%
 - control: pref value false: 50%

Expected Effect:
DESCRIBE THE EFFECT FOR THE USER IF ANY for purposes of diagnosing user problems.

Timeline:
PLEASE DEFINE WHEN TO LAUNCH AND STOP

Success Metrics:
WHICH TELEMETRY PROBES DO YOU CARE MOST

Uplift needed: 
"none" or PLEASE SPECIFY IF NEEDED

Responsible Party:
CONTACT PERSON (probably you?)
Flags: needinfo?(nicolas.b.pierron)
Thanks,

I reviewed the data that was already being collected and notice an issue.
I will send this email, as soon as the fix has landed (Bug 1376634).
Email sent to release-driver. The pref should be toggle to 5% of nightly users today. (from what I have been told)
Flags: needinfo?(nicolas.b.pierron)
The recipe has been enabled for 10% of nightly users (5% control, 5% experimental). Let us know once you are ready to increase the population size.
I think we can raise the nightly population. (15% experimental)

So far, Bug 1377369 (~160/day) would have been a top-crasher with multiple signatures, and it is fixed now.
Bug 1377681 (~40/day) is still around but not raising since Bug 1377369 got fixed.

I should already have a minimum of telemetry data to start customizing the heuristics.

Next week, I hope to land an optimization to reduce the size of the bytecode, and to tune the heuristics based on telemetry results.
Just bumped you to a 30% sample with 15% in the treatment group. Might be worth bumping your rel-drivers email letting them know you increased the sample. Let me know if there's anything else you need from us.
Depends on: 1379627
Depends on: 1379703
Depends on: 1379723
Depends on: 1381888
What's the status on this experiment? Do you need to change population sizes? Let me know if you've already had enough time to make a decision and we can end the experiment.
Flags: needinfo?(nicolas.b.pierron)
(In reply to Matt Grimes [:Matt_G] from comment #8)
> What's the status on this experiment?

I no longer have any crash-stat issues, and I do not expect raising the population would highlight any more at the moment.

The collection of telemetry data is going well, the problem is that the TIME_TO_* metrics I needed to evaluate if we can turn this feature on or not is not yet being reported on https://moz-experiments-viewer.herokuapp.com/ , I opened bug 1379703 to fix that, but the patch does not seems to have been pushed to the deployed version yet.

I evaluated the telemetry that I have in hand, and concluded that whatever results I got from the telemetry was not actionable yet.  Bug 1381888 landed yesterday, and should change the telemetry results in a way where I would be able to compare the parsing time with the decoding time.  Hopefully waiting an extra week should provide enough data to tune the heuristics.

For both of these reasons, I should delay the shutdown of this pref experiment, to gather enough information to tune it, and to double check the impact on users computers.

> Do you need to change population sizes?

I do not think this would be necessary.

> Let me know if you've already had enough time to make a decision and
> we can end the experiment.
Flags: needinfo?(nicolas.b.pierron)
We can shutdown this perf-flip study.

Crashes no longer seems to be a problem. Thanks to the Necko team for fixing these issues.

I aggregated the result and analysis of some telemetry probes & requests made on sql.telemetry.mozilla.org [1] in the following google doc:

https://docs.google.com/a/mozilla.com/spreadsheets/d/1pXtSdDmY4aneEmTgVSiqaK_g9R-C1OqHJSf9PPEYiko/edit?usp=sharing

The conclusion is that this optimization improves each page load by 43ms overall (counting cases where this optimization does not apply), while being applied to only ~50% of the requests.

I collected all the information I needed to continue my work and I will probably enable the JavaScript Start-up Bytecode Cache by default in the next release cycle, or later this week.

[1] https://sql.telemetry.mozilla.org/queries/5891/source#12325
That is awesome to hear! We'll end the experiment shortly.
The experiment being ended, I guess we can close this bug now?
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.