Closed Bug 2029129 Opened 4 months ago Closed 4 months ago

4.92% ares6 (Linux) regression on Wed April 1 2026

Categories

(Core :: JavaScript Engine, defect, P3)

Product:
Core
Component:
JavaScript Engine
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Milestone:
151 Branch
Tracking Flags:
Tracking Status
firefox-esr140 --- unaffected
firefox149 --- unaffected
firefox150 --- unaffected
firefox151 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: arai)

References

(Blocks 1 open bug, Regression)

Details

(Keywords: perf, perf-alert, regression)

Perfherder has detected a browsertime performance regression from push 19f0e5e88ab75d1203285f7466ced1beb8aefdf8. As author of one of the patches included in that push, we need your help to address this regression.

Please acknowledge, and begin investigating this alert within 3 business days, or the patch(es) may be backed out in accordance with our regression policy.

Regression Test Platform Options Absolute values [old vs new] Performance Profiles
5% ares6 (doc) linux1804-64-shippable-qr fission webrender 37.25 -> 39.09 Before/After

Support & Actionable Steps

For questions, please contact fbilt@mozilla.com. You can also find help on Slack (#perf-help) or Matrix (#perftest).

Action guide details:

Flags: needinfo?(arai.unmht)
Duplicate of this bug: 2028752

ARES-6 benchmark:

Each sample runs in a fresh iframe to simulate some of the effects of page navigation.

Maybe there is a problem with the navigation cache reuse, we should double check whether we bad interactions between iframes and the navigation cache, or whether we are evicting stencils too fast.

Set release status flags based on info from the regressing bug 2027803

status-firefox149: --- → unaffected
status-firefox150: --- → unaffected
status-firefox151: --- → affected
status-firefox-esr140: --- → unaffected

The regression happens only for the first iteration, so at least this is not something related to reusing cache across iterations within the same subtest.
Also, the regression happens only on linux1804:
https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=0&highlightCommonAlerts=0&highlightInitialDataPoints=0&replicates=0&series=autoland,3915302,1,13&series=autoland,5690948,1,13&series=autoland,5149776,1,13&series=autoland,5258087,1,13&timerange=604800

When I test it locally on ubuntu 24.04, I don't observe the regression, but actually an improvement, such as Basic_firstIteration from 14.85 to 13.39.
So this is not something consistently happening across environments, but more specific to the linux1804 environment.

Unless there's some other critical regression, I'm going to accept the regression.

(btw, the link for "5%" in the first comment, which is https://treeherder.mozilla.org/perfherderperfherder/graphs?timerange=1209600&series=autoland,3915286,1,13 , doesn't show anything)

Flags: needinfo?(arai.unmht)

some update from discussion in matrix.

The testcase runs 5 times, for 4 subtests, 60-200 iterations for each subtest,
and the regression happens on the first iteration of the 2nd+ runs.

In the "after" case, the 2nd+ runs don't trigger any network requests, which means the navigation cache is properly used.

The execution pattern of the first iteration itself doesn't show notable difference, as it's all done synchronously.
but there are some differences in the JIT compilation.
in the "after" with regressed case, JIT compilation seems to take longer, which happens off-main thread,
and also the timing of the JIT compilation differ.

One possible reason is the execution timing of the first iteration after the page load.
In the "before" case, the network requests and the compilation happens before the first iteration, that takes ~8ms.
In the "after" case, all the scripts are cached, and the first iteration immediately starts, possibly during or immediately after the iframe page load.
If there's other tasks running while that period, that may affect the off-thread JIT compilation.

Also, the JIT compilation immediately happens after the page load suggests JIT hint is used.
I'll check the combination of putting some delay before the first iteration, and disabling JIT hint.

try runs and the Basic_firstIteration score:

In (a), the problem is that, "after" case has 2 peaks, where the slower case doesn't exist in the "before" case.
(b) shows that disabling JIT hints doesn't reduce the gap, and it's ruled out.
Assuming this is timing issue and the contention with other off-thread tasks, added setTimeout to shift the executing timing in (c) and (d), and that makes both cases have 2 peaks, and both case shows similar characteristics. So, indeed this is timing issue, and the "before" case had been working well for some reason.

The main difference in the "after" case is that the 2nd+ iterations don't load scripts from necko, it shifts the execution timing earlier and that's affecting the performance. Added a dummy script with Date.now() query string, to always load it from necko, in (e). and it shows a change in the slower peak, where the frequency is reduced. So, indeed this is caused by the execution timing, where starting the first iteration earlier makes the slower case happen more frequently.

So, it's still unclear how exactly the slower case happens (most likely related to when the JIT compilation starts and finishes, which is affected by the number of off-thread tasks and other things), this is not a real regression in normal use case.
The testcase is now testing the performance in different setup and different timing, and the calculation starts earlier and that makes the calculation slightly slower for the first iteration.
The net performance and the time taken until the calculation ends aren't so much affected.

Blocks: sm-js-perf
Severity: -- → S3
Priority: -- → P3

Update:

Perfherder has detected a browsertime performance regression from push 19f0e5e88ab75d1203285f7466ced1beb8aefdf8. As author of one of the patches included in that push, we need your help to address this regression.

Please acknowledge, and begin investigating this alert within 3 business days, or the patch(es) may be backed out in accordance with our regression policy.

Regression Test Platform Options Absolute values [old vs new] Performance Profiles
306% twitch FirstVisualChange (doc) linux1804-64-shippable-qr fission warm webrender 91.34 ms -> 370.48 ms
305% twitch FirstVisualChange (doc) linux2404-64-shippable fission warm webrender 81.72 ms -> 330.78 ms Before/After
263% youtube FirstVisualChange (doc) linux1804-64-shippable-qr fission warm webrender 174.66 ms -> 633.37 ms
245% twitch FirstVisualChange (doc) macosx1470-64-shippable fission warm webrender 93.14 ms -> 321.54 ms Before/After
217% google-docs fcp (doc) linux1804-64-shippable-qr fission warm webrender 124.30 ms -> 394.26 ms Before/After
209% google-docs fcp (doc) linux2404-64-shippable fission warm webrender 112.43 ms -> 347.08 ms Before/After
209% google-docs FirstVisualChange (doc) linux2404-64-shippable fission warm webrender 119.30 ms -> 368.25 ms Before/After
204% google-docs fcp (doc) macosx1470-64-shippable fission warm webrender 108.65 ms -> 330.81 ms Before/After
204% youtube fcp (doc) linux2404-64-shippable fission warm webrender 217.35 ms -> 660.85 ms Before/After
201% youtube fcp (doc) linux1804-64-shippable-qr fission warm webrender 228.01 ms -> 685.85 ms
... ... ... ... ... ...
3% speedometer3 Editor-CodeMirror/Highlight/total (doc) linux1804-64-shippable-qr fission webrender 30.41 ms -> 31.25 ms Before/After
3% speedometer3 React-Stockcharts-SVG/ZoomTheChart/Sync (doc) linux2404-64-shippable fission webrender 79.09 -> 81.22 Before/After
2% speedometer3 TodoMVC-Svelte-Complex-DOM/Adding100Items/Sync (doc) macosx1500-aarch64-shippable fission webrender 0.98 ms -> 1.01 ms Before/After
2% speedometer3 perfstats-MinorGC (doc) linux2404-64-shippable fission webrender 699.35 -> 715.01 Before/After
2% speedometer3 Perf-Dashboard/total (doc) linux2404-64-shippable fission webrender 104.50 -> 106.62 Before/After
Improvements Test Platform Options Absolute values [old vs new] Performance Profiles
72% twitter loadtime (doc) linux2404-64-shippable bytecode-cached fission warm webrender 541.38 ms -> 150.39 ms Before/After
70% google-search LastVisualChange (doc) windows11-64-24h2-shippable fission warm webrender 774.87 ms -> 236.27 ms Before/After
68% twitter loadtime (doc) linux1804-64-shippable-qr bytecode-cached fission warm webrender 569.40 ms -> 179.59 ms
66% instagram loadtime (doc) macosx1470-64-shippable fission warm webrender 379.17 ms -> 128.62 ms Before/After
65% imgur SpeedIndex (doc) linux1804-64-shippable-qr fission warm webrender 490.45 ms -> 170.59 ms
... ... ... ... ... ...
3% imdb LastVisualChange (doc) android-hw-a55-14-0-aarch64-shippable warm webrender 6,118.03 ms -> 5,956.85 ms

Support & Actionable Steps

For questions, please contact fbilt@mozilla.com. You can also find help on Slack (#perf-help) or Matrix (#perftest).

Action guide details:

All of the percentage links lead to blank page.

Anyway, both the regressions and improvements are expected.

The regressions are happening mostly in the visual score.
This is caused by the fact that there's often a race between the JS resources and other resources (CSS, image, etc).
The stencil navigation makes the JS resources available earlier.
Making JS resources available earlier makes the JS resources processed earlier, such as, instantiating, and executing them,
which makes the other resources processed later.
In general, CSS and images contributes more to the visual, and also there are several cases where the page contains a placeholder elements,
which is shown until replaced with the actual content.
The "first" visual score often reflects the placeholder, but if the JS starts executing earlier, it can remove the placeholder and then the actual content will be populated after some calculation. If there's no chance to actually render the placeholder, the "first" visual scores get regressed.

Some of the improvement also come from the execution timing and scheduling, as it can populate the content earlier.

Anyway, the scheduling highly depends on the environment, such as the CPU, memory, disk speed, network speed, server performance, etc,
and the automation result is only one instance of them, and also the automation doesn't have network latency, which makes the scheduling characteristics different than the real case.
We're going to test the performance as an experiment on release channel once the code reaches the release channel (around May).

And thus, I'll accept the the regression around the browsertime.

I will second what Arai mentioned with additional information.

We have been tracking these metrics and regressions for months, and we are unable to answer whether these changes are real and whether they are impacting users.

Which is why Firefox now has much more telemetry instrumentation to help us answer these questions, that the stencil navigation cache is enabled by default on Nightly only, no beta nor release, and why we are planning to conduct some A/B testing experiments on the release channel in a few weeks. All to collect reliable data coming from users.

(the links have /perfherderperfherder/ in them, replace with /perfherder/ to get links to graphs)

I would recommend looking at the graphs for the SP3 regressions. A few of them look real (e.g. look at svelte)

(In reply to Matthew Gaudet (he/him) [:mgaudet] from comment #10)

(the links have /perfherderperfherder/ in them, replace with /perfherder/ to get links to graphs)

Thank you for the info!

I would recommend looking at the graphs for the SP3 regressions. A few of them look real (e.g. look at svelte)

Thanks!
So, this is happening mostly only on macOS. linux doesn't show any change. windows shows slight change, but not too obvious:

I'll investigate it.

Looks like the disk cache handling (encode, compress, and transfer) is happening during the benchmark, while the page is actively processing many things.

Currently the disk cache handling starts from the following:

  • when the load event is fired, and when there's no remaining requests
  • after the load event is fired, and the last request is processed

and the tasks are enqueued with EventQueuePriority::Idle, but apparently that's still happening too early, especially in the test's setup.

https://searchfox.org/firefox-main/rev/62b11cf9d978f8c6f8144156feebf42af5d71cf9/dom/script/SharedScriptCache.cpp#434-438

nsCOMPtr<nsIRunnable> updater =
    NewRunnableMethod("SharedScriptCache::UpdateDiskCache", this,
                      &SharedScriptCache::UpdateDiskCache);
(void)NS_DispatchToCurrentThreadQueue(updater.forget(),
                                      EventQueuePriority::Idle);

https://searchfox.org/firefox-main/rev/62b11cf9d978f8c6f8144156feebf42af5d71cf9/dom/script/SharedScriptCache.cpp#508-514

RefPtr<ScriptEncodeAndCompressionTask> encodeTask =
    new ScriptEncodeAndCompressionTask();
RefPtr<ScriptSaveTask> saveTask = new ScriptSaveTask();
saveTask->AddDependency(encodeTask);

TaskController::Get()->AddTask(encodeTask.forget());
TaskController::Get()->AddTask(saveTask.forget());

https://searchfox.org/firefox-main/rev/62b11cf9d978f8c6f8144156feebf42af5d71cf9/dom/script/SharedScriptCache.cpp#444-445,463

ScriptEncodeAndCompressionTask()
    : Task(Kind::OffMainThreadOnly, EventQueuePriority::Idle) {}
...
ScriptSaveTask() : Task(Kind::MainThreadOnly, EventQueuePriority::Idle) {}

We could look into tweaking the scheduling, to delay it more, for example until the idle phase continues for certain amount of period.

Anyway, this regression is very specific to the speedometer3's testing setup, where many pages are getting loaded and hot code runs continuously for each.
I don't think the similar case happens in wild.

Depends on: 2030623

Tested again on autoland,
and the svelte sync is not from stencil navigation cache, but something else.
(svelte async has some regression tho)

https://perf.compare/subtests-compare-results?baseRev=a9c1ac59fedf5ba43653bd227e69961b4422e907&baseRepo=autoland&newRev=19f0e5e88ab75d1203285f7466ced1beb8aefdf8&newRepo=autoland&framework=13&baseParentSignature=5352597&newParentSignature=5352597&test_version=mann-whitney-u&sort=delta%7Cdesc

also the local result seems to be very different than the automation result, to I should test more on automation.

For the speedometer3 regression, it's really hard to investigate further.

There are both improvements and regressions, across subtests

In the profile, each iteration is very noisy, for example, "NewsSite-Next/NavigateToUS/Sync" takes 5ms-14ms, mostly randomly, both in before and after. And for this subtest, the regression is 0.09ms, which is way smaller than the noise.

Also given that subtest interleaves each other and each iteration is very short like ~10ms, there's not much samples within that range. sometimes there's 0 samples, or sometimes 5 samples. So, extracting the execution characteristics is not really possible for each subtest, and thus comparing between before vs after is not possible.

I'll look into the entire characteristics some more, but most likely I'll accept the regression.
Anyway, I'm thinking backing out the patch for bug 2030588, until the memory reporting and the memory pressure handling are properly fixed.

Setting Fx151 to fixed since the regressor Bug 2027803 was backed out

status-firefox151: affected → fixed

It has been over 7 days with no activity on this performance regression.

:arai, since you are the author of the regressor, bug 2027803, which triggered this performance alert, could you please provide a progress update?

If this regression is something that fixes a bug, changes the baseline of the regression metrics, or otherwise will not be fixed, please consider closing it as WONTFIX. See this documentation for more information on how to handle regressions.

For additional information/help, please needinfo the performance sheriff who filed this alert (they can be found in comment #0), or reach out in #perftest, or #perfsheriffs on Element.

For more information, please visit BugBot documentation.

Flags: needinfo?(arai.unmht)
Status: NEW → RESOLVED
Closed: 4 months ago
Flags: needinfo?(arai.unmht)
Resolution: --- → FIXED
Assignee: nobody → arai.unmht
Target Milestone: --- → 151 Branch
You need to log in before you can comment on or make changes to this bug.