Open Bug 1898545 Opened 2 years ago Updated 1 year ago

Bryntum Components are 3.5x+ slower in Firefox than Chrome.

Categories

(Core :: JavaScript Engine, defect, P3)

Product:
Core
Component:
JavaScript Engine
Type:
defect

Tracking

()

People

(Reporter: mgaudet, Unassigned)

References

(Depends on 1 open bug, Blocks 1 open bug)

Details

Attachments

(3 files)

Chrome Trace 1970-01-06 08.03 profile.json.gz
420.03 KB, application/x-gzip
Details
Firefox 2024-12-11 08.50 profile.json.gz
1.76 MB, application/x-gzip
Details
demo.zip
1.52 KB, application/zip
Details

The below is copy pasted from some comments in Bug 1317690, as this is a better venue for this discussion

Bug 1317690 Comment #17

To examine things on real website, please do the following:

  1. Open the https://bryntum.com/products/gantt/examples/bigdataset/ page. (If you'd like to check other examples, see https://bryntum.com/products/gantt/examples/)
    2. In console, type: window.DEBUG = true
    3. Click the "5K Tasks" button in the top toolbar. This will switch the example gantt to a new dataset of 5K tasks.

You should see the following output in console:

For Chrome:

window.DEBUG = true
true
generate: 25.85107421875 ms
Initializing project
Populating project: 887.9541015625 ms
Time to visible: 5474.97509765625 ms
Finalize propagation: 708.40185546875 ms

For Firefox:

window.DEBUG = true
true
generate: 303ms - timer ended
Initializing project
Populating project: 18714ms - timer ended 
Time to visible: 168067ms - timer ended 
Finalize propagation: 21519ms - timer ended

What happens here is:

  • generate - the example data is generated. Generation uses generators, perhaps that explains the 10x slowdown compared to Chrome, however, this is not a primary metric we are interested in.
  • Populating project - example data is loaded into "reactive graph". This does not involve generators, but instantiates lots of small objects (like hundreds of thousands of them). 21x slowdown compared to Chrome, would be great to see it optimized to be comparable.
  • Time to visible - this metric is the topic of this issue. Here the "reactive graph" calculates the final state. 30x slowdown compared to Chrome. Calculations are generators-based. During calculations, it also allocates the edges in the graph (populates lots of Map instances).
    Finalize propagation - not related to generators, 30x slowdown.

Hopefully the numbers are self-explanatory, if you need any other information and/or benchmark, please let me know.

Probably one need to optimize the allocation speed too, because the most important metric Time to visible is a combination of generator calls + allocation. In particular, the allocation of Maps, and internal allocations by Maps. I'll open a new ticket for that.

Bug 1317690 Comment #17

Just opening https://bryntum.com/products/gantt/examples/bigdataset/
Nightly: https://share.firefox.dev/4b9raNf (3s)
Chrome: https://share.firefox.dev/4bb4Umm (1.8s)

Clicking on 5K
Nightly: https://share.firefox.dev/3UQdzVo (12s)
Chrome: https://share.firefox.dev/3WPzMVd (3.5s)

With window.DEBUG = true , I generally get 11s with Firefox. But one time, Firefox took 45s.

Severity: -- → S3
Priority: -- → P3

(One oddity looking at the Chrome vs Nightly profiles: The Chrome ones have what seem like real function names (Transaction2, etc), whereas the firefox one seems minified -- so it's not clear we're even running the same code as Chrome here)

Depends on: 1851662

Markus: Would you be able to generate a comparison report on this site? (Sort of curious if your tooling can take something odd like this too :) )

Flags: needinfo?(mstange.moz)

Hmm.

The page seems to be performing appreciably better for me now. Still a fair amount of jank and stutter, but where it was previously unusable, now it's just bad. Here's a new profile: https://share.firefox.dev/4bNMHuJ

I did two things:

  1. Caught up on a few nightly updates -- I was out of date by ~week
  2. Hard refreshed the site. Maybe I had an older version of this?

Mayank: You had previously reproduced slowness -- do you see it still?

(Thanks for the profiles by the way!)

Flags: needinfo?(mayankleoboy1)

A Release profile: https://share.firefox.dev/3Vd4jLj

Chrome window.DEBUG=1 numbers for 5K tasks

Populating project: 195.7890625 ms
Time to visible: 1257.27001953125 ms
Finalize propagation: 181.125 ms

Firefox window.DEBUG=1 numbers for 5K tasks

Populating project: 1127ms
Time to visible: 4800ms
Finalize propagation: 1113ms

So - Chrome is faster by

  • 5x for Populating Project
  • 3.8x for Time to Visible
  • 6x for Finalize Propagation.

(In reply to Matthew Gaudet (he/him) [:mgaudet] from comment #2)

Markus: Would you be able to generate a comparison report on this site? (Sort of curious if your tooling can take something odd like this too :) )

Report: https://github.com/jrmuizel/js-profile-compare/blob/main/reports/Bryntum-BrowserWin-May23.md
It seems to have worked ok except for the source code extraction. I wouldn't believe any of the source code that's displayed inline in the report.
You can get the actual start offsets of each compared function in the last column of the comparison table at the top.

Flags: needinfo?(mstange.moz)

(In reply to Matthew Gaudet (he/him) [:mgaudet] from comment #3)

Mayank: You had previously reproduced slowness -- do you see it still?

Just opening https://bryntum.com/products/gantt/examples/bigdataset/ : https://share.firefox.dev/4bTp4kr / https://share.firefox.dev/3UTfyHs (~3s)
Clicking on 5k : https://share.firefox.dev/3USpeSA (~12s)

These numbers are consistent with what i posted in https://bugzilla.mozilla.org/show_bug.cgi?id=1317690#c18

Flags: needinfo?(mayankleoboy1)

Any updates on this?

See Also: → 1895667

@Mayank - Any news? Are you planning to improve Firefox to be on par with Chrome?

This is backlogged work, so it's not an active investigation priority at the moment.

There are some things you could do which would be helpful and make this more actionable

  1. The level of obfuscation in your deployed code makes it challenging to investigate. It would be helpful if you had a non-obfuscated build you could provide (even password protected if you wanted). It would also be very interesting to know if your non-obfuscated build suffers from the same performance gap -- this well could be a problem in the obfuscator not your code.
  2. It would be great to have side-by-side profiles of the non-obfuscated builds. https://profiler.firefox.com/ & https://chromewebstore.google.com/detail/firefox-profiler/ljmahpnflmbkgaipnfbpgjipcnahlghn?pli=1
  3. In Comment #1 I highlighted that I was seeing a difference in code that Chrome was running vs the code Firefox was running. Ensuring that that is in fact not the case would be very helpful.
  4. A reduced test case that is less manual; i.e. a stand alone page which performs the operations and dumps timing into a div & console.log for example.

A few notes based on looking at the Firefox Profile & comparison report

  • So function __gab is the number one gap; 7% of the gap attributed to that, and of that and we're just straight-up 6x slower, even in Ion.
    _gab = function(d, e) {
        d = d - 0xbf;
        let f = c[d];
        return f;
    };

There's a few of these tiny functions where there's a big gap. This is the kind of thing where Ion is being out-performed by Turbofan codegen.

GetPropertyIC::update is being called in a lot of the hottest functions: several instances of _Model and *, getFieldDefinition, Entity</<,, finalizeCommitAsync, ... Of the functions with the biggest gap, I think the majority are hitting this problem. So I suspect that improving performance on that one issue would improve a lot of functions at once.

Here's an example (an arbitrary instance of _Model):

static get[_gaaw(0x2078, 0x1556)]() {
  function rs(a, b) {
    return _gaaw(b - -0x1dd, a);
  }
  return this[rs(0x50c, 0xb1f)][rs(0x280d, 0x1b84)][rs(0x42d0, 0x3770)];
}

This is all part of the obfuscator goop. This is _gaaw:

function _gaaw(a,b){
  return _gab(a-0x155,b);
}

This is how _gab is defined:

function _gab(a, b) {
  const c = _gaa();
  _gab = function(d, e) {
    d = d - 0xbf;
    let f = c[d];
    return f;
  };
  return _gab(a, b);
}

And _gaa returns a huge table of strings:

function _gaa() {
  const cuM = ['getRowFromElement', 'columnsDetacher', 'prototypeProperties', 'isCalculationStarted',
    'focusOnHover', 'onChipViewMousedown', 'll\x20LT', 'up.onDependencyChange', 'idx', 'calculateCritical',
    ....
 ];
  _gaa = function() {
    return cuM;
  };
  return _gaa();
}

So roughly speaking, the obfuscator is working by turning foo.x.y into foo[getPropName(x1, x2)][getPropName(y1,y2)]. Which is obviously terrible for performance, in a way that the engine is not going to be able to fix.

(Also, the second argument to _gaaw appears to be completely unused, which does not fill me with confidence about the quality of this obfuscator. In general, the performance overhead of this obfuscator relative to the ease with which a deobfuscator could be written doesn't seem like a great tradeoff to me.)

All that said: it's not actually clear to me why we're hitting the GetProp fallback here. Assuming that the input values are correct, we should compute an index into the string table, get a string back, and use it as a property key. As far as the GetProp is concerned, it's the same as var propName = "x"; return obj[propName], which should become GuardSpecificAtom + LoadSlot. So there's something else going on here.

I'm pretty convinced, though, based on this report, that the problems here are due to the obfuscator, not generators.

From a GC perspective:

First, the report shows 11.8x more time spent in Minor GC in Firefox vs Chrome. It would be nice to know if we're allocating at the same rate or the same number of objects.

The profile shows a ton of minor GCs. For the profile in comment 8, there are 837 minor GCs (and 2 major GCs).

Our nursery is quite small, usually 256-640KB, and mostly 256KB. But the promotion rate is very low, always under 1% and usually around 0.4%. That means lots of small, fast minor GCs, and indeed 79% of them are less than 50 microseconds. Correspondingly, 71% of the minor GCs have a nursery size less than 1MB. 99.6% of them tenure less than 0.5%; 86% tenure less than 0.05%.

So generally speaking, it's doing what we want it to do. The promotion rate is low, so we shrink the nursery down while it stays fast, and it stays fast down to very small sizes. Almost everything being allocated turns out to be garbage, and we're repeatedly using a pretty small chunk of memory to bump allocate and discard.

But 11.8x is not good, so perhaps we have some overhead, or perhaps we're allocating too much?

Short of missing scalar replacement opportunities, I think we should end up allocating roughly similar numbers of objects. I wonder if this is a case where V8 is doing something concurrent / parallel, and they're actually doing just as much work on a background thread.

(In reply to Matthew Gaudet (he/him) [:mgaudet] from comment #10)

This is backlogged work, so it's not an active investigation priority at the moment.

There are some things you could do which would be helpful and make this more actionable

  1. The level of obfuscation in your deployed code makes it challenging to investigate. It would be helpful if you had a non-obfuscated build you could provide (even password protected if you wanted). It would also be very interesting to know if your non-obfuscated build suffers from the same performance gap -- this well could be a problem in the obfuscator not your code.
  2. It would be great to have side-by-side profiles of the non-obfuscated builds. https://profiler.firefox.com/ & https://chromewebstore.google.com/detail/firefox-profiler/ljmahpnflmbkgaipnfbpgjipcnahlghn?pli=1
  3. In Comment #1 I highlighted that I was seeing a difference in code that Chrome was running vs the code Firefox was running. Ensuring that that is in fact not the case would be very helpful.
  4. A reduced test case that is less manual; i.e. a stand alone page which performs the operations and dumps timing into a div & console.log for example.
  1. We could send a non-obfuscated zip for usage on a local web server in private. I am not allowed to share it here publicly though
  2. I'll record profiles on non-obfuscated builds and post them here.
  3. I'll modify the demo to do that and post it here
Attached file demo.zip —

(In reply to johan from comment #16)

(In reply to Matthew Gaudet (he/him) [:mgaudet] from comment #10)

This is backlogged work, so it's not an active investigation priority at the moment.

There are some things you could do which would be helpful and make this more actionable

  1. The level of obfuscation in your deployed code makes it challenging to investigate. It would be helpful if you had a non-obfuscated build you could provide (even password protected if you wanted). It would also be very interesting to know if your non-obfuscated build suffers from the same performance gap -- this well could be a problem in the obfuscator not your code.
  2. It would be great to have side-by-side profiles of the non-obfuscated builds. https://profiler.firefox.com/ & https://chromewebstore.google.com/detail/firefox-profiler/ljmahpnflmbkgaipnfbpgjipcnahlghn?pli=1
  3. In Comment #1 I highlighted that I was seeing a difference in code that Chrome was running vs the code Firefox was running. Ensuring that that is in fact not the case would be very helpful.
  4. A reduced test case that is less manual; i.e. a stand alone page which performs the operations and dumps timing into a div & console.log for example.
  1. We could send a non-obfuscated zip for usage on a local web server in private. I am not allowed to share it here publicly though
  2. I'll record profiles on non-obfuscated builds and post them here.
  3. I'll modify the demo to do that and post it here

That demo is great. Thank you.

Please email a runnable package of the non-obfuscated version to myself (mgaudet@mozilla.com) and Iain (iireland@mozilla.com), and we will try to take a look over the next couple of weeks.

We spent a bunch of time talking about this today and we'd definitely like to try and figure out what can be done here. (If nothing else we should really nail down the Ion IC issue, which I will open a dependent bug for)

Flags: needinfo?(johan)
Depends on: 1936586

After some troubles including a zip, email with a non-obfuscated demo has been sent. Please let me know if it does not arrive 😅

Flags: needinfo?(johan)

OK, got the non-obfuscated code. Some notes:

Timing

Timing on my machine for Obfuscated:

  • generate: 5ms
  • Populating project: 1041ms
  • Time to visible: 5503ms
  • Finalize propagation: 1260ms

Timing on my machine for non-Obfuscated:

  • generate: 6ms
  • Populating project: 828ms (0.79x)
  • Time to visible: 3895ms (0.70x)
  • Finalize propagation: 1054ms (0.83x)

So obfuscation is costing about 30% time.

To compare against chrome (non obfuscated)

  • generate: 6ms
  • Populating project: 203ms
  • Time to visible: 1131ms
  • Finalize propagation: 177ms

and Chrome obfuscated

  • generate: 6ms
  • Populating project: 309ms
  • Time to visible: 1779ms
  • Finalize propagation: 278ms

Issues:

Unfortunately the Bryntum code breaks a number of assumptions that we tend to make about what is "common", inhibiting our optimizations.

I spot 11 calls to Object.setProrotypeOf which has the following warning on MDN:

Warning: Changing the [[Prototype]] of an object is, by the nature of how modern JavaScript engines optimize property accesses, currently a very slow operation in every browser and JavaScript engine. In addition, the effects of altering inheritance are subtle and far-flung, and are not limited to the time spent in the Object.setPrototypeOf(...) statement, but may extend to any code that has access to any object whose [[Prototype]] has been altered. You can read more in JavaScript engine fundamentals: optimizing prototypes.

Because this feature is a part of the language, it is still the burden on engine developers to implement that feature performantly (ideally). Until engine developers address this issue, if you are concerned about performance, you should avoid setting the [[Prototype]] of an object. Instead, create a new object with the desired [[Prototype]] using Object.create().

I -think- most of the prototype mutation in the code could be avoided with judicious use of Object.create and Object.assign (there's some funkiness about non-enumerable properties however which may explain why my quick attempt to try it out failed)

Huh. I did actually manage to remove setPrototypeOf... still no good.

Perfomrance didn't move almost at all:

  • generate: 5ms
  • Populating project: 826ms
  • Time to visible: 3834ms
  • Finalize propagation: 1058ms

https://share.firefox.dev/3Vx4m4s

We're still seeing a whole lot of IC failures though (search for ::update) (Bug 1936586)

Some additional info, that shows that generators in FF are still slower, compared to Chrome (and that causes the components slow down).

These are the benchmarks from chronograph - a reactive computations library, that makes heavy use of generators:

Firefox:

Deep graph changes - generators: 8.349ms ±0.429
Deep graph changes - generators big: 1185.667ms ±84.298
Deep graph changes - generators big, shared identifiers: 1160.167ms ±44.923
Shallow graph changes - generators: 6.512ms ±0.125
Shallow graph changes - generators big: 801.333ms ±39.786

Chrome:

Deep graph changes - generators: 2.23ms ±0.064
Deep graph changes - generators big: 291.175ms ±10.528
Deep graph changes - generators big, shared identifiers: 279.45ms ±8.024
Shallow graph changes - generators: 1.928ms ±0.055
Shallow graph changes - generators big: 223.034ms ±9.934

As you can see, there's a ~3.5x slow-down.

To reproduce:

git clone git@github.com:bryntum/chronograph.git
cd chronograph
git checkout generators-benchmark 
npm i
npx tsc

Open http://localhost/chronograph/tests/benchmark/suite.html and observe the results in console.

@Matthew Gaudet: Any updates or findings after trying Nick's test case above?

Haven't yet looked at the above, but am looking at https://bugzilla.mozilla.org/show_bug.cgi?id=1944081 which should cut some time off Bryntum loads

git clone git@github.com:bryntum/chronograph.git
cd chronograph
git checkout generators-benchmark
npm i
npx tsc

Open `http://localhost/chronograph/tests/benchmark/suite.html` and observe the results in console.

Needed npx http-server -p 8000 to get mime types correct so firefox would load.

Tweaked so that we don't log to devtools -- some optimizations can be disabled in devtools (e.g. wasm for sure)

So far (need to break for lunch though), nothing which looks like generators. Slower, yes!

Firefox

Deep graph changes - generators: 2.236ms ±0.024
Deep graph changes - generators big: 301.083ms ±14.769
Deep graph changes - generators big, shared identifiers: 282ms ±10.269
Shallow graph changes - generators: 1.867ms ±0.047
Shallow graph changes - generators big: 218.364ms ±4.994

Chrome

Deep graph changes - generators: 0.74ms ±0.003
Deep graph changes - generators big: 104.973ms ±1.812
Deep graph changes - generators big, shared identifiers: 107.015ms ±0.54
Shallow graph changes - generators: 0.734ms ±0.001
Shallow graph changes - generators big: 74.044ms ±2.202

Haven't dug into the methodology here yet tho.

Next step here is probably to see if Markus can get a performance comparison here (Test case linked in Comment 27)

Markus: This isn't super urgent, but I would love a comparison report if you have some time. (Use my forked repo for reproduction just because it means numbers get dumped into a div rather than requiring you to open devtools to see results)

Flags: needinfo?(mstange.moz)

It seems the original test case from bug 1565549 has improved significantly. It was ~9x slower and now its only ~2x slower.

Chrome: 931 ops/s ± 1.3%
Firefox: 516 ops/s ± 4.25%

Do you think it can be improved further?

Regarding the chronograph benchmarks - it performs a lot of Map allocations, so that might be another performance bottleneck. Already reported as bug 1895667.

Link to the benchmark: https://jsbench.me/eqm9lilg4s/1

I gave this one attempt but couldn't immediately build the development version on Windows and gave up. Matt is currently getting set up to create comparison reports on macOS so he may have higher chances of success here soon.

Flags: needinfo?(mstange.moz)

Needed more work to get this to work this time around. Some notes; I'll try to get a comparison report tomorrow probably.

  1. Install nvm-sh
  2. nvm install 18
  3. nvm use 18
  4. visit http://127.0.0.1:8000//tests/benchmark/suite.html
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: