415008 - eval Performance Regression

Status: RESOLVED WORKSFORME

(Core :: JavaScript Engine, defect)

Description

[John Resig]

A recent performance analysis revealed a serious regression (from Firefox 2 to nightly builds) relating to eval.

Here's the results:
                     FF3  FF2  Saf3
eval                 520   36     7

Here's the test case (see the URL, as well):
<script>
var start = (new Date).getTime();
for ( var i = 0; i < 1000; i++ ) {
	window.eval("function test1(){}");
}
document.write( (new Date).getTime() - start + "<br>");
</script>

I'm working on pulling together some Shark numbers, I will post them here when I have them.

Comment 1

[John Resig]

Per the request of Igor, I created a second eval test. This one uses less looping and longer evals. My results:
                     FF3  FF2  Saf3
eval                1927  109    36

The code:
<script>
var evalString = "";
for ( var i = 0; i < 100; i++ )
	evalString += "function test" + i + "(){} ";

var start = (new Date).getTime();
for ( var i = 0; i < 100; i++ )
	window.eval(evalString);
document.write( (new Date).getTime() - start + "<br>");
</script>

Every so often this test will trigger a massive garbage collection spike - which will cause it to run for 30-40 seconds (locking the browser, as well). Maybe that's just related to my setup, but that seems problematic.

Comment 2

[Igor Bukanov]

This should be related to more frequent invocation of the GC. If I run the  test case in an optimized thread-safe builds of jsshell, then on my laptop under Fedora 8 I have for the best value among 10 runs:

trunk:      67
181 branch: 81

With the browser the difference is that the GC would run periodically. So fixing the regression requires better GC scheduling.


Test for the shell:

var evalString = "";
for ( var i = 0; i < 100; i++ )
    evalString += "function test" + i + "(){} ";

var start = (new Date).getTime();
for ( var i = 0; i < 100; i++ )
    eval(evalString);
print((new Date).getTime() - start);

Comment 3

[Mike Wilcox]

This bug had minimal effect in Dojo, and I tracked down why. Dojo adds the following suffix to files that it evals: "//@ sourceURL=q" - 'q' is the url of the file, but for this test case, it can be anything but must be something. I've tried substituting or removing parts of that suffix, but only this seems to work. Try adding that to the end of John's test case and you'll see the difference.

Adding this suffix cuts the eval time down to 1/3 or 1/2. Hopefully this will add some insight to the performance problem. Currently IE6 evals ten to 30 times faster than FF3. That's no good.

http://bugs.dojotoolkit.org/ticket/7290

Comment 4

[Brendan Eich [:brendan]]

Need an owner -- mrbkap, igor?

/be

Comment 5

[Blake Kaplan (:mrbkap) (inactive)]

I'll take a look at this soon.

Comment 6

[Kris Zyp]

I believe this issue is almost entirely due to Firebug, not Firefox. According to my tests FF3 eval is actually almost twice as fast as FF2 eval (without Firebug running). One of the major features of Firebug 1.2 (FF3 version) is the ability to debug eval code, which means all evals must be processed by Firebug and added to a list of scripts, which is understandably expensive; making eval 10-20 times slower than in previous versions. However, this FB feature is a very valuable tool, although I think there may be an issue with turning it off, (disabling debugging doesn't seem to restore eval performance, but I think this is a FB issue, not FF).

Comment 7

[Adam L. Peller]

Mike Wilcox has shown that this seems to have something to do with Firebug parsing the eval buffer for //@ metadata.  This metadata helps establish a link between the anonymous eval buffer and its source.  For a loader like Dojo, this is critical for debugging.  iirc, this feature used to be optional, as it was known to carry a performance hit with it.  The metadata is best used at the end of the file so that it does not throw off line numbers by 1.  It's a bit surprising that without the metadata the file parses faster.

Comment 8

[Blake Kaplan (:mrbkap) (inactive)]

John, can you confirm Adam's hypothesis that Firebug is causing the apparent regression?

Comment 9

[John J. Barton]

If I understand Mike Wilcox's comment #3, eval is faster with //@ metadata. This would match my expectations.  With metadata, Firebug names the buffer by using the last line of the file. Else it names the file with an MD5 hash of the file. MD5 takes time comparable to the parser just because it has to visit all of the bytes and do some work. Profiling should show MD5 near the top of the list. 

Kris Zyp's comment 6 is correct, the eval() debugging is both critical and a significant preformance issue.

Complete analysis of the performance is difficult because there are lots of different kinds of cases. If you put 40,000 lines of JS in to one file, then the per/file handling overhead will be small but everytime you touch that buffer the machine huffs and puffs for while. Vice versa if you split the lines into 40 files. Nevertheless any other insights that we can use to improve the performance are most welcome.

Firebug 1.2 is actually much faster than 1.1 (which lucky for us does not work on FF3). But the size of JS is growing and growing (note the number of dojo users on this thread ;-).

Firebug 1.2 has four user controls for eval():
  1. //@ metadata for those who live by eval()
  2. script panel disable menu
  3. suspend/resume of entire Firebug 
  4. script - limiter menu to turn off eval and/or event script processing.
We need to do better documentation around these.

The Firebug team is also working with Mozilla to solve the non-MD5 parts of the performance problem.

Comment 10

[Rob Campbell [:rc] (:robcee)]

would it be possible to add some useful metadata at the end of an eval script before it gets parsed if none exists? The other option we discussed for Firebug is using some lighter checksum algorithm (CRC32) instead of the more expensive MD5 hashing.

Anything we can do to give firebug a performance boost that we can squeeze into Fx3.1 timeframe would be a huge bonus for us.

Comment 11

[John J. Barton]

What kind of metadata would be useful? 

Overall I think evals fall in to different categories that may need different solutions.

1) Simple evals from the old days: data URL or MD5 is just fine.
2) Massive dev-time evals (dojo): //@ src solution or extend eval() to have a filename arguement.
3) Production-time evals (pageads): CRC32 or similar. (here the developer is focused on something else, but needs to breakpoint in the eval code for some reason).
4) ?
The ideal algorithm for naming evals() does not exist: it would be a fast way of saying "this buffer means the same thing as before". But in general a machine can't tell if the buffers "mean the same thing". All of the hashing schemes (dataURI, MD5, CRC32) all fail if the user sends an edited eval buffer; we don't see that yet but we may soon. For these cases we need user-defined filenames (and the performance would be better in the bargain.

Comment 12

[John J. Barton]

Firebug 1.4 delays the MD5 hash on an eval buffer until the user has set a breakpoint in an eval() buffer. Thus all other times eval()  with Firebug active should be say twice as fast as Firebug 1.2.

Comment 13

[John J. Barton]

It would be great to have a retest with Firebug 1.4 + Firefox 3.0.4.

Comment 14

[Rob Campbell [:rc] (:robcee)]

given the work done on eval performance and JS optimization, I'm going to mark this closed. Reopen if needed.

Comment 15

[Bob Clary [:bc] (inactive)]

no patch -> wfm