576688 - IM: Performance on "new" JS A* benchmark somwhat volatile and between 30% and 100% slower than v8

Status: RESOLVED WORKSFORME

(Core :: JavaScript Engine, defect)

Description

[Boris Zbarsky [:bzbarsky]]

See url in url field.  Relevant numbers (moo branch has fix for bug 576398):

m-c:  491ms
v8:   167ms
moo: 1085ms

Comment 1

[Boris Zbarsky [:bzbarsky]]

With the fix for bug 576926 we're at 599ms on moo tip here.

Comment 2

[Jan de Mooij [:jandem]]

Current numbers with a TM nightly build from today:

Chrome: 147 ms
Safari: 282 ms
TM: 371 ms
JM+TM: 421 ms
JM: 428 ms

Although still slower than TM and competition, this is much better than the numbers bz posted two months ago.

Comment 3

[Nicholas Nethercote [inactive]]

See bug 579285.

Comment 4

[Boris Zbarsky [:bzbarsky]]

That bug isn't really related to this bug.  Sayrer's A* benchmark is totally different code (and structured very differently) from this one, last I checked.

What I want to know here is .... why are we not doing at least as well as TM?  Do I need to file a separate bug on the heuristics fail here?

Comment 5

[Jan de Mooij [:jandem]]

Attachment: fillStyle/fillRect test case

JM+TM runs this in 136 ms. Chrome and Safari are much faster (18 ms and 48 ms).

@bz: do you know what causes this?

Comment 6

[Jan de Mooij [:jandem]]

To clarify, the reduced test shows that ctx.fillRect is very slow. I can profile but I'm a bit lost in layout and painting, I don't know what numbers are reasonable there.

Comment 7

[Boris Zbarsky [:bzbarsky]]

Jan, on which OS?  I can confirm what you see on Mac, but graphics stuff can be much different on different platforms.

That testcase is the same size as the actual benchmark?  So it's costing us about 100ms of slowdown via Chrome, right?  The rest of the difference (and the JM+TM vs TM difference) are JS eng?

We really need to look into making fillRect on canvas faster; whether we make cairo faster for rect fills, or whatever.  And maybe we should particularly special-case 1x1 rects, if possible...  Sites tend to use them when they should just use imagedata.  :(

Comment 8

[Boris Zbarsky [:bzbarsky]]

Oh, and the FillRect profile shows the usual: us converting the whole thing to a DrawPath call and then _cairo_quartz_surface_fill spending forever in state setup and restoration, calling CGContextDrawPath, etc.

Comment 9

[Jan de Mooij [:jandem]]

(In reply to comment #7)
> Jan, on which OS?
On OS X..

> That testcase is the same size as the actual benchmark?  So it's costing us
> about 100ms of slowdown via Chrome, right? 
Both use a 100x100 canvas. But in the original benchmark there are even more ctx.fillRect calls, to draw the path, etc. If I comment out all "ctx.fillRect" lines in the benchmark it's at least 160 ms. faster. Chrome gets 50 ms faster, so yeah I suppose this may win at least 100 ms here. 

> The rest of the difference (and the JM+TM vs TM difference) are JS eng?>
If I comment out the fillRect calls, JM+TM is still 50 ms slower than Safari, but TM is only 5 ms. slower. I think most of this is in the JS-engine, but I haven't measured that yet.

Do we need a new bug for fillRect perf?

Comment 10

[Boris Zbarsky [:bzbarsky]]

For reference, bug 601176 was filed for the fillRect.

If you comment out the canvas style sets, btw, how does performance look across the browsers?

Comment 11

[Jan de Mooij [:jandem]]

(In reply to comment #10)
> If you comment out the canvas style sets, btw, how does performance look across
> the browsers?

Chrome: 77 ms
Safari: 204 ms.
FF4 TM: 211 ms
FF4 JM+TM: 248 ms

Comment 12

[Boris Zbarsky [:bzbarsky]]

Yeah, sounds like we have headroom here and that this bug as reported is still there.  ;)

Comment 13

[Jan de Mooij [:jandem]]

Some progress in the last month, and to summarize:

Chrome 8: 135 ms
Safari 5: 300 ms
Opera 10.63: 350 ms
TM: 358 ms
JM/TM/prof: 390 ms

With fillRect/fillStyle commented out:

Chrome 8: 71 ms
TM: 185 ms
JM/TM/prof: 210 ms
Safari 5: 211 ms
JM: 213 ms
Opera 10.63: 312 ms

So, the JS part is still slower than Chrome but faster than Opera and getting faster than Safari. fillRect/fillStyle is responsible for going from 2th place to 4th (last) place.

Comment 14

[Bas Schouten (:bas.schouten)]

On windows this is 280 ms -with- fillRect/fillStyle, and 30 ms without in Minefield. Chromium is 120 ms with fillRect and 10 ms without.

I'll check out what is making the graphics part of this so slow on windows.

Comment 15

[Jan de Mooij [:jandem]]

(In reply to comment #14)
> I'll check out what is making the graphics part of this so slow on windows.

Thanks! Also see bug 601176 for the fillRect issue.

Comment 16

[Boris Zbarsky [:bzbarsky]]

Note that just commenting out the canvas stuff doesn't just eliminate graphics time; it also removes the actual code needed to make the calls through xpconnect, possibly changes the results of -p, etc.  To do an apples-to-apples comparison, either you need to time the canvas stuff separately and subtract, or you need to replace the canvas stuff with equivalent (in terms of how the jseng treats them) but known-fast DOM method calls and property sets.

Comment 17

[Bas Schouten (:bas.schouten)]

So, I've been looking at this test case and my profiles. A couple of things are easily noticable:

We spend about 30% in setting the fillstyle, of which the majority is spent ... parsing the color string. Then we spend about 35% actually painting, where a lot of time is spent doing things like 'gfxContext::SetColor' or 'gfxContext::Rectangle' which are all just things that go 20 stacks deep in order to create all the right various cairo types. Less than 15% of our time is -actually- spent doing the FillRectangle call in Direct2D. Hence we are losing a -lot- of time in setup overhead.

One thing that definitely doesn't seem to be taking a lot of time in the profile is JS. The amount of things that JS does is just insignificant to the amount of painting calls that are executed (and setting up all the cairo state for it).

Comment 18

[Boris Zbarsky [:bzbarsky]]

Hmm.  Can someone file a bug blocking this one on the fillstyle set issue?  Ideally with a microbenchmark.  That should be not too hard to make faster.

Comment 19

[Boris Zbarsky [:bzbarsky]]

Er, nevermind.  The "reduced testcase" attached here is pretty minimal.

Comment 20

[Bas Schouten (:bas.schouten)]

(In reply to comment #19)
> Er, nevermind.  The "reduced testcase" attached here is pretty minimal.

I wonder if we can cut out the fillrect and still get the style string parsed?

Comment 21

[Brendan Eich [:brendan]]

(In reply to comment #19)
> Er, nevermind.  The "reduced testcase" attached here is pretty minimal.

Then this bug (not a metabug) is morphing into a gfx bug?

/be

Comment 22

[Jan de Mooij [:jandem]]

Comment on attachment 480036 [details]
fillStyle/fillRect test case

Renamed the test case to clarify that it's only about fillRect/fillStyle.

Comment 23

[Boris Zbarsky [:bzbarsky]]

No; the attached testcase just had nothing to do with this bug.

This bug is specifically about JM being slower than TM and others on the relevant testcase, not about the fact that TM is slower than others per se.

If that bug is fixed, fine.  This should be marked fixed.  But comment 13 doesn't sound like that's quite the case, and we definitely have work to do to catch up to V8 even for the JS only part.  Maybe that should happen in a separate bug, though.

I do think we need some additional gfx or DOM bugs here, and perhaps create a tracking bug for this testcase...

Comment 24

[Ryan VanderMeulen [:RyanVM]]

With JM+TI, I get 204-209ms vs. 137-151ms for Chrome on Win7 w/ D2D enabled.

Comment 25

[Till Schneidereit [:till]]

In the light of comment 23, this is invalid. I don't, however, know if any gfx or DOM bugs have been filed.

FWIW, Chromium is at ~60ms vs. our ~102ms, right now.

@bz: please reopen if you think more should be done here.

Comment 26

[Boris Zbarsky [:bzbarsky]]

> FWIW, Chromium is at ~60ms vs. our ~102ms, right now.

For the JS-only testcase or the one involving gfx calls?

Comment 27

[Till Schneidereit [:till]]

The one involving gfx calls. I didn't look into the other one as this bug's history says the js part is fixed. It might have regressed, but testing every bug-that-should've-been-closed-long-ago for regressions doesn't make sense to me. ;)

Comment 28

[Boris Zbarsky [:bzbarsky]]

> as this bug's history says the js part is fixed. 

Where, exactly?  The last comment I see with numbers for the JS part is comment 13, which shows that we had gotten it to almost TM perf, and were 3x slower than V8, right?

Comment 29

[Emanuel Hoogeveen [:ehoogeveen]]

FWIW I get the following progression of times on the linked page:
Firefox Nightly:  97ms, 68ms, 73ms, 58ms, 60ms
Chromium Nightly: 49ms, 50ms, 46ms, 53ms, 46ms

And on the fillStyle/fillRect test case:
Firefox Nightly:  19-20ms
Chromium Nightly: 8-9ms

So Firefox seems to have more of a warm-up period on the A* benchmark. After warmup it's still about 20% slower then Chromium. The ratio on the attached test case is much improved compared to comment 5, at least on my machine, but Chromium is still twice as fast.

Comment 30

[Till Schneidereit [:till]]

(In reply to Boris Zbarsky (:bz) from comment #28)
> > as this bug's history says the js part is fixed. 
> 
> Where, exactly?  The last comment I see with numbers for the JS part is
> comment 13, which shows that we had gotten it to almost TM perf, and were 3x
> slower than V8, right?

So I *might* have read too much into the comparison with TM, sorry about that.

Emanuel, thanks for those updated numbers!

Comment 31

[Guilherme Lima]

Setting the heuristic to 0 on the website benchmark, I get 112ms for Firefox 43 and 164ms for Chrome 47.