Closed Bug 329338 Opened 18 years ago Closed 16 years ago

"BenchJS" DHTML benchmark 50% slower with linux cairo build

Categories

(Core :: Graphics, defect, P3)

Product:
Core
Component:
Graphics
Platform:
x86
Linux
Type:
defect

Tracking

()

Status:
VERIFIED WORKSFORME

People

(Reporter: wbrana, Assigned: vlad)

References

(
URL
)

Details

(Keywords: perf)

Attachments

(1 file, 1 obsolete file)

BenchJS source
54.87 KB, application/octet-stream
Details 
User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9a1) Gecko/20060304 Firefox/1.6a1
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9a1) Gecko/20060304 Firefox/1.6a1

BenchJS returns 50% performance of non-cairo build

Reproducible: Always

Steps to Reproduce:
1. run BenchJS on cairo build
2. run BenchJS on non-cairo build
3. compare results

Actual Results:  
50% performance

Expected Results:  
>=100% performance
Could you post the benchmark results here for non-cairo and cairo build? (test 1 through test 7)
Severity: enhancement → normal
non-cairo 12.57 sec.
TEST 1 time: 1.727 sec.
TEST 2 time: 1.305 sec.
TEST 3 time: 1.738 sec.
TEST 4 time: 0.97 sec.
TEST 5 time: 0.48 sec.
TEST 6 time: 5.108 sec.
TEST 7 time: 1.238 sec.

cairo 24.3 sec.
TEST 1 time: 3.083 sec.
TEST 2 time: 2.597 sec.
TEST 3 time: 3.501 sec.
TEST 4 time: 4.709 sec.
TEST 5 time: 0.554 sec.
TEST 6 time: 7.392 sec.
TEST 7 time: 2.463 sec.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Keywords: perf
Summary: 50% performance → "BenchJS" DHTML benchmark 50% slower with cairo build
I ran BenchJS on Windows XP Home (32-bits) on an Athlon 64 3200+ Notebook and got the results below. This machine is almost two years old so it is by no means the fastest thing out there.

1.5.0.1 Official 6.89 seconds
1.5.0.1 K8B-X65 4.98
Latest Trunk Cairo 7.47
IE 7.78

So perhaps this is more of a Linux issue than a general issue.
yeah, we're a little slower on windows (9.52s vs 8.24s) in my builds, but not 50%
Summary: "BenchJS" DHTML benchmark 50% slower with cairo build → "BenchJS" DHTML benchmark 50% slower with linux cairo build
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9a1) Gecko/20060413 Firefox/3.0a1

non-cairo: 13.14 sec.
TEST 1 time: 1.927 sec.
TEST 2 time: 1.448 sec.
TEST 3 time: 1.833 sec.
TEST 4 time: 1.001 sec.
TEST 5 time: 0.481 sec.
TEST 6 time: 5.24 sec.
TEST 7 time: 1.21 sec.

cairo: 18.08 sec.
TEST 1 time: 1.96 sec.
TEST 2 time: 1.285 sec.
TEST 3 time: 2.184 sec.
TEST 4 time: 3.853 sec.
TEST 5 time: 0.491 sec.
TEST 6 time: 7.074 sec.
TEST 7 time: 1.231 sec
Depends on: cairoperf
Blocks: cairoperf
No longer depends on: cairoperf
Blocks: 334720
No longer blocks: cairoperf
With leak monitor 0.1.3 I get leaks every time test 2 is running,don't know if there's a bug filed for this.
and it gets worse...

ff 1.5 non cairo: 10.93 seconds
TEST 1 time: 2.196 sec.
TEST 2 time: 1.824 sec.
TEST 3 time: 0.69 sec.
TEST 4 time: 2.046 sec.
TEST 5 time: 0.267 sec.
TEST 6 time: 3.133 sec.
TEST 7 time: 0.77 sec.

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9a1) Gecko/20060609 Minefield/3.0a1 ID:2006060904 [cairo]:

36.53 seconds

TEST 1 time: 1.372 sec.
TEST 2 time: 3.416 sec.
TEST 3 time: 25.04 sec.
TEST 4 time: 1.952 sec.
TEST 5 time: 0.292 sec.
TEST 6 time: 3.599 sec.
TEST 7 time: 0.863 sec.

What happend to Test 3? (That is the 300 small images replacement)

Andreas, maybe you could find a regression range for when Test 3 became that much slower? (I'm not seeing that, but I'm on windows here)
(In reply to comment #8)
> Andreas, maybe you could find a regression range for when Test 3 became that
> much slower? (I'm not seeing that, but I'm on windows here)

The oldest nightly i can find is 2006-05-19-05-trunk and this one already has the speed problem on test3.


You can find older archives here:
http://archive.mozilla.org/pub/firefox/nightly/
Maybe this is a regression from bug 234455, in which case you should see that it regressed around 2006-03-07, but this is just a guess.
(In reply to comment #10)
> You can find older archives here:
> http://archive.mozilla.org/pub/firefox/nightly/
> Maybe this is a regression from bug 234455, in which case you should see that
> it regressed around 2006-03-07, but this is just a guess.

I feel a bit stupid right now. After testing lots of the older builds, i have now found that its something in my current profile that causes the slowdown. Running the latest nightly with a clean profile gives me:

10.29 seconds
TEST 1 time: 1.422 sec.
TEST 2 time: 1.809 sec.
TEST 3 time: 0.807 sec.
TEST 4 time: 1.806 sec.
TEST 5 time: 0.269 sec.
TEST 6 time: 3.378 sec.
TEST 7 time: 0.803 sec.

This is about the same speed as 1.5

sorry for noise.
(In reply to comment #11)
> This is about the same speed as 1.5

Wait, really?  Are you sure you're using a cairo build for that test?  I'd love to have that be true mind you, just wanted to make sure :)
(In reply to comment #12)
> Wait, really?  Are you sure you're using a cairo build for that test?  I'd love
> to have that be true mind you, just wanted to make sure :)

Yes, i am sure ;-).

The benchmark has a high variance between runs. Here are the number for the Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9a1) Gecko/20060611 Minefield/3.0a1 ID:2006061104 [cairo] build.

TEST 1 time: 1.304 sec.
TEST 2 time: 2.24 sec.
TEST 3 time: 0.794 sec.
TEST 4 time: 2.056 sec.
TEST 5 time: 0.292 sec.
TEST 6 time: 3.625 sec.
TEST 7 time: 0.871 sec.

Andreas
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9a1) Gecko/20060708 Minefield/3.0a1

cairo 23.91 sec.
TEST 1 time: 2.048 sec.
TEST 2 time: 1.905 sec.
TEST 3 time: 2.59 sec.
TEST 4 time: 4.962 sec.
TEST 5 time: 0.572 sec.
TEST 6 time: 10.341 sec.
TEST 7 time: 1.496 sec.

I changed Radeon from 9250 to 9600XT (stupid idea).
Both drivers (xorg radeon and ATI fglrx) don't support 2D acceleration  
on r300 and better Radeons. 3D performance is worse.

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1b1) Gecko/20060709 BonEcho/2.0b1

non-cairo 16.15 sec.
TEST 1 time: 1.838 sec.
TEST 2 time: 1.904 sec.
TEST 3 time: 1.741 sec.
TEST 4 time: 2.043 sec.
TEST 5 time: 0.506 sec.
TEST 6 time: 5.449 sec.
TEST 7 time: 2.664 sec.

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.5) Gecko/20060714 Firefox/1.5.0.5

non-cairo 14.38 sec.
TEST 1 time: 1.797 sec.
TEST 2 time: 1.198 sec.
TEST 3 time: 1.632 sec.
TEST 4 time: 1.232 sec.
TEST 5 time: 0.504 sec.
TEST 6 time: 5.34 sec.
TEST 7 time: 2.678 sec.



results from notebook with Intel graphics and full 2D acceleration

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1b2) Gecko/20060829 BonEcho/2.0b2

non-cairo: 6.24 sec.
TEST 1 time: 1.331 sec.
TEST 2 time: 0.744 sec.
TEST 3 time: 0.568 sec.
TEST 4 time: 0.298 sec.
TEST 5 time: 0.198 sec.
TEST 6 time: 2.329 sec.
TEST 7 time: 0.774 sec.

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9a1) Gecko/20060829 Minefield/3.0a1

cairo: 9 sec.
TEST 1 time: 1.551 sec.
TEST 2 time: 0.665 sec.
TEST 3 time: 0.815 sec.
TEST 4 time: 1.705 sec.
TEST 5 time: 0.249 sec.
TEST 6 time: 3.131 sec.
TEST 7 time: 0.889 sec.

can you just for fun set MOZ_DISABLE_IMAGE_OPTIMIZE=1 env var and rerun?
current cairo build isn't so slow: 7.28 sec.

TEST 1 time: 1.599 sec.
TEST 2 time: 0.654 sec.
TEST 3 time: 0.774 sec.
TEST 4 time: 0.358 sec.
TEST 5 time: 0.264 sec.
TEST 6 time: 2.725 sec.
TEST 7 time: 0.881 sec.

MOZ_DISABLE_IMAGE_OPTIMIZE seems to have no effect on result.
Flags: blocking1.9?
This sounds like it's fixed?
Test 3 and 4 are still much slower with cairo build.

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.2pre) Gecko/20070121 BonEcho/2.0.0.2pre

6.4 sec.
TEST 1 time: 1.246 sec.
TEST 2 time: 0.743 sec.
TEST 3 time: 0.587 sec.
TEST 4 time: 0.322 sec.
TEST 5 time: 0.213 sec.
TEST 6 time: 2.409 sec.
TEST 7 time: 0.878 sec.

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9a2pre) Gecko/20070121 Minefield/3.0a2pre

7.27 sec.
TEST 1 time: 1.33 sec.
TEST 2 time: 0.927 sec.
TEST 3 time: 0.958 sec.
TEST 4 time: 0.575 sec.
TEST 5 time: 0.258 sec.
TEST 6 time: 2.975 sec.
TEST 7 time: 0.249 sec.

test times: cairo vs. non-cairo
1: 107 %
2: 125 %
3: 163 %
4: 179 %
5: 121 %
6: 124 %
7: 28 %

Marking this is blocking; need to do a lot more investigation on this to see what's chewing up the time.  It could all be just fallout from generally slow linux text rendering.
Flags: blocking1.9? → blocking1.9+
Target Milestone: --- → mozilla1.9beta1
Testcase is no longer available:

"24fun.com expired on 05/27/2007 and is pending renewal or deletion."


Testcase seems to be cached in my PC. I still can get results.

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.5pre) Gecko/20070608 BonEcho/2.0.0.5pre
TEST 1 time: 1.235 sec.
TEST 2 time: 0.737 sec.
TEST 3 time: 0.604 sec.
TEST 4 time: 0.302 sec.
TEST 5 time: 0.206 sec.
TEST 6 time: 2.394 sec.
TEST 7 time: 0.781 sec.

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9a5) Gecko/20070605 GranParadiso/3.0a5
TEST 1 time: 1.333 sec.
TEST 2 time: 1.224 sec.
TEST 3 time: 0.818 sec.
TEST 4 time: 0.474 sec.
TEST 5 time: 0.235 sec.
TEST 6 time: 2.596 sec.
TEST 7 time: 0.277 sec.

wbrana@gmail.com, could you please attach the testcase/page to the bug? Thanks!
Test case is composed from many files. I will try to get them from cache.
Attached file BenchJS source (obsolete) — 
Files from www.24fun.com found in my cache.
(In reply to comment #28)
> Created an attachment (id=270490) [details]
> BenchJS source
> 
> Files from www.24fun.com found in my cache.
> 
Images seem to be corrupted.
Priority: -- → P3
Target Milestone: mozilla1.9 M8 → ---
wbrana, what are the current results?

(In reply to comment #29)
> (In reply to comment #28)
> > Created an attachment (id=270490) [details] [details]
> Images seem to be corrupted.
> 
Does attachment 270490 [details] work properly?


I have different PC.
Results from nightly builds:

BonEcho/2.0.0.9pre
TEST 1 time: 1.119 sec.
TEST 2 time: 0.322 sec.
TEST 3 time: 0.359 sec.
TEST 4 time: 0.192 sec.
TEST 5 time: 0.117 sec.
TEST 6 time: 1.348 sec.
TEST 7 time: 0.473 sec.

Minefield/3.0a9pre
TEST 1 time: 1.217 sec.
TEST 2 time: 0.963 sec.
TEST 3 time: 0.525 sec.
TEST 4 time: 0.197 sec.
TEST 5 time: 0.149 sec.
TEST 6 time: 1.591 sec.
TEST 7 time: 0.142 sec.
Attached file BenchJS source — 
Images should be fixed.
Attachment #270490 - Attachment is obsolete: true
thanks wbrana !
Finally dug into the source here, to figure out what these tests actually are:

- Test 1 counts to a 10000 and displays the figures inside the status bar. At the same time it displays a red progress bar.
- Test 2 opens eight red popup-windows and closes them again.
- Test 3 replaces 300 tiny images as fast as possible. It repeats this procedure 10 times.
- Test 4 manipulates a long text in different ways.
- Test 5 creates 2000 table-cells and calculates a random backgroundcolor for each table-cell.
- Test 6 creates a phrase out of 50 different layers that are pulled together.
- Test 7 calculates the days of the week for the next 10000 x-mas. IMPORTANT: The test considers only the calculating-time NOT the time required to display the result.

We're 3x slower on test 2, just under 50% slower on test 3, and the others look comparable.  A little slower on test 6, much faster on test 7.
Assignee: nobody → vladimir
Hope this is still needed if not I'm sorry for the bugspam,  I have seen tremendous improvements lately.

Here are the figures for my PC (Ubuntu 7.10 gutsy 32bit):

FF 2.0.0.11 Clean profile
TEST 1 time: 1.162 sec.
TEST 2 time: 1.039 sec.
TEST 3 time: 0.505 sec.
TEST 4 time: 1.173 sec.
TEST 5 time: 0.132 sec.
TEST 6 time: 2.027 sec.
TEST 7 time: 0.342 sec.

FF 3.0Beta2 Clean profile
TEST 1 time: 1.292 sec.
TEST 2 time: 0.829 sec.
TEST 3 time: 0.38 sec.
TEST 4 time: 0.606 sec.
TEST 5 time: 0.171 sec.
TEST 6 time: 1.473 sec.
TEST 7 time: 0.174 sec.

FF nightly (Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9b3pre) Gecko/2008012304 Minefield/3.0b3pre) Clean
TEST 1 time: 1.295 sec.
TEST 2 time: 0.765 sec.
TEST 3 time: 0.398 sec.
TEST 4 time: 0.598 sec.
TEST 5 time: 0.168 sec.
TEST 6 time: 1.431 sec.
TEST 7 time: 0.173 sec.
Thanks for those numbers, Martijn!
It seems that current trunk build is faster in every test. Can someone else confirm?
I just tested this (P4 2.8 GHz). New profiles, blank home page, pop-up blocking disabled. Run locally from benchjs/benchjs/www.24fun.com/downloadcenter/benchjs/benchjs.html Six sets of results, quit browser between each one, dropped the first, minimum and maximum results from each test and averaged the other 3. Here are the results.

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11

Test 1: 1.179
Test 2: 3.445
Test 3: 1.076
Test 4: 0.739
Test 5: 0.254
Test 6: 3.732
Test 7: 1.497
Total: 11.922

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9b3pre) Gecko/2008012304 Minefield/3.0b3pre

Test 1: 1.486
Test 2: 2.361
Test 3: 1.584
Test 4: 0.778
Test 5: 0.287
Test 6: 3.781
Test 7: 0.307
Total: 10.584

Change

Test 1: 26.039% slower
Test 2: 31.459% quicker
Test 3: 47.289% slower
Test 4:  5.232% slower
Test 5: 12.975% slower
Test 6:  1.304% slower
Test 7: 79.510% quicker
Total:  11.220% quicker

So, quicker overall, but a bit slow on test 5, quite slow on test 1 and almost 50% slower on test 3. Test 2 is quicker and test 7 is much quicker.

I think some more people need to test, as results are varied. Martijn K shows test 3 being quicker now whereas Vlad and I both show a 50% decrease in speed... Vlad and wbrana show a speed decrease in test 2 whereas I see a big increase (although this test probably has less to do with Firefox than the others do; maybe WM related?).

In response to comment 36, for me trunk is _slower_ in every test apart from tests 2 and 7. I'd say this bug should be a meta-bug and bugs filed on the issues causing slowdown. If anyone wants me to do some more work just ask.

Some more data this time From an Imac G5 1 GB ram. If I can test anything let me know should be able to get Windows XP data also if needed.

Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11
TEST 1 time: 1.516 sec.
TEST 2 time: 3.96 sec.
TEST 3 time: 3.211 sec.
TEST 4 time: 1.447 sec.
TEST 5 time: 0.62 sec.
TEST 6 time: 4.421 sec.
TEST 7 time: 1.402 sec.

Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.5; en-US; rv:1.9b2) Gecko/2007121014 Firefox/3.0b2
TEST 1 time: 3.077 sec.
TEST 2 time: 4.168 sec.
TEST 3 time: 3.961 sec.
TEST 4 time: 2.396 sec.
TEST 5 time: 0.546 sec.
TEST 6 time: 4.337 sec.
TEST 7 time: 0.784 sec

Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.5; en-US; rv:1.9b3pre) Gecko/2008012304 Minefield/3.0b3pre
TEST 1 time: 2.884 sec.
TEST 2 time: 3.771 sec.
TEST 3 time: 4.056 sec.
TEST 4 time: 2.303 sec.
TEST 5 time: 0.541 sec.
TEST 6 time: 3.851 sec.
TEST 7 time: 0.727 sec.
given current results, marking this blocking-
Flags: wanted1.9+
Flags: blocking1.9-
Flags: blocking1.9+
I tried with clean profile instead of my FX2 profile.

Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.13pre) Gecko/20080216 BonEcho/2.0.0.13pre
TEST 1 time: 1.107 sec.
TEST 2 time: 0.788 sec.
TEST 3 time: 0.345 sec.
TEST 4 time: 0.299 sec.
TEST 5 time: 0.105 sec.
TEST 6 time: 1.297 sec.
TEST 7 time: 0.452 sec.

Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9b4pre) Gecko/2008021604 Minefield/3.0b4pre
TEST 1 time: 1.123 sec.
TEST 2 time: 0.567 sec.
TEST 3 time: 0.433 sec.
TEST 4 time: 0.452 sec.
TEST 5 time: 0.115 sec.
TEST 6 time: 1.115 sec.
TEST 7 time: 0.13 sec.

Bug is fixed for me.
Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9pre) Gecko/2008041304 Minefield/3.0pre
TEST 1 time: 1.092 sec.
TEST 2 time: 0.459 sec.
TEST 3 time: 0.365 sec.
TEST 4 time: 0.243 sec.
TEST 5 time: 0.089 sec.
TEST 6 time: 1.017 sec.
TEST 7 time: 0.106 sec.

Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.15pre) Gecko/20080413 BonEcho/2.0.0.15pre
TEST 1 time: 1.113 sec.
TEST 2 time: 0.726 sec.
TEST 3 time: 0.403 sec.
TEST 4 time: 0.272 sec.
TEST 5 time: 0.104 sec.
TEST 6 time: 1.232 sec.
TEST 7 time: 0.429 sec.
Status: NEW → RESOLVED
Closed: 16 years ago
Resolution: --- → FIXED
->WORKSFORME since we're not sure exactly what patches fixed this.
Resolution: FIXED → WORKSFORME
I ran this again today using 3 Linux nightly builds from the 1st June 2008 (chosen so that there should be no difference between 3.0 and 3.1 builds and so the 3.0 build should be the same as RC2). Same method as in comment 37 with the same 2.8 GHz P4.

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.15pre) Gecko/20080601 BonEcho/2.0.0.15pre

Test  Average
1      1.184
2      3.425
3      1.073
4      0.773
5      0.267
6      3.632
7      1.495
Total 11.848

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9pre) Gecko/2008060104 Minefield/3.0pre

Test  Average
1     1.276
2     2.163
3     1.236
4     0.610
5     0.222
6     2.916
7     0.250
Total 8.675

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1a1pre) Gecko/2008060102 Minefield/3.1a1pre

Test  Average
1     1.265
2     2.168
3     1.254
4     0.621
5     0.227
6     2.893
7     0.253
Total 8.681

Difference between 3.1 and 3.0 builds is less than 2% (probably within noise) for every test and less than 0.1% overall which is good news:

Test  % Difference to 3.0
1     0.862 % quicker
2     0.231 % slower
3     1.402 % slower
4     1.803 % slower
5     1.949 % slower
6     0.789 % quicker
7     1.065 % slower
Total 0.073 % slower

Difference between 3.0 and 2.0:

Test  % Difference to 2.0
1      7.740 % slower
2     36.837 % quicker
3     15.258 % slower
4     21.053 % quicker
5     16.625 % quicker
6     19.712 % quicker
7     83.252 % quicker
Total 26.786 % quicker

So overall 3.0 is more than 25 % quicker than 2.0 so I'm verifying this bug ('"BenchJS" DHTML benchmark 50% slower with linux cairo build').

Test 3 is more than 15 % slower and test 1 is almost 8 % slower so I will try and minimise these and post new bugs soon. The other tests are significantly quicker. To quote vlad from comment 34:

"- Test 3 replaces 300 tiny images as fast as possible. It repeats this
procedure 10 times.
- Test 1 counts to a 10000 and displays the figures inside the status bar. At
the same time it displays a red progress bar."
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: