Closed Bug 482076 Opened 17 years ago Closed 14 years ago

TM: Investigate performance regression of test case for bug 458016 in 1.9.2 branch as compared to 1.9.1

Categories

(Core :: JavaScript Engine, defect)

Product:
Core
Component:
JavaScript Engine
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED WORKSFORME

People

(Reporter: duncan.loveday, Unassigned)

Details

(Keywords: perf, regression, testcase)

User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.0.6) Gecko/2009011913 Firefox/3.0.6 Build Identifier: Using Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2a1pre) Gecko/20090307 Minefield/3.2a1pre Performance of the test case from bug 458016 appears slower by a factor of 3 or more in the latest tracemonkey builds. Needs a regression range. Reproducible: Always
I went to the earliest tracemonkey build I could find here http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2009-02-04-02-tracemonkey/ and performance is about the same as the latest tracemonkey build from http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-tracemonkey/. Can I get earlier tracemonkey builds builds to try ?
Just to put some numbers on this: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b4pre) Gecko/20090307 Shiretoko/3.1b4pre 3779 3867 4469 4441 3776 4431 3779 4461 4439 3789 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2a1pre) Gecko/20090307 Minefield/3.2a1pre 9174 8598 9283 8578 9296 9356 8593 9469 8607 9372
FWIW Testing with JIT disabled
Keywords: perf, regression, testcase
There's a noticeable regression is between the two builds below, from 4-5 secs to 8-9 secs. 2 day range as there's no windows build for 8/1. I think there might be smaller perf regressions before this range also as the numbers for the earlier 1 are 4-5 seconds whereas the latest 1.9.1 branch builds 3-4. http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2009/01/2009-01-07-02-tracemonkey/ and http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2009/01/2009-01-09-02-tracemonkey/
(In reply to comment #4) > I think there might be smaller perf regressions before this range also as the numbers for the earlier 1 are 4-5 seconds whereas the latest 1.9.1 branch builds 3-4. The earliest tracemonkey 1.9.2 build seems to be 2008-12-06, before that they are 1.9.1. Performance here is in the range 4-5s, same as just before the regression above. However the tip of 1.9.1 seems a bit faster, 3.5-4.5. So it's as if some speedup has occurred in 1.9.1 but not 1.9.2 which is odd.
Flags: wanted1.9.2?
Flags: wanted1.9.1?
Flags: wanted1.8.1.x?
Flags: wanted-fennec1.0?
Flags: in-testsuite?
Flags: in-litmus?
Summary: Investigate performance regression of test case for bug 458016 in 1.9.2 branch as compared to 1.9.1 → TM: Investigate performance regression of test case for bug 458016 in 1.9.2 branch as compared to 1.9.1
Re Change of title: There's nothing here to suggest this is anything to do with JIT is there ?
Given that you're specifically mentioning the regression appearing in Tracemonkey builds, it's best to put the TM on the bug. Otherwise, the right people might not notice it.
Ryan: cc'ing is better than just adding TM: and hoping. It may be that this bug should lose the TM: prefix. Duncan, anyone: do we need a separate bug on lack of speedup with JIT enabled? /be
OS: Windows XP → All
Hardware: x86 → All
> Duncan, anyone: do we need a separate bug on lack of speedup with JIT enabled? The original bug 458016 and several other dependent bugs are about lack of speedup on the 1.9.1 branch when enabling JIT for this test case. I would hope that when this bug is fixed, performance of 1.9.2 tip (with or without JIT) will at least match 1.9.1 (ditto), if not then yes another bug would be called for. I suppose we could raise one now along the lines of "Make sure 1.9.2 perf matches 1.9.1 for test case from bug 458016 after bug 482076 is fixed." and have it dependent on this one ?
(In reply to comment #7) > Given that you're specifically mentioning the regression appearing in > Tracemonkey builds, it's best to put the TM on the bug. Otherwise, the right > people might not notice it. Ryan, my point is the slowdown if there even with JIT disabled albeit in a TM build it may be a change unrelated to TM that is the culprit, assuming there are other spidermonkey changes on the 1.9.2 branch.
This seems to be much better now JIT OFF JIT ON 3215 3310 3544 4545 4437 3956 3708 7011 3418 4960 3947 3935 3341 3972 3674 6948 4457 3729 3879 4336 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2a1pre) Gecko/20090406 Minefield/3.6a1pre
Here's a side by side comparison. The numbers are much much closer than they were before. 1.9.2 1.9.2 1.9.1 1.9.1 JIT OFF JIT ON JIT OFF JIT ON 3215 3310 3023 3585 3544 4545 2642 4375 4437 3956 3664 3213 3708 7011 3415 4621 3418 4960 4559 3900 3947 3935 3534 3387 3341 3972 4647 5600 3674 6948 4563 2842 4457 3729 3118 3490 3879 4336 4265 3215 ===== ===== ===== ===== 37620 46702 37430 38228 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2a1pre) Gecko/20090406 Minefield/3.6a1pre vs Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b4pre) Gecko/20090406 Shiretoko/3.5b4pre
No need to play with all the flags...
Flags: wanted1.9.2?
Flags: wanted1.9.1?
Flags: wanted1.8.1.x?
Flags: wanted-fennec1.0?
Flags: in-testsuite?
Flags: in-litmus?
Looking at the numbers in comment #12 I think this should be closed WFM. Perhaps I'll run the side by side comparison once more first.
Can this be closed now (did you run the test again)? Confirming as this is/was an investigative bug, perhaps needing attention.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Version: unspecified → Trunk
I'll try to re-run tomorrow, hopefully the results will show we can close it.
Testing on windows XP with a clean profile and a browser restart between each group of 10 runs, sorry to have to report that the results are not what we'd hoped. My crude interpretation is that whilst JIT has got progressively better - from adding 30% to adding 21% to saving 40% - the non-JIT numbers have got progressively slower, particularly so in 1.9.3 which shows a 50% slowdown compared to 1.9.1 in this latest test. 1.9.3 is substantially faster with JIT than without but the non-JIT timing is so much higher that it's still slower than 1.9.1 without JIT. 1.9.1 1.9.1 1.9.2 1.9.2 1.9.3 1.9.3 No JIT JIT No JIT JIT No JIT JIT 3922 5753 3611 5622 6583 3675 3680 5944 3700 6202 6833 4400 5097 4629 3930 4874 6152 4718 4861 5991 4285 4709 7129 5764 4022 5987 3805 3910 6835 4513 4427 5453 4415 5058 6551 3549 4165 5993 4804 5574 6472 4696 4799 5987 3649 5303 6068 4978 3929 4202 4005 4998 6509 4896 4070 5989 4746 3698 5012 5717 ===== ===== ===== ===== ===== ===== 42972 55928 40950 49948 64144 46906 0.00% +30.15% -4.71% +16.23% +49.27% +9.15% Versions used were 1.9.3: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.3a1pre) Gecko/20090817 Minefield/3.7a1pre (.NET CLR 3.5.30729) 1.9.2: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2a2pre) Gecko/20090817 Namoroka/3.6a2pre (.NET CLR 3.5.30729) 1.9.1: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.3pre) Gecko/20090817 Shiretoko/3.5.3pre (.NET CLR 3.5.30729)
I go from 2100 to 1300 on windows by turning the JITs on, on the test here, and that's with the slow windows Date behaviour still. Duncan: can you confirm that the trunk now rules?
There's no question that JIT is beneficial for this test case in the trunk builds, see below, but what I was actually reporting here was that the non-JIT times got a whole lot slower - see comment 2 and comment 4 - and that's still the case. I did notice just now that this test fills the error console with messages about undefined properties being referenced which I don't think used to happen. So perhaps that might be impacting performance ? Fx4 No JIT Fx4 JIT 7560 2239 8716 2481 9247 1613 7083 1646 8129 1768
Duncan, how does JM+TI fare these days?
Using Fx8, disabling trace and method JIT times on the same windows laptop as used for my previous tests are in the range 1.5 to 2.5 seconds so that is much faster than what I saw before with 1.9.x. With trace and method JIT enabled times are in the range 1 to 1.5 seconds which is also a whole lot better.
So JITs enabled leads to a perf increase. Going to go ahead and call this WORKSFORME unless you object.
Fine by me. The bug is not about JIT though, it is about a perf regression when JIT is disabled as per comment 3. Either way it is much better now so resolving as WFM seems appropriate.
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.