523400 - JIT performance drop with simple code change on Fuze

Status: VERIFIED WONTFIX

(Tamarin Graveyard :: Baseline JIT (CodegenLIR), defect, P3)

Description

[Brent Baker]

A small change in a performance test makes a huge difference when running on the Fuze. This is only happening with the jit enabled, using the interpreter shows no difference in performance.
 
The change:
In performance/scimark/LU.as I have this change:
 
VERSION 1:
else {// mobile
   print("metric time "+(getTimer()-starttime));
}

VERSION2:
else {// mobile
   var endtime:int = 12845;
   print("metric time "+ endtime);
}

When I run the two different versions I am getting drastically different numbers:
 
Version 1: ~26 seconds
Version 2: ~30 seconds


Setup Fuze for running acceptance/performance following directions in utils/wmremote/readme.txt

Steps to repo:
1) modify test/performance/dir.asc_args and set 'CONFIG::desktop=false'
2) run the following performance test on the Fuze scimark/LU.as
   should run in around 26 seconds
3) modify the test at line 318:
else {// mobile
    var endtime:int = 12845;
    print("metric time "+(getTimer()-starttime));
}
4) run the testcase, should run in 30 seconds

Comment 1

[Brent Baker]

Attachment: Base patch to testcase

Apply this patch as the base change. With this patch the testcase should run in ~26 seconds

Comment 2

[Brent Baker]

Attachment: base scimark/LU.abc

Comment 3

[Brent Baker]

Attachment: Modify testcase

One line change to the testcase, with this change the testcase will run in ~30 seconds

Comment 4

[Brent Baker]

Attachment: modified scimark/LU.abc

Comment 5

[Dan Smith]

Open to Edwin for further investigation..

Comment 6

[Edwin Smith]

on the beagleboard, the modified version is consistently 300ms faster.  (odd).  With softfloat vs VFP, the times differ between the two test cases as well.

tried another configuration by inserting

   if (A == null) { // always false
      print("never here")
   }

this only causes a different amount of code to be generated; all objects have the same slots compared to the baseline.  moving code around materially changes the test run time:

baseline:

    print("metric time "+(getTimer()-starttime));

example1:

    if (A == null) {
        print("never runs")
    }
    print("metric time "+(getTimer()-starttime));

example2:

    print("metric time "+(getTimer()-starttime));
    if (A == null) {
        print("never runs")
    }

example2 is consistenly slower than example1, and example1 and 2 are both faster than the baseline.  The only part of the system being poked is code size, and the JIT is known to insert jumps on demand between pages.  there's always a chance of a page jump being inserted anywhere, maybe inside a loop body.  so we could be just straddling a page size boundary that lands in the critical section of the benchmark.

I also used -Dnoincgc to make the test more deterministic but the effect still occurred.

Comment 7

[Edwin Smith]

Attachment: Hack to force every method to be jit'd on its own 4K page

This is a hack that makes each method start on a fresh page.  assuming no method is larger than 4K, no page breaks will land in critical sections.

with this patch, test times are no long sensitive to rearranging code.

a mystery remains: the "stable" test times are slower than the fastest time observed before the hack.

if this patch also stabilises performance on winmo (Fuze) then we can conclude it's due to code layout.

Comment 8

[Brent Baker]

Test results running tr rev 3331 with and without patch hack on a Fuze:

./ceremoteshell.exe -Dnoincgc c:/temp/LU_base.abc;
./ceremoteshell.exe -Dnoincgc c:/temp/LU_mod.abc;

without patch:
lu_base: 27440 [27879,29931,27440]
lu_mod:  28091 [28091,29895,30918]

with patch:
lu_base: 27657 [30462,27657,27736]
lu_mod:  26401 [26621,26401,26486]

Comment 9

[Edwin Smith]

can you run again, with lu_mod changed slightly?  instead of 

   end:int = 12354  (which inserts code and changes the global object size)

insert this:

   if (A == null) { print("never here") }

which inserts code that has no effect at all.

still, from the data above i think we're chasing a ghost.  clearly something is happening and we're affecting it, but code shape could still be a red herring.

Comment 10

[Brent Baker]

Rerun with code change from comment #9

without hack
lu_base: 28057 [28448, 28057, 28176]
lu_mod:  28046 [30135, 28046, 29810]


with hack
lu_base: 26044 [29209, 26484, 26044]
lu_mod:  25800 [26674, 26512, 25800]

Comment 11

[Trevor Baker]

bulk verifying resolved !fixed issues