523104 - Performance drop in scimark on windows mobile -Dinterp with changeset 2748

Status: VERIFIED WORKSFORME

(Tamarin Graveyard :: Virtual Machine, defect, P2)

Description

[Brent Baker]

There was a noticeable performance drop in the scimark testsuite with changeset 2748 when running on the winmo Fuze device with -Dinterp.

There was no noticeable change in hybrid or jit with this change and there was been no noticeable change on the desktop platforms either.

This was also tested on the TG01 and there was no performance drop. Issue is only reproducible on the Fuze device.

These results were generated on the HTC Fuze with test/performance/dir.asc_args set to CONFIG::desktop=false


-Dinterp                   r2747   r2748    %diff
scimark/FFT.as             33019   40118    -17.6%
scimark/LU.as              55802   67120    -16.7%
scimark/MonteCarlo.as      2976    3457     -13.6%
scimark/SOR.as             33831   40474    -16.4%
scimark/SparseCompRow.as   2855    3314     -13.9%

Comment 1

[Trevor Baker]

Change in question:
http://hg.mozilla.org/tamarin-redux/rev/2748

Comment 2

[Lars T Hansen]

Looking at the patch, there are only three things that i can think of that would result in a significant slowdown:

- one level of function call to get to strlen() that needed to be introduced
- 64-bit integer operations that needed to be introduced on string construction
  paths
- increased cache misses (notably icache)

The first two don't make sense to me.

The third is too much like "the hardware did it" for comfort.  But i can't think of anything else: even assuming string operations are hot (for example, if we missed cases of optimizing the int->string case for array indices, which is the only idea I've had so far), the first two items should not be that costly.

Comment 3

[Steven Johnson]

Are we measuring FLOPS or raw time here? (The Flash version of Scimark tests is coded in such a way that execution time may vary but FLOPS are accurate. I don't know if the same is true of the AS3 versions.)

Comment 4

[Brent Baker]

The version of the tests that we have measure time.

Comment 5

[Brent Baker]

re comment #4: The tests are setup to do a known amount of work/iterations. The amount of work/iterations is constant.

Comment 6

[Dan Smith]

Open to Lars for further investigation.

Comment 7

[Lars T Hansen]

Hm, there's a bug in that patch.  The use of ptrdiff_t is incorrect because ptrdiff_t is signed.  The correct test would use size_t.

(Not that I think this has an impact on the performance.)

Comment 8

[Lars T Hansen]

Looks like the experiences in bug #523400 could have bearing on this bug, even though this bug is interpreter-only and that's a JIT issue.

Comment 9

[Brent Baker]

Attempted to reproduce the data with having the eval feature disabled, but trying to run winmo without eval introduced a new issue: bug# 534908

Once this is resolved, we can try and back port the fix and attempt to get numbers for scimark on rev 2747 and 2748 with the eval feature disabled.

Comment 10

[Lars T Hansen]

Methodology: WinMo Release build running on my Fuze under control of Visual Studio with abc files preloaded on the device and located in the root directory.  The VM shell was built without eval support, no other changes.  Times reported below are middle-of-three, manually recorded from the program output.  Variations were small, no more than about 2% between fastest and slowest.

     r2747    r2748

FFT  27126    25996
LU   37711    38301
SOR  27206    27021

Note a couple of things:

 - No crashes were seen in eval-less builds
 - The numbers reported above are significantly faster than the ones
   reported by Brent earlier
 - In my measurements, 2748 is /faster/ for both FFT and SOR and the
   slowdown on LU is very slight.

I will now measure again with eval enabled to see if I can reproduce Brent's results.

Comment 11

[Lars T Hansen]

(I should note that the ABC files were taken from the current tamarin-redux, compiled with the latest ASC, and compiled with the latest ASC settings and test driver - so there's a variation point there.)

Comment 12

[Lars T Hansen]

The same revisions with eval included (but not used, of course); same conditions, but single runs only:

      r2747   r2748
FFT   30208   30893
LU    38333   39761
SOR   31386   29922

From this I conclude that:

 - code size can make a significant difference (look at FFT and SOR times
   in particular) even if that code is not involved at all in the execution
 - Brent's original slowdown is not observed in these experiments

Hypotheses I've not tested include slowdown by the benchmarking framework, or by different ROM version (my Fuze has not been reflashed umpteen times, it's pretty much in its original state) or by changes in the behavior of the phone when it's tethered vs when it's not.