571256 - Need asmicro/ (analogous to jsmicro/) AS3 microbenchmark directory

Status: RESOLVED FIXED

(Tamarin Graveyard :: Tools, enhancement, P3)

Description

[Felix S. Klock II [:pnkfelix, :fklock]]

Some micro-benchmarks should have AS3 typed variants, so that we can really dig into where time is going.

Lars mentioned that he's been meaning to add asmicro/ in the chat room; seems as good a name as any it.

Comment 1

[Felix S. Klock II [:pnkfelix, :fklock]]

Attachment: Initial asmicro/ directory

Ported jsmicro/ to asmicro/.

(This patch may or may not confuse bugzilla ... I'm having issues applying it via "patch -p1" ... let me know if it gives you trouble...)

Comment 2

[Felix S. Klock II [:pnkfelix, :fklock]]

(In reply to comment #1)
> (This patch may or may not confuse bugzilla ... I'm having issues applying it
> via "patch -p1" ... let me know if it gives you trouble...)

yeah, the combination of copying and then modifying the copy is confusing a
bunch of the parties involved here.  Tomorrow, I'll figure out if there's a way
to make this clearer.

("hg import --no-commit" should be a sane way to apply the patch to your tree
in the meantime...)

Comment 3

[Lars T Hansen]

Memo to self, need to copy over the benchmarks from bug #569321, pertaining to ...rest arguments - they can't go into the jsmicro suite.

Comment 4

[Felix S. Klock II [:pnkfelix, :fklock]]

Attachment: Use AVM environment variable if available rather than hardcoded path

Maybe the makefile here is obsolete, but it was useful for me to make this work for my setup, and I believe this is a generalization that should apply to you as well.

Unless you leave AVM environment variable set to a different value than the local shell...

Comment 5

[Felix S. Klock II [:pnkfelix, :fklock]]

Attachment: Initial copy from jsmicro/ into asmicro/

This is just an initial copy from jsmicro/ into asmicro/.  I have teased the copy out into its own patch in order to keep bugzilla less confused.

It is meant to be applied after the generalization of the jsmicro/Makefile.

Comment 6

[Felix S. Klock II [:pnkfelix, :fklock]]

Attachment: Add type annotations to asmicro/, and compile all in -strict mode

This is the meat of the work (and in essence the point of the ticket).

With these changes, every benchmark remains approximately as fast as it was before the type annotations were added, and some are much faster (that is, they perform many more iterations in the one second they run for).  For example, array-1 is able to perform 6x as many iterations.

The main exception (that I do not chalk to measurement noise) is array-sort-4.  I have not looked into what happened there.

I've pasted the measurement results below; I'll attach a csv file with the raw data separately.

" MICRO BENCHMARK             "	js avg	as avg	rel speedup
" alloc-1                     "	33.67	34.33	1.02
" alloc-2                     "	17.00	17.67	1.04
" alloc-3                     "	15.00	14.33	0.96
" alloc-4                     "	41.00	43.67	1.07
" alloc-5                     "	27.00	27.67	1.02
" alloc-6                     "	51.33	59.33	1.16
" alloc-7                     "	35.00	35.67	1.02
" alloc-8                     "	12.67	14.00	1.11
" alloc-9                     "	12.33	14.00	1.14
" alloc-10                    "	12.00	13.67	1.14
" alloc-11                    "	11.00	12.67	1.15
" alloc-12                    "	7.00	8.00	1.14
" alloc-13                    "	63.67	75.00	1.18
" alloc-13                    "	56.33	65.67	1.17
" array-1                     "	323.33	2051.33	6.34
" array-2                     "	230.00	606.00	2.63
" array-push-1                "	21.00	248.00	11.81
" array-pop-1                 "	57.33	375.67	6.55
" array-shift-1               "	52.00	122.00	2.35
" array-slice-1               "	13.00	16.00	1.23
" array-sort-1                "	22.33	24.00	1.07
" array-sort-2                "	4.00	4.00	1.00
" array-sort-3                "	18.00	18.33	1.02
" array-sort-4                "	10.00	9.00	0.90
" array-unshift-1             "	20.67	135.00	6.53
" do-1                        "	503.67	3951.00	7.84
" for-1                       "	496.67	3953.00	7.96
" for-2                       "	166.00	2660.33	16.03
" for-3                       "	79.00	2755.00	34.87
" for-in-1                    "	256.33	368.67	1.44
" for-in-2                    "	167.67	213.67	1.27
" funcall-1                   "	202.67	350.67	1.73
" funcall-2                   "	188.67	217.00	1.15
" funcall-3                   "	187.00	302.67	1.62
" funcall-4                   "	96.00	98.67	1.03
" isNaN-1                     "	217.00	489.00	2.25
" number-toString-1           "	7.00	7.00	1.00
" number-toString-1           "	51.00	54.33	1.07
" oop-1                       "	4.00	4.00	1.00
" parseFloat-1                "	33.00	74.00	2.24
" parseInt-1                  "	98.67	146.33	1.48
" string-casechange-1         "	16.67	23.00	1.38
" string-casechange-2         "	16.00	22.33	1.40
" string-charAt-1             "	76.33	461.33	6.04
" string-charAt-2             "	36.67	59.67	1.63
" string-charCodeAt-1         "	69.00	385.67	5.59
" string-charCodeAt-2         "	72.00	381.00	5.29
" string-fromCharCode-1       "	20.00	220.00	11.00
" string-fromCharCode-2       "	14.00	51.67	3.69
" string-indexOf-1            "	67.33	166.67	2.48
" string-indexOf-2            "	54.00	157.00	2.91
" string-indexOf-3            "	36.00	72.00	2.00
" string-lastIndexOf-1        "	69.00	528.67	7.66
" string-lastIndexOf-2        "	52.67	161.00	3.06
" string-lastIndexOf-3        "	64.67	356.00	5.51
" string-slice-1              "	42.00	107.00	2.55
" string-split-1              "	8.33	8.00	0.96
" string-split-2              "	8.00	8.00	1.00
" string-substring-1          "	43.33	111.67	2.58
" switch-1                    "	98.00	817.67	8.34
" switch-2                    "	34.67	37.67	1.09
" switch-3                    "	38.00	41.00	1.08
" try-1                       "	130.33	218.33	1.68
" try-2                       "	13.00	13.67	1.05
" try-3                       "	36.00	44.00	1.22
" while-1                     "	487.33	3952.33	8.11

Comment 7

[Felix S. Klock II [:pnkfelix, :fklock]]

Attachment: measurement data (exported from excel)

Original measurement data driving report in comment 6.

Comment 8

[Felix S. Klock II [:pnkfelix, :fklock]]

Attachment: Extend testconfig.txt with asmicro/

After adding, here's a useful bit a shell to get a printout of jsmicro/asmicro benchmarking:

% python runtests.py -f `for f in jsmicro/*-[0-9].as ; do echo $f ; echo $f | sed -e s/jsmicro/asmicro/ ; done | xargs`

(Its still inferior in the end to doing two separate runs and then post-processing the results to get raw numbers to feed into Excel, but the above is quick and dirty, which may be more appropriate for micro-benchmarks rather that attaching too much value to the gathered data.)

Comment 9

[Lars T Hansen]

Comment on attachment 450647 [details] [diff] [review]
Add type annotations to asmicro/, and compile all in -strict mode

This is OK, but as for jsmicro we must be aware that the test driver compiles with -strict -AS3 (if I am not mistaken).  So we want a dir.asc_args (if that's the name I'm looking for) to control switches in the asmicro directory as well as in the jsmicro repository.

I'll let you take care of the asmicro repository, I'll see if I can't deal with the other one myself.

Comment 10

[Felix S. Klock II [:pnkfelix, :fklock]]

Comment on attachment 450642 [details] [diff] [review]
Use AVM environment variable if available rather than hardcoded path

Lars: can you push this separately?  (I want to avoid issues with interactions between "hg copy".  (Or if you want, I can generate the necessary changeset via hg export, but I believe we're trying to avoid using that because it confuses bugzilla...)

Comment 11

[Felix S. Klock II [:pnkfelix, :fklock]]

(In reply to comment #10)
> (From update of attachment 450642 [details] [diff] [review])
> Lars: can you push this separately?  (I want to avoid issues with interactions
> between "hg copy". 

Alternatively, if you think it a mistake to relate jsmicro/Makefile and asmicro/Makefile via "hg copy", just say so and I'll make a big patch that seems to add everything from thin air.

Comment 12

[Lars T Hansen]

I think I like my files in trees, not graphs, even if it means I have to maintain them individually.

(Tell me what to push and I'll take care of it.)

Comment 13

[Felix S. Klock II [:pnkfelix, :fklock]]

(In reply to comment #12)
> I think I like my files in trees, not graphs, even if it means I have to
> maintain them individually.
> 
> (Tell me what to push and I'll take care of it.)

Heh.  The first part of your answer made me think that you wanted the big diff (that looks like its adding many files without isolating the deltas).  The second answer makes me think that you don't actually care, or rather, that you'll take whatever I hand you and strip out the copying actions and whatnot.

Anyway, I will simply upload the rebased patches and request a push.

Comment 14

[Felix S. Klock II [:pnkfelix, :fklock]]

Attachment: Use AVM environment variable if available rather than hardcoded path

(this should match patch that was checked off earlier; I just renamed it to fit the convention I am using to order all of these patches.)

Comment 15

[Felix S. Klock II [:pnkfelix, :fklock]]

Attachment: Copy from jsmicro/ into asmicro/

Copies all benchmarks from jsmicro into analogues in asmicro.  (Feel free to squash this in with all the other patches.)

Comment 16

[Felix S. Klock II [:pnkfelix, :fklock]]

Attachment: Add type annotations to asmicro/, and compile all in -strict mode

(just like the others, if you want to squash them together, feel free)

Comment 17

[Felix S. Klock II [:pnkfelix, :fklock]]

Attachment: Extend testconfig.txt with asmicro/

(If you want to squash with other patches, feel free.)

Comment 18

[Felix S. Klock II [:pnkfelix, :fklock]]

Attachment: a readme for the asmicro/ benchmarks.

This is maybe the only patch for this ticket from today that actually warrants a real review request.  I wrote up this readme a while ago and then revised it a touch today...

Lars: if you think it poor form to refer the reader back over to jsmicro/README.txt for the actual descriptions of the base test set, let me know.

Comment 19

[Lars T Hansen]

Comment on attachment 453139 [details] [diff] [review]
a readme for the asmicro/ benchmarks.

I may modify this file when I land it, there is no reason to *require* type annotations, in fact there are good reasons for many of the tests here to be untyped or partly-typed.

Comment 20

[Lars T Hansen]

Comment on attachment 453136 [details] [diff] [review]
Add type annotations to asmicro/, and compile all in -strict mode

I admit to only skimming this.

Comment 21

[Lars T Hansen]

(In reply to comment #18)
> 
> Lars: if you think it poor form to refer the reader back over to
> jsmicro/README.txt for the actual descriptions of the base test set, let me
> know.

It is poor form, I expect these test suites to diverge quite quickly.

Comment 22

[Lars T Hansen]

There's an issue (won't call it a bug yet) here: it's not possible to compile these benchmarks in strict mode.  The problem is not with the benchmarks but with the runtests.py driver: it does not import the shell_toplevel.abc file during compilation, so references to 'print' cause compilation errors.

cc'ing Brent.

Comment 23

[Lars T Hansen]

tamarin-redux:

changeset:   4866:739fbd5c4fc7
changeset:   4865:e0d649e402d4
changeset:   4864:a3f43a09bac4
changeset:   4863:97b62b255d2c
changeset:   4862:9993a590775f
changeset:   4861:fba5671e7e57

Comment 24

[Brent Baker]

(In reply to comment #22)
> There's an issue (won't call it a bug yet) here: it's not possible to compile
> these benchmarks in strict mode.  The problem is not with the benchmarks but
> with the runtests.py driver: it does not import the shell_toplevel.abc file
> during compilation, so references to 'print' cause compilation errors.
> 
> cc'ing Brent.

Should just change dir_aasc_args for asmicro to the following:
merge| -import $SHELLABC -strict

Comment 25

[Lars T Hansen]

(In reply to comment #24)
> (In reply to comment #22)
> > There's an issue (won't call it a bug yet) here: it's not possible to compile
> > these benchmarks in strict mode.  The problem is not with the benchmarks but
> > with the runtests.py driver: it does not import the shell_toplevel.abc file
> > during compilation, so references to 'print' cause compilation errors.
> > 
> > cc'ing Brent.
> 
> Should just change dir_aasc_args for asmicro to the following:
> merge| -import $SHELLABC -strict

Thanks!  I will attempt that.