487841 - update differential testing script avmdiff.py to return all PASSES/FAILS

Status: VERIFIED FIXED

(Tamarin Graveyard :: Virtual Machine, defect)

Description

[Dan Schaffer]

update avmdiff.py, if all configs match return the output, otherwise return the diff

there are still a few cases of scrubbing:
(1) strip runtime errors after Error: Error #[09]+ since release does not show error detail
(2) strip stack traces from runtime errors, only Debugger builds show stacktraces
(3) strip all lines '^metric', all time related or changing output can be prefixed with metric

Comment 1

[Dan Schaffer]

avmdiff.py does not pass all vm args to each vm, only passes the .abc name
e.g.  avmdiff.py -memstats test.abc should pass -memstats to each vm

Comment 2

[Chris Peyer]

Attachment: have avmdiff pass command line options to avmshell

There is a slight issue with patch when running avmdiff from the command line, if passing in a value for buildfile, the buildfile switch must use an =:

avmdiff --buildfile=file.txt

using a space will pass file.txt to avmshell

Comment 3

[Dan Schaffer]

Comment on attachment 465668 [details] [diff] [review]
have avmdiff pass command line options to avmshell

looks ok assuming all of the differential testing pass in the sandbox.  We will still have to skip tests when the vm args are not supported on all shells.  For example debugger shells have many vm args not supported on release shells.  The release shells will return known argument errors.  There are probably several testcases where these errors are hidden since currently the vm args are ignored by avmdiff.  running a sandbox build will show these problems.

Comment 4

[Chris Peyer]

Patch pushed to redux:
http://hg.mozilla.org/tamarin-redux/rev/723838f07125

Comment 5

[Chris Peyer]

Attachment: Followup fix for --buildfile usage

Previous patch didn't work well with the buildfile option.

This patch actually removes the vmargs var since the loop beforehand extracts out all of the vm args (and testname) into avmshell_args.

Also adds ability to now correctly pass application arguments to the testcase (using " -- ").

Comment 6

[Brent Baker]

Comment on attachment 466718 [details] [diff] [review]
Followup fix for --buildfile usage

'-f' switch does not work. The switch is currently not in use via the build system so the short form could either be removed or fixed, makes no difference to me. Currently using -f will fail to work and will just return the usage

- update the usage output to show the required equals sign in the buildfile switch '--buildfile='

Comment 7

[Chris Peyer]

Attachment: update avmdiff patch based on brents feedback

Removed -f option.
Added comments regarding what options are allowable in avmdiff.
Added = to end of buildfile switch.
Added a missing continue in for args loop for test_args.

Comment 8

[Chris Peyer]

Attachment: One more update

duh - no need for continue, just branch on if / elif

Comment 9

[Chris Peyer]

pushed to redux:

http://hg.mozilla.org/tamarin-redux/rev/724973cb001f