update differential testing script avmdiff.py to return all PASSES/FAILS

VERIFIED FIXED in Q3 11 - Serrano

Status

VERIFIED FIXED
10 years ago
8 years ago

People

(Reporter: dschaffe, Assigned: cpeyer)

Tracking

unspecified
Q3 11 - Serrano
x86
Windows 7
Bug Flags:
flashplayer-qrb +
flashplayer-bug -

Details

Attachments

(2 attachments, 2 obsolete attachments)

(Reporter)

Description

10 years ago
update avmdiff.py, if all configs match return the output, otherwise return the diff

there are still a few cases of scrubbing:
(1) strip runtime errors after Error: Error #[09]+ since release does not show error detail
(2) strip stack traces from runtime errors, only Debugger builds show stacktraces
(3) strip all lines '^metric', all time related or changing output can be prefixed with metric
Flags: flashplayer-qrb+
(Reporter)

Comment 1

10 years ago
avmdiff.py does not pass all vm args to each vm, only passes the .abc name
e.g.  avmdiff.py -memstats test.abc should pass -memstats to each vm

Updated

10 years ago
Target Milestone: --- → Future
(Assignee)

Comment 2

9 years ago
Created attachment 465668 [details] [diff] [review]
have avmdiff pass command line options to avmshell

There is a slight issue with patch when running avmdiff from the command line, if passing in a value for buildfile, the buildfile switch must use an =:

avmdiff --buildfile=file.txt

using a space will pass file.txt to avmshell
Assignee: nobody → cpeyer
Status: NEW → ASSIGNED
Attachment #465668 - Flags: review?(dschaffe)

Updated

9 years ago
Target Milestone: Future → flash10.2
(Reporter)

Comment 3

9 years ago
Comment on attachment 465668 [details] [diff] [review]
have avmdiff pass command line options to avmshell

looks ok assuming all of the differential testing pass in the sandbox.  We will still have to skip tests when the vm args are not supported on all shells.  For example debugger shells have many vm args not supported on release shells.  The release shells will return known argument errors.  There are probably several testcases where these errors are hidden since currently the vm args are ignored by avmdiff.  running a sandbox build will show these problems.
Attachment #465668 - Flags: review?(dschaffe) → review+
(Assignee)

Comment 4

9 years ago
Patch pushed to redux:
http://hg.mozilla.org/tamarin-redux/rev/723838f07125
OS: Mac OS X → Windows 7
(Assignee)

Comment 5

9 years ago
Created attachment 466718 [details] [diff] [review]
Followup fix for --buildfile usage

Previous patch didn't work well with the buildfile option.

This patch actually removes the vmargs var since the loop beforehand extracts out all of the vm args (and testname) into avmshell_args.

Also adds ability to now correctly pass application arguments to the testcase (using " -- ").
Attachment #466718 - Flags: review?(brbaker)

Comment 6

9 years ago
Comment on attachment 466718 [details] [diff] [review]
Followup fix for --buildfile usage

'-f' switch does not work. The switch is currently not in use via the build system so the short form could either be removed or fixed, makes no difference to me. Currently using -f will fail to work and will just return the usage

- update the usage output to show the required equals sign in the buildfile switch '--buildfile='
Attachment #466718 - Flags: review?(brbaker) → review-
(Assignee)

Comment 7

9 years ago
Created attachment 467046 [details] [diff] [review]
update avmdiff patch based on brents feedback

Removed -f option.
Added comments regarding what options are allowable in avmdiff.
Added = to end of buildfile switch.
Added a missing continue in for args loop for test_args.
Attachment #466718 - Attachment is obsolete: true
Attachment #467046 - Flags: review?(brbaker)
(Assignee)

Comment 8

9 years ago
Created attachment 467047 [details] [diff] [review]
One more update

duh - no need for continue, just branch on if / elif
Attachment #467046 - Attachment is obsolete: true
Attachment #467047 - Flags: review?(brbaker)
Attachment #467046 - Flags: review?(brbaker)

Updated

9 years ago
Attachment #467047 - Flags: review?(brbaker) → review+
(Assignee)

Comment 9

9 years ago
pushed to redux:

http://hg.mozilla.org/tamarin-redux/rev/724973cb001f
Status: ASSIGNED → RESOLVED
Last Resolved: 9 years ago
Resolution: --- → FIXED
(Assignee)

Updated

9 years ago
Status: RESOLVED → VERIFIED

Updated

8 years ago
Flags: flashplayer-bug-
You need to log in before you can comment on or make changes to this bug.