559995 - Test infrastructure to support versioned code in the AVM shell

Status: VERIFIED WONTFIX

(Tamarin Graveyard :: Virtual Machine, defect, P1)

Description

[Lars T Hansen]

Cloned from bug #535770.

The design for versioning will likely be that we want to pass an argument to the vm shell, eg "-bugcompat FP_10_0" (that's the current design anyway).  But the test cases we should run should only be those tests that were "defined" for that version; ie, if we fix a bug and add versioning support in the AS3 and C++ code, we will not run that code when running the shell on an earlier API, and so any test code that lands for the bugfix must also not be run.

This probably means a couple of things.  runtests.py must be version-aware, and
when providing eg --api=FP_10_0 to runtests.py it needs to do the following:

- if a test was added for a later bugfix then it either should not be run, or
  its buggy result should be considered the correct result for that version

- if a test case affects subsequent tests in its file (eg by crashing or
  throwing an exception) it needs to be factored into a separate file so that
  the running of version-oblivious tests is not affected by versioned tests.
  this really ought to be happening already, maybe it is, but we have to be
  more systematic about it.

- versioning info must be noted carefully when landing new test cases.

My general feeling is that the correct fix is to sequester bugs affected by versioning in individual files, so that it's easy to include / exclude the files.  If there are two versions of a file, one to run with the old version and one with the new version, then so be it - IMO the test has no business querying the shell to find out what version it is running against, the test should just be the test.

(Room for debate here.)

Comment 1

[Chris Peyer]

I believe that these issues have been addressed with the USES_SWFVERSION keyword for .avm_args files.  See https://developer.mozilla.org/En/Tamarin/Tamarin_Acceptance_Testing/Actionscript_Acceptance_Tests#testname.as.avm_args for details.

Comment 2

[Trevor Baker]

bulk verifying resolved !fixed issues