Need to have a way to run the acceptance suite using the -Dverifyonly switch. Issue at the moment is that the runtest code expects PASS/FAIL to be in the stdout and if it does not exist then it is assumed to have failed. The -Dverifyonly switch will only verify the code and never executes anything, so there is no output when verification passes. This mode should ONLY verify that the shell exits cleanly as all acceptance testcases should always verify. Could see if we could leverage the .err files for *.abc_ files that fail verification.
Once fixed add a simple/single test into the smoke phase to quickly verify that this stays working and then an additional pass in the deep phase
in mac-deep we are running the brightspot tests with -Dverifyonly using a separate script. it has some specific info about player so it is located in qe repo: projects/brightspot/verifyonly. It may be easier not to overload the acceptance/runtests.py script. it is a simple script. we could remove the player specific code and checkin a version into tr.
(In reply to comment #2) > in mac-deep we are running the brightspot tests with -Dverifyonly using a > separate script. it has some specific info about player so it is located in qe > repo: projects/brightspot/verifyonly. It may be easier not to overload the > acceptance/runtests.py script. it is a simple script. we could remove the > player specific code and checkin a version into tr. This is probably a much better idea. All that we need is a proxy script (python/bash) that makes the call to the real shell and just confirms that the exitcode is as expected and return PASS or FAIL.
Hoisted exitcode handling out of the runtests method so that it can be called separately when doing a verifyonly pass. Adds a --verifyonly option to runtests.py that will automatically add -Dverifyonly to the vm args unless already present. I also added a verifyonly passes to the deep phases and smoke phase.
Rebased patch to tip, changed smoke run to only test one file.
Comment on attachment 464226 [details] [diff] [review] updated patch Probably should be running a debug-debugger build in the smokes instead so that it would also hit any asserts that might be triggered (currently the --verifyonly smoke is running a release-debugger shell)
Attachment #464226 - Flags: review?(brbaker) → review+
Updated to run debug-debugger in smokes and pushed to redux: http://hg.mozilla.org/tamarin-redux/rev/ce701c85ec59
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.