Add --verifyonly switch to acceptance/runtests.py

VERIFIED FIXED

Status

defect
VERIFIED FIXED
9 years ago
9 years ago

People

(Reporter: brbaker, Assigned: cpeyer)

Tracking

unspecified
Bug Flags:
flashplayer-triage +

Details

Attachments

(1 attachment, 1 obsolete attachment)

Reporter

Description

9 years ago
Need to have a way to run the acceptance suite using the -Dverifyonly switch. Issue at the moment is that the runtest code expects PASS/FAIL to be in the stdout and if it does not exist then it is assumed to have failed. The -Dverifyonly switch will only verify the code and never executes anything, so there is no output when verification passes. This mode should ONLY verify that the shell exits cleanly as all acceptance testcases should always verify. Could see if we could leverage the .err files for *.abc_ files that fail verification.
Flags: flashplayer-triage+
Flags: flashplayer-qrb?
Reporter

Comment 1

9 years ago
Once fixed add a simple/single test into the smoke phase to quickly verify that this stays working and then an additional pass in the deep phase

Comment 2

9 years ago
in mac-deep we are running the brightspot tests with -Dverifyonly using a separate script.  it has some specific info about player so it is located in qe repo: projects/brightspot/verifyonly.  It may be easier not to overload the acceptance/runtests.py script.  it is a simple script.  we could remove the player specific code and checkin a version into tr.
Reporter

Comment 3

9 years ago
(In reply to comment #2)
> in mac-deep we are running the brightspot tests with -Dverifyonly using a
> separate script.  it has some specific info about player so it is located in qe
> repo: projects/brightspot/verifyonly.  It may be easier not to overload the
> acceptance/runtests.py script.  it is a simple script.  we could remove the
> player specific code and checkin a version into tr.

This is probably a much better idea. All that we need is a proxy script (python/bash) that makes the call to the real shell and just confirms that the exitcode is as expected and return PASS or FAIL.
Assignee

Updated

9 years ago
Assignee: nobody → cpeyer
Status: NEW → ASSIGNED
Assignee

Comment 4

9 years ago
Hoisted exitcode handling out of the runtests method so that it can be called separately when doing a verifyonly pass.  Adds a --verifyonly option to runtests.py that will automatically add -Dverifyonly to the vm args unless already present.

I also added a verifyonly passes to the deep phases and smoke phase.
Attachment #463630 - Flags: review?(brbaker)
Assignee

Updated

9 years ago
See Also: → 491934
Assignee

Comment 5

9 years ago
Posted patch updated patchSplinter Review
Rebased patch to tip, changed smoke run to only test one file.
Attachment #463630 - Attachment is obsolete: true
Attachment #464226 - Flags: review?(brbaker)
Attachment #463630 - Flags: review?(brbaker)
Reporter

Comment 6

9 years ago
Comment on attachment 464226 [details] [diff] [review]
updated patch

Probably should be running a debug-debugger build in the smokes instead so that it would also hit any asserts that might be triggered (currently the --verifyonly smoke is running a release-debugger shell)
Attachment #464226 - Flags: review?(brbaker) → review+
Assignee

Comment 7

9 years ago
Updated to run debug-debugger in smokes and pushed to redux:

http://hg.mozilla.org/tamarin-redux/rev/ce701c85ec59
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Assignee

Updated

9 years ago
Status: RESOLVED → VERIFIED
Reporter

Updated

9 years ago
Flags: flashplayer-qrb?
You need to log in before you can comment on or make changes to this bug.