Closed Bug 791033 Opened 8 years ago Closed 8 years ago

Marionette shouldn't log TEST-UNEXPECTED-FAIL for expected failures

Categories

(Testing :: Marionette, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED
mozilla18

People

(Reporter: jgriffin, Assigned: jgriffin)

References

Details

Attachments

(1 file)

Marionette has a number of tests with expected failures.  Currently, the failures in these tests are logged with TEST-UNEXPECTED-FAIL, which is misleading and gets picked up by TBPL as a real failure.
Blocks: 770769
Try run shows this patch solves this problem.
Comment on attachment 661372 [details] [diff] [review]
Allow tests to provide custom pass/fail strings for is/isnot/ok assertions

I don't love this approach but can't think of a better way to handle it.  This basically allows a test to define its own status strings for pass/fail.
Attachment #661372 - Flags: review?(mdas)
Try run for 0725231f95b6 is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=0725231f95b6
Results (out of 2 total builds):
    success: 2
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/jgriffin@mozilla.com-0725231f95b6
Comment on attachment 661372 [details] [diff] [review]
Allow tests to provide custom pass/fail strings for is/isnot/ok assertions

Review of attachment 661372 [details] [diff] [review]:
-----------------------------------------------------------------

Yeah, I'm not 100% happy with this either, but I can't think of a nicer way of doing it. If there isn't any other purpose to passing in your own logging lines than these expected-failures, perhaps we can add a function to toggle logging on and off, and log what we want for these particular tests, but I'm not sure that's any nicer.
Attachment #661372 - Flags: review?(mdas) → review+
(In reply to Malini Das [:mdas] from comment #5)
 perhaps we can add a function to toggle

> logging on and off, and log what we want for these particular tests, but I'm
> not sure that's any nicer.

Yeah, I'm not sure either.  I landed the existing patch as https://hg.mozilla.org/integration/mozilla-inbound/rev/1b9df2fa85a9
Assignee: nobody → jgriffin
Target Milestone: --- → mozilla18
https://hg.mozilla.org/mozilla-central/rev/1b9df2fa85a9
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.