1050178 - Marionette "TEST-UNEXPECTED-FAIL" now displays as "TEST-UNEXPECTED-ERROR"

Status: RESOLVED WONTFIX

(Remote Protocol :: Marionette, defect)

Description

[Ed Morley [:emorley]]

Compare old:
TEST-UNEXPECTED-FAIL | test_switch_remote_frame.py test_switch_remote_frame.TestSwitchRemoteFrame.test_we_can_switch_to_a_remote_frame_by_index | TimeoutException: TimeoutException: Connection timed out

With new:
TEST-UNEXPECTED-ERROR | test_switch_remote_frame.py TestSwitchRemoteFrame.test_we_can_switch_to_a_remote_frame_by_index | TimeoutException: TimeoutException: Connection timed out

Looking at the failures in eg bug 1008369, you can see the output change on ~9th July, which is when bug 956739 landed.

Was this intentional?
A lot of the horrid buildbot/mozharness regex sadly depend on the output strings to not change - this will quite possibly break things.

Comment 1

[Chris Manchester (limited bugmail, email directly)]

This was intentional in the sense that it was an effect of the change that I was aware of, but not a significant goal of it or anything. It could be argued that the difference between an assertion method failing and a test unexpectedly throwing an exception or the like is useful information to have, but I'm not really sure that's true in this particular case.

From reading the mozharness source and pushing to try, it seems the part to match is "TEST-UNEXPECTED" and the summary, so this change doesn't compromise our coverage.

We already have a one off formatter for marionette, so this change would be easy enough revert if necessary.

Comment 2

[David Burns :automatedtester]

SIgnificant changes have happened in this area and doesnt appear to have been something the sheriffs have complained about since the initial bug report