Closed Bug 1315747 Opened 3 years ago Closed 3 years ago

webrtc test failure goes unnoticed

Categories

(Testing :: web-platform-tests, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED INVALID

People

(Reporter: drno, Unassigned)

References

Details

While trying to figure out what I need to do to land bug 1315470 I started to wonder how the datachannel-emptystring test ever passed. It looks to me like the web-platform test framework actually reports a test failure back, which just doesn't get noticed from our test frameworks around it:

0:33.86 PROCESS_OUTPUT: ProcessReader (pid:57999) "1478544342836       Marionette      TRACE   conn7 <- [1,11,null,{"value":["/webrtc/datachannel-emptystring.html",0,null,null,[["Can send empty strings across a WebRTC data channel.",1,"assert_equals: Empty message not received expected \"\" but got \"done\"","onReceiveMessage<@http://web-platform.test:8000/webrtc/datachannel-emptystring.html:58:7\nTest.prototype.step@http://web-platform.test:8000/resources/testharness.js:1401:20\nTest.prototype.step_func/<@http://web-platform.test:8000/resources/testharness.js:1425:20\nEventHandlerNonNull*onReceiveChannel@http://web-platform.test:8000/webrtc/datachannel-emptystring.html:30:5\n"]]]}]"

As you can see it actually complains that it had not received the empty string message over the data channel, which at least is consistent with the currently broken implementation.
Guess what, when I run just the single test case like this:

./mach web-platform-tests --disable-e10s --test-types testharness --include webrtc/datachannel-emptystring.html

I actually see a test failure in the browser, but mach still does not realize/report the test failure after shutdown/test end.
It's annotated as expected failure:
https://dxr.mozilla.org/mozilla-central/source/testing/web-platform/meta/webrtc/datachannel-emptystring.html.ini
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → INVALID
You need to log in before you can comment on or make changes to this bug.