1315747 - webrtc test failure goes unnoticed

Status: RESOLVED INVALID

(Testing :: web-platform-tests, defect)

Description

[Nils Ohlmeier [:drno]]

While trying to figure out what I need to do to land bug 1315470 I started to wonder how the datachannel-emptystring test ever passed. It looks to me like the web-platform test framework actually reports a test failure back, which just doesn't get noticed from our test frameworks around it:

0:33.86 PROCESS_OUTPUT: ProcessReader (pid:57999) "1478544342836       Marionette      TRACE   conn7 <- [1,11,null,{"value":["/webrtc/datachannel-emptystring.html",0,null,null,[["Can send empty strings across a WebRTC data channel.",1,"assert_equals: Empty message not received expected \"\" but got \"done\"","onReceiveMessage<@http://web-platform.test:8000/webrtc/datachannel-emptystring.html:58:7\nTest.prototype.step@http://web-platform.test:8000/resources/testharness.js:1401:20\nTest.prototype.step_func/<@http://web-platform.test:8000/resources/testharness.js:1425:20\nEventHandlerNonNull*onReceiveChannel@http://web-platform.test:8000/webrtc/datachannel-emptystring.html:30:5\n"]]]}]"

As you can see it actually complains that it had not received the empty string message over the data channel, which at least is consistent with the currently broken implementation.

Comment 1

[Nils Ohlmeier [:drno]]

Guess what, when I run just the single test case like this:

./mach web-platform-tests --disable-e10s --test-types testharness --include webrtc/datachannel-emptystring.html

I actually see a test failure in the browser, but mach still does not realize/report the test failure after shutdown/test end.

Comment 2

[James Graham [:jgraham]]

It's annotated as expected failure:
https://dxr.mozilla.org/mozilla-central/source/testing/web-platform/meta/webrtc/datachannel-emptystring.html.ini