1436654 - SpecialPowers.loadChromeScript assertion failures don't fail tests

Status: NEW

(Testing :: Mochitest, defect)

Description

[Matthew N. [:MattN]]

I'm not sure if this is a regression since the assert.* ability landed in bug 965257 or whether it never worked properly but a simple test like the following does not cause a test failure (when run as a single file or the directory):

add_task(async function test_in_chrome() {
  let chromeScript = SpecialPowers.loadChromeScript(function test() {
    assert.ok(false, "This should fail the test");
  });
  ok(true, "This is to avoid the 'No checks actually run'");
});

I do get the following in the log:
> 0:26.09 GECKO(71672) TEST-UNEXPECTED-FAIL | test | This should fail the test - false == true - got false, expected true (operator ==)

I think there are multiple issues going on but I don't really understand how it's all supposed to work in order to fix it.

I believe the logging is happening because the code is falling back to dump() if it can't find the TestRunner: https://dxr.mozilla.org/mozilla-central/rev/4fe6f6560083f8c8257282bef1d4e0ced9d1b975/testing/specialpowers/content/specialpowersAPI.js#612
The assert.ok is not included in the pass count which I think is part of the problem. I'm not sure which API to use to report the pass/failure to have it count in the summary table and actually cause local failures.