1661513 - fail-if = fission makes ./mach test --enable-fission show failures as green PASS

Status: RESOLVED FIXED

(Testing :: Mochitest, defect, P2)

Description

[Henri Sivonen (:hsivonen)]

Today, I experienced a substantial productivity loss, because ./mach test --enable-fission shows green "PASS" upon assertion failure if mochitest.ini has fails-if = fission for the test.

I can understand that this behavior may be the easiest way to integrate with treeherder log parsing, but it's very confusing, and it would be much better to have ./mach test --enable-fission use the word "FAIL" (possibly in combination with another word) and the red color for assertion failures regardless of mochitest.ini content.

Comment 1

[Kris Maglione [:kmag]]

Note: Removing test.result = !test.result here seems to fix the problem. I'm not sure why I didn't do that the last time I touched the expected fail code...

Comment 2

[Chris Peterson [:cpeterson]]

assign to kmag

not a Fission bug, per se

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:ahal, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 4

[Chris Peterson [:cpeterson]]

M6b because kmag would like to fix this soon.

Comment 5

[Kris Maglione [:kmag]]

Attachment: Bug 1661513: Fix reporting of expected failures in fail-if tests. r=ahal

Without this change, failures in tests marked with fail-if are reported as
passes, though with messages and stacks that describe the failure. That works
as far as the automation machinery is concerned, but it is extremely confusing
to people who are trying to reproduce and fix the failures locally.

This change fixes the reporting so that they show up as expected failures in
logs, but have the same effect on final test status.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3b0e057ff3c3
Fix reporting of expected failures in fail-if tests. r=ahal

Comment 7

[Bogdan Tara[:bogdan_tara | bogdant]]

Backed out changeset 3b0e057ff3c3 (bug 1661513) for mochitest-webgl suite failures.

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&selectedTaskRun=S1vLepUgRqKuf4j9hpUUvA.0&searchStr=webgl&fromchange=583ecfdab160c513d7755b54d2f5c51f26ab0db1&tochange=15aeeeb2ff1fb2c2abeb4f2ff4a4642857e9d8d2

Backout link: https://hg.mozilla.org/integration/autoland/rev/15aeeeb2ff1fb2c2abeb4f2ff4a4642857e9d8d2

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=316113431&repo=autoland&lineNumber=5671

...
[task 2020-09-18T20:49:02.739Z] 20:49:02     INFO - TEST-OK | dom/canvas/test/webgl-conf/generated/test_2_conformance__uniforms__uniform-values-per-program.html | took 1275ms
[task 2020-09-18T20:49:02.987Z] 20:49:02     INFO - TEST-START | Shutdown
[task 2020-09-18T20:49:02.987Z] 20:49:02     INFO - Passed:  62900
[task 2020-09-18T20:49:02.987Z] 20:49:02     INFO - Failed:  1319
[task 2020-09-18T20:49:02.987Z] 20:49:02     INFO - Todo:    0
[task 2020-09-18T20:49:02.988Z] 20:49:02     INFO - Mode:    e10s
[task 2020-09-18T20:49:02.988Z] 20:49:02     INFO - Slowest: 24624ms - /tests/dom/canvas/test/webgl-conf/generated/test_2_conformance__uniforms__uniform-samplers-test.html
[task 2020-09-18T20:49:02.988Z] 20:49:02     INFO - SimpleTest FINISHED
[task 2020-09-18T20:49:02.988Z] 20:49:02     INFO - TEST-INFO | Ran 1 Loops
[task 2020-09-18T20:49:02.988Z] 20:49:02     INFO - SimpleTest FINISHED
[task 2020-09-18T20:49:03.140Z] 20:49:03     INFO - GECKO(1552) | JavaScript error: resource://gre/modules/AsyncShutdown.jsm, line 554: NotFoundError: No such JSWindowActor 'SpecialPowers'
[task 2020-09-18T20:49:03.535Z] 20:49:03     INFO - GECKO(1552) | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2020-09-18T20:49:03.535Z] 20:49:03     INFO - GECKO(1552) | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2020-09-18T20:49:03.871Z] 20:49:03     INFO - GECKO(1552) | JavaScript error: resource://gre/modules/Sqlite.jsm, line 1031: Error: Sqlite.jsm has been shutdown. Cannot open connection to: /var/folders/_j/pv6lflx97xd_jxg0fvdtlx2h000017/T/tmpOPTlgT.mozrunner/protections.sqlite
[task 2020-09-18T20:49:03.898Z] 20:49:03     INFO - GECKO(1552) | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2020-09-18T20:49:04.091Z] 20:49:04     INFO - GECKO(1552) | JavaScript warning: , line 0: Failed to allocate internal command buffer.
[task 2020-09-18T20:49:04.091Z] 20:49:04     INFO - GECKO(1552) | JavaScript warning: , line 0: WebGL context was lost.
[task 2020-09-18T20:49:04.633Z] 20:49:04     INFO - GECKO(1552) | 1600462144619	Marionette	TRACE	Received observer notification xpcom-will-shutdown
[task 2020-09-18T20:49:04.633Z] 20:49:04     INFO - GECKO(1552) | 1600462144620	Marionette	INFO	Stopped listening on port 2828
[task 2020-09-18T20:49:04.633Z] 20:49:04     INFO - GECKO(1552) | 1600462144621	Marionette	DEBUG	Marionette stopped listening
[task 2020-09-18T20:49:09.491Z] 20:49:09     INFO - TEST-INFO | Main app process: exit 0
[task 2020-09-18T20:49:09.491Z] 20:49:09     INFO - runtests.py | Application ran for: 0:06:43.491188
[task 2020-09-18T20:49:09.491Z] 20:49:09     INFO - zombiecheck | Reading PID log: /var/folders/_j/pv6lflx97xd_jxg0fvdtlx2h000017/T/tmpB4fplKpidlog
[task 2020-09-18T20:49:09.491Z] 20:49:09     INFO - ==> process 1552 launched child process 1554
[task 2020-09-18T20:49:09.492Z] 20:49:09     INFO - ==> process 1552 launched child process 1555
[task 2020-09-18T20:49:09.492Z] 20:49:09     INFO - ==> process 1552 launched child process 1556
[task 2020-09-18T20:49:09.492Z] 20:49:09     INFO - ==> process 1552 launched child process 1557
[task 2020-09-18T20:49:09.492Z] 20:49:09     INFO - ==> process 1552 launched child process 1609
[task 2020-09-18T20:49:09.492Z] 20:49:09     INFO - zombiecheck | Checking for orphan process with PID: 1609
[task 2020-09-18T20:49:09.492Z] 20:49:09     INFO - zombiecheck | Checking for orphan process with PID: 1554
[task 2020-09-18T20:49:09.492Z] 20:49:09     INFO - zombiecheck | Checking for orphan process with PID: 1555
[task 2020-09-18T20:49:09.492Z] 20:49:09     INFO - zombiecheck | Checking for orphan process with PID: 1556
[task 2020-09-18T20:49:09.493Z] 20:49:09     INFO - zombiecheck | Checking for orphan process with PID: 1557
[task 2020-09-18T20:49:09.493Z] 20:49:09     INFO - Stopping web server
[task 2020-09-18T20:49:09.533Z] 20:49:09     INFO - Stopping web socket server
[task 2020-09-18T20:49:09.680Z] 20:49:09     INFO - Stopping ssltunnel
[task 2020-09-18T20:49:09.844Z] 20:49:09  WARNING - leakcheck | refcount logging is off, so leaks can't be detected!
[task 2020-09-18T20:49:09.844Z] 20:49:09     INFO - runtests.py | Running tests: end.
[task 2020-09-18T20:49:10.133Z] 20:49:10     INFO - Buffered messages finished
[task 2020-09-18T20:49:10.133Z] 20:49:10     INFO -  0 INFO TEST-START | Shutdown
[task 2020-09-18T20:49:10.133Z] 20:49:10     INFO -  1 INFO Passed:  62900
[task 2020-09-18T20:49:10.133Z] 20:49:10     INFO -  2 INFO Failed:  1319
[task 2020-09-18T20:49:10.133Z] 20:49:10     INFO -  3 INFO Todo:    0
[task 2020-09-18T20:49:10.134Z] 20:49:10     INFO -  4 INFO Mode:    e10s
[task 2020-09-18T20:49:10.134Z] 20:49:10     INFO -  5 INFO SimpleTest FINISHED
[task 2020-09-18T20:49:10.134Z] 20:49:10     INFO - Buffered messages finished
[task 2020-09-18T20:49:10.134Z] 20:49:10     INFO - SUITE-END | took 405s
[task 2020-09-18T20:49:10.170Z] 20:49:10    ERROR - Return code: 1
[task 2020-09-18T20:49:10.171Z] 20:49:10     INFO - TinderboxPrint: mochitest-mochitest-webgl2-core<br/>495/0/0
[task 2020-09-18T20:49:10.171Z] 20:49:10    ERROR - # TBPL FAILURE #
[task 2020-09-18T20:49:10.171Z] 20:49:10  WARNING - setting return code to 2
[task 2020-09-18T20:49:10.171Z] 20:49:10    ERROR - The mochitest suite: mochitest-webgl2-core ran with return status: FAILURE
...

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9a18000e5ead
Fix reporting of expected failures in fail-if tests. r=ahal

Comment 9

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/9a18000e5ead