1474569 - Improve output for failing wpt tests in jstests harness

Status: RESOLVED FIXED

(Core :: JavaScript Engine, defect)

Description

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

Attachment: Patch v1

Comment 1

[Benjamin Bouvier [:bbouvier] (inactive)]

Comment on attachment 8990962 [details] [diff] [review]
Patch v1

Review of attachment 8990962 [details] [diff] [review]:
-----------------------------------------------------------------

Makes sense. Do you have an example of reporting in a try run? Thanks for the patch.

::: js/src/tests/lib/results.py
@@ +112,5 @@
>                  else:
>                      test_result = (cls.FAIL, test.message)
>                      result = cls.FAIL
> +                    test_output += "expected %s, found %s" % (expected, test.status)
> +                    if test.message:

Is message always guaranteed to exist, even if it's None/empty string? Otherwise, hasattr should be tested for here.

Comment 2

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

Thanks!

(In reply to Benjamin Bouvier [:bbouvier] from comment #1)
> Comment on attachment 8990962 [details] [diff] [review]
> Patch v1
> 
> Review of attachment 8990962 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Makes sense. Do you have an example of reporting in a try run? Thanks for
> the patch.

No, but locally it looks like this:

REGRESSION - wasm/jsapi/interface.any.js
[0|2|0|0]  40% =====================>                                 |   0.5s
## wasm/jsapi/interface.any.js: rc = 0, run time = 0.087217

Harness status: OK (Reported by harness: None)
Subtest "WebAssembly: property descriptor": expected FAIL, found PASS
Subtest "WebAssembly: constructing": as expected: PASS
Subtest "WebAssembly.validate": expected PASS, found FAIL (with message: "assert_true: enumerable expected true got false")


> ::: js/src/tests/lib/results.py
> @@ +112,5 @@
> >                  else:
> >                      test_result = (cls.FAIL, test.message)
> >                      result = cls.FAIL
> > +                    test_output += "expected %s, found %s" % (expected, test.status)
> > +                    if test.message:
> 
> Is message always guaranteed to exist, even if it's None/empty string?
> Otherwise, hasattr should be tested for here.

wpttest.SubtestResult.__init__ does always set it, yes.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/d8c824521e7b
Improve output for failing wpt tests in jstests harness; r=bbouvier

Comment 4

[Stefan Hindli [:stefan_hindli]]

Backed out 2 changesets (bug 1474569, bug 1474581) for wpt failures in test harness output on a CLOSED TREE

Problematic push: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=b39da1b094e9f89a42ce93f305d4cf1c09c8b4a5&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-classifiedState=unclassified
Failure: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&fromchange=383714d5acad1c17adb25b8145f0cf1c25f1261c&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-classifiedState=unclassified&selectedJob=187627918
Backout: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=25915c35842591babf7cfe5fc8ff0d5c3adbb327&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-classifiedState=unclassified
Log: https://treeherder.mozilla.org/logviewer.html#?job_id=187627918&repo=mozilla-inbound

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/9d11c83f1d68
Improve output for failing wpt tests in jstests harness; r=bbouvier

Comment 6

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/9d11c83f1d68