1503530 - [wpt-sync] Sync PR 13806 - [wptrunner] Improve errors from Selenium

Status: RESOLVED FIXED

(Testing :: web-platform-tests, defect, P4)

Description

[Web Platform Test Sync Bot (Matrix: #interop:mozilla.org)]

Sync web-platform-tests PR 13806 into mozilla-central (this bug is closed when the sync is complete).

PR: https://github.com/web-platform-tests/wpt/pull/13806
Details from upstream follow.

Mike Pennisi <[e-mail]> wrote:
>  [wptrunner] Improve errors from Selenium
>  
>  The `message` computed by the Selenium executor was mistakenly unused,
>  and in its place, a Python exception object was used to communicate
>  information on the failure. The string representation of Exceptions is
>  generally terse and less helpful during debugging. Replace with the
>  intended value which includes a complete stack trace and the name of the
>  exception's class.
>  
>  ---
>  
>  Prior to this patch, Python exception which were raised during Selenium-mediated testing would be reported without context. This made debugging difficult, particularly for `KeyError`s, (as in https://github.com/web-platform-tests/results-collection/issues/563):
>  
>      web-platform-test
>      ~~~~~~~~~~~~~~~~~
>      Ran 1 checks (1 tests)
>      Expected results: 0
>      Unexpected results: 1
>        test: 1 (1 error)
>      
>      Unexpected Results
>      ------------------
>      ERROR /dom/events/CustomEvent.html - 1
>  
>  With this patch applied, the reported message includes a stack trace and the name of the exception's class:
>  
>      web-platform-test
>      ~~~~~~~~~~~~~~~~~
>      Ran 1 checks (1 tests)
>      Expected results: 0
>      Unexpected results: 1
>        test: 1 (1 error)
>      
>      Unexpected Results
>      ------------------
>      ERROR /dom/events/CustomEvent.html - 1
>      Traceback (most recent call last):
>        File "/home/mike/projects/bocoup/google-wpt/web-platform-tests/tools/wptrunner/wptrunner/executors/executorselenium.py", line 259, in _run
>          self.result = True, self.func(self.protocol, self.url, self.timeout)
>        File "/home/mike/projects/bocoup/google-wpt/web-platform-tests/tools/wptrunner/wptrunner/executors/executorselenium.py", line 333, in do_testharness
>          done, rv = handler(result)
>        File "/home/mike/projects/bocoup/google-wpt/web-platform-tests/tools/wptrunner/wptrunner/executors/base.py", line 534, in __call__
>          self.logger.debug("Got async callback: %s" % result[1])
>      KeyError: 1

Comment 1

[Web Platform Test Sync Bot (Matrix: #interop:mozilla.org)]

Pushed to try https://treeherder.mozilla.org/#/jobs?repo=try&revision=48f865ef3908c83526f1a9eb2babc57c6af79d1e

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/b2dba02cdea3
[wpt PR 13806] - [wptrunner] Improve errors from Selenium, a=testonly

Comment 3

[Web Platform Test Sync Bot (Matrix: #interop:mozilla.org)]

Result changes from PR not available.

Comment 4

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/b2dba02cdea3