1447852 - web-platform-tests tasks should take screenshot on first failure like other tests

Status: NEW

(Testing :: web-platform-tests, enhancement)

Description

[Xidorn Quan [:xidorn] UTC+10]

Mochitests take screenshot when the first failure happens, while web-platform-tests don't.

Sometimes it is very helpful if there is a screenshot available.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I think that a lot of test harnesses take screenshots and upload them as artifacts when run in our CI.

I've had a couple of times when I wanted to see screenshots as well for failures that happen specifically in wdspec jobs. Especially when those are not reproducible locally like bug 1825501.

Potentially we should not let Marionette create the screenshot but do it similar like Mochitests and use mozscreenshot for it?

Comment 2

[James Graham [:jgraham]]

https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wptrunner/wptrunner/testrunner.py#671 is the code after we know a test is finished; this would be the place to do a system screenshot if we wanted to do that. A problem with depending on mozscreenshot is that it does add an extra dependency. If we're adding that in a way that will affect other vendors we'll need to make an wpt RFC.

Another problem with system screenshots is that if we're running tests in parallel we will get a screenshot with both the failing and non-failing browser instances. For that reason we might want to consider just using WebDriver to get a browser screenshot at the right time?

In any case this should be controlled by a command line flag (e.g. --screenshot-on-failure=never/first/always).