Closed Bug 1521072 Opened Last year Closed Last year

Make screenshot on fail the default for wpt reftests when run via mach

Categories

(Testing :: web-platform-tests, enhancement)

Version 3
enhancement
Not set

Tracking

(firefox66 fixed)

RESOLVED FIXED
mozilla66
Tracking Status
firefox66 --- fixed

People

(Reporter: jgraham, Assigned: jgraham)

References

Details

Attachments

(2 files)

Currently we only produce a screenshot on unexpected which is useful for performance and resource utilisation in CI since we have a large number of failing tests that would otherwise always produce a screenshot. But locally people debugging expected fails might expect a screenshot, so default to always producing one.

When the logging setup moved to earlier in the setup we ended up setting the
option to enable tbpl-style screenshots from mach after the loggers were already
initalised. Move this to earlier in the command so this option starts working again.

On CI we only want to log screenshots when something unexpected happens since anything
else is rather wasteful of resources. But locally getting screenshots for expected
failures seems helpful for debugging, so worth making the default. Hopefully this isn't
too confusing for people just checking if their patch regresses anything rather than
actively working on fixing failures.

Depends on D16973

Pushed by james@hoppipolla.co.uk:
https://hg.mozilla.org/integration/autoland/rev/ef8493b00486
Fix logging screenshots from wpt with mach logger, r=ato
https://hg.mozilla.org/integration/autoland/rev/b0377ec57ddd
Log wpt reftest screenshots on fail when running mach, r=ato

I'm not sure this is working correctly... I did a fresh build with autoland tip (updated directly to cset b0377ec57ddd from comment 3), but I'm still not seeing any screenshot data URIs output when running the following commands:

./mach test \
testing/web-platform/tests/css/vendor-imports/mozilla/mozilla-central-reftests/flexbox/flexbox-justify-content-horiz-001b.xhtml

./mach test --reftest-screenshot=fail \
testing/web-platform/tests/css/vendor-imports/mozilla/mozilla-central-reftests/flexbox/flexbox-justify-content-horiz-001b.xhtml

./mach test --reftest-screenshot=unexpected \
testing/web-platform/tests/css/vendor-imports/mozilla/mozilla-central-reftests/flexbox/flexbox-justify-content-horiz-001b.xhtml

Note that this particular test is annotated as "expected fail" with an .ini file right now[1] -- I tried performing these commands after removing the .ini file, though (as well as before removing it). So, I'm not seeing any reftest-screenshot data-URI logging in my terminal, with any of the above commands, regardless of whether the test-failure is annotated as expected or unexpected.

[1] https://searchfox.org/mozilla-central/rev/dac799c9f4e9f5f05c1071cba94f2522aa31f7eb/testing/web-platform/meta/css/vendor-imports/mozilla/mozilla-central-reftests/flexbox/flexbox-justify-content-horiz-001b.xhtml.ini

If I run the above ^^ commands with "--log-tbpl somefile.txt", then somefile.txt does have the reftest screenshot data URIs. So that's something! :)

I think they're expected to show up in terminal output, too, though -- right? (They don't for me right now, even when I pass --log-tbpl somefile.txt -- they only show up in the .txt file, not in my terminal output.)

Does it work with mach wpt? I didn't test with mach test, so that could be broken.

Flags: needinfo?(james)

(And in fact, it seems that even mach test [some-unexpectedly-failing-traditional-reftest] doesn't output screenshots, either! I guess mach test tries to be minimally verbose about the failures, or something, and you have to use the name of the actual harness if you really want all the failure information... Anyway: mach wpt seems to be what I needed here. Thanks!)

Status: NEW → RESOLVED
Closed: Last year
Resolution: --- → FIXED
Target Milestone: --- → mozilla66
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/15056 for changes under testing/web-platform/tests
You need to log in before you can comment on or make changes to this bug.