1422707 - Save screenshots to a file that's uploaded rather than outputting base64 encoded to the log

Status: NEW

(Testing :: General, enhancement, P3)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

• Screenshots taken when the test suite hit an error can be helpful to identify issues, especially for layout reftests.
    • Layout reftests store the screenshots for the expected and actual result (stored as plaintext in the log). These can be viewed together with a difference highlighter in the Reftest analyzer which has to be open separately from Treeherder and the first error to be clicked.
    • Tests with the browser visually open can also take a screenshot after a failure, but don’t have a reference available how it should look. These are usually stored as artifacts.
    • Speeding up the access to the screenshot(s) e.g. by showing it as a tooltip over a screenshot icon at the failure line and side-by-side if necessary will be a time saver when random tests fail with the same symtoms, e.g. https://bugzilla.mozilla.org/show_bug.cgi?id=1422706 and https://bugzilla.mozilla.org/show_bug.cgi?id=1371992

Comment 1

[Ed Morley [:emorley]]

I think the best way to fix this is to have the screenshots be saved to a file that's uploaded along with other files, rather than be dumped to the log base64 encoded. This is something that test harnesses will need to do.

Once that happens, Treeherder will automatically get the link to the uploaded screenshot under "Job details".

Comment 2

[(not currently active) Ted Mielczarek]

I think reftest is the only harness that dumps data URLs to the log, isn't it?