Support screenshots for tests

RESOLVED FIXED in Firefox 66

Status

enhancement
P3
normal
RESOLVED FIXED
8 months ago
8 months ago

People

(Reporter: Bebe, Assigned: Bebe)

Tracking

Version 3
mozilla66
Points:
---

Firefox Tracking Flags

(firefox66 fixed)

Details

Attachments

(1 attachment)

make Raptor support screenshots when running tests

This would be useful when testing on different platforms or verifying different tests
I did a little hacking today to see how we can get this to work.

it still needs some work but it's a good start

Try job:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=b9897656008c13ccad711b3bf6158a309a553491

You can find the screenshots in the job details artifacts.

This would be really helpful when reviewing a tp6 tests.
:rwood :davehunt please share your thought on this "feature" and how should it be implemented long term
Flags: needinfo?(rwood)
Flags: needinfo?(dave.hunt)
Assignee: nobody → fstrugariu
This is really cool, thanks :bebe! Please see the feedback in the phabricator review. Quick summary:

1. Taking screenshots should be an option in the test INI i.e. screenshot = true/false, so that we can turn this on/off for specific tests.

2. We should expand this to work with benchmark type of tests too - when the runner.js receives the benchmark result from benchmark-relay.js, the runner.js could take the screenshot at that time, same method as pageload tests.

3. In the screenshot.html artifact, please include the test name, and use 'pagecycles' instead of 'test' and start at 1. i.e. headings should be:

'raptor-tp6-google-firefox pagecycle 1' to 25.

That will match the logging and make it easier to map screenshots to the test logs.

4. Doesn't work in google chromium, needs a fix for that (see phabricator).

5. Please see the additional notes in-line in the phabricator review.

Thanks!
Flags: needinfo?(rwood)
See also bug 1512615.
Flags: needinfo?(dave.hunt)
Priority: -- → P3
(In reply to Dave Hunt [:davehunt] [he/him] ⌚️UTC from comment #5)
> See also bug 1512615.

Sorry, that should be bug 1490399.
As discussed with :davehunt, we will first land this just as an option in the test INI, with the default being off. Then if we want to test new or existing pages on all the platforms we could just set 'screenshots = True' in the test INI and push that to try, and that will use the current behaviour taking a screenshot on each pagecycle.

Down the road in follow-up bugs we can decide if/how to turn it on in production i.e. possibly take a screenshot on the first page cycle and after that only when there is a timeout (good suggestion :davehunt!).
Status: NEW → ASSIGNED
Pushed by fstrugariu@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/1a25a39e7fb4
Support screenshots for tests r=rwood
https://hg.mozilla.org/mozilla-central/rev/1a25a39e7fb4
Status: ASSIGNED → RESOLVED
Closed: 8 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla66
You need to log in before you can comment on or make changes to this bug.