make Raptor support screenshots when running tests This would be useful when testing on different platforms or verifying different tests
I did a little hacking today to see how we can get this to work. it still needs some work but it's a good start Try job: https://treeherder.mozilla.org/#/jobs?repo=try&revision=b9897656008c13ccad711b3bf6158a309a553491 You can find the screenshots in the job details artifacts. This would be really helpful when reviewing a tp6 tests.
:rwood :davehunt please share your thought on this "feature" and how should it be implemented long term
Assignee: nobody → fstrugariu
This is really cool, thanks :bebe! Please see the feedback in the phabricator review. Quick summary: 1. Taking screenshots should be an option in the test INI i.e. screenshot = true/false, so that we can turn this on/off for specific tests. 2. We should expand this to work with benchmark type of tests too - when the runner.js receives the benchmark result from benchmark-relay.js, the runner.js could take the screenshot at that time, same method as pageload tests. 3. In the screenshot.html artifact, please include the test name, and use 'pagecycles' instead of 'test' and start at 1. i.e. headings should be: 'raptor-tp6-google-firefox pagecycle 1' to 25. That will match the logging and make it easier to map screenshots to the test logs. 4. Doesn't work in google chromium, needs a fix for that (see phabricator). 5. Please see the additional notes in-line in the phabricator review. Thanks!
See also bug 1512615.
Priority: -- → P3
(In reply to Dave Hunt [:davehunt] [he/him] ⌚️UTC from comment #5) > See also bug 1512615. Sorry, that should be bug 1490399.
As discussed with :davehunt, we will first land this just as an option in the test INI, with the default being off. Then if we want to test new or existing pages on all the platforms we could just set 'screenshots = True' in the test INI and push that to try, and that will use the current behaviour taking a screenshot on each pagecycle. Down the road in follow-up bugs we can decide if/how to turn it on in production i.e. possibly take a screenshot on the first page cycle and after that only when there is a timeout (good suggestion :davehunt!).
Updated and created the try job: https://treeherder.mozilla.org/#/jobs?repo=try&revision=ad53228de2ebb2b475bfe9484f407432c0bcc721&selectedJob=216992237
Fixed the chrome errors: https://treeherder.mozilla.org/#/jobs?repo=try&revision=7794bbd49f898e08d3572d5e2a74ba49588d32f9
Pushed by firstname.lastname@example.org: https://hg.mozilla.org/integration/autoland/rev/1a25a39e7fb4 Support screenshots for tests r=rwood
You need to log in before you can comment on or make changes to this bug.