1512615 - Support screenshots for tests

Status: RESOLVED FIXED

(Testing :: Raptor, enhancement, P3)

Description

[Florin Strugariu [:Bebe]]

make Raptor support screenshots when running tests

This would be useful when testing on different platforms or verifying different tests

Comment 1

[Florin Strugariu [:Bebe]]

Attachment: Bug 1512615 Support screenshots for tests

Comment 2

[Florin Strugariu [:Bebe]]

I did a little hacking today to see how we can get this to work.

it still needs some work but it's a good start

Try job:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=b9897656008c13ccad711b3bf6158a309a553491

You can find the screenshots in the job details artifacts.

This would be really helpful when reviewing a tp6 tests.

Comment 3

[Florin Strugariu [:Bebe]]

:rwood :davehunt please share your thought on this "feature" and how should it be implemented long term

Comment 4

[Robert Wood [:rwood]]

This is really cool, thanks :bebe! Please see the feedback in the phabricator review. Quick summary:

1. Taking screenshots should be an option in the test INI i.e. screenshot = true/false, so that we can turn this on/off for specific tests.

2. We should expand this to work with benchmark type of tests too - when the runner.js receives the benchmark result from benchmark-relay.js, the runner.js could take the screenshot at that time, same method as pageload tests.

3. In the screenshot.html artifact, please include the test name, and use 'pagecycles' instead of 'test' and start at 1. i.e. headings should be:

'raptor-tp6-google-firefox pagecycle 1' to 25.

That will match the logging and make it easier to map screenshots to the test logs.

4. Doesn't work in google chromium, needs a fix for that (see phabricator).

5. Please see the additional notes in-line in the phabricator review.

Thanks!

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

See also bug 1512615.

Comment 6

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Dave Hunt [:davehunt] [he/him] ⌚️UTC from comment #5)
> See also bug 1512615.

Sorry, that should be bug 1490399.

Comment 7

[Florin Strugariu [:Bebe]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=8a4b940d27857b4872f8d35bf93c534a1f301c12

Comment 8

[Robert Wood [:rwood]]

As discussed with :davehunt, we will first land this just as an option in the test INI, with the default being off. Then if we want to test new or existing pages on all the platforms we could just set 'screenshots = True' in the test INI and push that to try, and that will use the current behaviour taking a screenshot on each pagecycle.

Down the road in follow-up bugs we can decide if/how to turn it on in production i.e. possibly take a screenshot on the first page cycle and after that only when there is a timeout (good suggestion :davehunt!).

Comment 9

[Florin Strugariu [:Bebe]]

Updated and created the try job: 
https://treeherder.mozilla.org/#/jobs?repo=try&revision=ad53228de2ebb2b475bfe9484f407432c0bcc721&selectedJob=216992237

Comment 10

[Florin Strugariu [:Bebe]]

Fixed the chrome errors: 
https://treeherder.mozilla.org/#/jobs?repo=try&revision=7794bbd49f898e08d3572d5e2a74ba49588d32f9

Comment 11

[Florin Strugariu [:Bebe]]

latest try run https://treeherder.mozilla.org/#/jobs?repo=try&revision=699b9f4cc9cfc3cce5cd4a071d53c367bda8a5b8

Comment 12

[Florin Strugariu [:Bebe]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c57848091afb4f8918de4abf8474915b907dca44

Comment 13

[Florin Strugariu [:Bebe]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=a2d42acef818399dc575df81417a9dc1e72bc3c1

Comment 14

[Florin Strugariu [:Bebe]]

rebase https://treeherder.mozilla.org/#/jobs?repo=try&revision=7c0e75f42796b076ebe7afe6a2c7ca2508f48424

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1a25a39e7fb4
Support screenshots for tests r=rwood

Comment 16

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/1a25a39e7fb4