1861731 - Determine if we can easily add screenshot/video as output to failing benchmark tests

Status: RESOLVED FIXED

(Testing :: Raptor, enhancement, P3)

Description

[Kash Shampur [:kshampur] ⌚EST]

currently we output screenshots/videos in the treeherder artifacts for pageload tests. This has been useful in identifying issues in the past and resolving the failing tests

A similar thing could be useful for benchmark tests, probably as an optional flag that could be passed with the custom raptor-extra-options in treeherder

Comment 1

[Greg Mierzwinski [:sparky]]

We could integrate mozscreenshot into raptor like we're doing for talos (it's always enabled in CI too): bug 1861641

Comment 2

[Kash Shampur [:kshampur] ⌚EST]

That's cool was not aware of this tool
I initially was thinking something in browsertime (I am assuming mozscreenshot would not work with other browsers?)

Comment 3

[Kash Shampur [:kshampur] ⌚EST]

As I was debugging A linux failure in CI for Bug 1689537, I ended up adding this (based on comment 1)
e.g.

a hacky implementation was smoother than I anticipated, so I will probably write a quick patch for this now

Comment 4

[Kash Shampur [:kshampur] ⌚EST]

Attachment: Bug 1861731 - Add a method to capture screenshots of failing browsertime tests. r?#perftest

This patch uses mozscreenshot to take a screenshot of the test right up
to the moment of failure. This should be helpful in diagnosing failures
in CI like benchmark tests (which don't output videos like pageload
tests) as well as pageload tests if they fail before visual metrics can
capture a video/screenshot to output.

Comment 5

[Kash Shampur [:kshampur] ⌚EST]

This might help with e.g. Bug 1774954

Comment 6

[Kash Shampur [:kshampur] ⌚EST]

Attachment: Bug 1861731 - Fix screenshot_on_failure typo in Talos. r?#perftest

This patch fixes the logic for --screenshot-on-failure in Talos that was
caught.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1e62a9735dfb
Add a method to capture screenshots of failing browsertime tests. r=perftest-reviewers,sparky
https://hg.mozilla.org/integration/autoland/rev/5640911065b0
Fix screenshot_on_failure typo in Talos. r=perftest-reviewers,sparky

Comment 8

[Cristina Horotan [:chorotan]]

https://hg.mozilla.org/mozilla-central/rev/1e62a9735dfb
https://hg.mozilla.org/mozilla-central/rev/5640911065b0