Closed Bug 1861731 Opened 9 months ago Closed 5 months ago

Determine if we can easily add screenshot/video as output to failing benchmark tests

Categories

(Testing :: Raptor, enhancement, P3)

enhancement

Tracking

(firefox125 fixed)

RESOLVED FIXED
125 Branch
Tracking Status
firefox125 --- fixed

People

(Reporter: kshampur, Assigned: kshampur)

References

(Blocks 1 open bug)

Details

(Whiteboard: [fxp])

Attachments

(2 files)

currently we output screenshots/videos in the treeherder artifacts for pageload tests. This has been useful in identifying issues in the past and resolving the failing tests

A similar thing could be useful for benchmark tests, probably as an optional flag that could be passed with the custom raptor-extra-options in treeherder

Component: Performance → Raptor

We could integrate mozscreenshot into raptor like we're doing for talos (it's always enabled in CI too): bug 1861641

See Also: → 1861641

That's cool was not aware of this tool
I initially was thinking something in browsertime (I am assuming mozscreenshot would not work with other browsers?)

As I was debugging A linux failure in CI for Bug 1689537, I ended up adding this (based on comment 1)
e.g.

a hacky implementation was smoother than I anticipated, so I will probably write a quick patch for this now

Assignee: nobody → kshampur
Status: NEW → ASSIGNED

This patch uses mozscreenshot to take a screenshot of the test right up
to the moment of failure. This should be helpful in diagnosing failures
in CI like benchmark tests (which don't output videos like pageload
tests) as well as pageload tests if they fail before visual metrics can
capture a video/screenshot to output.

Attachment #9382409 - Attachment description: WIP: Bug 1861731 - Add a method to capture screenshots of failing browsertime tests. r?#perftest → Bug 1861731 - Add a method to capture screenshots of failing browsertime tests. r?#perftest

This might help with e.g. Bug 1774954

See Also: → 1774954
Attachment #9382409 - Attachment description: Bug 1861731 - Add a method to capture screenshots of failing browsertime tests. r?#perftest → WIP: Bug 1861731 - Add a method to capture screenshots of failing browsertime tests. r?#perftest
Attachment #9382409 - Attachment description: WIP: Bug 1861731 - Add a method to capture screenshots of failing browsertime tests. r?#perftest → Bug 1861731 - Add a method to capture screenshots of failing browsertime tests. r?#perftest

This patch fixes the logic for --screenshot-on-failure in Talos that was
caught.

Attachment #9388121 - Attachment description: Bug 1861731 - Fix screenshot_on_failures typo in Talos. r?#perftest → Bug 1861731 - Fix screenshot_on_failure typo in Talos. r?#perftest
Blocks: 1882639
Pushed by kshampur@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/1e62a9735dfb
Add a method to capture screenshots of failing browsertime tests. r=perftest-reviewers,sparky
https://hg.mozilla.org/integration/autoland/rev/5640911065b0
Fix screenshot_on_failure typo in Talos. r=perftest-reviewers,sparky
Status: ASSIGNED → RESOLVED
Closed: 5 months ago
Resolution: --- → FIXED
Target Milestone: --- → 125 Branch
See Also: → 1883374
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: