1655554 - Debugging Firefox failures in raptor is too complicated and needs simplification

Status: RESOLVED FIXED

(Testing :: Raptor, task, P2)

Description

[Greg Mierzwinski [:sparky]]

This bug is for finding a way to make it simpler for devs to debug firefox when it's run through Raptor.

Recently, a couple people have been debugging failures on firefox through raptor but raptor is really not setup for this. See some of these conversations with :agi:
(1) https://matrix.to/#/!LfXZSWEroPFPMQcYmw:mozilla.org/$DIUPTlVCgvTBNRDtQu8Qxjad22TYuZIvzoho_oxJYPk?via=mozilla.org&via=matrix.org&via=rduce.org
(2) https://matrix.to/#/!LfXZSWEroPFPMQcYmw:mozilla.org/$kbvnMMip1feJpKAlPveTk80Jd1WAlLBLdpiw8ujb36A?via=mozilla.org

I think a quick win here would be to get a --verbose-try flag available in ./mach try fuzzy that would setup a highly verbose log (with MOZ_LOG set to capture everything). This would depend on bug 1494669 for setting the environment variables from the command line. It would also depend on the --verbose flag from bug 1653617.

Comment 1

[Greg Mierzwinski [:sparky]]

I've made a small entry in the wiki to hold the current methodology for debugging these failures: https://wiki.mozilla.org/TestEngineering/Performance/Raptor#Debugging_Desktop_Product_Failures

Comment 2

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1655554 - Fix debug-mode in Raptor-Browsertime. r?#perftest

This patch updates browsertime to add the following PR: https://github.com/sitespeedio/browsertime/pull/1798

It enables us to use a --debug flag in browsertime to pause test execution after each script iteration, and provides the ability for developers to pause it on their own by adding debug.breakpoint(name) calls to the test script. The execution can be unpaused by running window.browsertime.pause = false in the browser console.

The devtools panel is auto-opened when in debug-mode, and the browser is detached so it won't auto-close when you hit failures in raptor/browsertime.

This can only be used locally so we now raise a parser error when it's provided in CI.

Comment 3

[Greg Mierzwinski [:sparky]]

The work involving adding an easy way to run with verbose on try will be done in bug 1772541 as part of the try chooser rework (it's possible we end up with a different way of implementing this through that work).

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5718cf939e0d
Fix debug-mode in Raptor-Browsertime. r=perftest-reviewers,kshampur

Comment 6

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/5718cf939e0d