Closed Bug 1400380 Opened 7 years ago Closed 7 years ago

Doesn't seem to be possible to run Talos under a debugger

Categories

(Testing :: Talos, enhancement)

Product:
Testing
Component:
Talos
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(firefox59 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla59
Tracking Flags:
Tracking Status
firefox59 --- fixed

People

(Reporter: bzbarsky, Assigned: igoldan)

References

Details

(Whiteboard: [PI:December])

Attachments

(1 file, 2 obsolete files)

Bug 1400380 - run Talos with browser in debug mode
59 bytes, text/x-review-board-request
jmaher
: review+
Details 
"mach talos-test" doesn't support a --debugger option, as far as I can tell.  How am I supposed to debug something that fails during a talos run?
Flags: needinfo?(rwood)
Flags: needinfo?(jmaher)
I see bug 1237215 for using a --jsdebugger; what debuggers would be useful?  I don't think this is impossible, but there have been very few requests for this in the last 5 years- I put this in our next queue of work to consider.
Flags: needinfo?(jmaher)
Whiteboard: [PI:October]
> what debuggers would be useful? 

gdb and lldb come to mind, and whatever people use on Windows.

Basically, it should work just like all our unit test suites work: you pass in a debugger string and that gets used.
Oh, and the reason I expect there have been so few requests is that even just running Talos locally is so painful, and so broken, that people printf-debug on try instead.  :(
Flags: needinfo?(ionut.goldan)
Flags: needinfo?(rwood)
Assignee: nobody → igoldan
Flags: needinfo?(igoldan)
Implementation-wise: does this involve sticking gdb in the command line arguments that Talos prepares here
https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/ttest.py?q=path%3Atalos+file%3Attest.py&redirect_type=single#174
before it invokes the perf test?
Right before the browser path?
Flags: needinfo?(rwood)
(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #5)
> Implementation-wise: does this involve sticking gdb in the command line
> arguments that Talos prepares here
> https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/ttest.
> py?q=path%3Atalos+file%3Attest.py&redirect_type=single#174
> before it invokes the perf test?
> Right before the browser path?

Hi Ionut,

First of all I suggest you have a look at how other tests (i.e. mochitest) support debuggers (if you've already done that, awesome!).

Perhaps we could have talos options for each debugger that is implemented, i.e. '--gdb', '--lldb', '--jsdebugger', etc. I did a grep for 'jsdebugger' and here's where it is implemented in mochitest [1].

I would start there and follow the code through and see how the jsdebugger is started, etc. That will hopefully give you a good idea how to mirror similar functionality in talos perhaps? Maybe also look if gdb is supported in other unit tests?

When it comes time to adding debugger support to talos, you'd want it to work with ./mach talos-test for running talos and debugging locally. First step you'd have to add a talos command line option, i.e. in [2].

Then figure out where to start/stop the debugger etc, perhaps at or near the code link you asked about but I'm not sure.

[1] http://searchfox.org/mozilla-central/rev/1285ae3e810e2dbf2652c49ee539e3199fcfe820/testing/mochitest/mochitest_options.py#394
[2] http://searchfox.org/mozilla-central/rev/1285ae3e810e2dbf2652c49ee539e3199fcfe820/testing/talos/talos/cmdline.py#54
Flags: needinfo?(rwood)
Note that I was trying to run with --debugger and --debugger-args, because I wanted to run under rr, not one of the built-in debugger options.
(In reply to Boris Zbarsky [:bz] (no decent commit message means r-) from comment #7)
> Note that I was trying to run with --debugger and --debugger-args, because I
> wanted to run under rr, not one of the built-in debugger options.

Cool yes that would be ideal if talos could support those generic cmd line options for all debuggers instead of implementing one cmd line option for each debugger type.
:bz Can you please name some of the common mach subcommands you use, that have the desired CLI options? So I know precisely what kind of flow I need to integrate into Talos.
Flags: needinfo?(bzbarsky)
reftest, mochitest, wpt.
Flags: needinfo?(bzbarsky)
Comment on attachment 8923831 [details]
Bug 1400380 - start integrating debugger

https://reviewboard.mozilla.org/r/194996/#review200028

overall this looks good.  A few questions:
1) what did you do to test this?
2) how can you write unittests to ensure this doesn't break? (followup bug is ok)
3) does this break anything on try, and does it work with/without debugger args via ./mach?
Attachment #8923831 - Flags: review?(jmaher) → review+
Comment on attachment 8923831 [details]
Bug 1400380 - start integrating debugger

https://reviewboard.mozilla.org/r/194996/#review200028

This isn't currently functional. I wanted to be sure I'm on the right track.
I still need to prepare the final command line we use to run the browser. To do that,
I need to understand the mozdebug lib we have.
Whiteboard: [PI:October] → [PI:November]
We configure subprocess.PIPE here:
https://dxr.mozilla.org/mozilla-central/source/testing/mozbase/mozprocess/mozprocess/processhandler.py#742
so out threads defined here
https://dxr.mozilla.org/mozilla-central/source/testing/mozbase/mozprocess/mozprocess/processhandler.py#943
can process it when they like.
This messes the interaction with gdb.

I am able to give commands to gdb; the Talos test complete without problems. But it feels faulty and buggy.
Do you have suggestions for handling this? I will look for some options, but we need to agree on a special handling when using a debugger.
Flags: needinfo?(jmaher)
Flags: needinfo?(rwood)
(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #18)
> We configure subprocess.PIPE here:
> https://dxr.mozilla.org/mozilla-central/source/testing/mozbase/mozprocess/
> mozprocess/processhandler.py#742
> so out threads defined here
> https://dxr.mozilla.org/mozilla-central/source/testing/mozbase/mozprocess/
> mozprocess/processhandler.py#943
> can process it when they like.
> This messes the interaction with gdb.
> 
> I am able to give commands to gdb; the Talos test complete without problems.
> But it feels faulty and buggy.
> Do you have suggestions for handling this? I will look for some options, but
> we need to agree on a special handling when using a debugger.

Thanks Ionut. How do the other frameworks (like those listed in comment 10) handle/enable debuggers? Maybe there's some code there that will provide direction? Also maybe list specific issues that you are running into, that demonstrate "it feels faulty and buggy".
Flags: needinfo?(rwood)
mochitest uses --interactive on the command line:
https://searchfox.org/mozilla-central/source/testing/mochitest/runtests.py#538
Flags: needinfo?(jmaher)
(In reply to Robert Wood [:rwood] from comment #19)
> Thanks Ionut. How do the other frameworks (like those listed in comment 10)
> handle/enable debuggers? Maybe there's some code there that will provide
> direction? Also maybe list specific issues that you are running into, that
> demonstrate "it feels faulty and buggy".
Out of the 3 tests mentioned in comment 10, I looked over reftest; I investigate gtest's implementation too.
Both of these tools at some point launch a subprocess like |gdb firefox <various arguments>|.
It's plain simple as you get prompted into gdb directly and interract with it.

Problem is Talos handles things very differently. Our business logic involves Talos' output be consumed by a separate thread.
That thread reads the output line by line and then logs it again to the console. It acts like an intermediary.

The tricky part in this equation is this: gdb is interactive, so it prompts something like:
(gdb) <type command here>

Until you hit Enter, the thread won't be able to consume anything. You won't see that initial (gdb) prompt, so you'll type blindly.
This is the faulty behavior I was referring to.
in debugger mode we can avoid parsing the stdout- it will break some tests, but I think that is acceptable.  The goal here is to debug the code while firefox is running a test- vs when firefox is collecting metrics.

Another thing is we might want to skip the profile warmup when running with a debugger.  another random thought- we now have 'headless' mode for firefox and we could run the profile warmup in headless mode, this would make the user experience locally much less confusing as they wouldn't see a window popup and disappear quickly.
Attachment #8923831 - Attachment is obsolete: true
Attachment #8928533 - Attachment is obsolete: true
Comment on attachment 8923830 [details]
Bug 1400380 - run Talos with browser in debug mode

https://reviewboard.mozilla.org/r/194994/#review206476

this is looking really good, thanks for working on solving the problem of debugging talos.  I am sure there will be hiccups going forward.  A few thoughts:
1) it would be nice if all our harnesses has a mozdebug process that we could manage the cli properly, etc. (not required for this)
2) can you outline what commands (and os + build configs) you used to test this.  This will help when a developer runs into problems in the future
3) can you ensure we don't accidentally break anything on try
Attachment #8923830 - Flags: review?(jmaher) → review+
(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #24)
> Comment on attachment 8923830 [details]
> Bug 1400380 - run Talos with browser in debug mode
> 
> https://reviewboard.mozilla.org/r/194994/#review206476
> 
> this is looking really good, thanks for working on solving the problem of
> debugging talos.  I am sure there will be hiccups going forward.  A few
> thoughts:
> 1) it would be nice if all our harnesses has a mozdebug process that we
> could manage the cli properly, etc. (not required for this)
I don't follow. Can you elaborate on this?
> 2) can you outline what commands (and os + build configs) you used to test
You can get into debug mode using |mach talos-test -a <perf_test> --debugger gdb|.
Assuming the default debuggers are properly configured (as mozdebug is expecting here https://dxr.mozilla.org/mozilla-central/source/testing/mozbase/mozdebug/mozdebug/mozdebug.py#27), you should also be able to run the command above with the --debug/--debug_args flags. I didn't test this yet, but I don't expect issues from this part.
You get prompted into the debugger directly.
I only executed 'run' under gdb, to check that Firefox runs without problems and that Talos closes properly.
I tested this on Windows 10, using the gdb for this platform.
> this.  This will help when a developer runs into problems in the future
> 3) can you ensure we don't accidentally break anything on try
Before landing this, I'll do a thorough testing.
I am thinking we have mozprocess, mozprofile, etc. but each harness has custom code for managing the cli flags and setting up the debugger args- ideally we can have a common module that helps with that.  I view this as unrelated to this bug.

Please test all the cli flags locally and test on try to ensure no odd side effects.  When you are ready for landing request checkin-needed or ping myself or :rwood online.
Whiteboard: [PI:November] → [PI:December]
I tested this thoroughly here:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=5420c4afc6f27b73dc09bec17d76ec23eb607399
So far, the jobs are green, with only one orange.
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9137f72f3043
run Talos with browser in debug mode r=jmaher
https://hg.mozilla.org/mozilla-central/rev/9137f72f3043
Status: NEW → RESOLVED
Closed: 7 years ago
status-firefox59: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla59
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: