[Tracking] Workflow to run debuggers directly on interactive workers



a year ago
7 months ago


(Reporter: ahal, Unassigned)


(Depends on: 13 bugs, Blocks: 1 bug)

Firefox Tracking Flags

(Not tracked)


Just had a chat with Jonas, this might actually be less of a pipe dream than we originally thought, a lot of the leg work is done.

It's actually theoretically possible to e.g debug a test job right now with gdb using interactive jobs and the browser based shell. The only problems are that it's untested, undocumented and likely requires heavy modifications to the task definitions/docker scripts. (Also one-click loaner is already live and working).

One problem is that the task needs to pause after the initial setup. E.g, we want it to download all the relevant builds, tests, install the virtualenv, etc, but then delay execution of the payload (e.g test harness) until developers have had a chance to connect and attach a debugger. Armen has already started this work in bug 1250904.

Once that is solved, the remaining work is mostly workflow improvements and documentation. For example, we could create a class of tasks that are only interactive that automatically pause on start, print a helpful message of the day with instructions on resuming the payload, and use an image pre-loaded with all the debuggers a developer could possibly want.

The UX doesn't need to be perfect for a first iteration, people will be thrilled that it's possible at all. This bug should:
1. Make it possible to attach a debugger to a 1-click loaner without modifying the source.
2. Provide clear documentation.

It's also worth mentioning that Jonas has a GSoC project to create a react-based UI around gdb, and allow developers to debug tasks directly from this new browser-ui. This sounds awesome, but I don't think we should block on it. Creating a good debugging workflow for arbitrary debugging with the existing browser terminal is crazy useful as it stands.

For posterity, here are some notes from my chat with Jonas:
Depends on: 1262605
We've talked about making test invocation on our test machines more closely mirror what developers do in the past. If our test machines had a source checkout (which is nice for other reasons), and during setup they configured things such that just running `mach mochitest --debugger` from that source dir worked (pointing at the downloaded binaries, etc), that'd be pretty fantastic. That would boil this down to "make the mozharness scripts do all the setup and quit", and test invocation would look like it does on a developer machine with a local build.
I've changed TC interactive jobs in such way that Mozharness sets everything up and tells the developer what to do to run the tests.
Going to take a shot at finishing this. I'm hitting a couple problems when connecting with one-click loaner (haven't tried setting interactive in the task yet). Unfortunately the turn around time to test changes is pretty slow on this (as it depends on try jobs finishing and then connecting interactively).
Assignee: nobody → ahalberstadt
(That comment was meant for bug 1250904, but I'm going to be working on this one too)
Depends on: 1271677
Depends on: 1273134
Depends on: 1273158
Depends on: 1278313
Depends on: 1253658
Depends on: 1278540
Summary: Workflow to run debuggers directly on interactive workers → [Tracking] Workflow to run debuggers directly on interactive workers
FYI we've removed the following bug from the list of dependencies:
* Bug 1234728 - [meta] Run tests from source directory

It is not a requirement for completion but a "it would have made this easier".
We won't be tackling it inside of this project.
Depends on: 1284998
Depends on: 1285007
Depends on: 1285550
Depends on: 1285553
Depends on: 1285582
Depends on: 1290126
Depends on: 1248977
Depends on: 1292561
Depends on: 1293259
Depends on: 1293295
Depends on: 1296735
Depends on: 1300163
Depends on: 1299599
Assignee: ahalberstadt → nobody
Depends on: 1301382
Depends on: 1301431
Depends on: 1302518
Depends on: 1302519
Depends on: 1302520
Depends on: 1303031
Depends on: 1304484


7 months ago
Depends on: 1308511
You need to log in before you can comment on or make changes to this bug.