1262260 - [Tracking] Workflow to run debuggers directly on interactive workers

Status: NEW

(Testing :: General, defect)

Description

[Andrew Halberstadt [:ahal]]

Just had a chat with Jonas, this might actually be less of a pipe dream than we originally thought, a lot of the leg work is done.

It's actually theoretically possible to e.g debug a test job right now with gdb using interactive jobs and the browser based shell. The only problems are that it's untested, undocumented and likely requires heavy modifications to the task definitions/docker scripts. (Also one-click loaner is already live and working).

One problem is that the task needs to pause after the initial setup. E.g, we want it to download all the relevant builds, tests, install the virtualenv, etc, but then delay execution of the payload (e.g test harness) until developers have had a chance to connect and attach a debugger. Armen has already started this work in bug 1250904.

Once that is solved, the remaining work is mostly workflow improvements and documentation. For example, we could create a class of tasks that are only interactive that automatically pause on start, print a helpful message of the day with instructions on resuming the payload, and use an image pre-loaded with all the debuggers a developer could possibly want.

The UX doesn't need to be perfect for a first iteration, people will be thrilled that it's possible at all. This bug should:
1. Make it possible to attach a debugger to a 1-click loaner without modifying the source.
2. Provide clear documentation.

It's also worth mentioning that Jonas has a GSoC project to create a react-based UI around gdb, and allow developers to debug tasks directly from this new browser-ui. This sounds awesome, but I don't think we should block on it. Creating a good debugging workflow for arbitrary debugging with the existing browser terminal is crazy useful as it stands.

For posterity, here are some notes from my chat with Jonas:
https://public.etherpad-mozilla.org/p/jonasfj-ahal-ramblings-interactive

Comment 1

[(not currently active) Ted Mielczarek]

We've talked about making test invocation on our test machines more closely mirror what developers do in the past. If our test machines had a source checkout (which is nice for other reasons), and during setup they configured things such that just running `mach mochitest --debugger` from that source dir worked (pointing at the downloaded binaries, etc), that'd be pretty fantastic. That would boil this down to "make the mozharness scripts do all the setup and quit", and test invocation would look like it does on a developer machine with a local build.

Comment 2

[Armen [:armenzg]]

I've changed TC interactive jobs in such way that Mozharness sets everything up and tells the developer what to do to run the tests.

Comment 3

[Andrew Halberstadt [:ahal]]

Going to take a shot at finishing this. I'm hitting a couple problems when connecting with one-click loaner (haven't tried setting interactive in the task yet). Unfortunately the turn around time to test changes is pretty slow on this (as it depends on try jobs finishing and then connecting interactively).

Comment 4

[Andrew Halberstadt [:ahal]]

(That comment was meant for bug 1250904, but I'm going to be working on this one too)

Comment 5

[Armen [:armenzg]]

FYI we've removed the following bug from the list of dependencies:
* Bug 1234728 - [meta] Run tests from source directory

It is not a requirement for completion but a "it would have made this easier".
We won't be tackling it inside of this project.