Last Comment Bug 1262260 - [Tracking] Workflow to run debuggers directly on interactive workers
: [Tracking] Workflow to run debuggers directly on interactive workers
Status: NEW
Product: Testing
Classification: Components
Component: General (show other bugs)
: unspecified
: Unspecified Unspecified
-- normal (vote)
: ---
Assigned To: Nobody; OK to take it and work on it
Depends on: 1262605 1271677 1273158 1278313 1278540 1284998 1290126 1293259 1301382 1302518 1302519 1302520 1303031 1248977 1250904 1253658 1273134 1285007 1285550 1285553 1285582 1292561 1293295 1296735 1299599 1300163 1301431 1304484 1308511
Blocks: awesome-try
  Show dependency treegraph
Reported: 2016-04-05 13:15 PDT by Andrew Halberstadt [:ahal]
Modified: 2016-10-07 07:16 PDT (History)
9 users (show)
See Also:
Crash Signature:
QA Whiteboard:
Iteration: ---
Points: ---


Description User image Andrew Halberstadt [:ahal] 2016-04-05 13:15:56 PDT
Just had a chat with Jonas, this might actually be less of a pipe dream than we originally thought, a lot of the leg work is done.

It's actually theoretically possible to e.g debug a test job right now with gdb using interactive jobs and the browser based shell. The only problems are that it's untested, undocumented and likely requires heavy modifications to the task definitions/docker scripts. (Also one-click loaner is already live and working).

One problem is that the task needs to pause after the initial setup. E.g, we want it to download all the relevant builds, tests, install the virtualenv, etc, but then delay execution of the payload (e.g test harness) until developers have had a chance to connect and attach a debugger. Armen has already started this work in bug 1250904.

Once that is solved, the remaining work is mostly workflow improvements and documentation. For example, we could create a class of tasks that are only interactive that automatically pause on start, print a helpful message of the day with instructions on resuming the payload, and use an image pre-loaded with all the debuggers a developer could possibly want.

The UX doesn't need to be perfect for a first iteration, people will be thrilled that it's possible at all. This bug should:
1. Make it possible to attach a debugger to a 1-click loaner without modifying the source.
2. Provide clear documentation.

It's also worth mentioning that Jonas has a GSoC project to create a react-based UI around gdb, and allow developers to debug tasks directly from this new browser-ui. This sounds awesome, but I don't think we should block on it. Creating a good debugging workflow for arbitrary debugging with the existing browser terminal is crazy useful as it stands.

For posterity, here are some notes from my chat with Jonas:
Comment 1 User image Ted Mielczarek [:ted.mielczarek] 2016-04-12 09:42:17 PDT
We've talked about making test invocation on our test machines more closely mirror what developers do in the past. If our test machines had a source checkout (which is nice for other reasons), and during setup they configured things such that just running `mach mochitest --debugger` from that source dir worked (pointing at the downloaded binaries, etc), that'd be pretty fantastic. That would boil this down to "make the mozharness scripts do all the setup and quit", and test invocation would look like it does on a developer machine with a local build.
Comment 2 User image Armen Zambrano - Back on March 27th [:armenzg] (EDT/UTC-4) 2016-04-12 17:29:52 PDT
I've changed TC interactive jobs in such way that Mozharness sets everything up and tells the developer what to do to run the tests.
Comment 3 User image Andrew Halberstadt [:ahal] 2016-05-10 07:23:30 PDT
Going to take a shot at finishing this. I'm hitting a couple problems when connecting with one-click loaner (haven't tried setting interactive in the task yet). Unfortunately the turn around time to test changes is pretty slow on this (as it depends on try jobs finishing and then connecting interactively).
Comment 4 User image Andrew Halberstadt [:ahal] 2016-05-10 07:24:24 PDT
(That comment was meant for bug 1250904, but I'm going to be working on this one too)
Comment 5 User image Armen Zambrano - Back on March 27th [:armenzg] (EDT/UTC-4) 2016-07-05 14:42:27 PDT
FYI we've removed the following bug from the list of dependencies:
* Bug 1234728 - [meta] Run tests from source directory

It is not a requirement for completion but a "it would have made this easier".
We won't be tackling it inside of this project.

Note You need to log in before you can comment on or make changes to this bug.