559465 - Historical debugging

Status: RESOLVED WONTFIX

(Tamarin Graveyard :: Debugger, defect)

Description

[Mike Morearty]

Tracking bug for the historical debugging feature we are considering doing.  Here is the description from an internal document, posted here with permission:


Capture a complete (or mostly complete) trace of the execution of a run of a program, including every modification to memory, and provide the developer with rich tools for mining this information to detect and diagnose problems. Developers are exposed to this kind of forensic problem solving in profilers, but can use the same techniques to find faulty logic as well.

Imagine a developer faced with some symptom of a bug but no idea yet what the cause is. Typically, the developer will run their program multiple times, probing the system with breakpoints or traces to examine the state of the application at significant points, honing in on the problem with a binary search through the timeline of their session. Each time they gain a bit more information, they insert new probes - setting a breakpoint, or inserting more trace statements to expose more information -- and either re-running, or re-compiling their application, until the find the problem.

With historical debugging, A developer has all the information they need to determine the cause of the problem without ever running the program again. Rather than setting a breakpoint, they simply examine the state of the program at whatever point in time they think will be helpful. If it turns out the problem occurred earlier, they just scrub back in time to a new point. Since all data is captured, there is no need to insert trace statements and re-run the program - any value at any time can be examined with the one single recording.

'Un-reproducible' bug cases potentially become much less of a problem as well. QE engineers can submit recordings along side bug reports, allowing the developer to examine the state of the reporters bug directly without having to reproduce it.

The core of historical debugging is the difficult task of efficiently capturing the trace recording of the program. Once the recording is made, investments can be made in a variety of different tools for mining the data.

Comment 1

[Mike Morearty]

It seems to me that an obvious way to do this would be to take advantage of write barriers -- since we already have a write barrier every time an ActionScript variable gets assigned a new value (right?), we could modify the write barrier to record the assignment.

Edwin and Lars, what do you think of that?  Seems to me like that might actually be pretty easy.  (Of course, the devil may be in the details.)

Comment 2

[Lars T Hansen]

Right now write barriers are pretty heavyweight and call out of line and it wouldn't be too hard.  I suspect they're also not terribly well optimized so a lot of writes are handled.

However it's not true that barriers fire every time we change a variable; they only fire when we change a slot in a heap-allocated object (never for locals or stack-allocated variables).

An additional problem is that some C++ code skips the barriers for some system operations, because the code is known not to need them (the barrier wouldn't change anything).  That may be strictly speaking a bug though, that it would be good to fix.

Anyhow, in the (near) future we want to have fast inline barriers (a branchless sequence of instructions to store a byte in a map) and elide them altogether when the type information says we can.  Rather than piggyback a replay debugger on them a replay debugger should use its own mechanism, and in some cases and with some implementations it may be possible to merge the replay logger and the standard write barrier.

Comment 3

[Felix S. Klock II [:pnkfelix, :fklock]]

Related work I have personally seen:

Green Hills Time Machine:

  http://www.ghs.com/products/timemachine.html

This takes advantage of trace hardware; so the implementation technique is not likely to be relevant to us, but it is still worth reviewing.  (Just knowing that the hardware-support option is out there is a good thing.)

Omniscient Debugging:

  http://www.lambdacs.com/debugger/

This is a JVM-based approach.

Universal Reversible Debugger:

  http://urdb.sourceforge.net/

This is an impressive generic approach that works on native code; it relies on a relatively fast system checkpointing operation.  The version I saw was developed for Linux x86; I do not know if they support any non-llinux platforms yet.

Comment 4

[Simon Wilkinson]

I wrote a survey/report on this subject back in February. It covers 'replay', 'reversible' and 'omniscient' debuggers ('historical' is a bit fuzzy, but is probably closest to omniscient).

Ping me if anybody wants a copy. It contains 30+ references for further reading.

In the mean time, I'll add the following to Felix's list:
G. Pothier, É. Tanter, and J. Piquer. Scalable Omniscient Debugging.
http://pleiad.dcc.uchile.cl/papers/2007/pothierAl-oopsla2007.pdf
(You can get the Eclipse plugin from the University of Chile's TOD project too: http://pleiad.dcc.uchile.cl/tod/index.html)

It has some interesting discussion on how you would persist the trace information, run queries against it, and integrate it into an IDE. I think these are much harder questions than capturing the trace itself (the space and time optimization is still important though).

I've also been thinking about how all this stuff would work with the Player. Again, ping me if you want to discuss...

Comment 5

[Sylvestre Ledru [:Sylvestre]]

Tamarin isn't maintained anymore. WONTFIX remaining bugs.