Open Bug 636907 Opened 13 years ago Updated 2 years ago

SpiderMonkey should provide a low-level interface in JavaScript to JS debugging

Categories

(Core :: JavaScript Engine, defect)

defect

Tracking

()

mozilla2.0

People

(Reporter: jimb, Assigned: jimb)

References

(Depends on 5 open bugs, Blocks 1 open bug)

Details

(Keywords: dev-doc-needed)

SpiderMonkey should provide a debug object with functions providing a safe JavaScript interface to the JSDBGAPI facilities and the debugging hooks. This would be an alternative to JSD, a basis for the future debugging protocol server, and make it easy to test debugging facilities in SM's current JS-based test suite. It will begin life rough and rudimentary, so that we can release early and often as we improve it.

The debug object should be part of SpiderMonkey itself, not a separate directory like JSD. Embeddings would call something like JS_DefineDebugFunctions to add the debugging functions to their global if desired.

To avoid dependencies that may might be annoying to embedders, it should be in straight C++, not based on an IDL interface, to permit an idiomatic JS API (using objects the way Object.getOwnPropertyDescriptor does, for example).

The API should be safe, in that a buggy debugger should not be able to crash SpiderMonkey.

It should use compartments to enforce the boundary between the debugger and debuggee. It should protect the debugger from the debuggee's getters, setters, and so on; it should prevent the debuggee from referring to the debugger's objects at all; it should prevent the debugger from inserting breakpoints in its own code; and so on.

It should be nestable: it should support debugging debuggers. Placing debugger and debuggee in separate compartments makes it clear to whom each trap should be reported.

Having JSDBGAPI functionality available at the JS level will make it easier to test debugging functionality, since it will be testable via ordinary js/src/tests-style tests.

The initial cut will be as direct as practical a reflection of the existing JSDBGAPI functions and debug hooks, so that we can release early and often. The API will be unstable until we've done more hacking and have a better idea what we want.

Documentation wiki page here: https://wiki.mozilla.org/Debug_Object
Depends on: 637505
Blocks: 631377
Depends on: 611044
No longer blocks: 631377
Depends on: 631377
Now in wiki docs: overall Debug object, debug hook descriptions.

This is all pretty simple, but having the Debug object available in the shell is going to be a big win; already the 'error' hook gets us a way to test error message generation and the decompiler, which people have wanted for a long time.

Being able to test this stuff from the shell will also be exciting. Debugging is coming in from the cold!
Depends on: 637572
JJB, you mentioned a while back that you wanted the stack passed to every debug hook. The wiki sketch linked to in comment 0 doesn't pass a frame to the script creation and destruction hooks; I understand that you use the stack at that point in Firebug to reconstruct detailed origin information for scripts that are otherwise hard to attribute (appendChild and the like). If some idea like the one suggested in bug 637572 gets implemented, do you still need the stack at newScript and destroyScript?
Observation: This sounds an awful lot like jsdb, except without JSD, using compartments instead of runtimes, and present in web content.

Does this mean that remote debuggers are off the table? If not, how will the remote communicate with the debugger while the debuggee is running?
(In reply to comment #3)
> Observation: This sounds an awful lot like jsdb, except without JSD, using
> compartments instead of runtimes, and present in web content.

Thanks for reading this over!

jsdb includes user interaction stuff; this is just JS-level access to the underlying debug hooks. Yes, it's an alternative to jsd, but single-threaded; lacking the XPConnect dependency; and using compartments to check critical invariants, so we can sanely do things like debugging the debugger. It's meant to be enough to implement the debugging protocol server --- and write tests for the debugging support.

> Does this mean that remote debuggers are off the table? If not, how will the
> remote communicate with the debugger while the debuggee is running?

Not at all! Remote debugging is where we're headed. This is the bottom layer, on which a server can be written. I want something I can commit soon, play with, and revise, so as a starting point this is just a direct reflection of what's available via JSDBGAPI.

To keep the communication live while JS runs, I can imagine two kinds of architectures:

- Polling, where the running JS code's operation callback checks the communications channel (socket; pipe; inter-thread thingy) for requests from the client (the debugger UI); or

- Multi-threaded, where a separate thread watches the channel and interrupts the JS thread (using the operation callback stuff, or whatever the latest and greatest asynchronous interference API is).

In either case, we get to interrupt the debuggee and run debugger JS code, and the Debug object can do its job.

What's needed in the Debug object documentation for either of these scenarios is a way to get the debugger JS code the current stack frame of the interrupted code. Ideally it would receive it in a natural way, perhaps via a new hook? I think I need to understand the operation callback API better to do a good job with that.
This is exactly the right direction. Threading, remoting, etc. follow, using web technologies (not C++ libraries and OS-specific system calls!).

/be
(In reply to comment #2)
> JJB, you mentioned a while back that you wanted the stack passed to every debug
> hook. 

This request is based on the trouble we have with the jsd hooks that do not have stack arguments, onScriptCreated and onError. 

> The wiki sketch linked to in comment 0 doesn't pass a frame to the script
> creation and destruction hooks; 

Why not?

>I understand that you use the stack at that
> point in Firebug to reconstruct detailed origin information for scripts that
> are otherwise hard to attribute (appendChild and the like). If some idea like
> the one suggested in bug 637572 gets implemented, do you still need the stack
> at newScript and destroyScript?

 Stacks are just fundamental to debugging. When you call us, I guess you think about it as a call within a coherent environment and sequence of operations. But our experience is the opposite: we just get a call, "boom!" there it is.  We have to infer the environment and operations. The stack is what we need to make sense of the call.
(In reply to comment #4)
...
> > Does this mean that remote debuggers are off the table? If not, how will the
> > remote communicate with the debugger while the debuggee is running?
> 
> Not at all! Remote debugging is where we're headed. This is the bottom layer,
> on which a server can be written. I want something I can commit soon, play
> with, and revise, so as a starting point this is just a direct reflection of
> what's available via JSDBGAPI.

Firebug 1.8 will support at least prototype-quality remote JS debug, and probably production, depending on FF4.0+ schedules. So we're ready to start trying your api as soon as you land it. (1.8a1 ~two weeks).

> 
> To keep the communication live while JS runs, I can imagine two kinds of
> architectures:
> 
> - Polling, where the running JS code's operation callback checks the
> communications channel (socket; pipe; inter-thread thingy) for requests from
> the client (the debugger UI); or
> 
> - Multi-threaded, where a separate thread watches the channel and interrupts
> the JS thread (using the operation callback stuff, or whatever the latest and
> greatest asynchronous interference API is).

I guess 'event-driven' is half-way in between, with the channel-thread setting a flag that the running JS polls. Anyway, we have socket-based server now, but we also have experience with comet-like servers (holding on to an AJAX request until ready to reply).
(In reply to comment #6)
> (In reply to comment #2)
> > JJB, you mentioned a while back that you wanted the stack passed to every debug
> > hook. 
> 
> This request is based on the trouble we have with the jsd hooks that do not
> have stack arguments, onScriptCreated and onError. 

onError (the 'error' hook in the draft API) should clearly receive a stack frame. It's more the script creation and destruction stuff that I'm concerned about.

> > The wiki sketch linked to in comment 0 doesn't pass a frame to the script
> > creation and destruction hooks; 
> 
> Why not?

Primarily because the JSDBGAPI's JSDebugHooks members don't, but that's almost a non sequitur. A better argument is that JSScript creation isn't intrinsically related to any particular JS action; if some C++ code calls JS_CompileScript, that may not have anything to do with the top JS stack frame in the context. But see below.

> When you call us, I guess you think
> about it as a call within a coherent environment and sequence of operations.
> But our experience is the opposite: we just get a call, "boom!" there it is. 

This made me laugh really hard! Poor Firebug! Not sarcastic.

> We have to infer the environment and operations. The stack is what we need to
> make sense of the call.

Fair enough, but for the reason I gave above, whatever information you're inferring from the call stack, I think we should just be handing you directly via script location information; let's continue discussing this in bug 637572.
(In reply to comment #7)
> I guess 'event-driven' is half-way in between, with the channel-thread setting
> a flag that the running JS polls.

So the debuggee JS is checking this flag? Depending on the debuggee's cooperation seems sub-optimal; are there cases where this is the best choice available?
(In reply to comment #3)
> Observation: This sounds an awful lot like jsdb, except ... present in web
> content.

I wanted to address this: I don't think this should be present in web content. It should be added to globals only on request, like CTypes (one calls JS_InitCTypesClass if one wants the ctypes stuff available, as I understand it). The chrome global would have it; the shell global would have it; web globals would not.

This does raise an important point: debugging a compartment shouldn't magically permit any access that wouldn't otherwise be permitted by the debugger's and debuggee's compartments' relationship.
Depends on: 638053
When I mentioned jsdb above, I was specifically talking about js/jsd/jsdb/jsdrefl.cpp, not the UI written in JavaScript.

FWIW - since it never made it out of CVS, here's a link to my fork if you want to scan it (includes unreviewed patches from timeless) -- http://code.google.com/p/gpsee/source/browse/extras/gsrdb/jsd/jsdb/jsdrefl.cpp

Incidentally, by comparing to jsdb, I never meant to imply that this a bad idea or anything -- just pointing out that there may be opportunities for conceptual re-use.

JSD and jsdb also do not rely on XPConnect -- the only reliance is the way JSD is exposed to browser extensions, the jsdIDebuggerService.idl stuff.  JSD was written, IIUC, to be a high-level abstraction of the JSDBGAPI and then reflected into XPConnect, JavaScript, and Java.

I hadn't considered the operation callback in either scenario you described. That's clever.

Will it be tricky for debugger authors to handle the case where the operation callback interrupts the interpreter mid-statement?  (How does GDB handle getting an async signal mid-statement?)

As for API -- access to the debugger keyword hook, throw hook, etc, looks friendly. I'll assume "frame" carries with it filename and line number information. Does it carry the script object initially passed to the newScript hook?  How does the debugger request relevant source code from the throw/debugger/etc hooks?
(In reply to comment #9)
> (In reply to comment #7)
> > I guess 'event-driven' is half-way in between, with the channel-thread setting
> > a flag that the running JS polls.
> 
> So the debuggee JS is checking this flag? Depending on the debuggee's
> cooperation seems sub-optimal; are there cases where this is the best choice
> available?

Sorry I was not clear. In my world, both the debuggee and debugger are JS. The debugger starts socket event handlers and operates on the JS engine to control the debuggee. If a socket packet arrives, the engine passes control to the debugger on the next trip through the event loop which then operates on the JS engine to change the fate of the debuggee. We can use existing event logic for all of this (in fact we do). 

As nice improvement would be a way for the debugger to assert priority when the channel event arrives. This would allow a long running JS event in the debuggee to be paused. Normal JS event process would not allow this.  While the incremental work to support this feature does not seem difficult, but maintaining high performance with threads that share data is difficult. A unidirectional flag written by the channel handler and polled by the engine is all I am suggesting. In fact I was only really trying to paint this as small problem instead of a "Multi-threading architecture problem" ;-).
You know, it might be possible to achieve that behaviour by having the debuggee context yield from the operation callback.
(In reply to comment #12)
> Sorry I was not clear. In my world, both the debuggee and debugger are JS. The
> debugger starts socket event handlers and operates on the JS engine to control
> the debuggee. If a socket packet arrives, the engine passes control to the
> debugger on the next trip through the event loop which then operates on the JS
> engine to change the fate of the debuggee. We can use existing event logic for
> all of this (in fact we do). 
> 
> As nice improvement would be a way for the debugger to assert priority when the
> channel event arrives. This would allow a long running JS event in the debuggee
> to be paused. Normal JS event process would not allow this.

I understood that Firebug mostly worked via the event loop; I did not know that Firebug had no way (other than the operation callback) to interrupt long-running JS code. Now that I think about it, it's obvious --- nothing is listening while JS is running.

But that limitation is definitely something we want to ditch; it may be semi-acceptable for the browser, but it's certainly not okay for other embeddings. The ability to put the channel-watching code in another thread and then use the operation callback is another win for the debuggee-in-separate-thread approach.

> A
> unidirectional flag written by the channel handler and polled by the engine is
> all I am suggesting.

That's exactly what the operation callback is. JS_TriggerOperationCallback simply sets a bit in JSThreadData::interruptFlags.


> In fact I was only really trying to paint this as small
> problem instead of a "Multi-threading architecture problem" ;-).

Well, I hope that didn't make things seem more grandiose than they are. But that one little bit of cross-thread communication is critical. See also bug 638038.
(In reply to comment #11)
> JSD and jsdb also do not rely on XPConnect -- the only reliance is the way JSD
> is exposed to browser extensions, the jsdIDebuggerService.idl stuff.  JSD was
> written, IIUC, to be a high-level abstraction of the JSDBGAPI and then
> reflected into XPConnect, JavaScript, and Java.

The difference I was getting at was that JS will be able to get to debugging functionality without XPConnect --- at the moment, it has to go through js/jsd/idl/jsdIDebuggerService.idl, right?

> Will it be tricky for debugger authors to handle the case where the operation
> callback interrupts the interpreter mid-statement?  (How does GDB handle
> getting an async signal mid-statement?)

Well, any bytecode can throw an exception, so those states arise now, don't they?

> As for API -- access to the debugger keyword hook, throw hook, etc, looks
> friendly. I'll assume "frame" carries with it filename and line number
> information.

Yes.

> Does it carry the script object initially passed to the newScript
> hook?

Yes.

> How does the debugger request relevant source code from the
> throw/debugger/etc hooks?

The hook gets the stack frame; the stack frame provides the script (and possibly the function) it's running; the script provides its origin information --- and, if bug 637572 goes that way, the source code itself will be included in that information (a la V8). Either way, we'll want pretty-printed source accessible.
From https://wiki.mozilla.org/Debug_Object
>newScript(script, [function])
>    New code, represented by the Debug.Script instance script, has been loaded
>into the debuggee's compartment. If the new code is part of a function,
> function is a Debug.Object reference to the function object. 

How can |new code| not be part of a function?

>enterFrame(frame, call)
>    The stack frame frame is about to begin executing code. (Naturally, frame 
>is currently the youngest debuggee frame.) If call is true, it is a function 
>call; if call is false, it is global or eval code

In the false case, I guess you mean the function is what we call the "outer" or "file" function. That is the function created from the outermost scope in the compilation unit:
  var x = 1+1; // a statement in the outer most scope of a compilation unit
  function y() {
    return x+x;  // statement in the scope of y()
  };  

> If this hook function returns a function F, SpiderMonkey will call F when  
> execution of frame completes, passing one argument indicating how it 
> completed. If the argument is of the form { return: value }, then the code 
> completed normally, yielding value. If the argument is of the form { throw: 
> value }, then the code threw value as an exception. In either case, value is 
>a debuggee value. 

As we make deeper call stacks, the debugger enterFrame() function return functions are effectively interleaved with the debuggee frames.
So the debugger will probably want to close over the frame given in F, in order to keep track of which "F" is running.  That could be harmless and natural... or not.
(In reply to comment #16)

Thanks very much for reading this over!

> From https://wiki.mozilla.org/Debug_Object
> >newScript(script, [function])
> >    New code, represented by the Debug.Script instance script, has been loaded
> >into the debuggee's compartment. If the new code is part of a function,
> > function is a Debug.Object reference to the function object. 
> 
> How can |new code| not be part of a function?

Code passed to indirect eval is evaluated in the global scope. Code for DOM event handlers and top-level <script> code isn't part of any function.

> >enterFrame(frame, call)
> >    The stack frame frame is about to begin executing code. (Naturally, frame 
> >is currently the youngest debuggee frame.) If call is true, it is a function 
> >call; if call is false, it is global or eval code
> 
> In the false case, I guess you mean the function is what we call the "outer" or
> "file" function. That is the function created from the outermost scope in the
> compilation unit:
>   var x = 1+1; // a statement in the outer most scope of a compilation unit
>   function y() {
>     return x+x;  // statement in the scope of y()
>   };  

I'm trying to use ES5 terminology; the standard defines global, eval, and function code in section 10.1.

> > If this hook function returns a function F, SpiderMonkey will call F when  
> > execution of frame completes, passing one argument indicating how it 
> > completed. If the argument is of the form { return: value }, then the code 
> > completed normally, yielding value. If the argument is of the form { throw: 
> > value }, then the code threw value as an exception. In either case, value is 
> >a debuggee value. 
> 
> As we make deeper call stacks, the debugger enterFrame() function return
> functions are effectively interleaved with the debuggee frames.

In a sense --- although the functions the hook returns get called, not returned to. But if you're a Scheme fan, that may not seem like a critical distinction. :)

> So the debugger will probably want to close over the frame given in F, in order
> to keep track of which "F" is running.  That could be harmless and natural...
> or not.

Yes, exactly! There's no need for an explicit mechanism to carry data from entry function to exit function, because the latter just captures whatever data it needs naturally. Since the debugger's Debug.Frame objects are weak references to the debuggee's frames (that is, when the debuggee returns from the frame, it goes away regardless of whether there is a Debug.Frame object referring to it; operations on dead Debug.Frame objects just raise errors), it should be fine for it to get closed over.

Does this seem like it will introduce problems? I thought it was really neat.
(In reply to comment #17)
> (In reply to comment #16)
> Code for DOM
> event handlers and top-level <script> code isn't part of any function.
...
> I'm trying to use ES5 terminology; the standard defines global, eval, and
> function code in section 10.1.

That's great, as far as it goes. (Global unfortunately confusing, devs will think that functions that are properties of the global scope are "global"). But in terms of API we want to know if a function-that-isn't-a-function is a browser-generated event handler or a top-level script code.
(In reply to comment #17)
> Does this seem like it will introduce problems? 

We work with jsd now, it's not going to be worse than that ...

> I thought it was really neat.

...so go for it.
(In reply to comment #17)
> Code passed to indirect eval is evaluated in the global scope. Code for DOM
> event handlers and top-level <script> code isn't part of any function.

I believe that DOM event handlers are compiled and run as though the attribute's value were the body of an anonymous function taking one parameter named "event".

The string form of setTimeout will be a case like indirect eval, I think.  javascript: URLs as well.
Yes, event handlers are just functions.

  <input onclick="foo(event)" ...>

compiles to something like

  function onclick(event) { foo(event) }

in the scope of the receiver, named by its 'onclick' property.

/be
(In reply to comment #18)
> But
> in terms of API we want to know if a function-that-isn't-a-function is a
> browser-generated event handler or a top-level script code.

(In reply to comment #20)
> The string form of setTimeout will be a case like indirect eval, I think. 
> javascript: URLs as well.

(In reply to comment #21)
> Yes, event handlers are just functions.

I think all of these cases should be distinguished by getting the frame's script and looking at its origin information (bug 637572).
Debug.Frame docs now up on the wiki:
https://wiki.mozilla.org/Debug_Object#Debug.Frame
Depends on: 552488
Depends on: 639001
Debug.Object docs now up on wiki:
https://wiki.mozilla.org/Debug_Object#Debug.Object
Debug.Script docs now up on wiki:
https://wiki.mozilla.org/Debug_Object#Debug.Script

So all the major pieces are documented now.

There are things missing that jsd has:
- line/bytecode mappings for pretty-printed source
- watchpoints
- better cleanup when debugger is turned off
and I have a list of other issues I want to address. But it's probably time to start coding, and give people something to play with.
- Added watchpoint API.
- Debug instances now have an 'enabled' property; setting it to false takes out the instance's debug hooks, breakpoints, and watchpoints.
The Debug.Script api
https://wiki.mozilla.org/Debug_Object#Debug.Script
uses lines to identify source, but this is not adequate for real systems. Developers often have expressions in statements on a single line that need to be stepped through, especially expressions passed as arguments to functions. Also, lots of code is compressed so the entire script is on one line. Decompile is part of the answer for the compressed case, but that's really for the UI to decide.
(In reply to comment #27)
> The Debug.Script api
> https://wiki.mozilla.org/Debug_Object#Debug.Script
> uses lines to identify source, but this is not adequate for real systems.
> Developers often have expressions in statements on a single line that need to
> be stepped through, especially expressions passed as arguments to functions.
> Also, lots of code is compressed so the entire script is on one line. Decompile
> is part of the answer for the compressed case, but that's really for the UI to
> decide.

Yes. As a first cut, we're going to present the functionality JSDBGAPI currently has; the design on the wiki is simply fronting the JSScript we have now. Once that's usable, we'll evolve it. We certainly want column numbers (bug 568142), and the ability to use a pretty-printed source-to-bytecode map (no bug yet).
One thing which I've said before but which I should probably emphasize:

We want to make early versions of this interface available to hackers ASAP, but we want to *evolve* it. So names will change, arguments will change, semantic details will change, types will change, and so on. Because we're doing such a primitive initial cut, we'll value improvements to clarity, detail, and completeness over compatibility when evaluating changes for the initial period.

Perhaps the spec should make the Debug.foo.prototype properties writable, to allow monkey-patching...
Component: JavaScript Debugging/Profiling APIs → Build Config
monkeyPatching += Math.Infinity;
Component: Build Config → JavaScript Debugging/Profiling APIs
Assignee: nobody → jimb
Okay, spec is now monkey-patch friendly; I looked it over and didn't notice any areas where this was going to cause problems.

Also added frame arguments to a few places that were missing them.
(In reply to comment #29)
> One thing which I've said before but which I should probably emphasize:
> 
> We want to make early versions of this interface available to hackers ASAP, but
> we want to *evolve* it. So names will change, arguments will change, semantic
> details will change, types will change, and so on. Because we're doing such a
> primitive initial cut, we'll value improvements to clarity, detail, and
> completeness over compatibility when evaluating changes for the initial period.

This much is not a problem on the Firebug side. But what form might the early versions arrive in? All of our infrastructure relies on Firefox.
(In reply to comment #32)
> This much is not a problem on the Firebug side. But what form might the early
> versions arrive in? All of our infrastructure relies on Firefox.

I'm not sure I understand the question, so this may not be helpful:

At the JSAPI level, there'll be a JS_DefineDebugFunctions(cx, obj) function that adds a definition for Debug to the global object obj. The JavaScript shell will call it unconditionally. In Firefox, I'm not sure how we want to start shipping it; I don't think it should just be turned on, because then we may start getting clients and get locked in. Perhaps it could be enabled by a "enableDebugObjectWhoseInterfaceIUnderstandIsInFluxAndIAgreeIt'sMyProblemToKeepUp" preference. :) I don't know; I'd love suggestions.
In my opinion you should just ship it in Firefox. No one will build a lot of code on Debug with out reading docs or asking questions about the API, at which point the flux will be obvious.
I agree. There is no reason to make JSDBGAPI any stabler than JSAPI IMO.
I'd sure love to get the feedback from users.
Some spec work:
- Pulled out descriptions of completion and resumption values into their own sections.
- Added 'evalWithBindings' to Debug.Frame.prototype.
- Added 'call' and 'apply' methods to Debug.Object.prototype.

(The latter two were added in response to a question about how to do the analogous thing in Archer GDB. It turns out to be very clumsy to evaluate a source code expression using gdb.Value instances one has in Python. You can get values from the debuggee, but there's no way at all to get the values into the debuggee. Now we've avoided making the analagous mistake in the Debug interface.)
More spec work, based on suggestions from Jason Orendorff:
- Separated scope chain objects out from other objects: Debug.Environment vs. Debug.Object.
- "undefined" is now the resumption value that requests continuing execution normally.
- More minor monkey-patching friendliness.
Target Milestone: --- → mozilla2.0
Depends on: 659818
Depends on: 660039
Depends on: 662377
Component: JavaScript Debugging/Profiling APIs → JavaScript Engine
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.