914429 - [jsdbg2] We need measurements of Debugger's performance impact

Status: NEW

(Core :: JavaScript Engine, defect)

Description

[Jim Blandy :jimb]

At the moment, we make no regular measurements of the impact Debugger has on the performance of debuggee JavaScript. At the very least, we should know:

1) What is the performance impact of making a given benchmark a debuggee, but setting no hooks, versus letting it run at SpiderMonkey's fastest speed? Versus the Baseline compiler?

2) How expensive is the act of adding or removing a global as a debuggee, itself?

3) What is the further impact of setting an onEnterFrame hook? An onNewScript hook? etc.

Comment 1

[Jim Blandy :jimb]

I've put up some very rough code for measuring this here:
https://github.com/jimblandy/benchmark-debugger

The variance here is way too high; I can get the stddev down to around 3%, but here's what I have:

--ion,    debugger, onEnterFrame
warmup
measure
[Stats total: 206.26856884765598s, mean: 0.412537137695312s, stddev: 11%]

--no-ion, debugger, onEnterFrame
warmup
measure
[Stats total: 209.06723315429664s, mean: 0.4181344663085933s, stddev: 12%]

--ion,    debugger, no hooks
warmup
measure
[Stats total: 76.72733569335945s, mean: 0.1534546713867189s, stddev: 10%]

--no-ion, debugger, no hooks
warmup
measure
[Stats total: 76.6684204101562s, mean: 0.1533368408203124s, stddev: 17%]

--ion
warmup
measure
[Stats total: 14.706278320312501s, mean: 0.029412556640625004s, stddev: 11%]

--no-ion
warmup
measure
[Stats total: 41.867698242187494s, mean: 0.08373539648437499s, stddev: 17%]

Comment 2

[Jim Blandy :jimb]

Here's the full assortment of compilers vs. Debugger arrangements:

--ion,    debugger, onEnterFrame
warmup
measure
[Stats total: 238.5119206542969s, mean: 0.47702384130859377s, stddev: 4%]

--ion,    debugger, no hooks
warmup
measure
[Stats total: 77.43573144531248s, mean: 0.15487146289062498s, stddev: 9%]

--ion
warmup
measure
[Stats total: 20.861109374999998s, mean: 0.04172221875s, stddev: 95%]

--no-ion, debugger, onEnterFrame
warmup
measure
[Stats total: 246.48064135742163s, mean: 0.49296128271484324s, stddev: 1%]

--no-ion, debugger, no hooks
warmup
measure
[Stats total: 78.03183471679682s, mean: 0.15606366943359365s, stddev: 3%]

--no-ion
warmup
measure
[Stats total: 43.8149606933594s, mean: 0.08762992138671881s, stddev: 7%]

--no-ion --no-baseline, debugger, onEnterFrame
warmup
measure
[Stats total: 279.48205371093775s, mean: 0.5589641074218755s, stddev: 11%]

--no-ion --no-baseline, debugger, no hooks
warmup
measure
[Stats total: 138.29736279296876s, mean: 0.2765947255859375s, stddev: 1%]

--no-ion --no-baseline
warmup
measure
[Stats total: 118.26902050781254s, mean: 0.2365380410156251s, stddev: 1%]

Comment 3

[Jim Blandy :jimb]

That "--ion, no debugger" measurement has a startlingly high standard deviation. I'd like to see the distribution for those times...

Comment 4

[Jim Blandy :jimb]

Attachment: Plot of cost of Debugger under various compilation regimens.

Here's a CHART!

Comment 5

[Jim Blandy :jimb]

Attachment: Plot of cost of Debugger under various compilation regimens, after bug 1032869

Here's a new plot, showing the impact after Shu's work on disabling JITs on a per-frame and per-script basis.

Just making something a debuggee imposes no measurable overhead.

The onEnterFrame hook is still expensive, because it affects the behavior of every debuggee script.

Now we're in a position to ask more interesting questions:

- What is the overhead of using Debugger.Frame.prototype.eval and Debugger.Object.prototype.evalInGlobal? (This is the web console's usage pattern. I would expect no lasting impact, once the stack frame in which we performed the evaluation has returned.)

- What is the overhead of making a compartment a debuggee and calling Debugger.Memory.prototype.takeCensus? (I would expect none.)

- What is the overhead of setting Debugger.Memory.prototype.trackingAllocationSites? (I would expect this to still be significant, but perhaps less than before.)