Open Bug 603426 Opened 14 years ago Updated 2 years ago

We should warn users when they're running a known benchmark while the JS debugger is active

Categories

(Firefox :: General, defect)

Product:
Firefox
Component:
General
Platform:
x86
macOS
Type:
defect

Tracking

()

Tracking Flags:
Tracking Status
blocking2.0 --- -
status2.0 --- wanted

People

(Reporter: bzbarsky, Unassigned)

Details

Facts:

1)  When Firebug's script or console panel is activated for a site, our JS
    engine gets Very Slow.
2)  Firebug can be active for a site without being open; the Firebug icon
    indicates the state.
3)  The Firebug icon is located in the Add-On bar.
4)  The Add-On bar is currently not visible by default on trunk.
5)  Enabling and then disabling the console panel in Firebug leaves it in a
    state where the script panel and console panel both claim to be disabled
    but the JS debugger is active.

Assumptions:

1)  Many Firebug users have the script or especially console panel enabled, or
    are in the state one gets into due to fact #5.
2)  Lots of people who are likely to run Firefox 4 on Sunspider and such are
    likely to have Firebug installed.
3)  It's very easy to activate Firebug for a site (just open it there, and then
    close instead of deactivating).
4)  Most users have no idea that the Firebug icon is what tells them their JS
    is going to be slow.
5)  The Add-Ons bar is likely to continue being invisible by default.

Conclusion: Chances are, lots of people will think we're still really slow.

Proposal: When running a known benchmark suite (we can make a list) if jsd is on we should put up an infobar or doorhanger or whatever that says something along the lines of "Hey, if you aren't actually debugging this on purpose, you might want to consider turning off Firebug or whatever JS debugger you have running right this second".  Or some more polished message carefully crafted by someone who knows what he's doing, i.e. not me.

And yes, I know we're basically past string freeze.  I should have thought of this months ago...  Maybe we can work around this by having _Firebug_ do the warning?

I also know this is a total cop-out, but I really don't see a better solution; it's just too damn easy to put Firebug in slow mode and leave it there without realizing it.
blocking2.0: --- → ?
blocking2.0: ? → -
status2.0: --- → wanted
#5 is a Firebug bug that Firebug must fix.
Fact #5?  I agree, but even fixing that still leaves it pretty easy to enable jsd...
I cannot reproduce Fact 5 on Firefox 3.6 Firebug 1.7a3.  

In Firebug 1.6 (which will be the supported version for FF 4.0 and which is nearly equal to 1.7 at this point), the Console panel requires jsd to know when the page is ready for the console code injection. The Script panel is the control for jsd. Therefore, if the user disables the script panel (and thus jsd), the Console panel will be disabled. 

Note that this is roughly the opposite of the scenario in Fact 5.

I believe that the user experience issue here, illustrated in Bug 603050 is really a result of bug 595243. Firebug is not in control of jsd in FF 4.0b8pre nightly builds, so we can't make any sense of debugging now.

A simple fix here that would dramatically reduce the chance of user problems is to add a test to the performance web pages:
  if (window.console) { add a warning to the page to turn off firebug }
This is not fool proof because users can turn off the console and leave script panel on, but very few users do this in my experience.

A more drastic fix would color the border of Firefox when Firebug is active.

Also we will have to solve Assumption 5 and force the toolbar up in FF 4 when firebug is active so that may also make enough of a difference.
Firebug 1.7a4 and 1.6b2 will un-collapse the addon toolbar.
http://code.google.com/p/fbug/issues/detail?id=3532

The user can collapse the toolbar again by using Alt-Key followed by right click on the horizontal strip with the File menu and picking Addon Toolbar. Fortunately that spell will only last until they restart the browser.  One of the wizards will eventually complain that 'the browser keeps re-showing the toolbar when I tell it not to'.  Obeying the setting is a worse choice for the non-wizard who casts the spell. So I think we tell the wizard to hack the code if they want.

To be redundant: we have to wait for 595243 before evaluation here.
> I cannot reproduce Fact 5 on Firefox 3.6 Firebug 1.7a3.  

Try a nightly build?  That's where I was testing it... (with 1.7a3).  The Firebug icon remained colored after those steps, and jsd was certainly on.

> a result of bug 595243. Firebug is not in control of jsd in FF 4.0b8pre

I don't think bug 595243 is affecting your control of jsd (in the sense of not letting you turn it _off_), but maybe I missed something...  In any case, enabling and then disabling the Script panel properly disabled jsd, so it's just disabling the Console panel that has problems.

>  if (window.console) { add a warning to the page to turn off firebug }

window.console is now defined in more or less all browsers all the time.  And of course we don't control the web pages in question.

> To be redundant: we have to wait for 595243 before evaluation here.

I don't think that anything that happens in that bug will really address the fundamental issue that most people using Firebug don't understand that it slows things down and when it does so.
(In reply to comment #5)
> > I cannot reproduce Fact 5 on Firefox 3.6 Firebug 1.7a3.  
> 
> Try a nightly build?  That's where I was testing it... (with 1.7a3).  The
> Firebug icon remained colored after those steps, and jsd was certainly on.

I guess you mean FF 4.0 nightly build. But 4.0 is known to be broken for jsd:
http://getfirebug.com/testresults
Using the same Firebug code in FF 3.6 has no bug I can see.

...
> I don't think that anything that happens in that bug will really address the
> fundamental issue that most people using Firebug don't understand that it slows
> things down and when it does so.

We put a awful lot time in to enable/disable function in Firebug specifically to make it clear when debugging is active. Overall the gray/orange bug signal has been adequate. I don't think we should generalize from one user who was using a nightly build that disabled our user interface.
> But 4.0 is known to be broken for jsd:

And my point is that the known breakage in bug 595243 doesn't seem like it would account for the observed issue.  But ok, I'll wait for that to land before filing bugs on Firebug here...

> Overall the gray/orange bug signal has been adequate.

That hasn't been my experience; I've had dozens of conversations over the last year with people who were complaining about benchmarks being slow and were surprised when I pointed out the colored bug meant they were actively debugging the benchmark even when Firebug's UI is not otherwise visible.
From http://www.cs.auckland.ac.nz/~pgut001/ it should be clear that people misunderstand or completely overlook UI elements, all the time.

I agree with bz, we need something like what this bug's summary asks for.

/be
I'm hesitant to make a list of "known benchmarks" which seems fragile and like special-casing our browser in order to look better for specific press reasons. Dirty feelings aside, I think that it's brittle and won't protect us when the next Kraken becomes the thing about which we're concerned.

I'd much rather that when the JSD is enabled, Firefox indicate to the user that this will result in slower than normal JS performance. This could be a warning which can be dismissed globally or per site, depending on how we want to do it.
Firebug enables/disabled jsd on the fly on a per-site basis (that is, whether jsd is on depends on what tab is selected and on what the persisted "jsd should be on" preferences for that tab are)....
And comment 8 rejects user interface indicators, so I'm not sure we can make progress. 

One thing I think Firebug could/should do is provide a 'new-user experience' panel for our minimize function that Boris is complaining about.  The first time a user minimizes Firebug they could get a panel that explains the Firebug icon colors and the meaning of orange.
When other browsers have debugging available, as in this state in Firebug, does their performance suffer similarly?  Or is it only when they actually have breakpoints set, or are single-stepping/inspecting?
Note that "this state in Firebug" includes "the console is enabled".  I'm 99.99% sure that other browsers do not suffer a perf hit when their console is enabled.  ;)
> And comment 8 rejects user interface indicators,

Comment 8 rejects obscure user interface indicators that have to be interpreted.  This bug is asking for a very explicit piece of UI that flat-out says "your JS is being slow".

> The first time a user minimizes Firebug

Chances are, they'll forget about this a week later when they actually need to remember the information....
(In reply to comment #12)
> When other browsers have debugging available, as in this state in Firebug, does
> their performance suffer similarly? 

When I run
http://www2.webkit.org/perf/sunspider-0.9.1/sunspider.html
in Google Chrome 6.0.472.63, without their debugger up and with their debugger up, I get the results posted below. The Firefox values are in Bug 603050.

Based on these numbers, the debugger has a much bigger impact on Google than Firebug has on Firefox.
    




TEST                   COMPARISON            FROM                 TO             DETAILS

=============================================================================

** TOTAL **:           2.88x as fast     899.5ms +/- 1.4%   312.2ms +/- 1.9%     significant

=============================================================================

  3d:                  1.28x as fast      64.3ms +/- 5.1%    50.3ms +/- 3.1%     significant
    cube:              -                  21.1ms +/- 5.9%    20.6ms +/- 7.2% 
    morph:             1.47x as fast      22.1ms +/- 9.9%    15.0ms +/- 4.5%     significant
    raytrace:          1.44x as fast      21.1ms +/- 6.5%    14.7ms +/- 3.3%     significant

  access:              1.58x as fast      58.0ms +/- 3.6%    36.8ms +/- 6.5%     significant
    binary-trees:      -                   2.1ms +/- 10.8%     2.0ms +/- 53.3% 
    fannkuch:          1.65x as fast      26.1ms +/- 6.8%    15.8ms +/- 12.0%     significant
    nbody:             1.53x as fast      22.6ms +/- 3.1%    14.8ms +/- 8.5%     significant
    nsieve:            1.71x as fast       7.2ms +/- 7.8%     4.2ms +/- 7.2%     significant

  bitops:              1.62x as fast      50.4ms +/- 5.4%    31.1ms +/- 7.8%     significant
    3bit-bits-in-byte: 3.17x as fast       7.3ms +/- 12.3%     2.3ms +/- 15.0%     significant
    bits-in-byte:      1.89x as fast      14.9ms +/- 11.2%     7.9ms +/- 9.0%     significant
    bitwise-and:       -                  10.5ms +/- 5.8%    10.2ms +/- 17.4% 
    nsieve-bits:       1.65x as fast      17.7ms +/- 7.9%    10.7ms +/- 19.4%     significant

  controlflow:         1.52x as fast       3.5ms +/- 10.8%     2.3ms +/- 15.0%     significant
    recursive:         1.52x as fast       3.5ms +/- 10.8%     2.3ms +/- 15.0%     significant

  crypto:              1.40x as fast      31.7ms +/- 9.0%    22.6ms +/- 16.9%     significant
    aes:               1.29x as fast      12.8ms +/- 12.0%     9.9ms +/- 37.3%     significant
    md5:               1.43x as fast       9.3ms +/- 10.3%     6.5ms +/- 9.3%     significant
    sha1:              1.55x as fast       9.6ms +/- 12.3%     6.2ms +/- 17.9%     significant

  date:                15.8x as fast     514.0ms +/- 2.3%    32.6ms +/- 4.2%     significant
    format-tofte:      33.8x as fast     489.8ms +/- 2.4%    14.5ms +/- 3.5%     significant
    format-xparb:      1.34x as fast      24.2ms +/- 3.6%    18.1ms +/- 5.7%     significant

  math:                1.68x as fast      53.8ms +/- 5.1%    32.0ms +/- 3.0%     significant
    cordic:            1.68x as fast      19.2ms +/- 8.6%    11.4ms +/- 6.1%     significant
    partial-sums:      1.65x as fast      24.9ms +/- 4.6%    15.1ms +/- 2.7%     significant
    spectral-norm:     1.76x as fast       9.7ms +/- 12.1%     5.5ms +/- 24.7%     significant

  regexp:              ??                 15.8ms +/- 16.6%    16.2ms +/- 12.1%     not conclusive: might be *1.025x as slow*
    dna:               ??                 15.8ms +/- 16.6%    16.2ms +/- 12.1%     not conclusive: might be *1.025x as slow*

  string:              1.22x as fast     108.0ms +/- 1.7%    88.3ms +/- 2.5%     significant
    base64:            1.185x as fast      6.4ms +/- 5.8%     5.4ms +/- 9.3%     significant
    fasta:             1.33x as fast      19.0ms +/- 8.9%    14.3ms +/- 4.7%     significant
    tagcloud:          1.46x as fast      34.6ms +/- 2.4%    23.7ms +/- 6.4%     significant
    unpack-code:       ??                 30.8ms +/- 2.1%    31.0ms +/- 3.8%     not conclusive: might be *1.006x as slow*
    validate-input:    1.24x as fast      17.2ms +/- 6.7%    13.9ms +/- 3.8%     significant
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.