Closed Bug 534120 Opened 15 years ago Closed 15 years ago

jit remains disabled when jsd is paused, slowing Firefox when Firebug is installed

Categories

(Core :: JavaScript Engine, defect)

defect
Not set
normal

Tracking

()

VERIFIED FIXED
Tracking Status
status1.9.2 --- final-fixed

People

(Reporter: bzbarsky, Assigned: bzbarsky)

References

Details

(Whiteboard: [firebug-p1][firebug-blocks])

Attachments

(2 files, 1 obsolete file)

STEPS TO REPRODUCE:
1)  Install Firebug.
2)  Enable the console.
3)  Go to some site where Firebug is disabled (icon grayed out).
4)  Observe the crappy JS performance

EXPECTED RESULTS: When jsd is paused, jit should stay on.

ACTUAL RESULTS: When jsd is paused jit is off.

SOLUTION:  Unhook the runtime as much as we can while paused.

This is still somewhat suboptimal because we end up with script creation/destruction overhead in jsd, but to fix that we'd need Firebug to actually call off() on jsd, not just pause().
thanks for the cc. adjusting some flags for extra visibility.

Based on our discussion today it sounds like you have a good strategy for fixing this temporarily with a better solution for next point-release.
Flags: blocking1.9.2?
Whiteboard: [firebug-p1][firebug-blocks]
OS: Mac OS X → All
Hardware: x86 → All
OK, some bad news.  The object hook (which is one of the hooks that disables the jit) is in fact needed for internal jsd bookkeeping (and in particular for _destroyJSDObject.

Brendan, could we split the object hook into object creation and destruction hooks?  We could unhook the former and the latter wouldn't need to disable the jit, right?

Alternately, John, could Firebug in fact just call off() on the debugger whenever it's currently calling pause() to get it into the "quiet" state?

I think I have the rest of this working; will attach a work-in-progress diff in a second.
Attached patch wip (obsolete) — Splinter Review
Firebug can't call jsd.off() because that causes some of the meta-data Firebug has accumulated to become invalid. jsd destroys its internal state when off() is called.

I'm unclear what you mean by 'object hook', Firebug at present does not use such a thing.

As far as I understand it, the jit needs to be off if we do anything with the stack or pc. Is it really necessary to turn jit off during internal bookkeeping code for object lifecycle?
Summary: Firebug disables the jit → jit remains disabled when jsd is paused, slowing Firefox when Firebug is installed
> that causes some of the meta-data Firebug has accumulated to become invalid.

What, specifically?  That is, what meta-data is being accumulated between startup and when Firebug is actually opened by the user?

> I'm unclear what you mean by 'object hook'

It's a member called objectHook on the runtime.  It's notified when objects are created and destroyed, if it's set.  It also disables the jit, if it's set.

jsd uses the object hook for its object-tracing functionality, as well as for creating cached object representations that are used if you get objectValue from a jsdIValue.  So while there's no jsdIDebuggerService-api-level hook that corresponds to the object hook, the object hook is probably needed for correct functioning of jsdIValue.

> Is it really necessary to turn jit off during internal bookkeeping code for
> object lifecycle?

I believe so, because jit-generated code creates objects, and calling into the object hook, which can then ask for stack frame information, which the jit doesn't have, is bad.

But the point is, jit doesn't _destroy_ objects.  So a destroy object hook would not need to disable the jit, I think.
(In reply to comment #5)
> > that causes some of the meta-data Firebug has accumulated to become invalid.
> 
> What, specifically?  That is, what meta-data is being accumulated between
> startup and when Firebug is actually opened by the user?

For Firebug the answer is "none", we don't even need to call jsd.on() until later. The meta-data we need is accumulated on page load. So the scenario is:
1) User opens firebug on site A,
2) reloads site A, meta-data accumulates
3) user switches to gmail.com, Firebug suspends (pause jsd)
4) user switches back to site A, Firebug unsuspends and unpause jsd.

Firebug does a lot of analysis while scripts are loaded and if the corresponding jsdIScript are invalid, the web page has to be reloaded.

> 
> > I'm unclear what you mean by 'object hook'
> 
> It's a member called objectHook on the runtime.  It's notified when objects are
> created and destroyed, if it's set.  It also disables the jit, if it's set.
> 
> jsd uses the object hook for its object-tracing functionality, as well as for
> creating cached object representations that are used if you get objectValue
> from a jsdIValue.  So while there's no jsdIDebuggerService-api-level hook that
> corresponds to the object hook, the object hook is probably needed for correct
> functioning of jsdIValue.

Firebug specifically attempts to turn off object-tracing (due to historical info about performance hit).

I believe that Firebug does not need any jsdIValues which are created while jsd is paused.

> 
> > Is it really necessary to turn jit off during internal bookkeeping code for
> > object lifecycle?
> 
> I believe so, because jit-generated code creates objects, and calling into the
> object hook, which can then ask for stack frame information, which the jit
> doesn't have, is bad.

Except that we don't ask for the stack frame because we don't set the object hook:
            jsd.flags |= DISABLE_OBJECT_TRACE;
This if the flag is true, jit can continue correct?

> 
> But the point is, jit doesn't _destroy_ objects.  So a destroy object hook
> would not need to disable the jit, I think.
Sounds like we can just not have the object hook there, which is great.

Otherwise, if we only need to synchronize on object destruction, then jsd could install a GC hook and use JS_IsAboutToBeFinalized to check each of its remembered objects.  That would scale with the number of objects remembered, rather than the total number of objects finalized, which is probably better anyway.
OK, I was wrong.  We can't disable the object-creation hook while paused, in general.

In particular, getting objectValue only looks up cached jsdobjects.  So if there's any sort of meta-data you want preserved that can hand out jsdIValues that represent objects, and if those objects might get created while paused, then unhooking object creation would mean you get back broken jsdIValues.

Of course unhooking object destruction is simply a no-go, since gc can run and destroy objects while we're paused, and we need to know when that happens so we don't try to access that memory anymore.  But as I said, that hook should be jit-friendly.

> we don't even need to call jsd.on() until later.

Then why are you calling on() at startup and never calling off()?  Can you fix _that_, at least?  For the case when the user has never debugged anything in this session?

> 2) reloads site A, meta-data accumulates
> 3) user switches to gmail.com, Firebug suspends (pause jsd)
> 4) user switches back to site A, Firebug unsuspends and unpause jsd.

Right.

> I believe that Firebug does not need any jsdIValues which are created while
> jsd is paused.

So in the situation above if there are scripts created on site A while you're looking at gmail.com, you don't care about them?  I find that somewhat unlikely...

> This if the flag is true, jit can continue correct?

In theory, yes.  In practice, the JS engine doesn't know anything about that flag.  All it knows is that someone is going to run some sort of arbitrary code on every object creation.  The only way to not crash exploitably in that situation is to not have the jit running during object creation.

> Sounds like we can just not have the object hook there, which is great.

Sadly, no.  See above.
To be clear, I would be quite happy to not collect information on new objects in jsd if it's really not needed.  We can't do that for pause(), because that would, for example, break single-stepping (the debugger is paused while we're inside the breakpoint hook, so if objects got created while stepping we wouldn't be able to examine them).  But we can add a new API that does whatever we want in terms of not gathering new information.  I'm just trying to pin down exactly what we want: a precise list of all the information that Firebug expects to persist across steps 3 and 4 in John's example above, and a list of all the information Firebug expects to be gathered across those steps.
> that would, for example, break single-stepping

Er, it might not break stepping depending on how we implement it, but it'd break examination of objects (esp. of DOM properties that create new objects on get) at breakpoints for sure.  So still no good.
On a breakpoint Firebug disables JavaScript and events on the page.

When Firebug pauses jsd it wants no jsd data. The point is maximum preformance. But we want the data before the pause to remain valid.
(In reply to comment #8)

> 
> > This if the flag is true, jit can continue correct?
> 
> In theory, yes.  In practice, the JS engine doesn't know anything about that
> flag.  
Seems like an easy thing to fix.
> On a breakpoint Firebug disables JavaScript and events on the page.

Yes, I know, but new JSObjects can still be created as the user examines properties of DOM objects while stopped at a breakpoint.

> When Firebug pauses jsd it wants no jsd data.

That's demonstrably false.  You want certain jsd data to be retained.  You want jsd to drop data that references deleted memory.  What do you want to happen to new data that is created and referenced from the data that was already there when you paused the debugger?

> Seems like an easy thing to fix.

Could be, with some JS api changes.  But it doesn't help Firebug, per the first part of comment 8.
So, maybe this is a dumb question, but suppose we don't want any jsdIObject objects. Shouldn't it be possible to turn JSD's object hook off then?

I don't see anything in Firebug that's using that interface. Searches for creatorURL, creatorLine, constructorURL, constructorLine, and objectValue all come up empty.
Per discussion with Jason, the plan is to add a new debugger flag for "don't care about objectValue" (which Firebug in fact doesn't use).  If that flag is set and object tracing is disabled, we can in fact simply not install an object hook at all.  Patch will be coming up for that.
Attached patch Proposed fixSplinter Review
Don't install an object hook at all unless we're doing object tracing; unset the other hooks on pause when we can.  This seems to make jit happy when Firebug isn't active.
Attachment #417043 - Attachment is obsolete: true
Attachment #417174 - Flags: review?(jorendorff)
Attachment #417174 - Flags: review?(timeless)
Comment on attachment 417174 [details] [diff] [review]
Proposed fix

In JSD_SetContextFlags:
>+    if ((flags & JSD_COLLECT_PROFILE_DATA) ||
>+        !(flags & JSD_DISABLE_OBJECT_TRACE)) {
>+        // Need to reenable our call hooks now
>+        JS_SetExecuteHook(jsdc->jsrt, jsd_TopLevelCallHook, jsdc);
>+        JS_SetCallHook(jsdc->jsrt, jsd_FunctionCallHook, jsdc);
>+    }

Seems like we also need complementary code to disable them if
we're transitioning from, say, COLLECT to ~COLLECT, right?

>     /* clear hooks here */
>     JS_SetNewScriptHookProc(jsdc->jsrt, NULL, NULL);
>     JS_SetDestroyScriptHookProc(jsdc->jsrt, NULL, NULL);
>-    JS_SetDebuggerHandler(jsdc->jsrt, NULL, NULL);
>+    /* Have to unset these too, since jsd_DebuggerPause only unsets
>+       them conditionally */
>     JS_SetExecuteHook(jsdc->jsrt, NULL, NULL);
>     JS_SetCallHook(jsdc->jsrt, NULL, NULL);
>+    jsd_DebuggerPause(jsdc);
>     JS_SetObjectHook(jsdc->jsrt, NULL, NULL);

Just for symmetry, it'd be nice to call jsd_DebuggerPause first,
then clear the rest of the hooks.

r=me with those fixed, but note that I'm not a peer in this module.
Attachment #417174 - Flags: review?(jorendorff) → review+
(In reply to comment #13)
> > On a breakpoint Firebug disables JavaScript and events on the page.
> 
> Yes, I know, but new JSObjects can still be created as the user examines
> properties of DOM objects while stopped at a breakpoint.

But if jsd is paused (Firebug suspended) the user cannot interact with the DOM objects because they have no UI to do so. Recall that the pause call come when the user switches away from a debugged page and on to non-debug page: Firebug is not available when the jsd is paused.

> 
> > When Firebug pauses jsd it wants no jsd data.
> 
> That's demonstrably false.  You want certain jsd data to be retained.  You want
> jsd to drop data that references deleted memory.  What do you want to happen to
> new data that is created and referenced from the data that was already there
> when you paused the debugger?

This is hard to answer in the abstract. Perhaps firebug should disable JS and event handlers on pages it debugs when it suspends.  Because Firebug cannot get all the information it needs by API calls after the fact, it has to be live with the page. We really want 'suspend' to mean it.
> But if jsd is paused (Firebug suspended)

You seem to be under the impression that you're the only pause() caller.  That's not the case.  jsd pauses itself before calling out into one of your hooks, because otherwise manipulation you perform in your hook would cause jsd to call another one of your hooks, etc.

In particular, if you're stopped at a breakpoint, then jsd is paused.  If the user then examines the DOM in firebug, while stopped at the breakpoint, new JS objects are created and the jsd new object hooks called...

> This is hard to answer in the abstract.

The jsd code needs it answered in the abstract, unless you want to start crashing.  At least if you ever decide to enable object tracing (or if timeless doesn't like my patch).

> Perhaps firebug should disable JS and event handlers on pages it debugs when
> it suspends.

I'm not sure what you mean here....
(In reply to comment #19)
> > But if jsd is paused (Firebug suspended)
> 
> You seem to be under the impression that you're the only pause() caller. 
> That's not the case.  jsd pauses itself before calling out into one of your
> hooks, because otherwise manipulation you perform in your hook would cause jsd
> to call another one of your hooks, etc.

I did not know that. (There is no such info in the docs).

> 
> In particular, if you're stopped at a breakpoint, then jsd is paused.  If the
> user then examines the DOM in firebug, while stopped at the breakpoint, new JS
> objects are created and the jsd new object hooks called...

Ok, but that is not the case we are concerned with here. We are trying to improve Firebug's suspend mode, in which case the user cannot examine the DOM in Firebug because the UI is not available to do so.

> 
> > This is hard to answer in the abstract.
> 
> The jsd code needs it answered in the abstract, unless you want to start
> crashing.  At least if you ever decide to enable object tracing (or if timeless
> doesn't like my patch).

Based on the discussion here I would presume that we would not enable object tracing until we understood the issues better. But was also the case before this bug was opened: I've read multiple places that jsd object tracing is dubious and I have no information that it works or is good for anything.

> 
> > Perhaps firebug should disable JS and event handlers on pages it debugs when
> > it suspends.
> 
> I'm not sure what you mean here....

If we are debugging site A and the user switches to site B where Firebug is not open, we suspend Firebug. During that suspend we could disable JS and event handlers for Site B. When the user switches back to site A, then we resume. During that resume we could re-enable JS and event handlers.
> in which case the user cannot examine the DOM in Firebug because the UI is not
> available to do so.

Uh.. it's a breakpoint.  Can the user really not examine the values of local variables in the function?  What about properties of those local variables?

> During that suspend we could disable JS and event handlers for Site B.

I assume you mean Site A.  That would indeed be nice, if possible, and enable us to unhook the runtime without losing anything.  Can you file a but about making this possible.
(In reply to comment #21)
> > in which case the user cannot examine the DOM in Firebug because the UI is not
> > available to do so.
> 
> Uh.. it's a breakpoint.  Can the user really not examine the values of local
> variables in the function?  What about properties of those local variables?

The user may be on a breakpoint on site A, but they have switched to site B. Firebug has suspended. The UI is not available.

> 
> > During that suspend we could disable JS and event handlers for Site B.
> 
> I assume you mean Site A.  

Correct.

> That would indeed be nice, if possible, and enable
> us to unhook the runtime without losing anything.  Can you file a but about
> making this possible.

I'll try it.
Comment on attachment 417174 [details] [diff] [review]
Proposed fix

>+     * Set hooks here.  The new/destroy script hooks are on even when
>+     * the debugger is paused.  The destroy one so we'll clean up

change "one" to "hook" (it's consistent with 'newscript hook' below, and makes reading/parsing the comment easier)

>+     * internal data structures when scripts are destroyed, and the
>+     * newscript hook for backwards compatibility for now.  We'd like
>+     * to stop doing that.

>+    /* Have to unset these too, since jsd_DebuggerPause only unsets
>+       them conditionally */

While jsd so far generally has external and internal apis matching, I don't mind the internal function taking extra arguments if that makes the code cleaner/easier to read/understand.

>     JS_SetExecuteHook(jsdc->jsrt, NULL, NULL);
>     JS_SetCallHook(jsdc->jsrt, NULL, NULL);
>+    jsd_DebuggerPause(jsdc);

>+jsd_DebuggerPause(JSDContext* jsdc)
>+{
>+    JS_SetDebuggerHandler(jsdc->jsrt, NULL, NULL);
>+    if (!(jsdc->flags & JSD_COLLECT_PROFILE_DATA) &&
>+        (jsdc->flags & JSD_DISABLE_OBJECT_TRACE)) {
>+        JS_SetExecuteHook(jsdc->jsrt, NULL, NULL);
>+        JS_SetCallHook(jsdc->jsrt, NULL, NULL);
>+    }
>+    JS_SetThrowHook(jsdc->jsrt, NULL, NULL);
>+    JS_SetDebugErrorHook(jsdc->jsrt, NULL, NULL);
Attachment #417174 - Flags: review?(timeless) → review+
(In reply to comment #8)
...> > we don't even need to call jsd.on() until later.
> 
> Then why are you calling on() at startup and never calling off()?  Can you fix
> _that_, at least?  For the case when the user has never debugged anything in
> this session?

http://code.google.com/p/fbug/source/detail?r=5243
R5243 on branches/firebug1.5.

Honza, let's not forget to extend our activation test cases to cover this new feature. So we need to test TabWatcher.contexts.length going from 0->1->2->1->0->1, since 0 is now special and different from 1->2 or 2->1. We should verify jsd.isOn false when the contexts.length is zero.
> http://code.google.com/p/fbug/source/detail?r=5243

Does that also call off() once the user actually stops all debugging sessions?  Or just delay on() until the user starts debugging?  (Just the latter is already pretty good, fwiw.)
> change "one" to "hook" 

Done.

> I don't mind the internal function taking extra arguments

Done.
Pushed http://hg.mozilla.org/mozilla-central/rev/119c8036630f
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
Comment on attachment 417174 [details] [diff] [review]
Proposed fix

We want this on branch.
Attachment #417174 - Flags: approval1.9.2?
And pushed http://hg.mozilla.org/mozilla-central/rev/09233be76eec to fix Windows build bustage.
Comment on attachment 417174 [details] [diff] [review]
Proposed fix

a192=beltzner
Attachment #417174 - Flags: approval1.9.2? → approval1.9.2+
(In reply to comment #25)
> > http://code.google.com/p/fbug/source/detail?r=5243
> 
> Does that also call off() once the user actually stops all debugging sessions? 
> Or just delay on() until the user starts debugging?  (Just the latter is
> already pretty good, fwiw.)

R5245 on branches/firebug1.5, will be part of Firebug 1.5b7.
http://code.google.com/p/fbug/source/detail?r=5245
Is there a web page we can use to verify that the jit is running?
http://people.mozilla.com/~vladimir/ss/hosted/bitops-bitwise-and.html -- it should be on the order of 1-2ms if the jit is running, otherwise it'll be 40+ms.
You can also use http://web.mit.edu/bzbarsky/www/mandelbrot-clean.html -- on my machine it's at about 200ms if the jit is on, and over 4 seconds otherwise.
Whiteboard: [firebug-p1][firebug-blocks] → [firebug-p1][firebug-blocks][can land]
Depends on: 534676
Holding off on the branch landing pending landing the fix for bug 534676.
This checkin introduced this build warning:
> js/jsd/jsdebug.c:143 - C++ style comments are not allowed in ISO C90
> js/jsd/jsdebug.c:143 - (this will be reported only once per input file)

Source: http://office.smedbergs.us:8080/build?id=2015

The attached followup patch fixes this by converting the introduced comments to be C-style (/**/).
Attachment #417570 - Flags: review?(bzbarsky)
Comment on attachment 417570 [details] [diff] [review]
followup: use c-style comments in jsdebug.c, to fix build warning

r+, but please don't land this; it'd just merge-conflict with the crash fix followup.... I'll fix the comments as I land that.
Attachment #417570 - Flags: review?(bzbarsky) → review+
Sounds good.  Thanks!
This has eventually regressed bug 534755.
I don't think this blocks by itself, and due to the recent regression, we may end up having to back it out if the fix from bug 534676 doesn't resolve things.
Flags: blocking1.9.2? → blocking1.9.2-
Pushed http://hg.mozilla.org/releases/mozilla-1.9.2/rev/0627abf2bc18 with both of the patches initially pushed for this bug.

I rolled in attachment 417570 [details] [diff] [review] into the patch for bug 534676.
Verified this works with Firebug 1.5 @ r5262 in: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.3a1pre) Gecko/20091215 Minefield/3.7a1pre ID:20091215030817.
It's also worth double-checking with Firebug 1.4, just to make sure...
On Firefox 3.5.6 with Firebug 1.4.5, I get 1000ms for the test in comment 34, mandelbrot and 3ms for the test in comment 33, bitwise. These numbers are with firebug open.
Does all this mean that Firebug itself will never be JIT-ed?

If so, it would be a good idea to create another bug so we can have Firebug itself running fast, even when the page being debugged is not. Perhaps that is already the way it works, I just can't tell from this discussion. And I have no idea how to test the hypothesis.
> Does all this mean that Firebug itself will never be JIT-ed?

If the only fix here were on the Firebug side, that would be true.  But as it is, the jsd fix means that any code running inside a debugger callback will be JIT-ed.  See comment 19.  Firebug code that is running outside of debugger callbacks while the debugger is unpaused will in fact not be JIT-ed.  The right way to fix that is to not have it running in the same jsruntime as the thing being debugged (or to just have a better debugging API).
So if Firebug is enabled on a page, with jsd on looking to break on an error, for example, and no error yet occurring, and the user is simply using the HTML/CSS/DOM panels, then Firebug is still not JIT-ed?

In other words, if the script panel is active, but the user isn't really using it, that will cause the other panels to perform slower?

I thought there was always only one jsruntime, right? So to have more than one would be a big change. I guess if it were to ever happen it would be in a multi-process lifetime.

Is there a current bug for the debugging API that would allow Firebug to run at full speed when not actually stuck at a breakpoint? Another would be for Firebug itself to still run JIT-ed even when stopped at a breakpoint.
> then Firebug is still not JIT-ed?

Yes, but if you're using break-on-error I suspect that's the least of your performance worries, based on the profiles I've seen....

> In other words, if the script panel is active, but the user isn't really using
> it, that will cause the other panels to perform slower?

Yep.

> I thought there was always only one jsruntime, right?

Not at all.  Firefox only uses one jsruntime by default, and in particular the same one for chrome and content at the moment.  That will automatically change when they're in different processes, but anyone using jsapi can always just create another jsruntime and do stuff with it.

> Is there a current bug for the debugging API

There are several bugs about better debugging APIs (not necessarily targeted at Firebug).  I don't have the bug numbers offhand, though, sorry.  You'll have to search...

> Another would be for Firebug itself to still run JIT-ed even when stopped at
> a breakpoint.

If you're stopped at a breakpoint the debugger is paused and hence jit is enabled, no?
(In reply to comment #48)
> > then Firebug is still not JIT-ed?
> 
> Yes, but if you're using break-on-error I suspect that's the least of your
> performance worries, based on the profiles I've seen....

I doubt JIT will help Firebug much, we do too much DOM manipulation.

...

> > Is there a current bug for the debugging API
> 
> There are several bugs about better debugging APIs (not necessarily targeted at
> Firebug).  I don't have the bug numbers offhand, though, sorry.  You'll have to
> search...

Please use Bug 449452 to collect these.

> 
> > Another would be for Firebug itself to still run JIT-ed even when stopped at
> > a breakpoint.
> 
> If you're stopped at a breakpoint the debugger is paused and hence jit is
> enabled, no?

On a breakpoint we enterNestedEventLoop(), the debugger is not paused.

In the New Year we should discuss controlling the JIT and jsd by JSContext, in one of the newsgroups.
> I doubt JIT will help Firebug much, we do too much DOM manipulation.

"You" (the firebug code) do all sorts of nutty stuf; some of it might or might not be helped by the jit.

> On a breakpoint we enterNestedEventLoop(), the debugger is not paused.

Can you please just read comment 19?  I'm tired of repeating myself on this point...
I added 
 Bug 537199 -  document jsdIDebuggerService.pauseDepth changes by the platform
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 ID:20100115132715

Firebug 1.5.0 final installed and inactive.
Status: RESOLVED → VERIFIED
I was under the impression in Firefox 3.6 and Firebug 1.5 this issue was fixed. However with the Firebug console and script panel enabled I'm getting over 40ms on http://people.mozilla.com/~vladimir/ss/hosted/bitops-bitwise-and.html,
but with disabled I'm getting less than 5ms.

Firefox 3.6.4
Firebug 1.5.4
If you have the script panel enabled, jsd is not paused, and the JIT cannot run.
Marcus, this bug was about Firebug disabling the jit when not actively debugging.  But you're actively debugging.
Whiteboard: [firebug-p1][firebug-blocks][can land] → [firebug-p1][firebug-blocks]
Component: JavaScript Debugging/Profiling APIs → JavaScript Engine
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: