Closed Bug 950203 Opened 11 years ago Closed 11 years ago

ANR: Frequent hangs in GLController.updateCompositor

Categories

(Firefox for Android Graveyard :: Toolbar, defect)

ARM
Android
defect
Not set
normal

Tracking

(firefox26 unaffected, firefox27 unaffected, firefox28 fixed, firefox29 fixed, fennec27+)

RESOLVED FIXED
Firefox 29
Tracking Status
firefox26 --- unaffected
firefox27 --- unaffected
firefox28 --- fixed
firefox29 --- fixed
fennec 27+ ---

People

(Reporter: jchen, Assigned: bjacob)

References

Details

(Keywords: regression, Whiteboard: [ANR])

Attachments

(2 files)

Starting with the 11/23 Nightly, there has been a lot of Fennec hangs involving GLController.updateCompositor. I think it's a regression from bug 834243. The main thread stack is > java.lang.Object.wait > java.lang.Object.wait (line 401) > org.mozilla.gecko.GeckoAppShell.sendEventToGeckoSync (line 455) > org.mozilla.gecko.gfx.GLController.updateCompositor (line 195) > org.mozilla.gecko.gfx.GLController$1.run (line 168) > android.os.Handler.handleCallback (line 733) > android.os.Handler.dispatchMessage (line 95) > android.os.Looper.loop (line 137) > android.app.ActivityThread.main (line 5165) > java.lang.reflect.Method.invokeNative > java.lang.reflect.Method.invoke (line 515) > com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run (line 1013) > com.android.internal.os.ZygoteInit.main (line 829) > dalvik.system.NativeStart.main Other threads don't have very interesting stacks. For more information, see http://darchons.github.io/anr-dashboard and click on the top (red-colored) report.
Thanks a lot for bringing this to my attention. Great tool btw. I will have to look into this very quickly...
tracking-fennec: --- → ?
Can we get full symbolicated stacks of what the gecko thread and compositor thread are doing at the time of the ANR? I'm not sure I trust the stacks I see on the ANR dashboard at the moment; the gecko stack is reported as __futex_syscall3+4294832136 (/system/lib/libc.so) __pthread_cond_timedwait_relative+48 (/system/lib/libc.so) __pthread_cond_timedwait+64 (/system/lib/libc.so) deleted (/dev/ashmem/libnss3.so) org.mozilla.gecko.mozglue.GeckoLoader.nativeRun org.mozilla.gecko.GeckoAppShell.runGecko (line 373) org.mozilla.gecko.GeckoThread.run (line 177) Which doesn't say a whole lot. This was (and still is) a problem with all of the calls to sendEventToGeckoSync, such as from the LowMemoryMonitor - often Gecko is busy for 5+ seconds so sendEventToGeckoSync is inherently prone to ANRs. We should reduce using that function as much as possible but in cases where we can't we should try to find out why Gecko is taking so long and address that. Improving Gecko's responsiveness will help overall responsiveness as well.
Flags: needinfo?(nchen)
(In reply to Kartikaya Gupta (email:kats@mozilla.com) from comment #2) > Can we get full symbolicated stacks of what the gecko thread and compositor > thread are doing at the time of the ANR? I'm not sure I trust the stacks I > see on the ANR dashboard at the moment; the gecko stack is reported as > > __futex_syscall3+4294832136 (/system/lib/libc.so) > __pthread_cond_timedwait_relative+48 (/system/lib/libc.so) > __pthread_cond_timedwait+64 (/system/lib/libc.so) > deleted (/dev/ashmem/libnss3.so) > org.mozilla.gecko.mozglue.GeckoLoader.nativeRun > org.mozilla.gecko.GeckoAppShell.runGecko (line 373) > org.mozilla.gecko.GeckoThread.run (line 177) > > Which doesn't say a whole lot. This was (and still is) a problem with all of > the calls to sendEventToGeckoSync, such as from the LowMemoryMonitor - often > Gecko is busy for 5+ seconds so sendEventToGeckoSync is inherently prone to > ANRs. We should reduce using that function as much as possible but in cases > where we can't we should try to find out why Gecko is taking so long and > address that. Improving Gecko's responsiveness will help overall > responsiveness as well. Unfortunately the ANR reports don't have the full stack, and I'm not sure of the STR. I looked through a few different reports and they all have the Gecko and compositor threads as waiting when the hang happened, which is weird. Here's a logcat from one report that may help, > 12-08 20:36:40.195 31116 31116 W GeckoGLController: GLController::serverSurfaceChanged(800, 777) > 12-08 20:36:40.235 31116 31116 W GeckoGLController: GLController::updateCompositor with mCompositorCreated=false > 12-08 20:36:40.235 31116 31116 W GeckoGLController: done GLController::updateCompositor > … > 12-08 20:36:42.785 31116 31116 W GeckoGLController: GLController::serverSurfaceChanged(800, 665) > 12-08 20:36:42.815 31116 31116 W GeckoGLController: GLController::updateCompositor with mCompositorCreated=false > 12-08 20:36:42.815 31116 31116 W GeckoGLController: done GLController::updateCompositor > … > 12-08 20:36:45.045 31116 31116 W GeckoGLController: GLController::serverSurfaceDestroyed() with mCompositorCreated=false > 12-08 20:36:45.065 31116 31116 W GeckoGLController: done GLController::serverSurfaceDestroyed() > … > 12-08 20:36:45.495 31116 31116 W ManagedEGLContext: doTerminate failed: EGL count is 3 but managed count is 1 > … > 12-08 20:36:53.285 31116 31152 D GeckoLayerClient: Screen-size changed to (800,1205) > 12-08 20:36:53.285 31116 31152 D GeckoLayerClient: Window-size changed to (800,665) > 12-08 20:36:53.305 31116 31116 W GeckoGLController: GLController::updateCompositor with mCompositorCreated=false > 12-08 20:36:53.315 31116 31116 E GeckoGLController: eglCreateWindowSurface threw > 12-08 20:36:53.315 31116 31116 E GeckoGLController: java.lang.IllegalArgumentException: Make sure the SurfaceView or associated SurfaceHolder has a valid Surface > 12-08 20:36:53.315 31116 31116 E GeckoGLController: at com.google.android.gles_jni.EGLImpl._eglCreateWindowSurface(Native Method) > 12-08 20:36:53.315 31116 31116 E GeckoGLController: at com.google.android.gles_jni.EGLImpl.eglCreateWindowSurface(EGLImpl.java:92) > 12-08 20:36:53.315 31116 31116 E GeckoGLController: at org.mozilla.gecko.gfx.GLController.AttemptPreallocateEGLSurfaceForCompositor(GLController.java:303) > 12-08 20:36:53.315 31116 31116 E GeckoGLController: at org.mozilla.gecko.gfx.GLController.updateCompositor(GLController.java:186) > 12-08 20:36:53.315 31116 31116 E GeckoGLController: at org.mozilla.gecko.gfx.GeckoLayerClient$1.run(GeckoLayerClient.java:157) > 12-08 20:36:53.315 31116 31116 E GeckoGLController: at android.os.Handler.handleCallback(Handler.java:733) > 12-08 20:36:53.315 31116 31116 E GeckoGLController: at android.os.Handler.dispatchMessage(Handler.java:95) > 12-08 20:36:53.315 31116 31116 E GeckoGLController: at android.os.Looper.loop(Looper.java:137) > 12-08 20:36:53.315 31116 31116 E GeckoGLController: at android.app.ActivityThread.main(ActivityThread.java:4998) > 12-08 20:36:53.315 31116 31116 E GeckoGLController: at java.lang.reflect.Method.invokeNative(Native Method) > 12-08 20:36:53.315 31116 31116 E GeckoGLController: at java.lang.reflect.Method.invoke(Method.java:515) > 12-08 20:36:53.315 31116 31116 E GeckoGLController: at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:777) > 12-08 20:36:53.315 31116 31116 E GeckoGLController: at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:593) > 12-08 20:36:53.315 31116 31116 E GeckoGLController: at dalvik.system.NativeStart.main(Native Method) > … > 12-08 20:37:09.585 31116 31116 W ManagedEGLContext: doTerminate failed: EGL count is 3 but managed count is 1 > … > 12-08 21:09:07.515 31116 31116 W ManagedEGLContext: doTerminate failed: EGL count is 3 but managed count is 1 > … > 12-08 21:09:36.285 31116 31116 W GeckoGLController: GLController::serverSurfaceChanged(800, 740) > 12-08 21:09:36.285 31116 31116 D GeckoLayerClient: Window-size changed to (800,740) > 12-08 21:09:36.395 31116 31116 W GeckoGLController: GLController::updateCompositor with mCompositorCreated=false > 12-08 21:09:37.405 31116 31116 D GeckoAppShell: Gecko event sync taking too long: 1000ms > 12-08 21:09:38.405 31116 31116 D GeckoAppShell: Gecko event sync taking too long: 2001ms > 12-08 21:09:39.405 31116 31116 D GeckoAppShell: Gecko event sync taking too long: 3001ms > 12-08 21:09:40.405 31116 31116 D GeckoAppShell: Gecko event sync taking too long: 4001ms > 12-08 21:09:41.405 31116 31116 D GeckoAppShell: Gecko event sync taking too long: 5002ms > 12-08 21:09:42.405 31116 31116 D GeckoAppShell: Gecko event sync taking too long: 6002ms > 12-08 21:09:43.405 31116 31116 D GeckoAppShell: Gecko event sync taking too long: 7003ms > 12-08 21:09:44.405 31116 31116 D GeckoAppShell: Gecko event sync taking too long: 8003ms > 12-08 21:09:45.435 31116 31116 D GeckoAppShell: Gecko event sync taking too long: 9036ms > 12-08 21:09:46.435 31116 31116 D GeckoAppShell: Gecko event sync taking too long: 10036ms > 12-08 21:09:47.435 31116 31116 D GeckoAppShell: Gecko event sync taking too long: 11036ms > 12-08 21:09:47.475 31116 31129 I GeckoANRReporter: processing Gecko ANR
Flags: needinfo?(nchen)
bjacob: it's suspicious that mCompositorCreated is always false in the log above. It should get set to true at some point pretty early in the run and stay true forever after.
I would like to know, for how long have these firefoxes been running, when the ANR happens? I'm asking because the interpretation of mCompositorCreated being false depends on whether this is occurring on startup.
For this particular ANR, the uptime is 33 minutes. In general, uptimes are 42%: < 1 minute uptime (or error in getting uptime) 21%: 1-5 minutes uptime 16%: 5-30 minutes uptime 14%: 30-180 minutes uptime 5%: 3-24 hours uptime 2%: 1-7 days uptime
Is this something that enabling strict mode in the developer settings would show?
I just found something that might explain things, although it's not yet clear to me how it explains this particular ANR. When eglCreateWindowSurface fails, it returns EGL10.EGL_NO_SURFACE. It doesn't return null, although it conceivably could if there was a bug in Android. Before my patches, we used to check for both null and EGL10.EGL_NO_SURFACE. Now, we only check for null. So if eglCreateWindowSurface fails, and we now have EGL10.EGL_NO_SURFACE, we now mis-interpret it as a valid surface. We should at least fix that... whether that will somehow fix the bug is a different story, but it will at least make it easier to reason about, and will revert the code to what it did before.
Attachment #8348470 - Flags: review?(bugmail.mozilla)
Just ran into this on my Nexus 4, here's the Gecko thread stack. Looks like a deadlock that could be related to bug 937204 instead of bug 834243. > #0 0x400665d0 in __futex_syscall3 () > from /home/nchen/android-gdb/lib/016ec233976516f7/system/lib/libc.so > #1 0x400586ac in __pthread_cond_timedwait_relative () > from /home/nchen/android-gdb/lib/016ec233976516f7/system/lib/libc.so > #2 0x4005870c in __pthread_cond_timedwait () > from /home/nchen/android-gdb/lib/016ec233976516f7/system/lib/libc.so > #3 0x771cced8 in PR_WaitCondVar () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libnss3.so > #4 0x7aab3ff0 in mozilla::widget::GfxInfo::GLStrings::EnsureInitialized() () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #5 0x7aab41b2 in mozilla::widget::GfxInfo::EnsureInitialized() [clone .part.8] () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #6 0x7aab4c54 in mozilla::widget::GfxInfo::GetAdapterDescription(nsAString_internal&) () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #7 0x7a654df6 in NS_InvokeByIndex () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #8 0x7aaf57be in XPCWrappedNative::CallMethod(XPCCallContext&, XPCWrappedNative::CallMode) () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #9 0x7aaf5ade in XPC_WN_GetterSetter(JSContext*, unsigned int, JS::Value*) () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #10 0x7b425acc in js::Invoke(JSContext*, JS::CallArgs, js::MaybeConstruct) () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #11 0x7b425ed6 in js::Invoke(JSContext*, JS::Value const&, JS::Value const&, unsigned int, JS::Value*, JS::MutableHandle<JS::Value>) () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #12 0x7b425f64 in js::InvokeGetterOrSetter(JSContext*, JSObject*, JS::Value, unsigned int, JS::Value*, JS::MutableHandle<JS::Value>) () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #13 0x7b3d662a in js::Shape::get(JSContext*, JS::Handle<JSObject*>, JSObject*, JSObject*, JS::MutableHandle<JS::Value>) () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #14 0x7b3da682 in js::baseops::GetProperty(JSContext*, JS::Handle<JSObject*>, JS::Handle<JSObject*>, JS::Handle<int>, JS::MutableHandle<JS::Value>) () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #15 0x7b4244fe in Interpret(JSContext*, js::RunState&) () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #16 0x7b42585c in js::RunScript(JSContext*, js::RunState&) [clone .part.392] () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #17 0x7b425aaa in js::Invoke(JSContext*, JS::CallArgs, js::MaybeConstruct) () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #18 0x7b425ed6 in js::Invoke(JSContext*, JS::Value const&, JS::Value const&, unsigned int, JS::Value*, JS::MutableHandle<JS::Value>) () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #19 0x7b3c08fe in JS_CallFunctionValue(JSContext*, JSObject*, JS::Value, unsigned int, JS::Value*, JS::Value*) () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #20 0x7aaf38e8 in nsXPCWrappedJSClass::CallMethod(nsXPCWrappedJS*, unsigned short, XPTMethodDescriptor const*, nsXPTCMiniVariant*) () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #21 0x7aae98b0 in nsXPCWrappedJS::CallMethod(unsigned short, XPTMethodDescriptor const*, nsXPTCMiniVariant*) () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #22 0x7a655672 in _PrepareAndDispatch () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #23 0x7a654e10 in SharedStub () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #24 0x7a6407a4 in nsObserverList::NotifyObservers(nsISupports*, char const*, char16_t const*) () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #25 0x7a6407ec in nsObserverService::NotifyObservers(nsISupports*, char const*, char16_t const*) () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #26 0x7aab5766 in nsAppShell::ProcessNextNativeEvent(bool) () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #27 0x7aabe6f0 in nsBaseAppShell::DoProcessNextNativeEvent(bool, unsigned int) () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #28 0x7aabe7bc in nsBaseAppShell::OnProcessNextEvent(nsIThreadInternal*, bool, unsigned int) () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #29 0x7a65079e in nsThread::ProcessNextEvent(bool, bool*) () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #30 0x7a62933e in NS_ProcessNextEvent(nsIThread*, bool) () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #31 0x7a75c554 in mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #32 0x7a7544e2 in MessageLoop::RunInternal() () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #33 0x7a754594 in MessageLoop::Run() () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #34 0x7aabb182 in nsBaseAppShell::Run() () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #35 0x7b04ecf6 in nsAppStartup::Run() () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #36 0x7b0264b6 in XREMain::XRE_mainRun() () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #37 0x7b027548 in XREMain::XRE_main(int, char**, nsXREAppData const*) () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #38 0x7b0276c4 in XRE_main () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #39 0x7b02a2d8 in GeckoStart () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/assets/libxul.so > #40 0x7320251c in Java_org_mozilla_gecko_mozglue_GeckoLoader_nativeRun () > from /home/nchen/android-gdb/lib/016ec233976516f7/app/org.mozilla.fennec/lib/armeabi-v7a/libmozglue.so
Comment on attachment 8348470 [details] [diff] [review] Correctly handle EGL_NO_SURFACE Review of attachment 8348470 [details] [diff] [review]: ----------------------------------------------------------------- ::: mobile/android/base/gfx/GLController.java @@ +307,5 @@ > } > + // In failure cases, eglCreateWindowSurface should return EGL_NO_SURFACE. > + // We currently normalize this to null, and compare to null in all our checks. > + if (mEGLSurfaceForCompositor == EGL10.EGL_NO_SURFACE) { > + mEGLSurfaceForCompositor = null; I'd prefer to move this condition inside the try, right after the call to eglCreateWindowSurface. The second if condition you added can stay where it is.
Attachment #8348470 - Flags: review?(bugmail.mozilla) → review+
(In reply to Jim Chen [:jchen :nchen] from comment #10) > Just ran into this on my Nexus 4, here's the Gecko thread stack. Looks like > a deadlock that could be related to bug 937204 instead of bug 834243. > Indeed, this is a deadlock. The gecko thread is blocked in GLStrings::EnsureInitialized waiting for the InitializeGLStrings call to run on the compositor thread. That will never happen because InitializeGLStrings is triggered during compositor startup which is triggered from code on the gecko thread.
This is a net reduction of number of lines of code, and removes an artificial dependency between gfxinfo and compositor, and the motivation to do it now is that the above comments show that it was actually deadlocking. Now that I think about it, the deadlock is really easy to reproduce and an unavoidable flaw of the design we had: if by chance we run JS early enough, that tries to get GfxInfo (as is shown in the above stack) and the compositor creation messages haven't been processed on the main thread yet, then we stall. Taking a step back, the only reason why we were piggyback on the compositor's GL context was optimization. But the only place where GfxInfo initialization matters a lot is on startup, and on startup we already dodge the problem by early-returning in GetFeatureStatus on FEATURE_LAYERS_OPENGL as that is always available on Android. So really the optimization here (avoiding creating a GL context just for getting strings) is not very important. We're just saving a few ms on the first canvas getContext.
Attachment #8349087 - Flags: review?(jmuizelaar)
Note: I have a local patch that queries GfxInfo from the nsGlobalWindow ctor (which is about as early as any JS could run...), it reproduces the stall, and with the above GfxInfo patch, the stall is fixed.
Attachment #8349087 - Flags: review?(jmuizelaar) → review+
Comment on attachment 8349087 [details] [diff] [review] Let GfxInfo on Android create its own GL context, instead of waiting for the compositor's [Approval Request Comment] Bug caused by (feature/regressing bug #): bug 937204 User impact if declined: Top cause of hangs in fennec Testing completed (on m-c, etc.): m-i Risk to taking this patch (and alternatives if risky): Low risk: this patch is almost only removing code. String or IDL/UUID changes made by this patch: none
Attachment #8349087 - Flags: approval-mozilla-aurora?
Comment on attachment 8349087 [details] [diff] [review] Let GfxInfo on Android create its own GL context, instead of waiting for the compositor's pre-approving for aurora due to upcoming holidays and needing this in for crashdata collection this week - low risk, should be ok.
Attachment #8349087 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
flagging [leave open] for now, until we get confirmation that this is fixed.
Whiteboard: [ANR] → [ANR][leave open]
Blocks: 937204
No longer blocks: 834243
Jim, how quickly can we track if these ANRs are solved? With dozens of ANRs per day, we should already have statistically significant results after just a few hours today.
Flags: needinfo?(nchen)
It also depends on how fast people update to the new nightly. I'd wait until tomorrow at least.
Flags: needinfo?(nchen)
tracking-fennec: ? → 27+
NI'ing Jim for post-break bugmail
Flags: needinfo?(nchen)
ANR reports from the last two weeks are in. There is one report from the 2013-12-21 Aurora build that looks a lot like this one. However, since it's only one report versus the hundreds we were getting before, I want to say that report is due to another cause, so I'm marking this bug as fixed.
Status: NEW → RESOLVED
Closed: 11 years ago
Flags: needinfo?(nchen)
Resolution: --- → FIXED
Whiteboard: [ANR][leave open] → [ANR]
Many thanks Jim for reporting and tracking this really severe bug!
No, for all I know only central and aurora were affected.
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: