Closed Bug 616850 Opened 14 years ago Closed 12 years ago

Huge CC pauses when using AutoPagerize userscript on pixiv

Categories

(Core :: General, defect)

x86
Windows XP
defect
Not set
critical

Tracking

()

RESOLVED INCOMPLETE

People

(Reporter: andreasjunghw, Assigned: mccr8)

References

()

Details

(Keywords: hang, Whiteboard: [MemShrink:P3])

Attachments

(3 files)

User-Agent:       Mozilla/5.0 (Windows NT 5.1; rv:2.0b8pre) Gecko/20101205 Firefox/4.0b8pre
Build Identifier: Mozilla/5.0 (Windows NT 5.1; rv:2.0b8pre) Gecko/20101203 Firefox/4.0b8pre

When browsing on pixiv Firefox sometimes enters a state where it uses almost 100% CPU and you have to kill it because it never returns to normal.

I usually browse pixiv like this:
1) open a search and open the images I'm interested in new tabs
2) scroll down so that Autopagerize loads the next page
3) repeat 1)-2) 20-30 times
4) close the tab with the search results
5) save images I'm interested in, then close the tab
6) repeat 5)

Either it works and I'm able to go through all tabs, or Firefox hangs after a few tabs.

Reproducible: Sometimes




Mozilla/5.0 (Windows NT 5.1; rv:2.0b8pre) Gecko/20101203 Firefox/4.0b8pre
Built from http://hg.mozilla.org/mozilla-central/rev/0ff6d5984287

The thread that uses all the CPU is (according to Process Explorer):
   3  Id: 3f0.cc0 Suspend: 1 Teb: 7ffd9000 Unfrozen
ChildEBP RetAddr  
0162fca0 7c91cf7a ntdll!KiFastSystemCallRet
0162fca4 7c809b42 ntdll!NtAllocateVirtualMemory+0xc
0162fcf0 7c809b09 kernel32!VirtualAllocEx+0x47
0162fd0c 78135603 kernel32!VirtualAlloc+0x18
0162fd2c 78138cd9 MOZCRT19!chunk_alloc_mmap(unsigned int size = 0x800000, int pagefile = 0n-1957298293)+0x63 [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\obj-firefox\memory\jemalloc\crtsrc\jemalloc.c @ 2460]
0162fd38 7813928f MOZCRT19!chunk_alloc(unsigned int size = 0, int zero = 0n786919520, int pagefile = 0n786919520)+0x9 [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\obj-firefox\memory\jemalloc\crtsrc\jemalloc.c @ 2572]
0162fd4c 781399e3 MOZCRT19!huge_malloc(unsigned int size = 0, int zero = 0n786919520)+0x2f [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\obj-firefox\memory\jemalloc\crtsrc\jemalloc.c @ 4654]
0162fd54 101f7632 MOZCRT19!malloc(unsigned int size = 0x10c8e890)+0x53 [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\obj-firefox\memory\jemalloc\crtsrc\jemalloc.c @ 5873]
0162fd64 101f7969 xul!nsDeque::GrowCapacity(void)+0x22 [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\obj-firefox\xpcom\build\nsdeque.cpp @ 179]
0162fdb4 101f75bb xul!GraphWalker<scanVisitor>::DoWalk(class nsDeque * aQueue = 0x00800000)+0x2b9 [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\xpcom\base\nscyclecollector.cpp @ 1267]
0162fe10 101e88ce xul!GraphWalker<scanVisitor>::WalkFromRoots(struct GCGraph * aGraph = 0x0000000e)+0xab [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\xpcom\base\nscyclecollector.cpp @ 1255]
0162fea0 10246cc7 xul!nsCycleCollector::BeginCollection(int aForceGC = 0n8435820, class nsICycleCollectorListener * aListener = 0x00000000)+0xad [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\xpcom\base\nscyclecollector.cpp @ 2662]
0162feb8 10101eb6 xul!nsCycleCollectorRunner::Run(void)+0x47 [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\xpcom\base\nscyclecollector.cpp @ 3325]
0162ff24 10055fbc xul!nsThread::ProcessNextEvent(int mayWait = <Memory access error>, int * result = <Memory access error>)+0x266 [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\xpcom\threads\nsthread.cpp @ 632]
0162ff4c 0037bdf9 xul!nsThread::ThreadFunc(void * arg = 0x006f00c4)+0x8c [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\xpcom\threads\nsthread.cpp @ 278]
0162ff6c 0037e04d nspr4!_PR_NativeRunThread(void * arg = 0x008343c0)+0x169 [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\nsprpub\pr\src\threads\combined\pruthr.c @ 448]
0162ff74 78132c28 nspr4!pr_root(void * arg = 0x78132cb6)+0xd [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\nsprpub\pr\src\md\windows\w95thred.c @ 122]
0162ffac 78132cb6 MOZCRT19!_callthreadstartex(void)+0x48 [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\obj-firefox\memory\jemalloc\crtsrc\threadex.c @ 348]
0162ffb4 7c80b729 MOZCRT19!_threadstartex(void * ptd = 0x00000000)+0x66 [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\obj-firefox\memory\jemalloc\crtsrc\threadex.c @ 326]
0162ffec 00000000 kernel32!BaseThreadStart+0x37

The thread that causes the hang is (according to WinDbg):
00139c3c 7c91df5a 7c8025db 0000005c 00000000 ntdll!KiFastSystemCallRet
00139c40 7c8025db 0000005c 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc
00139ca4 7c802542 0000005c ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xa8
00139cb8 0037f3e9 0000005c ffffffff 00833f70 kernel32!WaitForSingleObject+0x12
00139cdc 0037af48 0086e184 00833f8c ffffffff nspr4!_PR_MD_WAIT_CV+0xc9 [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\nsprpub\pr\src\md\windows\w95cv.c @ 282]
00139cfc 0037b06b ffffffff 0081579c 008157a0 nspr4!_PR_WaitCondVar+0x58 [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\nsprpub\pr\src\threads\combined\prucv.c @ 205]
00139d14 10246cfc 0086e110 ffffffff 102035aa nspr4!PR_WaitCondVar+0x3b [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\nsprpub\pr\src\threads\combined\prucv.c @ 547]
00139d20 102035aa 00000001 62093080 00000002 xul!mozilla::CondVar::Wait+0xb [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\obj-firefox\dist\include\mozilla\condvar.h @ 101]
0013dbd4 102497e9 00000000 0013dc40 0013dc40 xul!nsCycleCollectorRunner::Collect+0x83 [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\xpcom\base\nscyclecollector.cpp @ 3362]
0013dbe4 10249798 00000000 1020cf1f 00000000 xul!nsCycleCollector_collect+0x1c [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\xpcom\base\nscyclecollector.cpp @ 3475]
0013dbec 1020cf1f 00000000 1025e2eb 01c4e800 xul!nsJSContext::CC+0x40 [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\dom\base\nsjsenvironment.cpp @ 3656]
0013dc00 1025e2c1 62093080 62093084 101d698e xul!nsJSContext::IntervalCC+0x27 [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\dom\base\nsjsenvironment.cpp @ 3760]
0013dc0c 101d698e 00000001 101d693c 1008bbf0 xul!nsJSContext::MaybeCC+0xad [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\dom\base\nsjsenvironment.cpp @ 3728]
0013dc18 1008bbf0 62093080 00000000 00817470 xul!GCTimerFired+0x52 [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\dom\base\nsjsenvironment.cpp @ 3785]
0013dc40 1013830c 008397c4 10102117 39f26e40 xul!nsTimerImpl::Fire+0xe0 [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\xpcom\threads\nstimerimpl.cpp @ 425]
0013dc48 10102117 39f26e40 2e3ab800 059ab280 xul!nsTimerEvent::Run+0x1c [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\xpcom\threads\nstimerimpl.cpp @ 519]
0013dca4 0037b422 0013dcb8 0037b422 a19502b6 xul!nsThread::ProcessNextEvent+0x4c7 [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\xpcom\threads\nsthread.cpp @ 626]
0013dcb4 1012b43e 00839780 00000000 0013dcdc nspr4!PR_AssertCurrentThreadOwnsLock+0x12 [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\nsprpub\pr\src\threads\combined\prulock.c @ 404]
0013dcf4 1028ccc8 0083c240 a19503e2 00839780 xul!mozilla::ipc::MessagePump::Run+0x6e [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\ipc\glue\messagepump.cpp @ 110]
0013dd30 1028cc92 00368b80 00000001 0013fc00 xul!MessageLoop::RunHandler+0x28 [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\ipc\chromium\src\base\message_loop.cc @ 203]
0013dd50 102759a9 10b0fd24 01c20ba0 1029f728 xul!MessageLoop::Run+0x15 [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\ipc\chromium\src\base\message_loop.cc @ 177]
0013dd5c 1029f728 01c20ba0 020f2280 00400000 xul!nsBaseAppShell::Run+0x34 [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\widget\src\xpwidgets\nsbaseappshell.cpp @ 198]
0013fcb0 1029f75c 01c20ba0 00000000 101c9bc1 xul!nsAppShell::Run+0x42 [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\widget\src\windows\nsappshell.cpp @ 264]
0013fcbc 101c9bc1 020f2280 00817300 00000003 xul!nsAppStartup::Run+0x1e [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\toolkit\components\startup\src\nsappstartup.cpp @ 192]
0013ff34 0040134c 00000003 00817300 00813380 xul!XRE_main+0xda8 [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\toolkit\xre\nsapprunner.cpp @ 3695]
0013ff80 004016f2 00000003 00839080 008110c0 firefox!wmain+0x34c [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\toolkit\xre\nswindowswmain.cpp @ 128]
0013ffc0 7c817077 0141df4c 00000018 7ffde000 firefox!__tmainCRTStartup+0x152 [e:\builds\moz2_slave\mozilla-central-win32-nightly\build\obj-firefox\memory\jemalloc\crtsrc\crtexe.c @ 591]
0013fff0 00000000 00401870 00000000 78746341 kernel32!BaseProcessStart+0x23
Attached file WinDbg log
Keywords: hang
Version: unspecified → Trunk
Hmm.  How big is your heap when this happens?  That stack is showing us cycle-collecting....
Attached image memory usage
I can't give details on the memory usage of Firefox because the UI stops responding when the bug occurs and so I can't access about:memory.

I hope the numbers displayed by Process Explorer help.
Yeah, that's what I was looking for.  So yeah, you're running at about the limit of memory that 32-bit processes on Windows can use.  Given the huge heap, I can definitely see cycle collection taking some time....

The real question here is why the heap is so big.  Either we're leaking, or some script is allocating memory and holding on to it, or something. :(
The situation improved a lot, because:
a) in recent builds Firefox's memory usage is much more reasonable again
b) the browser actually is able to recover from the situation 
   (you have to wait several minutes though)

I still rarely get this bug. 
When it happened I had logging active:

GC mode: full, timestamp: 1299536710618000, duration: 841 ms.
CC timestamp: 1299536718279000, collected: 16 (16 waiting for GC), suspected: 4153, duration: 5157 ms.
CC timestamp: 1299536731058000, collected: 0 (16 waiting for GC), suspected: 2214, duration: 6239 ms.
CC timestamp: 1299536961289000, collected: 230 (246 waiting for GC), suspected: 1525, duration: 200428 ms.
GC mode: full, timestamp: 1299537048394000, duration: 751 ms.
CC timestamp: 1299537337700000, collected: 4260 (4260 waiting for GC), suspected: 4617, duration: 287493 ms.
GC mode: full, timestamp: 1299537345992000, duration: 961 ms.
CC timestamp: 1299537349447000, collected: 3637 (3637 waiting for GC), suspected: 2314, duration: 3445 ms.
GC mode: full, timestamp: 1299537354484000, duration: 1021 ms.
CC timestamp: 1299537362345000, collected: 4574 (4574 waiting for GC), suspected: 9168, duration: 3374 ms.
GC mode: full, timestamp: 1299537362936000, duration: 540 ms.

GC mode: full, timestamp: 1299537684889000, duration: 531 ms.
CC timestamp: 1299537960365000, collected: 8525 (8525 waiting for GC), suspected: 3670, duration: 275466 ms.
GC mode: full, timestamp: 1299538011579000, duration: 9654 ms.
CC timestamp: 1299538226768000, collected: 98 (98 waiting for GC), suspected: 908, duration: 209681 ms.
GC mode: full, timestamp: 1299538233939000, duration: 581 ms.
CC timestamp: 1299538256641000, collected: 11894 (11894 waiting for GC), suspected: 1128, duration: 4636 ms.
GC mode: full, timestamp: 1299538264092000, duration: 1402 ms.
CC timestamp: 1299538286064000, collected: 5678 (5678 waiting for GC), suspected: 4382, duration: 3766 ms.
GC mode: full, timestamp: 1299538288197000, duration: 962 ms.
> duration: 275466 ms.

Uh... _that_ would do it, yes.

The question of how we get into that situation remains.  :(
Funky. I don't think there was a huge spike in allocations in between. Something just made the CC heap walk super slow. We don't have a delayed-marking mechanism in the CC that might become super slow.
I can't reproduce this on Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0) Gecko/20100101 Firefox/4.0
I can reproduce it.  Mozilla/5.0 (Windows NT 6.1; rv:2.0) Gecko/20100101 Firefox/4.0
I'm marking this as blocking bug 640452 because it's sounds leak-like, even if it's not actually a leak.
Blocks: mlk-fx5+
Summary: Hang when using AutoPagerize userscript on pixiv → Huge CC pauses when using AutoPagerize userscript on pixiv
No longer blocks: mlk-fx5+
Whiteboard: [MemShrink:P2]
Assignee: nobody → continuation
Now that there are per compartment memory reporters, I spotted something interesting.

Adding up the numbers per category resulted in the following:

number of tabs: 30

         compartments    memory
Ads       97             308 MB (50%)
Pixiv     33             210 MB	(34%)
Twitter   22              48 MB ( 8%)
Plusone   14              38 MB ( 6%)
Facebook  14              13 MB ( 2%)
--------------------------------------
         180             618 MB

Most of the memory is used by compartments belonging to ads.
Ads also use more compartments than everything else together.
Twitter/GooglePlusone/Facebook buttons use a lot of compartments, but not that much memory.

The memory used by images was less than 40 MB, the memory used by layout less than 20MB, so this is not a problem anymore (mainly thanks to fast discarding of images in background tabs, I think).

Note that these numbers were measured after some browsing around, so I'm not sure if all of these compartments really belong to the 30 open tabs or if some of them stayed alive after their tab closed.
I verified that all ad / button pages contain pixiv in the URL though, so all of them must have belonged or still belong to a pixiv tab.
That is interesting.  In my experience, Twitter, Plusone and Facebook don't have all that many ads, so maybe it is mostly from Pixiv.  Are you still experiencing the long pauses you described in your first message?
Just to make sure there is no misunderstanding between us,
all the mentioned compartments belong to pixiv tabs.
There are no tabs with Facebook / Twitter opened.

Basically the pixiv compartments are there (obviously) because in the tabs pixiv images + searches are opened.
The Twitter / Google PlusOne / Facebook compartments are there because on the pixiv (image) tabs the respective buttons (i.e. <iframe>s) are shown.
The ad compartments (google ads + pixiv own ads) are there because there are ads shown on pixiv.

(In reply to comment #12)
> Are you still experiencing the long pauses you described in your first message?

Like I already said in comment 5 it improved over time as Firefox's memory usage became more reasonable again and like I said in comment 11 the reactivation of fast discarding of images in background tabs also helped a lot.

Still, if you manage to get Firefox in massive memory pressure the cycle collector gets horribly slow. Just yesterday I got in this situation once; Firefox used almost all addressable memory (on 32-bit) and the CC took forever; After 5 minutes I gave up and killed Firefox.

What I don't understand is that this situation sometimes occurs at ca. 1.75 GB memory usage, sometimes not until 1.80 GB memory usage and sometimes not until a memory usage of ca. 1.95 GB...
> What I don't understand is that this situation sometimes occurs at ca. 1.75
> GB memory usage, sometimes not until 1.80 GB memory usage and sometimes not
> until a memory usage of ca. 1.95 GB...

It will depend what other programs on your machine are doing.  If they are using more of the available physical memory, then Firefox will start paging (and become hard to use) earlier.

Comment 11 seems like a good advertisement for AdBlock Plus.

Can we mark this bug as fixed?
(In reply to comment #14)
> > What I don't understand is that this situation sometimes occurs at ca. 1.75
> > GB memory usage, sometimes not until 1.80 GB memory usage and sometimes not
> > until a memory usage of ca. 1.95 GB...
> 
> It will depend what other programs on your machine are doing.  If they are
> using more of the available physical memory, then Firefox will start paging
> (and become hard to use) earlier.

Yes, but it still doesn't make sense to me that the behavior is basically:
(1) All of Firefox fits into physical memory => CC is fast
(2) Firefox doesn't fit completely into physical memory => CC becomes slower
                                                      (maybe 10x slower than (1))
(3) Firefox uses only a few MB more (~10MB?) => CC is much slower
                                                (up to 100x slower than (2))
[Note that these are numbers from my memory and not recently measured, the situation might have improved since]

Can 10MB more make really such a large difference?

The other thing is, while Firefox is hanging it uses 100% CPU.
A typical hang because of paging usually drains CPU usage to almost 0% or is that not always the case?

> Comment 11 seems like a good advertisement for AdBlock Plus.

I actually have Adblock Plus installed, but I have it disabled on sites which use unobtrusive ads. Therefore I have it disabled on pixiv.
If the conclusion of this bug is that it's the ads fault and not Firefox's I might have to reenable ABP on pixiv again though.
The cycle collector scans through a large chunk of memory, even things that may not have been use recently (like any kind of garbage collector ish thing), so it exacerbates the paging problem.  I'll investigate this a bit to see if I can make any use of it.  You are right about the CPU going to 100% being a little weird.
(In reply to comment #11)
> Note that these numbers were measured after some browsing around, so I'm not
> sure if all of these compartments really belong to the 30 open tabs or if
> some of them stayed alive after their tab closed.

I checked this again and all these compartments seem to belong to tabs that are alive (i.e. no zombie compartments).


(In reply to comment #16)
> You are right about the CPU going to 100% being a little weird.

Looking through the comments of this bug it seems I never mentioned that Process Explorer shows almost all of the CPU usage in red. The help file says: "Red in the CPU usage graph indicates CPU usage in kernel-mode whereas green is the sum of kernel-mode and user-mode execution."
This doesn't answer the question why the CPU usage goes too 100% and if Firefox or Windows is at fault, but I thought it's something that might be interesting.
(In reply to comment #17)
> (In reply to comment #11)
> > Note that these numbers were measured after some browsing around, so I'm not
> > sure if all of these compartments really belong to the 30 open tabs or if
> > some of them stayed alive after their tab closed.
> 
> I checked this again and all these compartments seem to belong to tabs that
> are alive (i.e. no zombie compartments).

The number of compartments is still incredibly high.
Using a profile with no add-ons I get the following numbers:

number of pixiv tabs: 30

         compartments     memory
Ads      3                43,6 MB
Pixiv    1                99,8 MB
Twitter  1                 5,5 MB
Plusone  1                23,0 MB
Facebook 1                23,0 MB
------------------------------------		
         7               195,0 MB

Compared to my default profile with add-ons (comment 11) there is a huge difference in the number of compartments and the memory usage.

With add-ons there are 173 more compartments and over 400 MB more memory is used.

Especially Greasemonkey + Autopagerize seems suspicious, because the Autopagerize userscript is injected and executed on every website.

Can that explain the huge difference in the number of compartments and memory usage?
Could be, if they create tons of sandboxes that then stick around...

Can you try enabling just autopagerize or just greasemonkey and seeing what happens?
Looking at the source code for the addon (located at https://github.com/swdyh/autopagerize/blob/master/autopagerize.user.js ), it looks like this is a Jetpack addon (I'm guessing that based on the existence of a isJetpack() function), so this may be an instance of Bug 672443.
But yes, it looks like Autopagerize can run without Greasemonkey, so if you could try with just that it would be helpful.
Blocks: 672443
No longer blocks: 672443
Depends on: 672443
So, I would guess that part of the problem is that there are a huge number of compartments, but I'm not sure how that is causing CC times to explode.
Well, not just large number of compartments but also a few hundred megs extra RAM usage, right?
(In reply to comment #19)
> Can you try enabling just autopagerize or just greasemonkey and seeing what
> happens?

Yes, I tried it (always with the same set of 5 pixiv pages) and this are the numbers I got:

number of opened pixiv tabs: 5

number of compartments
======================

no add-ons
----------
pixiv   pixiv ads    google ads   plusone   twitter   facebook
1       1            1            1         1         1

[System Principal] moz-nullprincipal
1                  1


Greasemonkey + Autopagerize userscript
--------------------------------------
pixiv   pixiv ads    google ads   plusone   twitter   facebook
5       11           10           2         2         1

[System Principal] moz-nullprincipal
1                  1


Autopagerize add-on
-------------------
pixiv   pixiv ads    google ads   plusone   twitter   facebook
5       11           10           2         1         2

[System Principal] moz-nullprincipal
62                 1


memory usage
============
            private   resident   vsize
no add-ons  150 MB    155 MB     250 MB
GM+AP       444 MB    366 MB     584 MB
AP          183 MB    180 MB     299 MB


used versions
=============
Mozilla/5.0 (Windows NT 5.1; rv:8.0a1) Gecko/20110801 Firefox/8.0a1
http://hg.mozilla.org/mozilla-central/rev/f92e021f1f44

Greasemonkey 2011.08.02.nightly
https://arantius.com/misc/gm-nightly/

AutoPagerize userscript version 0.0.59
http://userscripts.org/scripts/show/8551

AutoPagerize add-on version 0.8.6
http://autopagerize.net/
This is a excel file containing:
1) the overview from comment 24
2) the output of about:memory?verbose for the 3 cases (no add-ons, GM+AP, AP)
Should this be downgraded to a MemShrink:P3?  That's what we usually use for problems that only affect add-ons (except for really popular add-ons like Firebug or AdBlock Plus, but I don't think AutoPagerize is that popular?)
Whiteboard: [MemShrink:P2] → [MemShrink:P3]
Is this still a problem?  We've landed some major CC improvements since this was last updated.
I'm going to close this for now. Please reopen if you are still having problems.
Status: UNCONFIRMED → RESOLVED
Closed: 12 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: