Closed Bug 51028 Opened 24 years ago Closed 15 years ago

memory usage for images is ~46% worse than nc4.75/ie5.5

Categories

(Core :: Graphics: ImageLib, defect, P3)

Product:
Core
Component:
Graphics: ImageLib
Type:
defect

Tracking

()

Status:
RESOLVED WORKSFORME

People

(Reporter: timeless, Unassigned)

References

(
URL
)

Details

(Keywords: memory-footprint, perf, testcase, Whiteboard: [adt needinfo])

Attachments

(3 files)

stress.html saved from http://timeless.haque.net/mozilla/stress.html
125.09 KB, text/plain
Details
another HTML showing the problem (using Yahoo images)
8.22 KB, text/html
Details
Another testcase: png image (72 KB) with 10k-15k size
71.39 KB, image/png
Details 
the bug url is a bit extreme, ~65k html file which re
oops
... -quires 885MB of [virtual] memory in nc4.75/ie5.5
moz needed 1300+MB.

I will generate a more reasonable testcase.

http://wam.umd.edu/~soref/mstress.html ~1K,
ie5.5: 26832K/19544K
moz:   42604K/35224K
          mem/vm
Keywords: 4xp, footprint, perf
talkback info, albeit sparse:

Incident ID 16672285 
 Trigger Time 
                2000-09-01 01:48:33 
 Email Address 
                timeless@bemail.org 
 User Comments 
 Build ID
                2000082815 
 Product ID
                Netscape6 
 Platform ID
                Win32 
 Stack Trace

nsFileTransport::Process
[d:\builds\seamonkey\mozilla\netwerk\base\src\nsFileTransport.cpp, line 511] 
here's a better trace, ignore the one above.

Incident ID 16672002 
 Trigger Time 
                2000-09-01 01:38:39 
 Email Address 
                timeless@bemail.org 
 User Comments 
                please attach to
http://bugzilla.mozilla.org/show_bug.cgi?id=51028 
 Build ID
                2000082815 
 Product ID
                Netscape6 
 Platform ID
                Win32 
 Stack Trace

nsFileTransport::Process
[d:\builds\seamonkey\mozilla\netwerk\base\src\nsFileTransport.cpp, line 511] 
nsFileTransport::Run
[d:\builds\seamonkey\mozilla\netwerk\base\src\nsFileTransport.cpp, line 363] 
nsThreadPoolRunnable::Run
[d:\builds\seamonkey\mozilla\xpcom\threads\nsThread.cpp, line 694] 
nsThread::Main [d:\builds\seamonkey\mozilla\xpcom\threads\nsThread.cpp, line 96] 
_PR_NativeRunThread [pruthr.c, line 421] 
0x006f006d 
KERNEL32.DLL + 0x37cd (0x77e837cd) 
the crashes happen by loading http://wam.umd.edu/~soref/ostress.html ~2k [well 
mozilla "finish"es but doesn't render it all], and then clicking reload.
Just a guess from the stack trace ->networking:cache
Assignee: pnunn → neeti
Component: ImageLib → Networking: Cache
QA Contact: paw → tever
-->gagan
Assignee: neeti → gagan
9/13-12: on my large testcase: <1100mb vm needed which is only 23% worse 
than nc4.75/ie5.5 (that's a 200+mb savings)
24772/31704K (4mb saved), we still don't load all images on mstress.

Gagan, if you can't reproduce the crash, kick this back to imagelib or layout 
or compositor; I can't.
this is really an overall memory usage issue. ->warren.
Assignee: gagan → warren
taking
Assignee: warsome → waterson
Changing component.  It doesn't look cache specific.
Component: Networking: Cache → Browser-General
libpr0n fixes this, right pav?
Assignee: waterson → pavlov
Lib pr0n makes this much wrose :-( i was at ~1.95GB of ram for the large 
testcase [119% worse than competition, and browser doesn't actually respond] 
this test run was done in a w2k terminal session w/ [peak mem usage 2510864, 
current 550456 - after killing mozilla] @256 colors.
Component: Browser-General → ImageLib
There is definitely something wrong here, even at 24bit color we shouldn't be
eating that much memory.
removing 'crash' keyword.  This is an out of memory problem potentially 
resulting in a crash, but not a crash due to anything specific.
Keywords: crash
Target Milestone: --- → mozilla1.0
Is the huge memory uptake and long load time (I haven't let it finish yet on my
machine before killing mozilla) on
http://www.12see.de/hpm_design01.cgi?user=funnylady (Select the "Gif's Gif's
Gif's" link) this bug?

Loads quickly and takes only 11MB in Opera 5.0 for Linux, takes 100% CPU usage
and over 70MB in Mozilla 2001060408-trunk
I am using Mozilla Build ID: 2001060703 on NT4.0. The start up memory image of
this build is 17346KB while that of IE6.0 (the beta release) is only 6000 and
odd KB. I had mozilla on last night and when I came into to work this morning, I
found that it had used up 76MB of memory space! Now that is an astronomical
figure, in contrast, IE rarely takes more than 20MB. Someone needs to look at
the way mozilla handles memory and why it isn't flushing unused memory blocks.
Windows 2000
P3 500
128 MB


IE 5.5

mem: 82 M
VM:  866 M


IE 6

mem: 35 M
VM:  1,317 M


Mozilla 2001062708 (0.9.2)

mem: 53 M
VM:  1,381 M


IE6 and Mozilla are very close...

Anyone have XP?

NS 4.7something on win2k:

mem: 37 M
VM:  798 M

http://wam.umd.edu/~soref/mstress.html

Testing build 2001070321 (gnucc-295) on Linux 2.4.x gives:
  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
24055 kmaraas    9   0 53624  49M  9392 S     0,0 26,4   2:14 mozilla-bin

Which amounts to ~40 MB?
Fizilla build from 0_9_2 branch under Mac OS X 10.0.4: 1,890,164 kbytes

IE 5.whatevercomeswith Mac OS X 10.0.3               : 1,806,004 kbytes

http://wam.umd.edu/~soref/mstress.html
 on Windows 2000, 16 bit color, RAM = 256MB

NN4.7       10M
IE6 (beta)  25M
M 0.9.2     44M

It's still the problem.
I agree.  Moz 0.9.2+ freezes up for many seconds after the browser dispenses
with one page and fills with another.  It's really annoying.  IE 5.0 doesn't do
this.  I prefer open source.
i am using build 2001082608 on WinNT and tried to load the pages mentioned

http://wam.umd.edu/~soref/stress.html
http://wam.umd.edu/~soref/mstress.html

there was nothing displayed whatsoever, this was even if i tried to reload the
said pages
didn't work for me either the first time, but the second attempt in a new window
worked. Odd

qa -> tpreston
QA Contact: tever → tpreston
*** Bug 98930 has been marked as a duplicate of this bug. ***
Additionally, if you use many browser windows (control-N) in IE, memory usage
with IE (any version 5 or above, I'm using 6) is much lower than Mozilla
(currently using 0.9.5). 

In fact, just browsing with about 6 Moz windows open uses up all of my 192M
within about 1/2 hour to 1 hour (W2K) while I can browse all day with IE and
have no problems.

Sorry about the lack of a test case, but this is really obvious if you do it for
a while.
Keywords: mozilla1.0
bug 104999 (that has just been fixed) should help with this abit.
Depends on: 95810, 98835, 110048
No longer depends on: 95810, 98835, 110048
Blocks: 124608
46% ? i would say 900% ! in bugzilla, request ALL unconfirmed bugs ... ie and ns
both use like 40 megs, mozilla 0.9.8 requires 400! megs. thats just hillarious
for 0.9.8:

as you know windows 2000 will swap mozilla out overnight leading to extremely 
slow response each morning. today I was watching the task manager display
as mozilla was being paged in.

it showed VM size for mozilla.exe at 66MB. mozilla was totally unresponsive
(no screen repaint, no nothing) until the Memory Usage field got up to 45MB.
As I loaded the bugzilla.mozilla.org it got to 55MB.

This seems to show that mozilla has a huge working set (perhaps the heap gets
fragmented over time).

Then after checking some mails, moving them aroung, doing a few bugzilla queries
to find this bug it shows both Memory at 118MB and VM at 121MB

The machine has 512 MB RAM and shows 256MB used in task manager.
sorry guys, you want some other bug, this is an *IMAGELIB* bug
Summary: memory usage is ~46% worse than nc4.75/ie5.5 → memory usage for images is ~46% worse than nc4.75/ie5.5
Target Milestone: mozilla1.0 → mozilla1.1
Keywords: nsbeta1
dp, cathleen: can you guys help/comment on this one? [adt needinfo]
Whiteboard: [adt needinfo]
We need a new valid test case for this bug!  Otherwise, this bug is useless.

these links do not work
http://wam.umd.edu/~soref/mstress.html
http://wam.umd.edu/~soref/ostress.html 

comment #30
http://bugzilla.mozilla.org/buglist.cgi?bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&email1=&emailtype1=substring&emailassigned_to1=1&email2=&emailtype2=substring&emailreporter2=1&bugidtype=include&bug_id=&changedin=&votes=&chfieldfrom=&chfieldto=Now&chfieldvalue=&short_desc=&short_desc_type=allwordssubstr&long_desc=&long_desc_type=allwordssubstr&bug_file_loc=&bug_file_loc_type=allwordssubstr&status_whiteboard=&status_whiteboard_type=allwordssubstr&keywords=&keywords_type=anywords&field0-0-0=noop&type0-0-0=noop&value0-0-0=&cmdtype=doit&order=Reuse+same+sort+as+last+time
is not an imagelib issue, this is a query generating a bugzilla list of 23461
bugs, which took a huge load of memory.  There is already an bug filed, see bug
49539, which has nothing to do with image.

The URL link provided on top of bug report
http://timeless.haque.net/mozilla/stress.html works.  However, nothing gets
rendered after page gets done loading.  This is same with IE.  So, I saved the
page, and look at all image links in page source, and found there is no valid
access to those image files.  I will attatch stress.html for people to view.
no access to any of the images specified in this page.	--> test invalid.
aww that sucks. ok, i think i found someone with a webserver who can host the 
files. i should have more info in 24hrs
ok, the files are now up on the server in the url, if you can mirror them instead of constantly hitting the server, that'd probably be good.

index.html mstress.html ostress.html slurp-stress.html slurp.html stress.html

you should be able to use wget or netscape composer to slurp the files.
URL: http://timeless.haque.net/mozilla/str... → http://www2.imperialnetwork.org:8080/...
another HTML showing the problem (using Yahoo images)

Mozilla takes twice of IE's footprint to display this HTML file

BTW JPEG images seem to be worse than GIF images
I did a spacetrace analysis using attachment 72809 [details] and it showed 100 calls to
imglib, which holds around 120MB.

task manager showed us using 120+MB to load this page and IE5 took 42+MB to load.

saari is going to take a look at this.  he guessed implementing paletted gifs
will help, 3 to 1 ratio.
Those images are PNGs and the ones on the old testcase were JPEGs.  GIF fixes
are irrelivant to this.
Palettized image storage in general then. I'm guessing PNG uses a palettized
format anyway, and if you look at the content of the images, you can see where
IE is getting the win; it is an obvious 3:1 ratio.

We can teach each decoder to go directly to native paletized formats when
appropriate (my choice), or we can take a temporary decode to full 3 byte per
pixel, keep color statistics, and then make a decision on if it should be
palettized (< 256 colors). My vote is the first option; more work but better
results. I'm willing to punt on JPEGs as they're meant to be used when GIF isn't
the best choice: large amounts of colors often seen in photographs where LZW
starts falling down anyway.
If GIF-s are not relevant to this, should a new bug to filed for the GIF problem?
Keywords: mozilla1.0 → mozilla1.3, testcase
*** Bug 116257 has been marked as a duplicate of this bug. ***
Bug 143046 is the windows only (currently) GIF solution to this bug.  1 bit/8
bit PNGs should also use the framework that Bug 143046 is trying to set up.
Depends on: slowGIF
*** Bug 122581 has been marked as a duplicate of this bug. ***
Keywords: mozilla1.3
Target Milestone: mozilla1.1alpha → ---
(from comment 16 of bug 122581:) Page with 141 1280x960 images, each shrunk to
160x120 using WIDTH and HEIGHT attributes in IMG tag, requires 1GB RAM
(allocated to Mozilla while decoding, X afterwards).  Netscape 4.05 requires
only 50MB (Netscape+X11 combined) for the same page, and loads 2-3 times faster
as well.  Shouldn't Mozilla only keep the shrunk image in memory, not the full one?
adt: nsbeta1-
Keywords: nsbeta1 → nsbeta1-
Can someone please update this bug with more example web pages?  Thank you much.
Please slurp only once. I will try to retest stress locally, but i currently
have a mozilla which crashed due to OOM after using about 700mb of vm, so I
don't think mozilla has a chance of loading this page.
URL: http://www2.imperialnetwork.org:8080/... → http://217.160.130.21/~timeless/stres...
I set up a small (~250kB including images) test page at
http://achurch.org/51028/ .  NC4.78 doesn't even break a sweat; Mozilla chews
through memory like crazy without displaying anywhere near all of the images
(and dies from OOM on both a 512MB Windows box and a 1.5GB Linux box, both
Mozilla 1.6a).
Using Mozilla 1.4 (20030821) on Linux I tried the site mentioned in comment #50.
I had a window open showing top (sorted according to mem usage) and noticed it
wasn't mozilla-bin eating my memory but X. 
Since Netscape doesn't cause X to take up such huge amounts of memory, I'd argue
that Mozilla is at fault.
This affects Windows to, and on Windows it does bloat Mozilla process memory, so
I'd agree Mozilla is at fault.
andrew church: could you get a stack trace w/ gdb? :)
I will try to fix any oom crashes for which you provide a stack.
You're welcome to file such stack traces in new bugs and reference them here,
since we probably shouldn't clutter this bug with them (although I did cause
some clutter in comments 2 and 3).
OS: Windows 2000 → All
Hardware: PC → All
Unfortunately, it's killed by the OOM killer, so I can't get any stack traces.
Blocks: 215491
Any progress on that?
I think the URL example results in a "memory cache overflow". Each image is
stored decoded in the memory cache even if the memory cache is already full.
While loading the page about:cache increases and "Storage in use" will exceed
"Maximum storage size". 

I have seen this also in bug 213391 comment 5. If bug 213391 is a dupe of this
then other bugs with this memory cache behaviour (like bug 105370) could also be
dupes of this bug.
The account from timeless had moved to our new server so I changed the URL.
URL: http://217.160.130.21/~timeless/stres... → http://itkombinat.de/~timeless/stress...
I affected by the problem also, and IE don't suffer from this, may be we should
put the image in disk rather than at Memory??
No longer blocks: 215491
PNG Image as Testcase that shows huge memory usage (over 400 MB). Memory Usage
of IE with same PNG is far less.
Not really sure this is the right Bug, there are a lot Bugs about images and
huge memory usage: is Bug 240369 only about .bmp?
Tested with: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8b4)
Gecko/20050831 Firefox/1.0+ ID:2005083106
even worse for me, moz 1.8b on win2k, while rendering memory usage goes over 1.1
gig, then calms down to 550megs once image is shown. stalls the system for a
good minute tho...
This bug has been quiet for a while, with no apparent solution in sight. This analysis seems relevant:

  http://primates.ximian.com/~federico/news-2005-11.html

While I think the scheme suggested by the author is sound enough, I would also suggest memory-mapping the in-memory images to the cache, at least for non-visible/inactive tabs; in my experience it's a good idea to let the kernel dynamically distribute memory to apps.

Bug 240369 looks like a dupe of this bug, can anyone confirm?
(In reply to comment #62)
> This bug has been quiet for a while, with no apparent solution in sight. This
> analysis seems relevant:
> 
>   http://primates.ximian.com/~federico/news-2005-11.html

see bug 259672
(In reply to comment #63)
> see bug 259672

Not talking about X; the excessive memory usage problem also exists on Windows.
Assignee: pavlov → nobody
QA Contact: tpreston → imagelib
When trying to view the attachment "Another testcase", I get this error message :

The image “https://bugzilla.mozilla.org/attachment.cgi?id=194498” cannot be displayed, because it contains errors.

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3
for reference, bug 296818 has significantly improved things. someone will comment here with numbers.
Depends on: 296818
All with emptied disk cache set to 404800 running the stress test;

Page was loaded, then slowly scrolled to the bottom.

Camino Version 2007102701 (2.0a1pre)
294MB real, 656MB virtual.
Cache: Storage in use: 	130418 KiB

Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.9a9pre) Gecko/2007102804 Minefield/3.0a9pre
323MB real, 681MB virtual.
Cache: Storage in use: 	130093 KiB


For completeness:

Safari Version 2.0.4 (419.3)
625.75MB real, 1.03GB virtual.

Oh, with a beach ball of death that lasted nearly 10 minutes.
Note that the numbers may vary over time. EG, when I ran the testcase with Minefield the VM size was ~1.1GB immediately after running the test. It then dropped to ~700MB after a bit (296818's work), and then scrolling through the whole page increased it back again.
Depends on: 398983
The original testcase requires 885MB of [virtual] memory in nc4.75/ie5.5,
but now in Firefox 3.5b4pre it requires less than 450K (with some other tabs open and a lot of addons active).
So, I would say the original reported problem that that page requires more memory than nc4.75 is no longer applicable...
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
No longer depends on: 398983
Resolution: FIXED → WORKSFORME
You need to log in before you can comment on or make changes to this bug.