Open Bug 319143 Opened 16 years ago Updated 6 years ago

large plain text file (150MB) uses all available memory and poor performance until hanging

Categories

(Core :: General, defect)

x86
All
defect
Not set
critical

Tracking

()

People

(Reporter: tech, Unassigned)

References

(Depends on 1 open bug, Blocks 3 open bugs, )

Details

(Keywords: perf)

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; fr; rv:1.7.12) Gecko/20051007 Debian/1.7.12-1
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; fr; rv:1.7.12) Gecko/20051007 Debian/1.7.12-1

On every OSs I use, Mozilla uses more and more memory while using it. The system becomes slower and slower until it crashes : It's nearly impossible to use a system while mozilla's running on it.

Reproducible: Always

Steps to Reproduce:
1.start mozilla
2.use mozilla
3.reboot the crashed system

Actual Results:  
The memory is completely exhausted, so the system slows down and finally crashes

Expected Results:  
Mozilla should free the memory it doesn't use, ans mozilla shouldn't use so much memory
OS: Linux → All
Version: unspecified → 1.7 Branch
Works for me well enough on SeaMonkey trunk - and 1.7 Branch won't see any big stuff happening anyways.
Status: UNCONFIRMED → RESOLVED
Closed: 16 years ago
Resolution: --- → WORKSFORME
DOESN'T WORKSFORME
(In reply to comment #1)
> Works for me well enough on SeaMonkey trunk - and 1.7 Branch won't see any big
> stuff happening anyways.
perhaps that 1.7 Branch won't see any big stuff happening anyways.
but for the moment it's the only stable branch if it's not possible to debug the stable branch, it's no use to switch to the next

Status: RESOLVED → UNCONFIRMED
Resolution: WORKSFORME → ---
Exactly the same happens on my Slackware boxes. As far as i use Composer it runs OK. But when I start up WebBrowser it leaks. In my case system does not crash though in oder i start it up from terminal I have to close it and rerun it.
See Bug 320915, and see bugs pointed by it or bugs in dependency tree of it.
> Bug 320915 : 1.8 memory leak campaign
tech@mediaforest.net and MrEvening, can you check/understand what occurs in your environment, and find workaround?
(In reply to comment #4)
> See Bug 320915, and see bugs pointed by it or bugs in dependency tree of it.
> > Bug 320915 : 1.8 memory leak campaign
> tech@mediaforest.net and MrEvening, can you check/understand what occurs in
> your environment, and find workaround?
> 
I installed mozilla 1.8beta which is the last 1.8 branch version avaible online, it seems to work a bit better, but the leak detector located in the debug menu doesn't outputs anything, so it's not easy to locate memory leaks. 
There's another strange thing with this version : the tuneup utilities task manager indicates no more than 60Mo of memory used by mozilla, while my 512Mo of ram are completely consummed as soon as I start mozilla. at this moment, the taskmanager reports 184Mo of ram used out of 512 (59Mo used by mozilla) but my memory manager announces only 35 Mo of free ram. And I already know that it will soon reach 0 free mem and will crash.
I don't really know how to find a single workaround,because as I explained, I get the same symptoms on windows 98 , windows xp and linux mandrake and debian...
How long does it take to use up the memory?

The free memory will always be 0 because Linux & Windows use unused memory for cache.  In any case, if Mozilla/SeaMonkey use up all your memory, it shouldn't mean that your system crashes or you have to reboot.  You should be able to close Mozilla/SeaMonkey and your OS should reclaim the memory.  If it doesn't or can't, that's a bug in the OS.
I've found a very good example for this problem : 
just open two tabs and try to see ftp://ftp.debian.org/debian/dists/stable/Contents-i386.gz in the first and ftp://ftp.debian.org/debian/dists/unstable/Contents-i386.gz in the second
and wait for your system to hang up.   
I can confirm this in version 1.5.0.3, mem usage on win2000 is increasing by 10Mb every few minutes (40Mb -> 70Mb and increasing). I'd say this is a critical bug, firefox is unusable in this state. I’m running java 1.5 runtime update 6 (if you can’t reproduce I’d install the jre if you haven’t already).
I am using the current (version 2.0.0.2) and still having a similar problem.  System does not crash but in normal use, the Firefox application continuously grows.  Most of my activity night and day involves use of Firefox and I am running XP-PRO.  I will admit I do not shut down Firefox (or my system) until I am forced to so it can be running for days (if you can actually believe Windows can stay up that long...) but eventually Firefox has swelled to a size that the system becomes very slow.  At that point I simply shut down Firefox (not the OS), wait a minute for the system to quiet down and then relaunch Firefox and everything is cool again.  OS appears to be doing it's job correctly but Firefox is not releasing memory in all cases upon task completion.
(In reply to comments #9)
> I am using the current (version 2.0.0.2)... 

(In reply to comments #8)
> I can confirm this in version 1.5.0.3... 

There seems to be a mistake : this bug report concerns "Mozilla Application Suite" which is now Seamonkey. the last Seamonkey release is the 1.1.2.
I'm afraid you're mentionning Firefox release numbers so your remarks might not be useful here ;)

For Wayne Mery : there's now a long time, I have no more this kind of critical memory problems with Seamonkey ;)

bz, Per my crude analysis below, the time to load the page quite strange.

bug 216418 is not related here per your comment in bug 367116 (I guess I jumped the gun there) :(  so we won't go there.  Is bug 367177 relevant?

Bug 210943 is the oldest "plain text" bug I could find.  bug 252694 and bug 78911 are probably not related, but I don't understand this dom+layout territory stuff. and bug 252694 is about tables despite the summary (and is also missing a testcase)


(In reply to comment #7)
> I've found a very good example for this problem : 
> just open two tabs and try to see
> ftp://ftp.debian.org/debian/dists/stable/Contents-i386.gz in the first and

afaict this file is plain text - NO valid or partial html - 150MB file with 1,976,664 lines of text.  it's just a unix directory list with lots of slashes and removing them out doesn't seem to matter

TESTS:
* IE - hangs (shows "not responding") but not as memory hungry as as seamonkey.  Used about 250MB
* SM trunk** - hangs, killed it after an hour. UI was toast, it used 600+MB and still wasn't done, and windows was behaving badly.  No crash btw
* tested top 19K lines (1% of the original file) == 1.3MB (also about 1%) and it loads like a rocket. 

Other samplings

<1 sec for first 19k lines
13 sec for first 190k lines (OK, 10x time for 10x size)
45-50 sec for first 380k lines (hmm, 3x time for 2x size) (several sections of the file sampled the same)
110 sec for first 760k lines (OK, approx 2x time for 2x size)
65 sec for second 760k
110sec for third 760k

but, the full file chugs on for an hour when it should be on the order of 5 minutes.  (garbage collection?)


**Build identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a6pre) Gecko/2007060809 SeaMonkey/2.0a1pre

=-=-=-

Geoff, Jim, if you did not use comment 7 for your testing or similar type files then your problem is something else and not this bug.

All, please avoid making comments that do not include or cite a specific URL or testcase. This is a critical information.
Summary: Mozilla uses all avable memory until crashing the system → large plain text file (150MB) uses all available memory and poor performance until hanging
part 2

there is a clear performance degradation of the full file when you look at memory usage over time.  test done on an idle system just after reboot. memory deltas for the first 10 minutes ...
190k
60k
30k
7k
6k
6k
12k
15k
12k
11k

at 10 minutes seamonkey has rendered only 62% of the file - it's gotten to line 1,229,772 of the 1,976,664 lines
Blocks: 384260
This seems like a bigger version of bug 367116.  Given the memory numbers there, I'd expect we start swapping pretty early in this testcase...

I have no idea what the numbers in comment 12 mean, but I seriously doubt whatever is being reported is in fact measured in kilo-anything.  ;)

The profile shows:

Total hit count: 287996
Count %Total  Function Name
30191   10.5     IsWordBoundary(unsigned short)
23322   8.1     gfxTextRunWordCache::MakeTextRun(unsigned char const*, unsigned int, gfxFontGroup*, gfxTextRunFactory::Parameters const*, unsigned int, int*)
22421   7.8     gfxTextRun::CopyGlyphDataFrom(gfxTextRun*, unsigned int, unsigned int, unsigned int, int)
21354   7.4     gfxTextRun::gfxTextRun(gfxTextRunFactory::Parameters const*, void const*, unsigned int, gfxFontGroup*, unsigned int)
19426   6.7     nsTextFrameUtils::TransformText(unsigned char const*, unsigned int, unsigned char*, int, unsigned char*, gfxSkipCharsBuilder*, unsigned int*)
17852   6.2     gfxTextRunWordCache::CacheHashEntry::KeyEquals(gfxTextRunWordCache::CacheHashKey const*) const

Looking top-down it looks pretty much the same as bug 367116.
Depends on: 367116
(In reply to comment #13)
> This seems like a bigger version of bug 367116.  Given the memory numbers
> there, I'd expect we start swapping pretty early in this testcase...
> 
> I have no idea what the numbers in comment 12 mean, but I seriously doubt
> whatever is being reported is in fact measured in kilo-anything.  ;)

quite right - should be MB increase for a 1 minute period as reported by task manager memory usage column

190 MB - first minute
 60 MB - second minute
 30 MB - 3rd minute 
  7 MB - 4th minute   
  6 MB - 5th minute 
  6 MB - 6th minute 
 12 MB - 7th minute 
 15 MB - 8th minute 
 12 MB - 9th minute 
 11 MB - 10th minute 
30 second intervals, *additional* lines displayed (thousands):

301
182
145
130
129
102
55
35  

i.e. going into the 5th minute, performance is decreased ~90% compared to first 30 seconds

Status: UNCONFIRMED → NEW
Ever confirmed: true
Keywords: perf
> i.e. going into the 5th minute, performance is decreased ~90% compared to first
> 30 seconds

Well, the perf was O(N^1.5) up until the last two data points (not too bad) and then was ~O(N^3).  Perhaps SeaMonkey exceeded the amount of RAM Windows was willing to give it.
(In reply to comment #16)
> > i.e. going into the 5th minute, performance is decreased ~90% compared to first
> > 30 seconds
> 
> Well, the perf was O(N^1.5) up until the last two data points (not too bad) and
> then was ~O(N^3).  Perhaps SeaMonkey exceeded the amount of RAM Windows was
> willing to give it.

I don't know windows to comment from an internals perspective, but today's test results parallels yesterday's and those tests were done after rebooting on an idle system with plenty of memory, virtual and real.  

hopefully roc's description in bug 367116 matches up with these degradation symptoms.
Assignee: general → nobody
Product: Mozilla Application Suite → Core
QA Contact: general → general
Version: 1.7 Branch → unspecified
Whiteboard: [MemShrink]
I'll pull a profile on this
File is 226MB unpacked now, 282K lines

After opening it (and FF got very unresponsive in the process, took a few minutes), mem usage with a few other tabs open was:

2,444,542,156 B (100.0%) -- explicit
├──1,806,366,487 B (73.89%) -- heap-unclassified
├────545,208,560 B (22.30%) -- layout
│    ├──530,203,904 B (21.69%) -- shell(file:///tmp/Contents-i386)
│    │  ├──530,120,064 B (21.69%) -- arenas
│    │  └───────83,840 B (00.00%) -- styledata

...

      173,940 B -- gfx-surface-image
      181,360 B -- gfx-surface-xlib
2,391,121,436 B -- heap-allocated
    2,023,424 B -- heap-dirty
   24,788,386 B -- heap-unallocated
              2 -- js-compartments-system
              6 -- js-compartments-user
   42,991,616 B -- js-gc-heap
    2,211,016 B -- js-gc-heap-arena-unused
    1,048,576 B -- js-gc-heap-chunk-clean-unused
   23,982,272 B -- js-gc-heap-chunk-dirty-unused
         63.36% -- js-gc-heap-unused-fraction
            649 -- page-faults-hard
      3,658,595 -- page-faults-soft
2,369,794,048 B -- resident
3,381,243,904 B -- vsize

The O(n^3) slowdown reported before was probably the process running short of VM and/or RAM (mine is a Fedora 15 x64 system with 16GB of ram).  Also interesting is that the tab has no scroll bar, at all.  (Overflow?)
If I leave the tab focused the scroll bars stay and work.

Profile shows nothing shocking; almost all the time is used in ReflowText() (~67%) and associated routines.  If anyone wants I'll upload it.

FYI after navigating away, etc mem usage (RSS) dropped to 200MB.
Whiteboard: [MemShrink] → [MemShrink:P2]
I just tried the file in comment 11.  It took a while to load and once it did I could only see the first page -- I wasn't able to scroll down.  Not sure about that.  Anyway, about:memory said:

1,268,566,087 B (100.0%) -- explicit
├──1,063,868,248 B (83.86%) -- window-objects
│  ├──1,061,424,232 B (83.67%) -- top(file:///Users/njn/Contents-i386, id=7)
│  │  ├──1,061,135,136 B (83.65%) -- active
│  │  │  └──1,061,135,136 B (83.65%) -- window(file:///Users/njn/Contents-i386)
│  │  │     ├────547,409,440 B (43.15%) -- layout
│  │  │     │    ├──547,260,000 B (43.14%) ── arenas
│  │  │     │    └──────149,440 B (00.01%) ── style-sets
│  │  │     └────513,725,696 B (40.50%) ── dom [2]

"heap-unclassified" was only 13%.
Removing the MemShrink tag because it blocks tracking bug 689769 which also has the MemShrink tag.
Whiteboard: [MemShrink:P2]
Blocks: 210943
You need to log in before you can comment on or make changes to this bug.