319143 - large plain text file (150MB) uses all available memory and poor performance until hanging

Status: RESOLVED INCOMPLETE

(Core :: General, defect)

Description

[tech]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; fr; rv:1.7.12) Gecko/20051007 Debian/1.7.12-1
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; fr; rv:1.7.12) Gecko/20051007 Debian/1.7.12-1

On every OSs I use, Mozilla uses more and more memory while using it. The system becomes slower and slower until it crashes : It's nearly impossible to use a system while mozilla's running on it.

Reproducible: Always

Steps to Reproduce:
1.start mozilla
2.use mozilla
3.reboot the crashed system

Actual Results:  
The memory is completely exhausted, so the system slows down and finally crashes

Expected Results:  
Mozilla should free the memory it doesn't use, ans mozilla shouldn't use so much memory

Comment 1

[Robert Kaiser]

Works for me well enough on SeaMonkey trunk - and 1.7 Branch won't see any big stuff happening anyways.

Comment 2

[tech]

DOESN'T WORKSFORME
(In reply to comment #1)
> Works for me well enough on SeaMonkey trunk - and 1.7 Branch won't see any big
> stuff happening anyways.
perhaps that 1.7 Branch won't see any big stuff happening anyways.
but for the moment it's the only stable branch if it's not possible to debug the stable branch, it's no use to switch to the next

Comment 3

[MrEvening]

Exactly the same happens on my Slackware boxes. As far as i use Composer it runs OK. But when I start up WebBrowser it leaks. In my case system does not crash though in oder i start it up from terminal I have to close it and rerun it.

Comment 4

[WADA:World Anti-bad-Duping Agency]

See Bug 320915, and see bugs pointed by it or bugs in dependency tree of it.
> Bug 320915 : 1.8 memory leak campaign
[e-mail] and MrEvening, can you check/understand what occurs in your environment, and find workaround?

Comment 5

[tech]

(In reply to comment #4)
> See Bug 320915, and see bugs pointed by it or bugs in dependency tree of it.
> > Bug 320915 : 1.8 memory leak campaign
> [e-mail] and MrEvening, can you check/understand what occurs in
> your environment, and find workaround?
> 
I installed mozilla 1.8beta which is the last 1.8 branch version avaible online, it seems to work a bit better, but the leak detector located in the debug menu doesn't outputs anything, so it's not easy to locate memory leaks. 
There's another strange thing with this version : the tuneup utilities task manager indicates no more than 60Mo of memory used by mozilla, while my 512Mo of ram are completely consummed as soon as I start mozilla. at this moment, the taskmanager reports 184Mo of ram used out of 512 (59Mo used by mozilla) but my memory manager announces only 35 Mo of free ram. And I already know that it will soon reach 0 free mem and will crash.
I don't really know how to find a single workaround,because as I explained, I get the same symptoms on windows 98 , windows xp and linux mandrake and debian...

Comment 6

[Andrew Schultz]

How long does it take to use up the memory?

The free memory will always be 0 because Linux & Windows use unused memory for cache.  In any case, if Mozilla/SeaMonkey use up all your memory, it shouldn't mean that your system crashes or you have to reboot.  You should be able to close Mozilla/SeaMonkey and your OS should reclaim the memory.  If it doesn't or can't, that's a bug in the OS.

Comment 7

[tech]

I've found a very good example for this problem : 
just open two tabs and try to see ftp://ftp.debian.org/debian/dists/stable/Contents-i386.gz in the first and ftp://ftp.debian.org/debian/dists/unstable/Contents-i386.gz in the second
and wait for your system to hang up.   

Comment 8

[Geoff Simpson]

I can confirm this in version 1.5.0.3, mem usage on win2000 is increasing by 10Mb every few minutes (40Mb -> 70Mb and increasing). I'd say this is a critical bug, firefox is unusable in this state. I’m running java 1.5 runtime update 6 (if you can’t reproduce I’d install the jre if you haven’t already).

Comment 9

[Jim Powell]

I am using the current (version 2.0.0.2) and still having a similar problem.  System does not crash but in normal use, the Firefox application continuously grows.  Most of my activity night and day involves use of Firefox and I am running XP-PRO.  I will admit I do not shut down Firefox (or my system) until I am forced to so it can be running for days (if you can actually believe Windows can stay up that long...) but eventually Firefox has swelled to a size that the system becomes very slow.  At that point I simply shut down Firefox (not the OS), wait a minute for the system to quiet down and then relaunch Firefox and everything is cool again.  OS appears to be doing it's job correctly but Firefox is not releasing memory in all cases upon task completion.

Comment 10

[tech]

(In reply to comments #9)
> I am using the current (version 2.0.0.2)... 

(In reply to comments #8)
> I can confirm this in version 1.5.0.3... 

There seems to be a mistake : this bug report concerns "Mozilla Application Suite" which is now Seamonkey. the last Seamonkey release is the 1.1.2.
I'm afraid you're mentionning Firefox release numbers so your remarks might not be useful here ;)

For Wayne Mery : there's now a long time, I have no more this kind of critical memory problems with Seamonkey ;)

Comment 11

[Wayne Mery (:wsmwk)]

bz, Per my crude analysis below, the time to load the page quite strange.

bug 216418 is not related here per your comment in bug 367116 (I guess I jumped the gun there) :(  so we won't go there.  Is bug 367177 relevant?

Bug 210943 is the oldest "plain text" bug I could find.  bug 252694 and bug 78911 are probably not related, but I don't understand this dom+layout territory stuff. and bug 252694 is about tables despite the summary (and is also missing a testcase)


(In reply to comment #7)
> I've found a very good example for this problem : 
> just open two tabs and try to see
> ftp://ftp.debian.org/debian/dists/stable/Contents-i386.gz in the first and

afaict this file is plain text - NO valid or partial html - 150MB file with 1,976,664 lines of text.  it's just a unix directory list with lots of slashes and removing them out doesn't seem to matter

TESTS:
* IE - hangs (shows "not responding") but not as memory hungry as as seamonkey.  Used about 250MB
* SM trunk** - hangs, killed it after an hour. UI was toast, it used 600+MB and still wasn't done, and windows was behaving badly.  No crash btw
* tested top 19K lines (1% of the original file) == 1.3MB (also about 1%) and it loads like a rocket. 

Other samplings

<1 sec for first 19k lines
13 sec for first 190k lines (OK, 10x time for 10x size)
45-50 sec for first 380k lines (hmm, 3x time for 2x size) (several sections of the file sampled the same)
110 sec for first 760k lines (OK, approx 2x time for 2x size)
65 sec for second 760k
110sec for third 760k

but, the full file chugs on for an hour when it should be on the order of 5 minutes.  (garbage collection?)


**Build identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a6pre) Gecko/2007060809 SeaMonkey/2.0a1pre

=-=-=-

Geoff, Jim, if you did not use comment 7 for your testing or similar type files then your problem is something else and not this bug.

All, please avoid making comments that do not include or cite a specific URL or testcase. This is a critical information.

Comment 12

[Wayne Mery (:wsmwk)]

part 2

there is a clear performance degradation of the full file when you look at memory usage over time.  test done on an idle system just after reboot. memory deltas for the first 10 minutes ...
190k
60k
30k
7k
6k
6k
12k
15k
12k
11k

at 10 minutes seamonkey has rendered only 62% of the file - it's gotten to line 1,229,772 of the 1,976,664 lines

Comment 13

[Boris Zbarsky [:bzbarsky]]

This seems like a bigger version of bug 367116.  Given the memory numbers there, I'd expect we start swapping pretty early in this testcase...

I have no idea what the numbers in comment 12 mean, but I seriously doubt whatever is being reported is in fact measured in kilo-anything.  ;)

The profile shows:

Total hit count: 287996
Count %Total  Function Name
30191   10.5     IsWordBoundary(unsigned short)
23322   8.1     gfxTextRunWordCache::MakeTextRun(unsigned char const*, unsigned int, gfxFontGroup*, gfxTextRunFactory::Parameters const*, unsigned int, int*)
22421   7.8     gfxTextRun::CopyGlyphDataFrom(gfxTextRun*, unsigned int, unsigned int, unsigned int, int)
21354   7.4     gfxTextRun::gfxTextRun(gfxTextRunFactory::Parameters const*, void const*, unsigned int, gfxFontGroup*, unsigned int)
19426   6.7     nsTextFrameUtils::TransformText(unsigned char const*, unsigned int, unsigned char*, int, unsigned char*, gfxSkipCharsBuilder*, unsigned int*)
17852   6.2     gfxTextRunWordCache::CacheHashEntry::KeyEquals(gfxTextRunWordCache::CacheHashKey const*) const

Looking top-down it looks pretty much the same as bug 367116.

Comment 14

[Wayne Mery (:wsmwk)]

(In reply to comment #13)
> This seems like a bigger version of bug 367116.  Given the memory numbers
> there, I'd expect we start swapping pretty early in this testcase...
> 
> I have no idea what the numbers in comment 12 mean, but I seriously doubt
> whatever is being reported is in fact measured in kilo-anything.  ;)

quite right - should be MB increase for a 1 minute period as reported by task manager memory usage column

190 MB - first minute
 60 MB - second minute
 30 MB - 3rd minute 
  7 MB - 4th minute   
  6 MB - 5th minute 
  6 MB - 6th minute 
 12 MB - 7th minute 
 15 MB - 8th minute 
 12 MB - 9th minute 
 11 MB - 10th minute 

Comment 15

[Wayne Mery (:wsmwk)]

30 second intervals, *additional* lines displayed (thousands):

301
182
145
130
129
102
55
35  

i.e. going into the 5th minute, performance is decreased ~90% compared to first 30 seconds

Comment 16

[Andrew Schultz]

> i.e. going into the 5th minute, performance is decreased ~90% compared to first
> 30 seconds

Well, the perf was O(N^1.5) up until the last two data points (not too bad) and then was ~O(N^3).  Perhaps SeaMonkey exceeded the amount of RAM Windows was willing to give it.

Comment 17

[Wayne Mery (:wsmwk)]

(In reply to comment #16)
> > i.e. going into the 5th minute, performance is decreased ~90% compared to first
> > 30 seconds
> 
> Well, the perf was O(N^1.5) up until the last two data points (not too bad) and
> then was ~O(N^3).  Perhaps SeaMonkey exceeded the amount of RAM Windows was
> willing to give it.

I don't know windows to comment from an internals perspective, but today's test results parallels yesterday's and those tests were done after rebooting on an idle system with plenty of memory, virtual and real.  

hopefully roc's description in bug 367116 matches up with these degradation symptoms.

Comment 18

[Randell Jesup [:jesup] (needinfo me)]

I'll pull a profile on this

Comment 19

[Randell Jesup [:jesup] (needinfo me)]

File is 226MB unpacked now, 282K lines

After opening it (and FF got very unresponsive in the process, took a few minutes), mem usage with a few other tabs open was:

2,444,542,156 B (100.0%) -- explicit
├──1,806,366,487 B (73.89%) -- heap-unclassified
├────545,208,560 B (22.30%) -- layout
│    ├──530,203,904 B (21.69%) -- shell(file:///tmp/Contents-i386)
│    │  ├──530,120,064 B (21.69%) -- arenas
│    │  └───────83,840 B (00.00%) -- styledata

...

      173,940 B -- gfx-surface-image
      181,360 B -- gfx-surface-xlib
2,391,121,436 B -- heap-allocated
    2,023,424 B -- heap-dirty
   24,788,386 B -- heap-unallocated
              2 -- js-compartments-system
              6 -- js-compartments-user
   42,991,616 B -- js-gc-heap
    2,211,016 B -- js-gc-heap-arena-unused
    1,048,576 B -- js-gc-heap-chunk-clean-unused
   23,982,272 B -- js-gc-heap-chunk-dirty-unused
         63.36% -- js-gc-heap-unused-fraction
            649 -- page-faults-hard
      3,658,595 -- page-faults-soft
2,369,794,048 B -- resident
3,381,243,904 B -- vsize

The O(n^3) slowdown reported before was probably the process running short of VM and/or RAM (mine is a Fedora 15 x64 system with 16GB of ram).  Also interesting is that the tab has no scroll bar, at all.  (Overflow?)

Comment 20

[Randell Jesup [:jesup] (needinfo me)]

If I leave the tab focused the scroll bars stay and work.

Profile shows nothing shocking; almost all the time is used in ReflowText() (~67%) and associated routines.  If anyone wants I'll upload it.

FYI after navigating away, etc mem usage (RSS) dropped to 200MB.

Comment 21

[Nicholas Nethercote [inactive]]

I just tried the file in comment 11.  It took a while to load and once it did I could only see the first page -- I wasn't able to scroll down.  Not sure about that.  Anyway, about:memory said:

1,268,566,087 B (100.0%) -- explicit
├──1,063,868,248 B (83.86%) -- window-objects
│  ├──1,061,424,232 B (83.67%) -- top(file:///Users/njn/Contents-i386, id=7)
│  │  ├──1,061,135,136 B (83.65%) -- active
│  │  │  └──1,061,135,136 B (83.65%) -- window(file:///Users/njn/Contents-i386)
│  │  │     ├────547,409,440 B (43.15%) -- layout
│  │  │     │    ├──547,260,000 B (43.14%) ── arenas
│  │  │     │    └──────149,440 B (00.01%) ── style-sets
│  │  │     └────513,725,696 B (40.50%) ── dom [2]

"heap-unclassified" was only 13%.

Comment 22

[Nicholas Nethercote [inactive]]

Removing the MemShrink tag because it blocks tracking bug 689769 which also has the MemShrink tag.

Comment 23

[BugBot [:suhaib / :marco/ :calixte]]

In the process of migrating remaining bugs to the new severity system, the severity for this bug cannot be automatically determined. Please retriage this bug using the new severity system.

Comment 24

[Andrew McCreight [:mccr8]]

IIRC, the memory usage of layout of large text files has been improved in the last decade, so let's just close this for now.