210985 - Mail & News not closing summary files for newsgroups

Status: RESOLVED FIXED

(MailNews Core :: Database, defect)

Description

[Roope Lehmuslehto]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.4) Gecko/20030624
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.4) Gecko/20030624

Mozilla loads Usenet group indexfiles (.msf) to memory, when browsing group.
Naturally, memory load increases amount of how large index is. Unfortunately
Mozilla 'forgets' to unload group indexfile when moving to next group, that
results huge memory consumption especially when reading large groups.

In worst case one group indexfile takes ~10MB of diskspace resulting 100MB
memory usage after reading 10 groups. Memory usage returns to normal level only
after *ALL* mozilla windows have been closed. 

As you can see in screenshot, mozilla takes ~120MB of memory and ~120MB of VM
after ~10 groups, none of them are binarygroups. This bug leads to severe
overall system performance slowdown even on high-end machines, and could crash
low-end systems.

Reproducible: Always

Steps to Reproduce:




This bug is new to Mozilla 1.4 RC2.

Comment 1

[David :Bienvenu]

actually, this isn't a memory leak. It's bloat, since we do free the memory on
shutdown. (which is not to diminish the seriousness of this). Do you have
Mozilla configured to check news groups for new messages?

Comment 2

[David :Bienvenu]

*** Bug 214324 has been marked as a duplicate of this bug. ***

Comment 3

[David :Bienvenu]

I think we're leaking the downloaded headers. I can't reproduce this problem in
newsgroups that I'm up to date in, but when I click on a newsgroup that has to
download headers, I can reproduce this.

Comment 4

[Felix Miata]

Mozilla/5.0 (OS/2; U; Warp 4.5; en-US; rv:1.4) Gecko/20030624

Not sure how to tell so precisely how much RAM I lose to newsgroups, but the two
groups I frequent are netscape.public.mozilla.os2, 6 MB .msf, and
netscape.public.mozilla.wishlist, 10 MB .msf. I just shut Mozilla down after 98
hours of uptime, and regained 170 MB of RAM. In the past four days, I probably
opened and closed mailnews 15-20 times, going to the smaller group each time,
and the larger maybe three times. After restarting browser and CZ, I had 130 MB
free. After starting mailnews and opening npmw, free was down to 100 MB. After
opening npmo, free dropped another 10 MB. Several minutes after closing
mailnews, no free RAM has been recovered. A mere 30 minutes after opening
Mozilla, it's using 99 MB for one CZ window with 6 tabs and one browser window
with 14 tabs.

OS -> all

Comment 5

[k b]

i can confirm this memory eating problem.
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.5b) Gecko/20030720
RH9, kernel 2.4.20.

after having the browser run for about 5 days, from time to time having up to 4
windows with around 10 tabs in each. (pages plain htlm, no plugins etc).
even though i closed tabs and windows, memory usage did not go down conciderably.
memory usage slowley but stady rose to 130MB, then i shut mozilla down since on
my poor computer with only 390MB RAM, i hardly can allow applicatoins to eat 1/3
of the memory resources. (memory useage max was 160MB)

now after running 12 hours it uses 60MB.

Comment 6

[David :Bienvenu]

Attachment: proposed fix

this was caused by a ref-counting problem, in particular, this part of the
diff:

@@ -3569,7 +3549,7 @@
	  else
		  rv = CreateMsgHdr(hdrRow, key, &msgHdr);
   }
-  NS_IF_ADDREF(*aHdr = msgHdr);
+  *aHdr = msgHdr; // already addreffed above.
   return NS_OK; // it's not an error not to find a msg hdr.
 }

the rest is formatting cleanup.

Comment 7

[David :Bienvenu]

fix checked in, r/sr=mscott.

However, I believe that the bloat caused by opening newsgroups is a "one-time"
bloat. Once you've paid the cost of opening the newsgroup, you don't pay it
again in the same session, no matter how many times you open the newsgroup (or
mail folder). So I suspect you're seeing other bloat. But please try with
today's build and let us know.

Comment 8

[Felix Miata]

Checked into the 1.4 branch?

Comment 9

[David :Bienvenu]

no, checked into trunk. But I'm trying to nominate for 1.4.1

Comment 10

[Mike Kaply [:mkaply]]

Can you create a 1.4 patch that only includes the fix please?

Thanks

Comment 11

[David :Bienvenu]

I'm pulling a 1.4 tree, but as I said before, this is what the patch would look
like:

@@ -3569,7 +3549,7 @@
	  else
		  rv = CreateMsgHdr(hdrRow, key, &msgHdr);
   }
-  NS_IF_ADDREF(*aHdr = msgHdr);
+  *aHdr = msgHdr; // already addreffed above.
   return NS_OK; // it's not an error not to find a msg hdr.
 }

Comment 12

[Mike Kaply [:mkaply]]

Sorry, missed that comment

a=mkaply checkin of the simple fix to 1.4.

Comment 13

[David :Bienvenu]

np, thx for the a=, fix checked onto branch.

This regressed 08/07/2002, so almost a year ago.

Comment 14

[Felix Miata]

Mozilla/5.0 (OS/2; U; Warp 4.5; en-US; rv:1.4) Gecko/20030731

free|after action
185|none
127|open CZ/browser
123|open mailnews
100|open group netscape.public.mozilla.wishlist (.msf=10MB)
87|open group netscape.public.mozilla.os2 (.msf=6MB)
87|close mailnews

Total Mozilla consumption: 98MB

I don't see any improvement. Did this actually land on the branch yesterday?