185634 - Mozilla News consumes unbearable amounts of memory in big newsgroups.

Status: NEW

(MailNews Core :: Networking: NNTP, defect)

Description

[mrosen]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.3a) Gecko/20021212
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.3a) Gecko/20021212

Open a newsgroup with *lots* of messages (in the 10thousands), and let Mozilla
download all headers (not only 500). Look at taskmanager, ans watch Mozilla
consuming lots of memory until it finally loaded all messages. If the newsgroup
was downloaded previously, just watch the memory footprint of Mozilla skyrocket
when opening the newsgroup. It seems to load the complete .msf file into memory.
Now odo the very same with NW4.X. You'll notice that Netscape hardly needs *any*
additional memory, regardless how much headers the newsgroup holds.

Reproducible: Always

Steps to Reproduce:
1. Create a new account on news://support-forums.novell.com
2. Subscribe to the group novell.community.chat. Currently, this group holds
300.000 messages
3. Let Mozilla download all headers in this group. Go on a long vacation, don't
expect yor PC to ever come back from this task. It'll start trashing memory and
finally hang at around 70.000 downloaded headers (depending on the memory
installed in your workstation, 70.000 was the max I could achieve with 256MB). 

Actual Results:  
Mozilla will use virtually infinite amounts of memory, and finally crash and
burn a W2K PC with 512MB of physical memory and no other apps running. This is
at least 25 times the memory footprint of NS4.8, without ever completing the
task succesfully, and without any additional functionality Mozilla offers over
Netscape in News.

Expected Results:  
The same as Netscape 4.8. NS4.X opens, downloads and displays above example
group without ever using more than 20MB of memory.

Comment 1

[Andrew Hagen]

Maybe related to bug 145441.

Comment 2

[mrosen]

I'm not convinced that this is related, it sounds like a different issue to me. 
But bug 191491 (which was marked - IMHO incorrectly - as duplicate of bug 
145441), definitely is a duplicate of this one. The symptoms match exactly.

Comment 3

[Mark Stier]

This 'feature' is really boring. Additionally it seems to have the effect that
loading the message list of huge newsgroups takes way too much time.

My mozilla regularly uses 300 megs of RAM when syncing newsgroups for offline
reading.

Comment 4

[vadiml8]

I was using Moz 1.2,1.3,1.4,1.5,1.6 for downloading and reading newsgroups.
One problem remains - when downloading newsgroups for offline use, Mozilla eats 
memory like crazy - now it's 1 Gb. News folder now is 4 Gb in size.
RAM usage is increasing each time. So 1GB isn't the last stop, i think...

My PC has 384 Mb RAM and WinXP as OS.

Comment 5

[Mark Stier]

Was there ANY Mozilla release that did not have that problem? As far as I know,
Netscape 4.x did not have that issue.

Comment 6

[Mark Stier]

The problem is obviously Mozilla's internal database (Mork) which needs to load
the complete .msf files before being able to access their contents. Can't we
just integrate/update Netscape 4.8's corresponding storage algorithms?

Comment 7

[mrosen]

(In reply to comment #5)
> Was there ANY Mozilla release that did not have that problem? As far as I know,
> Netscape 4.x did not have that issue.

No. All Mozilla versions, as well as all Thunderbirds show this behaviour. This
is a major showstopper for any *serious* newsreader usage of the two.
Unfortunately, this 100% and easily reproducible Bug isn't even confirmed after
almost two years. So I doubt this will ever change.

Comment 8

[Mark Stier]

Or we need news-specific storage methods.

Comment 9

[Wayne Mery (:wsmwk)]

(In reply to comment #0)
> User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.3a)
> Gecko/20021212
> Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.3a)
> Gecko/20021212
> 
> Open a newsgroup with *lots* of messages (in the 10thousands), and let Mozilla
> download all headers (not only 500). Look at taskmanager, ans watch Mozilla
> consuming lots of memory until it finally loaded all messages. If the newsgroup
> was downloaded previously, just watch the memory footprint of Mozilla skyrocket
> when opening the newsgroup. It seems to load the complete .msf file into
> memory.
> Now odo the very same with NW4.X. You'll notice that Netscape hardly needs
> *any*
> additional memory, regardless how much headers the newsgroup holds.
> 
> Reproducible: Always
> 
> Steps to Reproduce:
> 1. Create a new account on news://support-forums.novell.com
> 2. Subscribe to the group novell.community.chat. Currently, this group holds
> 300.000 messages
> 3. Let Mozilla download all headers in this group. 

now over 580k messages.

Using TB 1.5 - started at 26,451k and ended at 449,944k

Comment 10

[Jonathan Pritchard]

Memory usage for, 3 newsgroups containing hundreds of thousands of messages each, meant that Thunderbird 2 was starting up with 130MB of memory usage on Windows XP. Unsubscribing from these newsgroups and restarting dropped it down to 28MB. There's got to be a solution to this.

Comment 11

[Joshua Cranmer [:jcranmer]]

Two problems exist:

1. We hold a reference to all msf's loaded so far in a cache
2. Each msf is fully loaded in memory.

#2 will be hopefully fixed by bug 11050, but what to do about #1? We probably need to see what happens with the Moz2 memory stuff.

Comment 12

[Wayne Mery (:wsmwk)]

Accessed novell.community.chat and hit 144MB, cycled through several newsgroups/servers and now I'm at 120MB.  That's a far cry from the 450MB of comment 9

(In reply to comment #11)
> Two problems exist:
> 
> 1. We hold a reference to all msf's loaded so far in a cache
> 2. Each msf is fully loaded in memory.
> 
> #2 will be hopefully fixed by bug 11050, but what to do about #1? We probably
> need to see what happens with the Moz2 memory stuff.

bienvenu, Joshua, do you agree comment 10 would likely have been addressed by  the "close unused db" fixes done in the past few months?  If so, given my test, I think we can demote this but to minor or normal

Comment 13

[Lapo Luchini]

Thunderbird version 2.0.0.21 (20090504) compiled on FreeBSD7/AMD64 from FreeBSD Ports.
Being subscribed to some 25 gmane newsgroups (but actually opening only 2-3 of them to read messages) my Thunderbird usually shows in top(1) as 1060MB size, 880MB resident… which seems pretty "major" to me, but maybe the changes you refer to are not yet in 2.0.0.21?
That value does NOT increase opening (for the first time in the session) new newsgroups, it seems they get loaded during the initial scan-for-new-messages and kept for good.

Comment 14

[Wayne Mery (:wsmwk)]

(In reply to comment #13)
> ... maybe the changes you refer to are not yet in 2.0.0.21?

correct. Sorry Lapo, I failed to state I tested with 3.0b3pre 20090515 nightly build.  These fixes will not appear in version 2 (1.8 branch). 

So you will want to test nightly build if you are able.  Don't need to uninstall Thunderbird 2, just backup your profile and install the nightly in it's own directory.
ftp://ftp.mozilla.org/pub/thunderbird/nightly/latest-trunk/

Comment 15

[David :Bienvenu]

there's nothing can do about one giant newsgroup given Mork's all-in-memory implementation - but we can close db's when the user isn't using the folder, as you say, Wayne.

Comment 17

[Tom Roberts]

Obviously this is a long-standing issue, dating back to 2002. But I never noticed it until Thunderbird 3.1.2 (which I tried in September 2010). I can accept a few hundred MB, or even 1 GB of memory, but I cannot accept the >>30 minutes. Note, however, in 2.0.0.14 I only see 80 MB.

Thunderbird 2.0.0.14 works OK for a newsgroup with 600,000 messages, only a few hundred of which are unread, and it's deleting messages older than 30 days -- 80 MB of "Real memory" on my Mac.

Something very bad happened between Thunderbird 2.0.0.14 and 3.1.2 -- when clicking on this newsgroup in 3.1.2, CPU goes to 100%, memory remains at 89 MB, with no network usage. So I suspect 3.1.2 has an infinite loop somewhere; probably related to sorting messages (see my description in bug 597688, which has been marked a duplicate of this one).

In the newsgroup preferences there is an option to "delete message older than 30 days" -- can't that be used to reduce the size of the DB? Indeed it looks like 2.0.0.14 did just that (when I discovered it a few weeks ago before upgrading, its speed went up considerably).

Comment 19

[Aaron Burgemeister]

Bug# 595145 was mine and while there are some similarities I haven't noticed this as primarily a memory issue... who knows.

The issue that I see is a minimal amount of network traffic but a huge amount of processor activity by the Thunderbird (3.1.7 currently) process.  Thunderbird becomes almost completely worthless for thirty or more seconds but it always comes back (my box is 64-bit and I have 8 GB RAM so maybe I'm past the memory problem by brute force).  Either way this single bug has me on the brink of changing because it seems (not tested... just seems) that Thunderbird decides to do its refresh of the newsgroup messages at the exact time I decide to use Thunderbird (not always, but regularly).  My workaround is to go away and come back in a couple minutes when all is well for a while longer.  It is likely that the original reporter and I have the same forums involved so duplicating this should still be pretty straight-forward.

It may be worth mentioning that my ~/.thunderbird directory is over 1 GiB in size so I suspect this has something to do with the disk more than anything else, perhaps involving the CPU as it thrashes the disk.

Comment 20

[Aaron Burgemeister]

Attachment: Output of 'ps aux' for the Thunderbird process with timestamps.

The purpose of this attachment is to show memory usage for my Thunderbird setup over several hours.  As you can see the memory fluctuates periodically and I believe those fluctuations, though not extreme, correspond to the time when Thunderbird takes over one of my CPUs/cores.  What this also shows is that Thunderbird is taking an average of 3.6% of my total CPU time (over several hours) through its high utilization during those times of high activity (ps's %CPU shows total cpu time divided by total runtime).  The math probably makes more sense than my description... 99% * 30 seconds averages out to 3.6% over about 13.75 minutes, which is close to my 13-minute refresh interval.

Comment 21

[Pascal Peyremorte]

I'm not sure it is really the same bug but it looks like close.

I am subscribed to another large nntp forum : news://server.amis.tv:119/nzn.fr.delphi, that currently holds around to 132800 posts.

Periodically Thunderbird halts to responds, whatever I am doing in it, and restart a few time later. It is extremely boring when I am writing some message.

During these halts, the task manager show that Thunderbird is using 100% time of one core. The amount of ram eaten is stable, around 150 Mb (from 148.5 to 151.5 Mb) and do not show large variations.

That problem was also discussed in forum mentionned above, under the subject "§§ Thunderbird n'aime pas server.amis.tv.Delphi ? §§" starting on 2010 november 27th.
It was reported that many Thunderbird users has the same problem. One of them even reported that he leaved Thunderbird because of this and he now uses SeaMonkey that does not have these halts, despite it is based on the same sources than Thunderbird.

Comment 22

[Ludovic Hirlimann [:Usul]]

Pascal could you get a stack trace when this happens  (see https://developer.mozilla.org/en/how_to_get_a_stacktrace_with_windbg) ?

Comment 23

[Aaron Burgemeister]

As mentioned before I do not think my bug matches this one (not a high memory issue... more like Pascal's it's a high processor utilization issue) but today I found another clue.  When I run something like `find ./ -mmin -4 | xargs ls -ld` from my user's .thunderbird directory I see that one file per subscribed forum is updated every time Thunderbird polls and finds either some or no changes.  EVEN IF there are no changes every file is changed, at least timestamp-wise.

Thinking this may be just a 'touch' of some sort I backed up a file and then compared it to the original after a change and found the new file, even though the forum had no new posts.  The lines added to the file, despite a lack of new posts to the forum, follow:

<quote>

@$${3D0F0{@
[1:^9F(^A9^709A)]
@$$}3D0F0}@

@$${3D0F1{@
@$$}3D0F1}@

@$${3D0F2{@
@$$}3D0F2}@

@$${3D0F4{@
[1:^9F(^A9=0)]
@$$}3D0F4}@
</quote>

I have no idea what these lines are doing but they happen every polling interval of Thunderbird (fifteen minutes currently) and appear to mean almost nothing.  I see more data added when something shows up in the forum to which this file corresponds.

Can this be changed/fixed?  Why doesn't Thunderbird just compare what it receives as far as forum counts via NNTP to the contents of the (in my case) ../.thunderbird/u9a8njvw.default/News/newsrc-forums.novell.com file?

Comment 24

[chicagolovr]

For the second time in recent months I am unable to download one large newsgroup, because *two* applications of the news server show up in task manager when I click to download it, and they utilize 100% of the CPU.  It only happens with this one ng on that server and it is the largest one.

The last time I had to re-subscribe and enter all my filters again which is quite a chore on that group, but I would really like to avoid this again.  Are there any solutions anywhere to this problem with SeaMonkey 2.0.10?