Closed Bug 54491 Opened 24 years ago Closed 24 years ago

Subscribe dialog, slow, 100% CPU, mega bloat (23 MB to 88 MB), leaks

Categories

(SeaMonkey :: MailNews: Message Display, defect, P1)

Product:
SeaMonkey
Component:
MailNews: Message Display
Type:
defect

Tracking

(Not tracked)

Status:
VERIFIED FIXED
Milestone:
mozilla1.0

People

(Reporter: adamlock, Assigned: sspitzer)

References

Details

(Keywords: perf, Whiteboard: [dogfood-][rtm-]) 

The newsgroup subscribe dialog builds a tree containing every single newsgroup 
for the specified server. This takes a long time, overloads the CPU and leads to 
massive memory consumption.

Steps to reproduce:

1. Subscribe to a news server (e.g. news.netscape.com) which has a large 
newsgroup count.
2. Select the server from the account panel, right mouse click
   and choose "Subscribe...".
3. Once all the newsgroup names are fetched, quit Mozilla.
4. Start up Mozilla, note the memory consumption and choose
   "Subscribe..." again.

Here are the memory consumption figures for displaying newsgroups on 
news.netscape.com on my box:

Before opening dialog: 19900K
After loading groups:  66152K
Difference:            46252K

news.netscape.com has approximately 38000 newsgroups so memory consumption works 
out about 1.2K per newsgroup. It takes over 2 minutes on my 450Mhz machine to 
load the entire list.

After the dialog is dismissed the memory consumption does not drop indicating 
either a leak or a need for garbage collection.

Wouldn't it be better for this dialog to only build a tree containing groups 
that are visible? Newsgroups should only be populated when the parent group is 
expanded.
Adding keywords
Keywords: dogfood, perf
yes, these problems are all known.

subscribe is something that I'll fix in the next point release.
Status: NEW → ASSIGNED
or whatever release.  (I'm not sure how they are going to work.)

changing QA to suresh.
QA Contact: esther → suresh
This sounds like it borders on making subscription to news groups unusable.  Is
that the case? ...or only when the news-group server has a large list? (and is
that unusual?).
Comments?

I have no problem using the dialog with news.mozilla.org that has a couple of 
hundred groups but that is not typical. Most ISP newsgroup servers have upwards 
of 15000 groups, and sometimes as many as 40000 groups.

The combination of bloat, slowness & CPU usage certainly make it unusable at 
those numbers. Possibly 15000 groups would be just tolerable but beyond that 
and it is unusable - and dangerous. I crashed my NT box the first time I opened 
this dialog because it exhausted all physical memory, probably causing the 
graphics driver to fail somehow.
Newzbot is a handy site for seeing statistics such as how many groups a server 
contains:

http://www.newzbot.com/

Some servers listed contain 80000 groups!
yes, yes, yes, the subscribe dialog is a beast.

using it against news.mcom.com is painful, since it takes a long time to
populate and layout.

and yes, the more the newsgroups on the server, the worse it gets because all
the groups are put into the tree, instead of populating it lazily.

but it in its current state,  it is acceptable against imap servers and
news.mozilla.org which means it was usable for dogfood.

it's on the list of things to fix.  not going to happen for b3 or rtm.

..and also note that typedown performance has improved a looot!! yeah, from O(n) 
to log(n).
Per spitzer's analysis on news servers need for Netscape folks to do work... I'm
marking this dogfood-minus.
Whiteboard: [dogfood-]
I know this is basically being put on hold for now, but did want to mention that
in my case, there is not only slowness, but a crash of the browser.

I did get a small server to work such as the netscape/mozilla one, but not for
the one that comes with my ISP, which has a little over 70,000 groups.

It gets a bit farther than 2 megs downloaded, and then then it crashes (talkback
still isn't working for me, but it crashed in xpcom.dll).

I suppose it is moot, since it sounds like the whole thing needs to be
re-written, but I worry that a lot of people's isps have that many (would tend
to have more than any of the public ones because it is paid for, etc.), and if
it crashes while d/ling the groups, they may think it doesn't work at all.

And thanks Adam for the newzbot link. =)

Marking for rtm (Yes I know this it late, and )

Reason:

It makes the news part of Netscape 6 unuasble! for most users (>90%)

here are some number:...
Memory usage before subscribe : 23 MB
Memory usage after subscribe  : 88 MB
Number of groups:               around 35000

this is with 26/10-2000 on win2000 (modern skin).

I know all danish isp's have these news servers with large amounts of groups, 
and I bet every ISP in the rest of the world has it.
Thus this bug is making the news  part unusable for the majority of users.

Saying that it is okay because it work with news.mozilla.org (with a VERY low 
number of groups) is good enough simply doesn't cut it. IMO.

I'm upping priority and severity
Severity: normal → critical
Keywords: rtm
Priority: P3 → P1
Summary: Subscribe dialog, slow, 100% CPU, mega bloat, leaks → Subscribe dialog, slow, 100% CPU, mega bloat (23 MB to 88 MB), leaks
I agree, it is barely usable.  it is really too late in rtm for this.

marking mozilla 1.0
Target Milestone: --- → mozilla1.0
rtm- based on the comments already in the bug.
Whiteboard: [dogfood-] → [dogfood-][rtm-]
*** Bug 47459 has been marked as a duplicate of this bug. ***
*** Bug 52351 has been marked as a duplicate of this bug. ***
*** Bug 55996 has been marked as a duplicate of this bug. ***
updating summary.

this will be my catch all bug for fixing the subscribe dialog.
Keywords: crash
Summary: Subscribe dialog, slow, 100% CPU, mega bloat (23 MB to 88 MB), leaks → Subscribe dialog, slow, 100% CPU, mega bloat (23 MB to 88 MB), leaks, can crash
*** Bug 39396 has been marked as a duplicate of this bug. ***
*** Bug 57540 has been marked as a duplicate of this bug. ***
*** Bug 57876 has been marked as a duplicate of this bug. ***
'my' bug has just been duped to this one, so adding myself to cc to keep track.
However I notice this one is OS WinNT .. whereas mine was Win98+Linux.
I wouldn't want to make that alteration myself to this bug, but just airing the 
info.
Major caveat here that I don't know what I'm talking about.... however Moz only 
manages to not crash when subscribing.. (downloading list) if I do _nothing_ at 
all. And a quit and back to subscribe means I have to resume the list download. 
Does Moz keep a copy of the downloaded list? Doesn't seem to.. 'cause it if did 
it shouldn't be so slow.
marking all to avoid confusion.
OS: Windows NT → All
Hardware: PC → All
Adding self. Can you please fix this bug very very soon so I can do beta testing
of the newsgroups I like to subscribe to?

I've landed the big change to fix this.  there are two more things I need to fix
before I declare victory.

1) fix bloat, see #60507
2) crash in #58238

but you should notice the performance and memory improvement.


Changing QA Contact per request, adding self to cc.

FWIW, using build 111621 Linux on a K6-2 300 w/ 256Mb RAM, it takes 5min 26secs
to load my ISP's group list of over 65,000 groups. Immediately after opening a
top node the whole app segfaults... Here's hoping Seth is on top of this one :)
QA Contact: suresh → stephend
the crash is probably #58238

was the slowness when building up subscribe dialog when talking to the
server, or from a local file?

I'm finishing up some last fixes for performance and bloat.  when I land those
(and fix the crash) I'll mark this bug fixed.

putterman found that the second time you use subscribe in a session, it gets
much slower.  the last fixes I'm working on will address that.
Depends on: 58238
Just tried news on Mozilla Build 2000111708 and it still crashed on me.
here's part of the console message...

clearing PRIMARY clipboard
stopping meteors 1
************************************************************
* Call to xpconnect wrapped JSObject produced this error:  *
[Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) 
[nsIStringBundle.GetStringFromName]"  nsresult: "0x80004005 
(NS_ERROR_FAILURE)"  location: "JS frame :: 
chrome://messenger/content/mailWindow.js :: anonymous :: line 259"  data: no]
************************************************************
root subscribe tree at: news://ogdensp@news.rmplc.co.uk
JavaScript error: 
 line 0: goDoCommand is not defined

JavaScript error: 
 line 0: goDoCommand is not defined

JavaScript error: 
 line 0: goDoCommand is not defined

I believe these last few javascript errors occured _as_ I was trying to type 
into the subscribe box.. rather than search out the group I wanted, i was 
trying to type it in... couldn't even get that far. I'm guessing that the 
command to 'find' the group as one types doesn't work.. even though I _did_ see 
it start to jump as I began to type...
I repeat, the crash is bug #58238

those js execptions / console warnings are not related to the crash.

see bug #53106
This seems a lot faster to me now and the memory usage is *much* better and 
looks like it's freed when the dialog is dismissed. Great work Seth!
the crash is covered in 58238.

this bug covers general performance and bloat.

performance is way up, and bloat is way down.

marking fixed.
Status: ASSIGNED → RESOLVED
Closed: 24 years ago
Keywords: crash
Resolution: --- → FIXED
Summary: Subscribe dialog, slow, 100% CPU, mega bloat (23 MB to 88 MB), leaks, can crash → Subscribe dialog, slow, 100% CPU, mega bloat (23 MB to 88 MB), leaks
a few things i would like to add here:

i'm on a 640k/7mbit adsl line, my isp has about 35k groups on it's news server
and the group file is a little over 1.6megabytes.
with "tin", it takes less than 5 seconds to fetch the groups, network meter
shows speeds around 800kbytes/s when getting the groups.

with mozilla, the groups come about 20kbytes/s and it takes several (2-3)
minutes to refresh the group list. during the refresh, cpu load is near 0%.
why is this so slow, when cpu speed and network speed are clearly not the issue?

also, when i'm in the subscribe window, all other mozilla windows are not
responding to mouse clicks, keypresses etc..
The subscribe window is "modal" which means that you can't bring focus to
background windows until the subscribe window is OK'd or CANCEL'led...
Before I re-open this, I'll have to see what seth says..(for performance)
Re: modalness... That's another issue entirely and thus should be another bug
id. I think I've heard talk about such issues but I don't have a bug id handy.

Modalness in the Subscribe window is bug # 41470.  So you can add your comments
to that bug, but I'm currently waiting for the client spec to be clarified.
OK filling with my groups (loading from disk) is down from over 5mins to under
10 secs. I call that fixed :). Using moz build 2108 linux. Now to fix the 'alt'
crasher... :-).

Marking VERIFIED FIXED, due to my own experience on Windows NT and user comments
in this report.
Status: RESOLVED → VERIFIED
refreshing groups they still come around 20kB/s with build 2000112208 when
there's 7mbits of available bandwidth. other news clients do it in a few
seconds. cpu usage is near zero during fetch. what's wrong? don't others see this?
You can file a new bug with the specifics of your problem. That's better than to 
bring it up here in a closed bug which doesn't directly covers what you are 
seeing. Please cc me in the new bug.

Not so quick on the Verified Fixed :( See bug 65468.
When Seth checked in his changes, News' Subscribe dialog was substantially
improved.  Granted, some issues remain.  But this bug was covering the majority
of the issues, all in one.  It's better to file smaller, more concentrated bugs
than overall.  This remains VERIFIED FIXED, as filed.
Product: Browser → Seamonkey
Component: MailNews: Subscribe → MailNews: Message Display
QA Contact: stephend → search
You need to log in before you can comment on or make changes to this bug.