Closed Bug 63115 Opened 24 years ago Closed 23 years ago

Mozilla chokes with 100 000 new headers to download

Categories

(MailNews Core :: Backend, defect)

Product:
MailNews Core
Component:
Backend
Platform:
x86
Windows ME
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
VERIFIED WORKSFORME

People

(Reporter: matxdr, Assigned: sspitzer)

References

(
URL
)

Details

(Keywords: perf) 

From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Windows; U; Win 9x 4.90; en-US; m18) Gecko/20001215
BuildID:    2000121520

I downloaded around 100 000 headers in a newsgroup.  The download is fine, but
after the download Mozilla begins to scratch the disk and takes forever to do
it's job.  I had to stop it with a end task after 5 (yes five!) minutes of work
on a 7200 rpm drive. Now, it I click on this group, Mozilla restarts to scratch
the disk the same way it did after the download.

Reproducible: Always
Steps to Reproduce:
Subscribe to a newsgroup which has an insane posting volume like
alt.binaries.cd.image.highspeed and get all the headers

Actual Results:  Mozilla download fine but take forever to build its DB		

Expected Results:  It should be as fast as with other mail/news programs

 I know that 100 000 headers isn't usual and that
alt.binaries.cd.image.highspeed should not be read, but it's a nice performance
test case...!
actually, it's not really a database problem. It's all the stuff that goes on
above the database to display the headers for the newsgroup that consumes the
lion's share of the CPU. Reassigning to Putterman for triage - presumably, this
is a dup of someone's bug - not sure who's. Seth? Mscott?
Assignee: bienvenu → putterman
Component: Mail Database → Mail Back End
QA Contact: esther → stephend
matxdr@yahoo.com, what are the specs of your machine?  Do you have at least 64
megs of RAM? XUL is known to be slow rendering trees..
I have an AMD Athlon 700 with 128 megs ram and a Quantum 7200 RPM.  Vid card is
Diamond Viper 2
I only had about 30,000 headers total for this group, and I'm really not
choking, yes it's slow, but we've got XUL improvements coming up hopefully
before the next release.  I'm also checking Outlook Express 5.5 and it's just as
slow downloading the headers, so the performance problems you see are not likely
specific to news, they are most likely XUL tree problems in general.  David, am
I correct? 
Fabian, could you check this? I've been unsuccessful at reproducing this, thanks!
Poisoned gift, having to download 100k headers! ;) I'll try it, I'll let you 
know the results.

Looks like I can't find a news server with alt.binaries.* on it. Can you provide 
me with one?
reassigning to sspitzer, but this is probably just a dup of our large folders 
take a long time to load bugs.
Assignee: putterman → sspitzer
I suggest to put this one on ice since it's a possible dup and since we're in
the hollydays (you won't find enough volume to test the case).  On my side, I
have max 30-45k headers.  I suggest to wait for the optimization fix for the
trees and I'll test this again.
Ok guys, I tested this again with build 2001010404.  Again, I had to end-task
Mozilla after 43 minutes of work (plus 7 mins of download time... I timed it,
not kidding :-(.  For 43 mins the light of the HD never turned off and the HD
was spinning like hell.  I have downloaded 101422 headers.  In Netscape 4.76 it
took ages just to download the headers so I didn't dare testing. Forte Agent did
the job in 10 mins (download plus displaying) and OE5 in 14 for download plus 3
before it's usable (first screen is displayed but cannot scroll or download
before 3 mins),  Also took 5 mins to quit the app...! :-).
This is done on a freshly defraged hard drive with over 3G left on.  The
alt.binaries.cd.image.highspeed.msf was 39 megs after the end task.

IMHO, the I/O disk rate is clearly the bottle neck here.  With other apps, I
don't hear the hard disk that much, but with mozilla, the disk goes wild and
never stops.  The problem starts only when the download of the headers is
complete and the disk goes crazy.

I tried to give it a second chance so I deleted the MSF files, reopened mozilla
and clicked on the newsgroup.  I crashed in MORK.DLL.  Talkback event
TB24105749W and TB24105453Q.
Yes, on alt.binaries.cd.image.highspeed

If any machine would fail on this, it would be my Pentium 133, so I'll try it
out and report my findings here.
Other than being slow, I downloaded all the headers, (was only around 14,000)
this time.  Is this from a migrated profile or a freshly created Account?
It was with a migrated profile, but I subscribe to this NG only after the
migration.  I downloaded today's build and tried it with a fresh profile.  It
seems to be better.  I had to end-task it after 17 mins of HD scratching, but
the MSF was 42 megs already.  
Comments from Matt:
I did some test yesturday and today (with today
build) and it seems to be morte a performance problem
than a bug...

Yesturday:

# header        time to process    avg per header
---------------------------------------------------
20000            23 secs            0.00115 sec/header
33000            1 min 12 secs      0.002181~
50000            8 mins 30 secs     0.0102 

closing app took ~ 10 mins first time

Today:
# header        time to process    avg per header
---------------------------------------------------
50000            7 mins 55 secs     0.0095 sec/header

When I closed the app today, crashed with TB24339777Q.

As you see it takes more time to process per header as
the total grows (probably normal).  Also, note that
they are practicly no trees to build because these
group are mostly a single post without reply.

If you want, I can send you the MSF of 50k header so
you can compare (3 megs compressed).

His talkback crash was here: 
Call Stack:    (Signature = nsCOMPtr_base::~nsCOMPtr_base 0f1cfc93)  
     nsCOMPtr_base::~nsCOMPtr_base  
[d:\builds\seamonkey\mozilla\xpcom\base\nsCOMPtr.cpp, line 49] 
     nsElementMap::ReleaseContentList  
[d:\builds\seamonkey\mozilla\rdf\content\src\nsElementMap.cpp, line 138] 
     nsElementMap::~nsElementMap  
[d:\builds\seamonkey\mozilla\rdf\content\src\nsElementMap.cpp, line 119] 
     nsXULDocument::~nsXULDocument  
[d:\builds\seamonkey\mozilla\rdf\content\src\nsXULDocument.cpp, line 556] 
     nsXULDocument::`scalar deleting destructor'    
     nsXULDocument::Release  
[d:\builds\seamonkey\mozilla\rdf\content\src\nsXULDocument.cpp, line 660] 
     nsJSUtils::nsGenericFinalize  
[d:\builds\seamonkey\mozilla\dom\src\base\nsJSUtils.cpp, line 439] 
     FinalizeXULElement  
[d:\builds\seamonkey\mozilla\rdf\content\src\nsJSXULElement.cpp, line 292] 
     js_FinalizeObject  [d:\builds\seamonkey\mozilla\js\src\jsobj.c, line 1630] 
     js_GC  [d:\builds\seamonkey\mozilla\js\src\jsgc.c, line 1200] 
     js_ForceGC  [d:\builds\seamonkey\mozilla\js\src\jsgc.c, line 932] 
     js_DestroyContext  [d:\builds\seamonkey\mozilla\js\src\jscntxt.c, line 
225] 
     JS_DestroyContext  [d:\builds\seamonkey\mozilla\js\src\jsapi.c, line 886] 
     nsJSContext::~nsJSContext  
[d:\builds\seamonkey\mozilla\dom\src\base\nsJSEnvironment.cpp, line 389] 
     nsJSContext::`scalar deleting destructor'    
     nsJSContext::Release  
[d:\builds\seamonkey\mozilla\dom\src\base\nsJSEnvironment.cpp, line 394] 
     nsCOMPtr_base::assign_with_AddRef  
[d:\builds\seamonkey\mozilla\xpcom\base\nsCOMPtr.cpp, line 59] 
     nsWebShell::Destroy  
[d:\builds\seamonkey\mozilla\docshell\base\nsWebShell.cpp, line 1446] 
     nsXULWindow::Destroy  
[d:\builds\seamonkey\mozilla\xpfe\appshell\src\nsXULWindow.cpp, line 330] 
     nsWebShell::Destroy  
[d:\builds\seamonkey\mozilla\docshell\base\nsWebShell.cpp, line 1446] 
     nsXULWindow::Destroy  
[d:\builds\seamonkey\mozilla\xpfe\appshell\src\nsXULWindow.cpp, line 330] 
     nsWebShellWindow::Destroy  
[d:\builds\seamonkey\mozilla\xpfe\appshell\src\nsWebShellWindow.cpp, line 1752] 
     nsWebShellWindow::Close  
[d:\builds\seamonkey\mozilla\xpfe\appshell\src\nsWebShellWindow.cpp, line 347] 
     nsWebShellWindow::HandleEvent  
[d:\builds\seamonkey\mozilla\xpfe\appshell\src\nsWebShellWindow.cpp, line 419] 
     nsWindow::DispatchEvent  
[d:\builds\seamonkey\mozilla\widget\src\windows\nsWindow.cpp, line 691] 
     nsWindow::DispatchWindowEvent  
[d:\builds\seamonkey\mozilla\widget\src\windows\nsWindow.cpp, line 708] 
     nsWindow::DispatchStandardEvent  
[d:\builds\seamonkey\mozilla\widget\src\windows\nsWindow.cpp, line 729] 
     nsWindow::ProcessMessage  
[d:\builds\seamonkey\mozilla\widget\src\windows\nsWindow.cpp, line 2842] 
     nsWindow::WindowProc  
[d:\builds\seamonkey\mozilla\widget\src\windows\nsWindow.cpp, line 944] 
     KERNEL32.DLL + 0x3613 (0xbff63613)    
     KERNEL32.DLL + 0x248f7 (0xbff848f7)    
 

Just finished a test with 75k headers.  It took 36 mins 40 secs, so 0.0293~ secs
/ headers.  Last test for at least a week, I don't want to kill my drive ;-)
Maybe Netscape can provide a free Giganews account and a spare drive for Mozi
testers? ;-)
I'm tired of seeing this being UNCONFIRMED status, and since it's performance +
news, I'm confirming.  Note: I haven't hit 100,000 headers yet to be able to
tell though.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Keywords: perf
Blocks: 63759
Interesting...
New test with 2001012904.  Major speed improvement.  It took 5 mins 10 secs for
50k headers, which is a 3 mins 20 secs improvement over build 200010109xx.  Nice
work guys.
Okay, so I guess we can mark it FIXED when the reporter is happy with the speed
(the reason this bug was filed in the first place)?
You still tested disk thrashing speed, not download speed, right? If so, I'll
mark this WORKSFORME.
Yes, only disk processing.  But it's still pretty slow to exit the news or
changing group.  I'll couldn't test with 100K, not enough volume today.  I'd
still wait a bit before closing, i'd like to see how it is with 100K.
New test using build 2001031904, using the new cache system I guess.  I still
can't test with 100k headers because my news server itself is choking... So, I
tested again with 50k, and wow, I'm impressed and very happy about the perfs. 
Beside the download time, NO more hard drive thrashing at all, display speed is
instant for the messages (scrolling too), and closing the mail/news is also
instant and won't cause any damage.  It now seems that Mozilla is the fastest
mail/news client currently available.  So, I'm marking this one WORKSFORME.  Did
I mentionned that I'm happy?  Again, great work!
Status: NEW → RESOLVED
Closed: 23 years ago
Resolution: --- → WORKSFORME
verified.  if we see this again, we can open a new bug (but I agree, performance
has improved greatly.)
Status: RESOLVED → VERIFIED
I'd like a few others to revisit this issue and see if it should be reopened.

From my recent experience with build 2002061707 I would have to say that Mozilla
flunks the newsgroup header download test.

I went to comp.graphics.algorithms today and attempted to download all
10581 headers with build 2002061707.  Using an Athlon 600 with 256 MB 
of RAM.   Mozilla was completely unresponsive while chugging away at
this and I terminated it with NT's Task Manager after 55 minutes.   
For comparison I did the same thing with Netscape 4.79 and it took about 
93 seconds to download all of the headers, sort them, and regain full
responsiveness.

Downloading only 500 headers from that newsgroup works OK and only takes
about one minute.  Unfortunately, once I have browsed those 500 headers
Mozilla does not provide me with a way to tell it to get the next 500 
headers.
Product: MailNews → Core
I'm still experiencing this bug on the latest nightly build of Thunderbird and Seamonkey.

Using alt.binaries.cd.image, there are over 100k headers to download.  The download is still running for me.  I have had it complete before but then the usability of the news window is horrendous (very slow to load messages, etc.
Product: Core → MailNews Core
You need to log in before you can comment on or make changes to this bug.