Closed Bug 6873 Opened 25 years ago Closed 24 years ago

Mailnews folder loading performance tracker.

Categories

(MailNews Core :: Backend, defect, P1)

Product:
MailNews Core
Component:
Backend
Type:
defect

Tracking

(Not tracked)

Status:
VERIFIED DUPLICATE of bug 26456

People

(Reporter: cyrus, Assigned: sspitzer)

References

(
URL
)

Details

(Keywords: perf) 

Using IMAP, I get a bunch of messages about how mailnews is going to the server
and the timing between gets progressively slower and slower. Looks like there's
some sort of slow linear process here that it does for each message. After a while
it REALLY starts to crawl. I'm not sure if this has to do with the fact that my
mailbox has lots of messages, or that there are lots of mailboxes in my IMAP
mailspace, but it sure is annyoing.
Assignee: phil → hyatt
Unfortunately, this is expected behavior with the current code. We need to have
some table layout improvements from hyatt before we expect this to get better.
Reassigning to hyatt for now.
Status: NEW → ASSIGNED
Target Milestone: M8
Whiteboard: [Perf]
OS: Mac System 8.5 → All
Priority: P3 → P1
Hardware: Macintosh → All
Adding myself and waterson to the cc list since we are both involved in this.
Changed Platform and OS to All since the Mac shouldn't get all the fun, and
changed priority to P1 since we aren't a product without this fix.

This will happen on Pop3, and news, as well as Imap.  Although we may have
performance problems with regard to getting the messages from the Imap Server,
that's most likely not the case(or at least the bottleneck) since we've seen the
same slowing down affect on both Pop and News.

Chris and I have been able to improve performance a bunch with some other
changes we've made but at the moment it looks like the number of times we
relayout the table is what is causing our problem.

See following threads in the netscape.public.mozilla.mail-news for discussions
of this:

"Profiling results for displaying headers in the thread pane." 5/22/99
"More profiling results for displaying headers in the Messenger thread pane"
5/26/99
"Messenger Performance update" 6/2/99 which I will use to track our progress

Chris and David, we should talk about how this is going to work at some point.
One of my concerns is that David's code will only ask for 10 messages but that
the RDF code will still build up the content model for 10,000 messages.  In that
case GetTarget will be a huge bottleneck since it looks like we can process
about 100 messages in .5 sec currently(i.e. 10,000 messages would take 50
seconds by default).  Even if I can speed up GetTarget, we still have another 5
or so columns that haven't been added, so at best those cancel each other out.
What we'll probably need to do is change RDF-generated content to be fully lazy,
i.e., make it so that it has a finer level of granularity regarding the building
of children.  Right now whenever we need to find out information about the
children of a node, RDF goes off and builds all of the children.

RDF should only build children as they are asked for, and we should annotate
nodes with visible child counts, so that we don't have to make all the children
when we ask for the child count of a node.
*** Bug 8472 has been marked as a duplicate of this bug. ***
Summary: Mailnews Performance Sucks with Large Mailboxes → [DOGFOOD] Mailnews Performance Sucks with Large Mailboxes
Added dogfood distinction to summary.
We will have some numbers for loading POP folders soon.  The machine we're
using is a 166mhz with 48mb RAM.  Recently, it took about 64 seconds to load
1000 messages (.msf file already created) on Win32.
I'm still (6/28) seeing unacceptable performance when deleting messages via
IMAP. The particular scenario is:
1. Open mozilla. Load mailbox with 650 messages. Quit.
2. An another machine, delete some messages, get some new messages, etc...
3. Open mozilla. Watch mozilla crawl as it attempts to sync up the mailbox.
I have numbers for loading Pop folders on the mail-news newsgroup in various
threads called "Messenger Performance update".  The machine may be fast, but you
can still see progress made with the numbers.  A machine with less memory is
useful also since it will help us measure the performance hit we take because of
it.

Getting new messages is always going to be worse than just loading a folder
with our current architecture.  That's one of the reasons why this bug exists
and is not marked fixed yet. You can view the various threads on the newsgroup
to see why the asynchronous case is worse.  Therefore I'd also expect things
like deleting and marking read to be slow as well.  Admittedly I haven't used
imap very much with 5.0, but until we get folder performance fixed, I'm not sure
we can attribute anything to the IMAP code yet (unless of course you are running
IMAP tests outside of Apprunner or you are synching with a folder that is not
the currently loaded folder in which case we could attribute this to the IMAP
code).
Assignee: hyatt → waterson
Status: ASSIGNED → NEW
Target Milestone: M8 → M9
Moving to M9 and handing off to waterson. My task here is done. :)
Boy oh boy!
Status: NEW → ASSIGNED
Blocks: 9161
Checked in some improvements for the template builder (lazily build template
content as frame system asks for it). This should give a 2-3x speedup for
folders that have already been indexed.
I'm going to change the meaning of this bug.  We're getting a big push right now
on the async case, I guess people higher up suddenly started seeing our numbers.
So, since a large portion of the work on this bug has gone to .msf case, I'd
like to leave this to represent that.  I will open up another bug we can track
for the async case.  I will cc all of the people on this bug.
Blocks: 11091
Assignee: waterson → putterman
Status: ASSIGNED → NEW
scottip is owner of this "meta bug" now.
Depends on: 11400
Depends on: 11402
Summary: [DOGFOOD] Mailnews Performance Sucks with Large Mailboxes → [DOGFOOD] Mailnews folder loading performance tracker.
Target Milestone: M9 → M15
This bug goes on as long as the product does.  Marking M15.  Changing summary.
Whiteboard: [Perf] → [Perf] [PR1]
Obviously, we need to get faster performance by Beta 1.

Scott: If the target milestone is M15, will this bug stay on your radar as we go
from one pre-Beta 1 milestone to the next?
This bug will never leave my radar.  I made it M15 because I'm sure it will be
ongoing until we ship.  Plus, the bugs it depends upon are the more important
ones since those are where the real work is being done.
Blocks: 12176
No longer blocks: 11091
Status: NEW → ASSIGNED
Blocks: 10791
No longer blocks: 12176
I'm going to change the dependency bug from 12176 (dogfood) to 10791
(feature) tracking bugs since we're already at dogfood, I believe, since folks
are using the product internally.  Since this bug is always "on-going," I
think having it on the feature tracking bug is better.  If anyone disagrees, pls
change back.  Thanks.
Summary: [DOGFOOD] Mailnews folder loading performance tracker. → Mailnews folder loading performance tracker.
Remove [dogfood] from bug summary
*** Bug 21419 has been marked as a duplicate of this bug. ***
Depends on: 23071
Depends on: 23070
No longer depends on: 23070
No longer depends on: 23071
Keywords: perf
Bulk add of "perf" to new keyword field.  This will replace the [PERF] we were
using in the Status Summary field.
Removing [PR1] since we're using keywords now. Nominating for beta1 since we are
currently more than 2x slower than 4.x (according to Suresh's Jan 14 performance
testing results). Not much more than 2x, but more.
Keywords: beta1
Whiteboard: [Perf] [PR1]
Putting on PDT+ radar for beta1.
Whiteboard: [PDT+]
why is a top-level tracking bug marked PDT+? this is a meta-bug, not related to 
any task AFAICT. Removing "PDT+" and "beta1". What am I missing here?
Keywords: beta1
Whiteboard: [PDT+]
Mass moving to M16 to get these off the M15 radar.  Please let me know if this
is really an M15 stopper.
Target Milestone: M15 → M16
Not M16 stopper.  Marking M17.
Target Milestone: M16 → M17
moving to M18, though to be honest it's not as if I'm going to forget this bug 
exists :-)
Target Milestone: M17 → M18
Downloading 50 additional messages into INBOX (which already has 1000+) messages
takes around 202 secs, thats pretty slow. 
Tested using build 2000-08-02-04, M18 on Linux (performance system). 

Other Platforms (same build dates)
Mac: 10.20 seconds. 
Windows: 20.72
QA Contact: lchiang → stephend
Putterman, you haven't forgotten about this bug have you?  It's got a milestone
of M18, is apparently a tracker and has no unresolved dependencies.  Come on,
you said you wouldn't forget about it.  =)
Rest assured, we haven't forgotten about it, in fact we are working very
dilligently on this.  Wish I could say more.
reassigning to sspitzer.  This is in the process of being fixed.
Assignee: putterman → sspitzer
Status: ASSIGNED → NEW
For those of you interested, check out
http://www.mozilla.org/mailnews/performance/speed.html and keep track of the
newsgroups on news.mozilla.org:119 (groups netscape.public.mozilla.mail-news and
netscape.public.mozilla.performance)
please reconsider the milestone on this bug; it's kinda hard to go back in time. :)
This *will* be vastly improved with the thread pane re-write.  See the URL above.
URL: http://www.mozilla.org/mailnews/perfo...
Keywords: nsbeta1
Target Milestone: M18 → ---
this is a dup (the dup is later but it's already got all of the markings on it.)

*** This bug has been marked as a duplicate of 26456 ***
Status: NEW → RESOLVED
Closed: 24 years ago
Resolution: --- → DUPLICATE
verified dup, "Folder load performance is slow" bug 26456
Status: RESOLVED → VERIFIED
Product: MailNews → Core
Product: Core → MailNews Core
You need to log in before you can comment on or make changes to this bug.