52270 - Scrolling thread pane consumes memory without bounds (other than the size of the folder)

Status: VERIFIED FIXED

(MailNews Core :: Backend, defect, P1)

Description

[David :Bienvenu]

If you scroll the thread pane (either by page or a line at a time), it consumes
memory as each row is put in the dom.

Comment 1

[David :Bienvenu]

Dave's comment from 51359:

RDF builds the content nodes for the treecells and then XBL goes off and builds
the content nodes for the internals of the cells.  These are cached so that
scrolling speed will improve.

I could flush the XBL binding info quite easily if it would help, but scrolling
will take a hit from a change like that.

Comment 2

[David :Bienvenu]

How hard would it be to limit the number of content nodes cached? Caching the
equivalent of maybe 100 rows would handle the case of the user scrolling around
in a limited area, which is the important case to handle. When you say flushing
the XBL binding info would hurt scrolling performance, do you mean because we
would have to recreate the binding info, or because flushing the XBL binding
info is itself a performance hit?

Comment 3

[Michael Collette]

With the very real possibility of getting lynched I just have to ask this.  How 
did 4.x handle the thread pane?  For all the things that the 4.x series of 
Communicator did wrong, this was certainly one area that had stellar 
performance, and handled memory quite well.  I know this is probably one of 
those kinds of questions better asked around M3 rather than M18, but I really 
liked how 4.x performed in this regard on both NT and Linux.  Can any of that 
code be salvaged, or are we looking at problems that go deep into XPCom?

Comment 4

[Alec Flett]

unfortunately, 4.x had a much more direct connection between the mail backend
and the UI. When the frontend needed to paint a row, it would ask the backend
for the data required to throw text on the screen.

in the mozilla world, there is the mail backend, the DOM which represents the
thread tree's content, and the frame tree which describes the actual interface
that should be drawn for the given content. Each layer is populated by the layer
underneath it.

it would sure be nice if there was a way to make some of these layers thinner or
even have queries to each layer pass through directly to the mail backend, but
you're right, it's kind of a difficult thing at this stage in the game :(

Comment 5

[Michael Collette]

At this point in time probably the single most noticeable bug in all of Mozilla 
is the speed of that thread pane.  That's what got me looking at the memory 
usage in the first place.  It's worse, in my mind, than a crash since even when 
it is working it's just barely usable.  A nice hard crash at least goes pretty 
quickly! :)

Considering how glaring this bug is to someone testing Mozilla, I'd like to 
recommend it get moved to an nsbeta3+ status.  Not my place to schedule such 
things, so I'll leave that decision to one of the Mozilla team.

Comment 6

[Alec Flett]

the thread pane being slow is something we've been working on for well over a 
year, there's no one bug on it because there's no one thing that will fix it - 
it's an iterative process.

Comment 7

[David Hyatt]

Sorry, we can try to make small changes, but you're literally talking about 
weeks of work on the part of multiple engineers to try to fix this problem.

Comment 8

[Michael Collette]

Considering the complexity of actually fixing this bug in the short term, might
there at least be a possibility of having the memory used by mail/news reclaimed
when the window shuts down?  It seems that this should happen anyway, and would
at least put a band-aid on the wound.

Comment 9

[David :Bienvenu]

to answer the last question, that's a memory leak of an nsXULDocument that I've
tried to track down without success. I'm pretty sure it has to do with event
handling and the scroll bar - if I just click anywhere in the scroll bar, the
leak happens. If I navigate with next unread, the leak doesn't happen. It's not
a design issue, per se, just a bug.

Comment 10

[David :Bienvenu]

marking p2 - we should fix this after rtm, but it is a serious problem,
especially when coupled with the leaks that cause this memory to never get freed up.

Comment 11

[David :Bienvenu]

I *believe* that the leak that caused none of this memory ever to get freed up
has been fixed - so, now we still bloat like crazy, but at least the memory is
freed up when you close the window.

Comment 12

[David :Bienvenu]

reassigning to Hyatt for now...

Comment 13

[Peter Trudelle]

->putterman, this will require mail/news work and rdf work before our part can 
be done.

Comment 14

[scottputterman]

reassigning to sspitzer. hyatt, what rdf or mailnews work needs to be done first?

Comment 15

[scottputterman]

marking nsbeta1+ and moving to mozilla0.8

Comment 16

[(not reading, please use [e-mail] instead)]

accepting.

moving to p1.

Comment 17

[scottputterman]

moving to mozilla0.9

Comment 18

[Håkan Waara]

bug 68365 may be related, they both touch the same code.

Comment 19

[scottputterman]

marking nsbeta1- and moving to future.  This should get solved with the thread
pane rewrite.

Comment 20

[(not reading, please use [e-mail] instead)]

fixed.

we may still have some small memory leaks (I'm not sure, I haven't quantified),
but when scrolling things look good to me (using gtop)

Comment 21

[[e-mail] (gone - use [e-mail] instead)]

Seth - would this bug need to be quantified by me to be verified?  Knowing our
new <outliner> widget and it's efficiency in regards to memory usage (especially
when scrolling), and due to the fact that this bug was originally filed against
building RDF trees (which we no longer use) it seems it doesn't apply.  

Comment 22

[David :Bienvenu]

you can just bring up the windows task  manager performance tab and scroll
through a large folder to note that we don't consume megabytes of memory.

Comment 23

[[e-mail] (gone - use [e-mail] instead)]

Verified FIXED.  I started with ~144336k when browser+mail/news was opened, with
my default IMAP account logged in.  When I logged into my performance account
(also IMAP), the usage numbers increased to ~151796k opening the inbox.  When I
scrolling a 1,000 message folder, memory went to ~152203k (sometimes going to
154420k, but never above that.)  Also, memory usage+RDF trees were what made us
slow (see the results for build 2001-03-16 in
http://www.mozilla.org/mailnews/win_performance_results.html.)