206502 - Checking all IMAP folders for new mail uses lots of memory

Status: RESOLVED DUPLICATE of bug 266679

(MailNews Core :: Backend, defect)

Description

[Marc Branchaud]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4b) Gecko/20030516
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4b) Gecko/20030516

This bug is spawned from work done in bug 18266.

I have a large number of IMAP folders (about 150) and some of them are quite big
(10,000+ messages).

When I enable mail.check_all_imap_folders_for_new, Mozilla's memory footprint
explodes to about 120+ MB when it checks for mail.  The memory size does not
decrease over time.

In bug 18266, bienvenu commented:

  Marc, how many folders do you have? And how big are they? The most likely
  cause I can think of is that the .msf files for all of your folders are
  opened and in memory, which will eat up a lot of memory. You should be
  able to tell which .msf files mozilla has open after the get new mail
  check runs. If that turns out to be the case, we need to make sure the
  .msf files are closed after get new mail runs (or limit the number that
  are kept opening - if you have large folders, opening and closing the
  .msf files could get expensive as well)


Reproducible: Always

Steps to Reproduce:
1. Use a lot of IMAP folders
2. Enable mail.check_all_imap_folders_for_new
3. Check for new mail

Actual Results:  
Memory use skyrockets.


Expected Results:  
Memory use should remain fairly stable.

Comment 1

[Marc Branchaud]

How can I tell which .msf files Mozilla has open?

Comment 2

[David :Bienvenu]

I'm not sure on linux, though there must be a way to tell what the open file
handles are for a process...

Comment 3

[Marc Branchaud]

Yep, it's the lsof command.

And yep, Mozilla has all the .msf files open.

Comment 4

[http://tinymailto.com/oliversl]

Confirmed too.
This happend in TB 1.0.2 and WinXP SP2.
Using 2 imap accounts.

To reproduce:
- have bug folders, with more tan 5000+ emails in it
- activate Junk Folders and have a training.dat bigger than 12MB
- start TB
- click inbox, wait until JunkControl stops
- click en each folder you have and let Thunderbird check for new messages
- meanwhile, keep looking at task manager
- notice the increment in the memory usage

Comment 5

[David :Bienvenu]

can you try a recent trunk build, e.g., a 1.1a build?
 I believe this has been fixed on the trunk.


ftp://ftp.mozilla.org/pub/mozilla.org/thunderbird/releases/1.1a1/

Comment 6

[Wayne Mery (:wsmwk)]

Marc, oliversl, can you try 1.1 or trunk   (in follow up to comment 5) 

http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-trunk/ 


is this related to bug 216535?

Comment 7

[Marc Branchaud]

Things have changed over the years -- I no longer have folders with 10,000+ messages, and I've switched to Thunderbird.  I'll try to find some time to put together a test or two, but since I don't experience the problem anymore it's just not high on my list of priorities.  Sorry!

Regarding bug 216535, the issues do seem similar.  I've always done server-side filtering of my IMAP mail, so that bug's filtering-or-not discussion might be a bit of a red herring.  If these two bugs are the same problem, it's probably something lower down.

Comment 8

[Wayne Mery (:wsmwk)]

fixed by bug 266679 and available through TB 1.5.
If you still have a problem please file a new bug (unless you are the reporter).

*** This bug has been marked as a duplicate of 266679 ***