997857 - Thunderbird uses a large amount of memory - immediately after launch it shoots up to 800MB

Status: RESOLVED INVALID

(Thunderbird :: Untriaged, defect)

Description

[thunderbird]

Attachment: msgdb.log

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36

Steps to reproduce:

Launch TB.


Actual results:

Immediately a large amount of memory is grabbed. I had a local database that was larger than 3GB which I have archived into databases of around 1GB each, and if anything the problem has got worse: TB used to grab 500-600MB, now it's up to 750-800MB straight away.



Expected results:

TB should use a lot less memory - I see that anything more than 300MB is unusually excessive. I attach a MSGDB log.

Comment 1

[Wayne Mery (:wsmwk)]

(In reply to thunderbird from comment #0)
> TB used to grab 500-600MB, 

when exactly was this, and with which version?

> now it's up to 750-800MB straight away....

What is the size of your inbox?
size of Junk folder?
size of trash folder?

> anything more than 300MB is unusually excessive. 

~3GB of messages may have as much as 450MB of index files (.msf files). And if all folders are actively used, you are not going to see less than about 4

Comment 2

[thunderbird]

Thanks for the quick reply, and sorry for the delay - I am in a patchy connectivity area.

> > TB used to grab 500-600MB, 

> when exactly was this, and with which version?

Early December I started measuring it, but I had been experiencing reported memory problems maybe a few months before. I don't remember what version of TB it was, but TB is configured to update automatically, so it would have been the latest release version at the time.

Current version is 24.4.0.

> > now it's up to 750-800MB straight away....

> What is the size of your inbox?

IMAP inbox (on server) is 1.54 GB. POP3 inbox (locally) is 2.11GB (brought down from 4GB by archiving older messages in two archives, now 700MB and 1GB respectively).

> size of Junk folder?

Junk folder on server = 0; there is no local Junk folder

> size of trash folder?

IMAP (server) 'Deleted' folder = 1MB (I clear it out regularly)
POP3 (locally) 'Deleted' folder = empty

> > anything more than 300MB is unusually excessive. 

> ~3GB of messages may have as much as 450MB of index files (.msf files). And if all folders 
> are actively used, you are not going to see less than about 4

Hmm. I think we're on to something here. I went looking for the .msf files and found one HUGE one (578MB), C:\Users\koen\AppData\Roaming\Thunderbird\Profiles\44ihn6is.default\ImapMail\grey.mail.cix.co.uk\INBOX.msf

I have no idea why it's in a subfolder called 'grey.mail.cix.co.uk' but that folder seems to be associated with the IMAP mailbox. There is another one 
C:\Users\koen\AppData\Roaming\Thunderbird\Profiles\44ihn6is.default\Mail\mail.cix.co.uk\Inbox.msf
which is the same folder as the local POP3 copy (an Inbox file of 2.2GB), but that .msf file is just 88MB.
There are a few more .msf files that are smaller still for various smaller mail files.

The problem is quite likely to be related to this huge .msf file. Why would an index file associated with an IMAP mailbox of 1.64Gb be > 500GB, while the index file associated with a local mailbox of 2.2GB is only 88MB?

Would a global rebuild of the index help here? Intriguingly, I've just had a look at the Activity Manager in TB and it shows "Indexing 55 of 29876 messages" - constantly, no movement. I restarted TB and immediately opened the Activity Manager again. I saw it indexing the unread messages on the IMAP server, and then starting to index the whole lot, only to get stuck at 122 this time.

Hope this helps!

Comment 3

[Wayne Mery (:wsmwk)]

1. you have no gmail accounts?

2. " I've just had a look at the Activity Manager in TB and it shows "Indexing 55 of 29876 messages""  can you determine which account and folder that is happening in ?

3. It would be great if you could further reduce the size of the pop inbox by at least have. Better yet, move all messages out of there that no longer require action on your part.

Comment 4

[thunderbird]

Thanks for your continuing help! Sorry for the delay in coming back - I've been away for a short break in a place with very poor connectivity...

1. Yes, I have some gmail accounts, but I am not managing them with TB

2. I'm not sure how to do that. (Actually, I don't know how to do that is more accurate :-)). I just had a look now, and it says "Indexing 165 of 29985 messages). I presume it is the main (IMAP) account, in the main inbox - there is a total of 30,301 messages there, while in the POP3 local copy there are 38,000 messages

3. Would it be OK if I simply archive more messages and leave just the last (say) six months' worth in there?

Comment 5

[Wayne Mery (:wsmwk)]

(In reply to thunderbird from comment #0)
> Expected results:
> 
> TB should use a lot less memory - I see that anything more than 300MB is
> unusually excessive. I attach a MSGDB log.

I suspect you read https://wiki.mozilla.org/Thunderbird:Testing:Memory_Usage_Problems#Memory_Overview ... I guess I need to reword that. One of the operative words there is "may", as in "is over 300MB, then you may have a problem that needs analysis."  

Anyway, 

2. constant or frequent "Indexing xx of xxxx messages" is going to cause you performance issues. Reducing the number of messages in the affected imap folder will help reduce the performance hit. Even better would be to find out the repeating indexing is happening in the first place - which is typically caused either by a) messages being redownloaded from your ISP, or b) a uid roll on your ISP's server. 

3. Archiving messages from Inbox that are only needed for historical reasons is always a good idea. Furthermore, the msf file for a 2gb pop Inbox will be 200-300MB and requires a corresponding amount of memory. A significant reduction in message count will reduce your memory usage.

Comment 6

[thunderbird]

> > Expected results:
> > 
> > TB should use a lot less memory - I see that anything more than 300MB is
> > unusually excessive. I attach a MSGDB log.
>
>
> I suspect you read https://wiki.mozilla.org/Thunderbird:Testing:Memory_Usage_Problems#Memory_Overview 
> ... I guess I need to reword that. One of the operative words there is "may", as in "is over 300MB,
> then you may have a problem that needs analysis." 

Haha, yes, that's where I got it from.

2. I'm afraid my knowledge of IMAP is somewhat limited, so any pointers for finding out the reason for the reindexing (or even establishing whether it's the reindexing that it responsible for the memory glut - could it be that a lot of memory is needed for this, which would be given back to the OS once it's done, but since it gets stuck the memory remains blocked? That would explain the fact that the large memory usage problem occurred relatively suddenly, as I've always had between 1.2 and 2GB of IMAP mail on the server, even before TB started eating my memory).

I had a look for 'UID roll' and found this in an old bug (264800) report:

> Asa's problem turned out to be a bug with his server, triggered by the fact that
> he was accessing the same inbox with pop3 check for new mail, which panicked the
> server and caused it to roll uid validity...

The POP3 and IMAP mail boxes are actually the same - I use IMAP because I want to be able to manipulate read flags and deletions from various devices (phone, tablet, PC), and I use POP because I want a single, secure local copy of all my incoming mail. Could this be related to the indexing problem? If so, how can I demonstrate this?

3. I'm continuing with the archiving. So far it doesn't seem to make much difference to the memory usage though. The archives themselves are of the order of 1GB - is that potentially a problem?

Thanks again!

Comment 7

[Wayne Mery (:wsmwk)]

(In reply to thunderbird from comment #6)
> 2. I'm afraid my knowledge of IMAP is somewhat limited, so any pointers for
> finding out the reason for the reindexing (or even establishing whether it's
> the reindexing that it responsible for the memory glut - could it be that a
> lot of memory is needed for this, which would be given back to the OS once
> it's done, but since it gets stuck the memory remains blocked? That would
> explain the fact that the large memory usage problem occurred relatively
> suddenly, as I've always had between 1.2 and 2GB of IMAP mail on the server,
> even before TB started eating my memory).
> 
> I had a look for 'UID roll' and found this in an old bug (264800) report:
> 
> > Asa's problem turned out to be a bug with his server, triggered by the fact that
> > he was accessing the same inbox with pop3 check for new mail, which panicked the
> > server and caused it to roll uid validity...
> 
> The POP3 and IMAP mail boxes are actually the same - I use IMAP because I
> want to be able to manipulate read flags and deletions from various devices
> (phone, tablet, PC), and I use POP because I want a single, secure local
> copy of all my incoming mail. Could this be related to the indexing problem?
> If so, how can I demonstrate this?

are you accessing the same account with pop and imap in the same thunderbird client, i.e. on the same machine?  Or different machines.

Comment 8

[thunderbird]

The same machine (Windows 7 64bit). After archiving the local inbox (pop) and pruning the IMAP mailbox TB's memory usage has reduced to about 300MB which is a lot better, but still somewhat high.

Comment 9

[Wayne Mery (:wsmwk)]

for a 3gb folder the index will be on the order of 450MB, so 450MB+ memory usage just for the folder. So your memory usage is not unusual

Comment 10

[thunderbird]

Thanks - some more pruning is in order it seems. 
Just a quick question: a significant contributor to the size of the folders is the email attachments. Does that affect the size of the index, or is the number of messages the main/only parameter affecting the memory usage?

Comment 11

[Wayne Mery (:wsmwk)]

Attachments do not affect the size of the folder index. Only the number of messages and the choice of message headers being indexed.

Comment 12

[thunderbird]

Thanks, that is indeed what I suspected.
But I don't think I've quite understood how indexing affects working memory. If I archive more, then the active 'inbox' files will come down in size, but the archives in which the excess messages are placed of course increase in size. Is that going to help with my memory problem, or is there something else I can do? For instance, my 2012 Archive (local only) is 1.4GB in size - admittedly with some attachments still in there. Will that affect the memory usage?
Thanks again for you help.

Comment 13

[Wayne Mery (:wsmwk)]

I failed to answer some of your questions. So ...

(In reply to thunderbird from comment #12)
> Thanks, that is indeed what I suspected.
> But I don't think I've quite understood how indexing affects working memory.
> If I archive more, then the active 'inbox' files will come down in size, but
> the archives in which the excess messages are placed of course increase in
> size. 

Archiving messages should make the original folder smaller - if you or thunderbird compact the folders of the account to physically "squeeze out" the messages that were logically removed/deleted.  It is possible however for the compact operation to fail.  If the size of the folder and it's .msf do not go down then this should be suspected.

As for total memory usage, it will go down but only if the "archive folder"(s) aren't opened in your normal usage of Thunderbird.  


> is there something else I can do? For instance, my 2012 Archive (local only) is 1.4GB in size -
> admittedly with some attachments still in there. Will that affect the memory usage?

Attachments do not affect memory usage


> The problem is quite likely to be related to this huge .msf file. Why would an index file associated with an 
> IMAP mailbox of 1.64Gb be 500GB, while the index file associated with a local mailbox of 2.2GB is only 88MB...

I assume you mean 500MB, not GB.  And yes, a 500MB Inbox.msf is a problem, in that it will require >500MB to keep that .msf in memory.