1058026 - (tbbigfolder) [meta] Thunderbird big folder / high mail volume performance issues, high CPU and/or high IO

Status: NEW

(MailNews Core :: Database, defect)

Description

[Wayne Mery (:wsmwk)]

This meta bug is to collect the "class" of performance issues involving large folders or high message volume [1] as backend activity (not manual message moves and the like) and that significantly impacts UI response or startup time. Source of performance hit is not limited to specific protocol or component. The goal is to give them focused attention and organization.  

Non-goal: we're not going to cover every possible performance bug here.

If you are a user with a problem that involves very large folders or high message volume, and don't see your problem described in one of the bugs blocking this then please do not comment here, but please file a new bug and make it block this bug.

This set is just the first pass.

[1] Definition of very large folders or high message volume - I don't have a great defintion ready, it somewhat like, "I'll know it when I see it".  I'll update the user story eventually. For now let's say an approximation is"

* large folder - probably focused mainly on message count, say 5,000+ messages.  But maybe also by message size, say >1GB
* high message volume - one or more folders gets more than 50 messages on startup or in a short interval during the day

Comment 1

[Wayne Mery (:wsmwk)]

(mozilla-central bug 872497 was fixed 2013-05-19, TB24)

I certainly don't expect these issues to be limited to enterprises, but I expect they will disproporrtionately affect enterprises, so blocking bug 564148

Comment 2

[Ed I]

Tbird 60.4.0 (64-bit)
Linux Mint 19 tara 64bit
Kernel Linux 4.15.0-45-generic x86_64

Large profile folder full of decades' worth of mail

Very high cpu and 300+MB memory
raised idle setting by factor of 100
still high both mem and cpu.
Followed section 9 here: https://wiki.mozilla.org/Thunderbird:Testing:Memory_Usage_Problems
Global messages db was 940MiB - renamed and restarted
Global messages now 580MiB

Problems solved

Thanks!

Comment 3

[Wayne Mery (:wsmwk)]

(In reply to Ed I from comment #2)

Tbird 60.4.0 (64-bit)
Linux Mint 19 tara 64bit
Kernel Linux 4.15.0-45-generic x86_64

Large profile folder full of decades' worth of mail

Very high cpu and 300+MB memory
raised idle setting by factor of 100
still high both mem and cpu.
Followed section 9 here: https://wiki.mozilla.org/Thunderbird:Testing:Memory_Usage_Problems
Global messages db was 940MiB - renamed and restarted
Global messages now 580MiB
Problems solved

This would be related to Bug 484646 - some day gloda may want to VACUUM or auto-VACUUM

Comment 4

[donatetosorry]

Thunderbird 60.8.0 64-bit
Fedora 30

I do often have an ongoing high CPU load due to thunderbird. Memory usage at least currently is at about 500 MiB (but idc, 500 MiB memory usage in my case is no issue in my opinion)

I have three folders with 29 000 messages and way more than 100 000 messages in total. The whole inbox is about 3,8 GiB.

Comment 5

[ISHIKAWA, Chiaki]

(In reply to donatetosorry from comment #4)

Thunderbird 60.8.0 64-bit
Fedora 30

I do often have an ongoing high CPU load due to thunderbird. Memory usage at least currently is at about 500 MiB (but idc, 500 MiB memory usage in my case is no issue in my opinion)

I have three folders with 29 000 messages and way more than 100 000 messages in total. The whole inbox is about 3,8 GiB.

Interesting. I thought you have a few rather big folders, but when I look at the number of messages in my folders on my home PC, I found
Inbox has 28955 mails (many are spams, I am afraid) and it is about 1.2 GB.
I have mailing list folders which store rather busy mailing list messages.:
binutils - 22531 messages as of now.
libc-alpha - 44771 messages.
mozilla dev platform has 25484, etc.
Actually there are many folders that have more than 10,000 messages.
I also see 500+ MB memory usage.

Generally speaking, I try to break a folder into smaller pieces when it goes over 2GB. There used to be a bug (32 bit vs 64 bit integer) regarding the handling of folder whose size goes over 2GB, and I made the habit of breaking them into smaller chunks so that not many folders go over this size. This is true on my home PC and office PC. My office PC has probably way more than 100,000 messages. Inbox was divided into YYYY-1Q, YYYY-2Q, etc. (YYYY is the year number) and I think I have more than 14 years worth of e-mails.

Yes memory usage can go high, especially after searching seems to visit and load folder databases.
But I have not felt that memory usage is bad. Compared to memory hogger like Firefox, TB's usage is peanut.

After searching for messages using search function (the last 30 days, subject has "xxx" in it), TB now uses 537MB on my home PC.
Next to TB, FF uses 1.8GB (with 12 processes, I think.). Of course the comparison is "oranges vs apples".
I have used FF to access Facebook and youtube videos since last evening, so FF naturally uses large amount of memory.

I am posting this because the current memory usage of TB is NOT that bad for DESKTOP PC at home and at the office.
(I have 24 GB of memory on the home PC and 16GB on my office desktop and notebook if I recall ccorrectly.)

I don't know if same can be said for usage on pad devices or smatphones.

I can live with the speed of TB for now.
But for any improvement work of the performance, I think we need a better performance measurement tool.
I mean I tried to use performance measurement tools built into mozilla software, but it did not work as advertized when it was tried against TB.
Maybe the situation has improved? I have no idea.

Is there a meta bug/entry for referring to the current status/request for improvement of
performance measurement tools?

Something like a very crude gprof-like statistics goes a long way to find a hot spot which needs attention for performance improvement.
But I never figured out if gprof-like compiler flag/linker flag can work with TB or for that matter with FF.

It would be very interesting to see something like DTRACE facility for toolkits used inside mozilla software. Then
we can monitor the operation of these inside FF and TB. Of course, TRACE facility for TB would be really nice, but
I doubt if the TB developer community has ever the man-power resouces to do that.

TIA for people's attention.

Comment 6

[donatetosorry]

Ok, now I found something else: My profile uses 133 GB (while my inbox is 3.8). I could free up quite some storage running repair on some folders.

There are also 52 GB worth of nstmp files (up to number > 1000).

For some folders there is also the following:

FolderName
FolderName-1
FolderName-2
FolderName-3
FolderName-4

That is not being fixed by using the repair folder functionality.