534520 - Significant lag selecting Inbox Smart/unified Folder (or any very large folder)

Status: RESOLVED INCOMPLETE

(Thunderbird :: Folder and Message Lists, defect)

Description

[Larkin Lowrey]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5 (.NET CLR 3.5.30729)
Build Identifier: 3.0 - Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.5) Gecko/20091204 Lightning/1.0b2pre Thunderbird/3.0

When I select the Inbox smart folder I see "Searching folder..." and one core of my CPU goes to 100% for about 10 seconds. While this is happening, I cannot open any messages in that folder. If I select an individual inbox, this process takes less than a second. The time seems proportional to folder size based on the lag for the Sent smart folder.

Inbox #1: ~13,000
Inbox #2: ~25,000
Inbox #3: < 1,000

Reducing the sizes of my inboxes is not the answer I'm looking for. Since each folder takes less than 1 second to access, I would expect the smart folder to take less than 3s, not 10s.

Reproducible: Always

Steps to Reproduce:
1. Click on Inbox smart folder
2.
3.
Actual Results:  
Functions correctly, after 10 second delay (and 100% CPU usage).

Expected Results:  
No, or very short delay (1-2sec max)

Comment 1

[Ludovic Hirlimann [:Usul]]

Does this happens in -safe-mode (http://kb.mozillazine.org/Safe_mode) ?

Anything in Tools -> Error console when this happens ?

Comment 2

[Larkin Lowrey]

(In reply to comment #1)
> Does this happens in -safe-mode (http://kb.mozillazine.org/Safe_mode) ?

Yes. No change.

> Anything in Tools -> Error console when this happens ?

Error console is empty.

Comment 3

[Wayne Mery (:wsmwk)]

ultimately this will likely be a dup of an existing virtual folder bug

Comment 4

[Wayne Mery (:wsmwk)]

Larkin, do you also see the following memory behavior?

chatted with kithpom today in #thunderbird where the time to load the smart virtual "sent" of 52k messages was not linear to the individual files (10 sec total instead of expected 3), and memory was as much as 150M greater than the total for each individual folder.  lightning is installed. v3.0.3. not memory constrained (4gb)

I wasn't able to find a dup amongst https://bugzilla.mozilla.org/buglist.cgi?value1-1-0=crash%20send%20body%20dele%20repl&type1-0-0=anywordssubstr&field1-2-0=component&type1-2-0=notsubstring&field0-0-0=short_desc&bug_severity=major&bug_severity=normal&bug_severity=minor&type0-0-1=substring&field0-0-1=keywords&resolution=---&classification=Client%20Software&classification=Components&type1-1-0=nowordssubstr&chfieldto=Now&query_format=advanced&chfieldfrom=600d&value1-2-0=compos&value1-0-0=virt%20saved%20smart%20fold&field1-1-0=short_desc&longdesc=smart%20virt%20sav&value0-0-1=perf&type0-0-0=anywordssubstr&value0-0-0=memory%20%20slow%20lag%20long%20cpu%20process&field1-0-0=short_desc&longdesc_type=anywordssubstr&product=MailNews%20Core&product=Thunderbird or https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced&short_desc=smart&bug_severity=major&bug_severity=normal&bug_severity=minor&short_desc_type=anywords&type0-0-0=nowords&value0-0-0=count%20counts&resolution=---&product=MailNews%20Core&product=Thunderbird

Comment 5

[Larkin Lowrey]

Performance is improved for me. I am now down to ~5sec for my combined inbox and ~3sec for Sent. I am running 3.0.3 but I don't know for sure what to attribute the improvement to. I have not used the combined folders since the performance has been so poor so I can't tell if it was the update to 3.0.3 or the fact that I also in the mean time removed an e-mail account.

At 5 sec I find the performance to still be too slow to be tolerated on a daily use basis.

Comment 6

[Wayne Mery (:wsmwk)]

(In reply to comment #5)
> At 5 sec I find the performance to still be too slow to be tolerated on a daily
> use basis.

Larkin, I'd agree. How are things in version 3.1?  

Mike, rsx11m, Roland, can you find descriptions of this in gsfn or forum?  I'm pretty sure I've seen some.

Comment 7

[Larkin Lowrey]

I'm at 8 seconds for 42,000 messages. It acts like it's trying to merge and sort on access. If that's the case then it's entirely reasonable that it would take something on the order of 8 seconds. I imagine that a collection of inboxes totaling 42,000 messages is probably an edge case and therefore not worth addressing.

Comment 8

[Wayne Mery (:wsmwk)]

Larkin, how much memory is being used by thunderbird, and how much memory available in your machine [1]? and what type and speed of PC?

bug 520272 might help here (larkin is all imap with imap accounts with 28,500, 14,200, and 500 messages in the inboxes.)

[1] memory details at https://wiki.mozilla.org/Thunderbird:Testing:Memory_Usage_Problems#Overview

Comment 9

[Larkin Lowrey]

Memory is ranging from 150MB to 160MB and fluctuates within that range. Windows ResourceMonitor shows 1.3GB free physical memory.

My present machine is dual core AMD Turion Ultra X2 2.3GHz. The host is Windows 7 x64 w/ 4GB of RAM. Disk is SSD, if it matters. I am traveling so i can't test my desktop speed atm.

Comment 10

[Michael A. Pasek]

(In reply to comment #6)
> Mike, rsx11m, Roland, can you find descriptions of this in gsfn or forum?  I'm
> pretty sure I've seen some.

Larkin's observation that it's size related meshes with observations in GS for
large folders.  For Inboxes or Sent folders that are large (byte-wise) but
contain relatively few messages (i.e., have not been compacted or cannot be
compacted), lag to open can be minutes, regardless of folder view.

(In reply to comment #7)
> I'm at 8 seconds [...] 42,000 messages is probably an edge case and therefore not
> worth addressing.

I this a reversal of the opinion in Comment #5 that 5 seconds is "too slow to be tolerated on a daily
use basis" ?  I don't know that 42,000 messages is an "edge" case; most long-time users (e.g., those migrating from TB2 or other clients) will have a significant number of saved messages.

Comment 11

[Larkin Lowrey]

No reversal. I still cannot use the combined inbox... without being driven mad. I do use the "Unified Folders" view but have the combined inbox expanded to reveal the individual inboxes. I find it convenient to have all the inboxes in one spot. Navigating to the individual inboxes is instantaneous.

My edge case opinion is based on the belief that many, if not most, users divide their mail into multiple smaller folders. I personally find that tedious but I feel that I am in the minority.

Comment 12

[Wayne Mery (:wsmwk)]

(In reply to comment #9)
> Memory is ranging from 150MB to 160MB and fluctuates within that range. Windows
> ResourceMonitor shows 1.3GB free physical memory.

sounds like nominal/normal memory usage.


> My present machine is dual core AMD Turion Ultra X2 2.3GHz. The host is Windows
> 7 x64 w/ 4GB of RAM. Disk is SSD, if it matters. I am traveling so i can't test
> my desktop speed atm.

hmm, mkanat has SSD Intel X25-M G2 (CPU: Intel Core 2 Duo E6700 (2.66Ghz), RAM: 2GB CL2 DDR2)  and was also complaining about performance with his big bugmail folder. coincidence?  

My main interested here is the imap case, which bug 520272 will help. 
But there may be non-imap related issues (or even other imap issues).

Speculatively adding bug 347837 as a potential helper. 
xref bug 588952 - Extremely bad performance at startup or when reading messages with very big IMAP folders

Comment 13

[Wayne Mery (:wsmwk)]

Unless we are going to tell gmail users or anyone with a large inbox that they shouldn't keep there stuff in a big folder, 50k is certainly not an edge case.

Additionally, we now have archive folders :)

Comment 14

[Wayne Mery (:wsmwk)]

blocking request for 3.2 on evaluating the questions posed above, and whether the bugs that presumably should fix this (bug 347837, bug 520272 and perhaps others) should be targeted for 3.2

Comment 15

[Wayne Mery (:wsmwk)]

what's the consensus on these items?

(In reply to comment #14)
> blocking request for 3.2 on evaluating the questions posed above, and whether
> the bugs that presumably should fix this (bug 347837, bug 520272 and perhaps
> others) should be targeted for 3.2

switching request to 3.3

Comment 16

[David :Bienvenu]

Don't think this is going to block 3.3, since it didn't block 3.1.

Comment 17

[Maximilian Mehnert]

Perhaps it's not a good idea to display all messages all the time.
If you try a folder with about 10^6 messages, Thunderbird will be quite busy for a while figuring out how to display that.
Perhaps there should be an option to have a paged view if the message number goes over a limit + some indexing / caching / searching magic that essentially provides a view to operate on a subset of messages.

Comment 18

[Mark Banner (:standard8)]

I don't think there's anything specific enough here that we can block 3.3 on. Perf improvements are always wanted though.

Comment 19

[Wayne Mery (:wsmwk)]

Is this better for you in version 5 (or 6) now that Bug 520272 was fixed in v5? ( Optimize perceived speed of switching IMAP folders)

Is speed now acceptable?

Comment 20

[Wayne Mery (:wsmwk)]

please comment if you have additional information, including whether or not you still see this problem using a newer version.