636306 - Improve virtual folder / saved search performance - doesn't scale for large numbers of virtual folders

Status: NEW

(MailNews Core :: Backend, defect)

Description

[Wayne Mery (:wsmwk)]

Improve virtual folder / saved search performance - doesn't scale for large numbers of virtual folders.  

This isn't about the impact of entering a folder (such as Bug 534520 - Significant lag selecting Inbox Smart Folder (or any very large folder) but rather, Thunderbird becomes slower in normal usage when there are many virtual folders.

I don't have precise steps to reproduce nor a measure for what "many" means.
But I've encountered this myself - probably starting in the range of 10+ virtual folders. Pretty sure it's dependent on the aggregate size of the folders being spanned (not surprising) - so the number of virtual folders where one hits the breaking point may vary.  

Were it not for this performance issue I'd probably have more virtuals defined to make my life easier. And (dmose) iirc it's one of the reasons I abandoned the mailing list manager addon. 

http://getsatisfaction.com/mozilla_messaging/topics/smart_folders_that_only_search_when_you_click_on_them#reply_4015258 cites this performance issue (user created 100+ virtuals).  I see it often mentioned in the context of people's small businesses, even for modest numbers of folders, so I'm going to mark this blocking Bug 564148.  And there are many possible applications such as http://getsatisfaction.com/mozilla_messaging/topics/can_we_please_have_a_combined_unified_inbox_view_in_thunderbird


Perhaps this should be a meta bug blocked by several bugs which address the technical issues.

Comment 1

[Wayne Mery (:wsmwk)]

(In reply to comment #0)
> Improve virtual folder / saved search performance - doesn't scale for large
> numbers of virtual folders.  
> 
> This isn't about the impact of entering a folder (such as Bug 534520 -
> Significant lag selecting Inbox Smart Folder (or any very large folder) but
> rather, Thunderbird becomes slower in normal usage when there are many
> virtual folders.
> 
> I don't have precise steps to reproduce nor a measure for what "many" means.
> But I've encountered this myself - probably starting in the range of 10+
> virtual folders. Pretty sure it's dependent on the aggregate size of the
> folders being spanned (not surprising) - so the number of virtual folders
> where one hits the breaking point may vary.  
> 
> Were it not for this performance issue I'd probably have more virtuals
> defined to make my life easier. And (dmose) iirc it's one of the reasons I
> abandoned the mailing list manager addon. 
> 
> http://getsatisfaction.com/mozilla_messaging/topics/
> smart_folders_that_only_search_when_you_click_on_them#reply_4015258 cites
> this performance issue (user created 100+ virtuals). 

dudley, on Mac 10.4 machine with 1GB memory posted that gsfn topic, and wrote me "I have 508 "smart (virtual) folders" in Macmail"   I suppose we can't rule out that 508 folders might have caused high memory usage.

He blew away all the virtual folders, so we no longer have his testcase.

Comment 2

[Wayne Mery (:wsmwk)]

is cache working 100% for saved search? 

for reference purposes, here are examples of fixed bugs perf from 3.1/3.0 era:
Bug 533865 - Doing a quick search in a cross-folder saved search clears cached results
Bug 537820 - quick search in smart folders/saved searches should prefer offline when it has a choice
Bug 466048 - Thunderbird freezes/slow when selecting threaded Saved Search/Virtual Folder

Comment 3

[Wayne Mery (:wsmwk)]

(In reply to Wayne Mery (:wsmwk) from comment #0)
> This isn't about a) the impact of entering a folder (such as Bug 534520 -
> Significant lag selecting Inbox Smart Folder (or any very large folder) but
> rather, b) Thunderbird becomes slower in normal usage when there are many
> virtual folders.

backtracking slightly, it's possible that a) and b) are tightly linked.  so we could explore general virtual perf issues, and split bugs off for specific use cases if needed

Comment 4

[WADA:World Anti-bad-Duping Agency]

If slowness started to occur in Tb 3, it's perhaps due to "MsgDB open takes long if .msf is big or many .msf is read".
   From Tb 3, "close msgDB after inactivity period" is executed. 
   Virtual Folder(Search Folder, United Folder/Smart Folder) opens many msgDB at once.
If slowness started to occur after landing of patch for bug 764306(landed on Tb 16), and if IMAP folder is relevant, slowness of ListAllKeys() may be involved.
   http://mxr.mozilla.org/comm-central/source/mailnews/db/msgdb/src/nsMsgDatabase.cpp#1402
      nsMsgDatabase::OpenMDB calls  DumpContents();
   http://mxr.mozilla.org/comm-central/source/mailnews/db/msgdb/src/nsMsgDatabase.cpp#5004
      nsMsgDatabase::DumpContents() calls ListAllKeys(keys);
   ListAllKeys() has problem of Bug 872869 after patch for bug 764306.
       See test result of bug 872869 comment #33
           With patch for bug 764306                   : 266 seconds.
           After backout of patch for bug 764306 : 265 milliseconds for same job.
 
To bug opener, how big is your Virtual Folder?
     # of Search Target Folder ]
         # of vlocal Mbox, # of real imap Mbox with Offline-Use=Off, # of real imap Mbox with Offline-Use=Off
     # of mails in each Search Target Folder.

Once a Virtual Folder is opened, Search Target Folders(real Mbox) is opened at least a while.
If other than Virtual Folder is clicked at folder pane and the Virtual Folder is clicked again, open of Virtual Folder is executed bu open of real Mbox is not needed.

To bug opener: Do you see your "Significant lag" issue in this "second open of Virtual Folder"?
Call "# of Search Target Folder" "N". Is the "Significant lag"  O(  N ^ 2 ) like one? Or O(  N * logN ) like one?  Or O( N )?

I can't imagine  other than O( N ) in Virtual Folder open where N = "# of Search Target Folder", although I can imagine O(  M * logM ) folder open time of each Search Target Folder where M="number of mails in Search Target Folder".

> Bug Summaary : doesn't scale for large numbers of virtual folders

To bug opener: Actually scalability issue in "numbers of virtual folders"? Scalability of what?

 

(const char *dbName, bool create, bool sync)

, 872869  in Tb 3, it's perhaps due to "MsgDB open takes long if .msf is big or many .msf is read".
   From Tb 3, "close msgDB after inactivity period" is executed. 
   Virtual Folder(Search Folder, United Folder/Smart Folder) opens many msgDB at once.

Comment 5

[WADA:World Anti-bad-Duping Agency]

(In reply to Wayne Mery (:wsmwk) from comment #0)
> but rather, Thunderbird becomes slower in normal usage when there are many virtual folders.

Each Virtual Folder opens all real folder requested as Search Target Folder. And all Virtual Folders does do it.
This produces state of "very many mail boxes are opened at same time and kept open" by single folder click at folder pane.
This surely increases Virtual Memory consumption.
All Unified Folder are opened by Tb upon startup of Tb.
.
Unified Folder(aka Smart Folder) hooks events such as msgDBHdrAdded of real Mboxes to reflect new mail in Inbox, sent mail copy to Sent etc., because it's "Unified Folder"(user expects real Inbox, real Sent like presentation) instead of "Search Folder"(display of a Search Result).
This kind of activites may increase CPU utilization, Virtual Memory  consumption.