autosync should use new async folder opening to avoid extremely bad performance at startup or when reading messages with very big IMAP folders (like gmail All Mail)
Categories
(MailNews Core :: Backend, defect)
Tracking
(Not tracked)
People
(Reporter: ehsan.akhgari, Unassigned)
References
(Blocks 4 open bugs)
Details
(Keywords: perf, Whiteboard: [tbperf-key][has stack trace])
Attachments
(1 file)
|
6.26 KB,
text/plain
|
Details |
|
Reporter | |
Comment 1•15 years ago
|
||
|
||
Comment 2•15 years ago
|
||
|
||
Comment 3•15 years ago
|
||
|
||
Comment 4•15 years ago
|
||
|
||
Comment 5•15 years ago
|
||
|
||
Comment 6•15 years ago
|
||
|
|
Reporter | |
Comment 7•15 years ago
|
||
|
|
||
Comment 8•15 years ago
|
||
|
|
Reporter | |
Comment 9•15 years ago
|
||
|
|
||
Comment 10•15 years ago
|
||
|
||
Comment 11•15 years ago
|
||
|
|
||
Comment 12•15 years ago
|
||
|
|
||
Comment 13•15 years ago
|
||
|
|
Reporter | |
Comment 14•15 years ago
|
||
|
|
Reporter | |
Comment 15•15 years ago
|
||
|
|
Reporter | |
Comment 16•15 years ago
|
||
|
|
||
Comment 17•15 years ago
|
||
|
|
||
Comment 18•15 years ago
|
||
|
|
Reporter | |
Comment 19•15 years ago
|
||
|
|
Reporter | |
Comment 20•15 years ago
|
||
|
|
Reporter | |
Updated•15 years ago
|
|
|
Reporter | |
Comment 21•15 years ago
•
|
||
| important | ||
|
|
Reporter | |
Comment 22•15 years ago
|
||
|
|
||
Comment 23•15 years ago
|
||
|
||
Comment 24•15 years ago
|
||
|
|
Reporter | |
Comment 25•15 years ago
|
||
|
|
Reporter | |
Comment 26•15 years ago
|
||
|
|
||
Comment 27•15 years ago
|
||
|
|
Reporter | |
Comment 28•15 years ago
|
||
|
|
Reporter | |
Comment 29•15 years ago
|
||
|
|
||
Comment 30•15 years ago
|
||
|
|
||
Comment 31•14 years ago
|
||
|
|
Reporter | |
Comment 32•14 years ago
|
||
|
||
Comment 33•14 years ago
|
||
|
|
||
Comment 35•14 years ago
|
||
|
||
Comment 36•13 years ago
|
||
|
|
||
Updated•11 years ago
|
|
|
||
Updated•8 years ago
|
|
|
||
Updated•6 years ago
|
|
|
||
Comment 37•5 years ago
|
||
Ben, thus far the async open has not been utilized. Given what you've seen so far in your backend work, is there anything to suggest a) that we shouldn't do this to alleviate some of our performance issues, and b) are there any backend changes that should be done before this is attempted?
|
||
Comment 38•5 years ago
|
||
I'm not sure what you mean by async folder opening. There is nsIMsgDatabase.asyncOpenFolderDB() for opening the msgdb. Is that what you meant?
asyncOpenFolderDB() doesn't appear to be used anywhere currently (other than in a couple of unit tests), so I'd be a little wary of it.
Stepping back a little, if it's the msgdb that's the speed bottleneck, then I'd say that real issue is that it's too slow (for whatever reason). The locally-stored db of metadata should be lightening-quick (relatively!) to open/scan. Even if you've got colossal quantities of email, it should really be on the order of seconds, not minutes...
Anyway - have we got replication steps for this one? If I set up a dummy local IMAP server with huge quantities of emails, do you think that would trigger it? Are we sure it's IMAP-only?
|
|
||
Comment 39•5 years ago
|
||
(In reply to Ben Campbell from comment #38)
I'm not sure what you mean by async folder opening. There is
nsIMsgDatabase.asyncOpenFolderDB()for opening the msgdb. Is that what you meant?
Yes
asyncOpenFolderDB()doesn't appear to be used anywhere currently (other than in a couple of unit tests), so I'd be a little wary of it.
Yes, it would be wise to proceed cautiously, especially if we don't have tests in this area.
Stepping back a little, if it's the msgdb that's the speed bottleneck, then I'd say that real issue is that it's too slow (for whatever reason). The locally-stored db of metadata should be lightening-quick (relatively!) to open/scan. Even if you've got colossal quantities of email, it should really be on the order of seconds, not minutes...
Yes, async is a workaround for improving the speed of opening the meta - but there's no other choice AFAIK until some distant future date when the backend gets reworked. In the mean time (including the past decade), message stores have on average gotten considerably bigger and we haven't improved our methods/performance in this area to keep pace.
Anyway - have we got replication steps for this one? If I set up a dummy local IMAP server with huge quantities of emails, do you think that would trigger it? Are we sure it's IMAP-only?
Yes, it would be imap-only - born out of https://wiki.mozilla.org/MailNews:Better_Faster_IMAP_Plan and Bug 470624 - RFE: On-demand Auto-Sync. Note however, the request for this feature (i.e. this bug 588952) predates the version 3 concept of autosync for better faster imap, which is partly driven to service to gloda search ... where we also have Bug 551209 - Gloda should asynchronously enter/open folders during indexing
I'm not aware of a testcase. I would think your idea of triggering should work. Jcranmer or Gene might have thoughts.
|
||
Comment 40•5 years ago
|
||
TB starts up fine for me even with lots of test accounts and messages. Debug trunk builds are a bit slower, as expected. I usually have several folder scattered around with 17000+ messages and those don't cause a problem.
I want to get back to looking at the copying of large number of messages between imap accounts and to local so maybe I'll notice something that causes a major slowdown, but I doubt it.
|
|
||
Updated•5 years ago
|
|
|
||
Comment 41•5 years ago
|
||
| important | ||
(In reply to gene smith from comment #40)
TB starts up fine for me even with lots of test accounts and messages. Debug trunk builds are a bit slower, as expected. I usually have several folder scattered around with 17000+ messages and those don't cause a problem.
I want to get back to looking at the copying of large number of messages between imap accounts and to local so maybe I'll notice something that causes a major slowdown, but I doubt it.
The comments suggest this is more likely to be visible at 100k messages and more.
(In reply to Wayne Mery (:wsmwk) from comment #39)
(In reply to Ben Campbell from comment #38)
...
Stepping back a little, if it's the msgdb that's the speed bottleneck, then I'd say that real issue is that it's too slow (for whatever reason). The locally-stored db of metadata should be lightening-quick (relatively!) to open/scan. Even if you've got colossal quantities of email, it should really be on the order of seconds, not minutes...Yes, async is a workaround for improving the speed of opening the meta - but there's no other choice AFAIK until some distant future date when the backend gets reworked. In the mean time (including the past decade), message stores have on average gotten considerably bigger and we haven't improved our methods/performance in this area to keep pace.
Anyway - have we got replication steps for this one? If I set up a dummy local IMAP server with huge quantities of emails, do you think that would trigger it? Are we sure it's IMAP-only?
Yes, it would be imap-only - born out of https://wiki.mozilla.org/MailNews:Better_Faster_IMAP_Plan and Bug 470624 - RFE: On-demand Auto-Sync. Note however, the request for this feature (i.e. this bug 588952) predates the version 3 concept of autosync for better faster imap, which is partly driven to service to gloda search ... where we also have Bug 551209 - Gloda should asynchronously enter/open folders during indexing
I'm not aware of a testcase. I would think your idea of triggering should work. Jcranmer or Gene might have thoughts.
Rereading the bug report, I'm inclined to think db load is not the core issue and that async open would only help a little. I suspect the high mork activity seen by Ehsan was more symptom than cause, and the more likely cause is imap. Ref comment 11, comment 16, and comment 35. I also recall that Ehsan had server side filters, and we know from other reports that imap syncing related to server side filters can be costly.
We don't have all of Ehsan's traces but perhaps another possible factor is automatic compact, because compact can happen immediately on startup (AFAIK not delayed, unlike the purge service which delays itself on startup by 5 minutes). We might determine whether this is true, and determine the possible impact.
Comment 23 and comment 25 cite nsMsgPurgeService related to retention settings. The purge service should not greatly impact startup because the code states "if we've spent more than .5 seconds in this loop, don't apply any more retention settings because it might lock up the UI." - ref https://searchfox.org/comm-central/rev/dfafa758b061c4b4052899dd51d4304c066b249f/mailnews/base/src/nsMsgPurgeService.cpp#111 Those comments also state the purge service should not be invoked until 5 minutes after startup, which could clearly happen 5 minutes into Ehsan's "long startup". Also interesting is purge is invoked during "compact all folders" - which I had not known - but I think that is not the same as automatic compact. (purge checking every 5 minutes seems excessive, but that's a different matter)
|
|
||
Comment 42•5 years ago
|
||
(In reply to Wayne Mery (:wsmwk) from comment #41)
(In reply to gene smith from comment #40)
TB starts up fine for me even with lots of test accounts and messages. Debug trunk builds are a bit slower, as expected. I usually have several folder scattered around with 17000+ messages and those don't cause a problem.
I want to get back to looking at the copying of large number of messages between imap accounts and to local so maybe I'll notice something that causes a major slowdown, but I doubt it.The comments suggest this is more likely to be visible at 100k messages and more.
(In reply to Wayne Mery (:wsmwk) from comment #39)
(In reply to Ben Campbell from comment #38)
...
Stepping back a little, if it's the msgdb that's the speed bottleneck, then I'd say that real issue is that it's too slow (for whatever reason). The locally-stored db of metadata should be lightening-quick (relatively!) to open/scan. Even if you've got colossal quantities of email, it should really be on the order of seconds, not minutes...Yes, async is a workaround for improving the speed of opening the meta - but there's no other choice AFAIK until some distant future date when the backend gets reworked. In the mean time (including the past decade), message stores have on average gotten considerably bigger and we haven't improved our methods/performance in this area to keep pace.
Anyway - have we got replication steps for this one? If I set up a dummy local IMAP server with huge quantities of emails, do you think that would trigger it? Are we sure it's IMAP-only?
Yes, it would be imap-only - born out of https://wiki.mozilla.org/MailNews:Better_Faster_IMAP_Plan and Bug 470624 - RFE: On-demand Auto-Sync. Note however, the request for this feature (i.e. this bug 588952) predates the version 3 concept of autosync for better faster imap, which is partly driven to service to gloda search ... where we also have Bug 551209 - Gloda should asynchronously enter/open folders during indexing
I'm not aware of a testcase. I would think your idea of triggering should work. Jcranmer or Gene might have thoughts.
Rereading the bug report, I'm inclined to think db load is not the core issue and that async open would only help a little. I suspect the high mork activity seen by Ehsan was more symptom than cause, and the more likely cause is imap. Ref comment 11, comment 16, and comment 35. I also recall that Ehsan had server side filters, and we know from other reports that imap syncing related to server side filters can be costly.
however, comment 12 and comment 13 suggest async open as the solution (btw, the purge code comments indicate purge is also invoked on folder open.)
|
|
||
Comment 43•5 years ago
|
||
Sounds like fundamental problems in areas I've never tread, e.g., sync vs. async of autosync. However, with many messages in a folder the time to open the folder can get long due to having to fetch the folder flags for every message. The use of imap CONDSTORE can help in this regard but it's disabled by default. A while back a user had a problem and enabling CONDSTORE and fixed the "time to open large folder" problem. I had to make some tweaks to the code for it to work but don't remember what they were. Then again, this may be another problem and not the same as this bug.
|
|
||
Updated•4 years ago
|
|
|
||
Comment 44•4 years ago
|
||
So condstore (bug 1747311) and other imap issues should help more.
|
|
||
Updated•3 years ago
|
|
|
||
Updated•2 years ago
|
Description
•