I'm a dialup user from home. Here are some wall-clock timings that illustrate the problems I am seeing (which I really think are critical for dialup users, and probably affect non-dialup users as well, but are less noticable). 7 seconds for it to display a dialog, "You are 92% over quota". 23 *more* seconds before the first content displays in the message window, ad content from Netscape.com. 55 *more* seconds until that content loads. 65 *more* seconds before message summary pane content displays. During this time there is a bunch of data receive indications on modem, so I suppose this is traffic to the imap server. My INBOX has on the order of 9000 messages. Perhaps excessive but no means impossible. And, this is made worse by the fact that if I open up the folder using Pine, I can login, dismiss the quota message, open INBOX, and see the last 25 or so message headers all in a matter of 15 or so seconds. I'm not sure what we are doing during that last 65 seconds, but it sure seems like more than would be necessary to grab the 25 or so message headers that are needed to fill the message header pane. -- a second or two at most to get the number of messages in the INBOX. -- some small amount of time to compute based on my viewing pref, e.g., sort by date, and the UIDs of the IMAP messages that must be retrieved to fill the message pane based upon what will be viewable. -- a few seconds to retrieve the headers corresponding to those UIDs. Now, I gather that all the messages in the mailbox have to be loaded into the tree or outliner in order for the scrollbar size to be correct. I wonder if there is some possible workaround there. Still, it shouldn't take a minute to build that content. And it doesn't explain the blinky lights on my modem, which imply we are reading headers for all of that content. Or something. Note: I made sure there is nothing else happening on the system, specifically, the Instant Messenger is not loading in the sidebar at the same time, and the browser is not loading content in another window. ~
reassigning to cavin. If you load the mail start page in the browser, how long does it take for you? If you have a 9000 message local folder, how long does that take?
Assignee: mscott → cavin
Note, that this only happens on a clean start. If I start the browser, bring up mail, then I will see this. If I close down mail, then bring it back up with the browser still in memory, the message summary pane shows almost immediately, and the time it takes to get new messages seems reasonable to me.
for that 65 seconds, we're syncing any possible flag changes to the folder, i.e., fetching all the flags for all 9000 messages. If you generate an imap protocol log, you'll get a much better idea of what we're doing during each phase. In the past, we've talked about syncing the flags by doing three separate searches instead of the one fetch of all flags - search for unread messages, search for deleted messages, and one other search, iirc, and then deduce the flags for each message from the search results. The theory is that the searches would produce less data coming back from the server, and thus be faster over slow connections. We could also sync the flags after displaying the message headers, but then we might have to delete headers out from under the view, update flags, etc, which would produce extra complications.
I'm not sure what those complications might be, or if they are worse in terms of usability than seeing nothing in the message header pane for over 1 minute on dialup. Anyway, for certain flags, is it possible that it is only necessary to have them correctly sync'd if the header comes into view? This certainly would seem to be the case for "read" status. From RFC 2060, section 6.4.5: Example: C: A654 FETCH 2:4 (FLAGS BODY[HEADER.FIELDS (DATE FROM)]) S: * 2 FETCH .... S: * 3 FETCH .... S: * 4 FETCH .... S: A654 OK FETCH completed this shows that flags can be fetched for a range of message numbers (in the above example, 2:4 fetches for messages with 2, 3, and 4). UIDs could also be used if you are maintaining them. To find out what headers are in view, you could (approximately) compute the set based upon a query of the scrollbar position, the font height, and the size of the message header pane window. Then you could get flags for messages in some neighborhood (a window) of the viewable headers, say, 150% (25% before and 25% behind). You could keep track of what message number or UID ranges have had their flags retrieved (a list of windows), and listen for scrollbar changes, and then compute each time to see if there is a need to go get more flags because a message that falls outside of the ranges in the window list has been scrolled into view. We should be able to detect a slow connection somehow (based on the connection setup or the IMAP authentication phase). We also can easily determine if the mailbox is large (like, 3000 messages or so, a number that we could figure out experimentally). Using those two variables, perhaps we could determine that it is better to grab flags on demand as opposed to grabbing them in bulk at startup time. Or put up (and later dismiss) a dialog (with a cancel button perhaps) that shows the status better (e.g., "Grabbing flags for messages" and a progress bar) than the status bar does (which gets overridden with a "Document: Done (x,y secs)" message when the ad content is loaded into the message view pane)? Otherwise, this looks like my mail session has hung.
Syd, we don't have time to implement what you describe in the near term. What you describe has been 90%implemented before (in 4.x), but cut from the product, and never has made the cut for 6.x. If I remember correctly, it was called imap headers on demand, and was a bit simpler than you describe - when asked to paint a line, if we didn't have the coresponding header, we would go fetch it and paint it when it had been returned - no need to deal with listening for scroll bar changes, etc.
Syd, if you have the time to measure this, what kind of times do you see for a 1000 message folder?
there are lots of complications that would ensue if we have the wrong flags for an existing header. Remember that IMAP is a server-based mail store, designed so that you can access the same folder from multiple machines, and have clients seemlessly handle changes made by other clients. Unfortunately, IMAP doesn't give you any way of knowing that no other client has made changes since our client last accessed the folder, so we have to assume the worst. For example, if another client changed the read status of some headers, our read/unread counts would be off unless we synced the flags.
Syd, please try accessing the same account with Outlook or Outlook Express (IMAP mode) and see if that's faster.
taking; I'm working on syncing the flags after getting new headers and displaying the thread pane.
Assignee: cavin → bienvenu
Status: ASSIGNED → NEW
Bienvenu in comment #10: > taking; I'm working on syncing the flags after getting new headers and > displaying the thread pane. David, was the above finished? Is there a bug to dupe to? FWIW, related is bug 117560 and Syd is gone.
No, that didn't happen...
QA Contact: grylchan → networking.imap
Summary: IMAP startup over a 56k dialog connection with a big folder is unusably slow → IMAP startup over a 56k dialog connection with a big folder is unusably slow due to syncing flags
We're going to support CONDSTORE instead, and hope imap servers support that in droves. bug 435153
Status: NEW → RESOLVED
Last Resolved: 10 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.