117506 - IMAP startup over a 56k dialog connection with a big folder is unusably slow due to syncing flags

Status: RESOLVED WONTFIX

(MailNews Core :: Networking: IMAP, defect)

Description

[Syd Logan]

I'm a dialup user from home. Here are some wall-clock timings that illustrate
the problems I am seeing (which I really think are critical for dialup users,
and probably affect non-dialup users as well, but are less noticable).

7 seconds for it to display a dialog, "You are 92% over quota".
23 *more* seconds before the first content displays in the message window, ad
content from Netscape.com.
55 *more* seconds until that content loads.
65 *more* seconds before message summary pane content displays. During this
time there is a bunch of data receive indications on modem, so I suppose this
is traffic to the imap server.

My INBOX has on the order of 9000 messages. Perhaps excessive but no means
impossible. And, this is made worse by the fact that if I open up the folder
using Pine, I can login, dismiss the quota message, open INBOX, and see the
last 25 or so message headers all in a matter of 15 or so seconds.

I'm not sure what we are doing during that last 65 seconds, but it sure seems
like more than would be necessary to grab the 25 or so message headers that
are needed to fill the message header pane.

-- a second or two at most to get the number of messages in the INBOX.
-- some small amount of time to compute based on my viewing pref, e.g., sort
   by date, and the UIDs of the IMAP messages that must be retrieved to fill
   the message pane based upon what will be viewable.
-- a few seconds to retrieve the headers corresponding to those UIDs.

Now, I gather that all the messages in the mailbox have to be loaded into the
tree or outliner in order for the scrollbar size to be correct. I wonder if
there is some possible workaround there. Still, it shouldn't take a minute to
build that content. And it doesn't explain the blinky lights on my modem, which
imply we are reading headers for all of that content. Or something.

Note: I made sure there is nothing else happening on the system, specifically,
the Instant Messenger is not loading in the sidebar at the same time, and the
browser is not loading content in another window.
~

Comment 1

[scottputterman]

reassigning to cavin.

If you load the mail start page in the browser, how long does it take for you? 
If you have a 9000 message local folder, how long does that take?

Comment 2

[Syd Logan]

Note, that this only happens on a clean start. If I start the browser, bring up
mail, then I will see this. If I close down mail, then bring it back up with the
browser still in memory, the message summary pane shows almost immediately, and
the time it takes to get new messages seems reasonable to me.

Comment 3

[Syd Logan]

See also http://bugzilla.mozilla.org/show_bug.cgi?id=117560

Comment 4

[David :Bienvenu]

for that 65 seconds, we're syncing any possible flag changes to the folder,
i.e., fetching all the flags for all 9000 messages. If you generate an imap
protocol log, you'll get a much better idea of what we're doing during each phase.

In the past, we've talked about syncing the flags by doing three separate
searches instead of the one fetch of all flags - search for unread messages,
search for deleted messages, and one other search, iirc, and then deduce the
flags for each message from the search results. The theory is that the searches
would produce less data coming back from the server, and thus be faster over
slow connections.

We could also sync the flags after displaying the message headers, but then we
might have to delete headers out from under the view, update flags, etc, which
would produce extra complications.

Comment 5

[Syd Logan]

I'm not sure what those complications might be, or if they are worse in terms of
usability than seeing nothing in the message header pane for over 1 minute on
dialup. 

Anyway, for certain flags, is it possible that it is only necessary to have them
correctly sync'd if the header comes into view? This certainly would seem to be
the case for "read" status.

From RFC 2060, section 6.4.5:

 Example:    C: A654 FETCH 2:4 (FLAGS BODY[HEADER.FIELDS (DATE FROM)])
               S: * 2 FETCH ....
               S: * 3 FETCH ....
               S: * 4 FETCH ....
               S: A654 OK FETCH completed

this shows that flags can be fetched for a range of message numbers (in the
above example, 2:4 fetches for messages with 2, 3, and 4). UIDs could also be
used if you are maintaining them.

To find out what headers are in view, you could (approximately) compute the set
based upon a query of the scrollbar position, the font height, and the size of
the message header pane window. Then you could get flags for messages in some
neighborhood (a window) of the viewable headers, say, 150% (25% before and 25%
behind). You could keep track of what message number or UID ranges have had
their flags retrieved (a list of windows), and listen for scrollbar changes, and
then compute each time to see if there is a need to go get more flags because a
message that falls outside of the ranges in the window list has been scrolled
into view.

We should be able to detect a slow connection somehow (based on the connection
setup or the IMAP authentication phase). We also can easily determine if the
mailbox is large (like, 3000 messages or so, a number that we could figure out
experimentally). Using those two variables, perhaps we could determine that it
is better to grab flags on demand as opposed to grabbing them in bulk at startup
time. 

Or put up (and later dismiss) a dialog (with a cancel button perhaps) that shows
the status better (e.g., "Grabbing flags for messages" and a progress bar) than
the status bar does (which gets overridden with a "Document: Done (x,y secs)"
message when the ad content is loaded into the message view pane)? Otherwise,
this looks like my mail session has hung.

Comment 6

[David :Bienvenu]

Syd, we don't have time to implement what you describe in the near term. What
you describe has been 90%implemented before (in 4.x), but cut from the product,
and never has made the cut for 6.x. If I remember correctly, it was called imap
headers on demand, and was a bit simpler than you describe - when asked to paint
a line, if we didn't have the coresponding header, we would go fetch it and
paint it when it had been returned - no need to deal with listening for scroll
bar changes, etc.

Comment 7

[scottputterman]

Syd, if you have the time to measure this, what kind of times do you see for a
1000 message folder?

Comment 8

[David :Bienvenu]

there are lots of complications that would ensue if we have the wrong flags for
an existing header. Remember that IMAP is a server-based mail store, designed so
that you can access the same folder from multiple machines, and have clients
seemlessly handle changes made by other clients. Unfortunately, IMAP doesn't
give you any way of knowing that no other client has made changes since our
client last accessed the folder, so we have to assume the worst. For example, if
another client changed the read status of some headers, our read/unread counts
would be off unless we synced the flags. 

Comment 9

[Henrik Gemal]

Syd, please try accessing the same account with Outlook or Outlook Express 
(IMAP mode) and see if that's faster.

Comment 10

[David :Bienvenu]

taking; I'm working on syncing the flags after getting new headers and
displaying the thread pane.

Comment 11

[Wayne Mery (:wsmwk)]

Bienvenu in comment #10:
> taking; I'm working on syncing the flags after getting new headers and
> displaying the thread pane.

David, was the above finished?  Is there a bug to dupe to?

FWIW, related is bug 117560 and Syd is gone.

Comment 12

[David :Bienvenu]

No, that didn't happen...

Comment 13

[David :Bienvenu]

We're going to support CONDSTORE instead, and hope imap servers support that in droves. bug 435153