129495 - Copying mail to large IMAP sent folder takes unusually long

Status: RESOLVED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[Jason Tackaberry]

This bug has been present for quite some time.  I originally mistook it for bug
#89285, but it's different because it never actually fails.

When the Sent folder is a remote IMAP folder with thousands of messages, after
sending an email, copying that email to the sent folder can take anywhere from
10-40 seconds.  On the client end, it sits idle at 0% CPU with the "Copying to
Sent folder" progress bar, apparently waiting for the server.   The IMAP server
process is running at 100% CPU.  It is Courier IMAP 1.3.5.  The Sent folder
holds 6000 emails.

It does seem like a problem on the server side, except that no other email
client, including Netscape Messenger 4.7x, behaves this way.  All other clients
copy the email to the large sent folder instantly.

An exception to the "the client idles at 0%" above is when Sent.msf does not
exist.  Then the process takes an extra 10-20 seconds and the CPU usage maxes out.

Comment 1

[Jason Tackaberry]

After looking at the IMAP session, it looks like the slowness is coming from a
"uid SEARCH SEEN HEADER" command.  (Disclaimer: I don't really know the IMAP
protocol.)

Here is a capture of the commands issued by Mozilla (text returned by the server
deleted):

1 login "tack" "<CENSORED :)>"
2 append "INBOX.Sent2" (\Seen) {391}
  < ... insert message text here ...>
3 select "INBOX.Sent2"
4 UID fetch 1:* (FLAGS)
5 UID fetch 13641 (UID RFC822.SIZE BODY.PEEK[HEADER] FLAGS)
6 uid SEARCH SEEN HEADER Message-ID [e-mail]

Note that line 6 is what takes all the time (10-40 seconds).  Compare this to
Netscape Messenger which sends only lines 1 and 2, and not the last 4.

Also interesting to note that line 4 causes the server to generate 1 line for
every message in the mailbox that looks like this for example:
 
   * 2 FETCH (UID 7461 FLAGS (\Seen))

In my Sent folder which has 6200 emails, that means 6200 lines of text.  The
entire session from the above commands resulted in 250k of text.  I don't notice
it over my local network, but to a modem user the added time to transfer the
session compounded with the 10-30 seconds from line 6 would make Mozilla mail
completely unusable.

Comment 2

[Jim Laurent]

During this time the number of "Total Messages" in the mail folders pane is
incrementing with the number of messages in the Sent folder as it's counting.

Comment 3

[David :Bienvenu]

I'll look into this. I have an idea what's going on here.

Comment 4

[David :Bienvenu]

the code that tries to handle appending to drafts and templates folders (which
needs to know the uid of the appended message) is unfortunately getting invoked
for other appends to local folders, thus slowing down all appends to folders.
I'll try to find the fix that allows things like drafts and templates to keep
working.

Comment 5

[Karen Huang]

*** Bug 116018 has been marked as a duplicate of this bug. ***

Comment 6

[David :Bienvenu]

Attachment: proposed fix

the fix is not to get into the code that selects and searches the folder,
unless we're saving a draft/template. I've tried various operations (cross
server imap move/copy, undo, etc) and they work the same as they did without
the change. Which is to say that undo of a cross server imap move/copy doesn't
remove the dest messages in either case. I believe the code I've removed might
have been trying to handle that case by saving away the msg keys in the dest
folder, but it doesn't work anyway. A better way of handling this would be to
actually do the search for message-ids at the time the user tries to undo.

Comment 7

[David :Bienvenu]

adding Navin for review, since this might come into play later for undo of cross
imap server move/copy.

Comment 8

[(not reading, please use [e-mail] instead)]

Comment on attachment 91979 [details] [diff] [review]
proposed fix

sr=sspitzer

Comment 9

[Navin Gupta]

Comment on attachment 91979 [details] [diff] [review]
proposed fix

r=naving, but please don't
check this is in now, only 
when the tree opens after 1.1 beta. we could be missing some case here.

Comment 10

[David :Bienvenu]

yes, that was my thought, exactly. I've run with this for a while with
breakpoints set in the interesting places and haven't seen any cases, but I was
planning on waiting until the tree opens for general checkins.

Comment 11

[Boris 'pi' Piwinger]

Are those bugs dupes or related in another way?

bug 89285
bug 102816
bug 103936
bug 104481
bug 147285

pi

Comment 12

[David :Bienvenu]

fixed on trunk.

Comment 13

[Navin Gupta]

This fix broke undo move msgs from local folder to imap folder. 

Comment 14

[Navin Gupta]

logged bug 162008 about it.

Comment 15

[Henry Jia]

*** Bug 103936 has been marked as a duplicate of this bug. ***

Comment 16

[Henry Jia]

*** Bug 102816 has been marked as a duplicate of this bug. ***

Comment 17

[R.K.Aa.]

*** Bug 163027 has been marked as a duplicate of this bug. ***

Comment 18

[Boris 'pi' Piwinger]

*** Bug 104481 has been marked as a duplicate of this bug. ***

Comment 19

[grylchan]

can't verify this on courier since we don't have
access to a courier server.

will verify on cyrus when bug 27002 is fixed.