55126 - Mozilla unable to handle unsolicited IMAP responses.

Status: VERIFIED FIXED

(MailNews Core :: Networking: IMAP, defect, P2)

Description

[Sam Varshavchik]

From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux 2.2.16-22smp i686; en-US; m17) Gecko/20000821
BuildID:    2000082115


I'm going to start by pasting in an strace log of an IMAP server talking to
Mozilla.  This is an strace of the IMAP server, with everything removed
except the read/write calls to the socket.  Also, I'm manually wrapping the
long lines in the log.  A detailed explanation follows.

read(0, "9 select \"INBOX\"\r\n", 8192) = 18
write(1, "* FLAGS (\\Answered \\Flagged \\Deleted \\Seen \\Recent)\r\n* OK
[PERMANENTFLAGS (\\Answered \\Flagged \\Deleted \\Seen)] Limited\r\n
* 2 EXISTS\r\n* 0 RECENT\r\n* OK [UIDVALIDITY 939609418]\r\n9 OK [READ-WRITE]
Ok\r\n", 196) = 196
read(0, "10 UID fetch 1:* (FLAGS)\r\n", 8192) = 26
write(1, "* 1 FETCH (UID 10163 FLAGS (\\Seen))\r\n* 2 FETCH (UID 10387 FLAGS
(\\Seen))\r\n10 OK FETCH completed.\r\n", 98) = 98
read(0, "11 noop\r\n", 8192)            = 9
write(1, "* 3 EXISTS\r\n* 1 RECENT\r\n* 3 FETCH (FLAGS (\\Recent))\r\n11 OK
NOOP completed\r\n", 75) = 75
read(0, "12 UID fetch 10388:* (FLAGS)\r\n", 8192) = 30
write(1, "* 3 FETCH (UID 10479 FLAGS (\\Recent))\r\n12 OK FETCH
completed.\r\n", 63) = 63
read(0, "13 UID fetch -1,10479 (UID RFC822.SIZE FLAGS ENVELOPE
BODY.PEEK[HEADER.FIELDS (Priority X-Priority References
Newsgroups)])\r\n", 8192) = 125
write(1, "13 NO Error in IMAP command received by server.\r\n", 49) = 49

Mozilla is, apparently, gets confused if it receives unsolicited IMAP
replies (as defined by RFC 2060) during the processing of the NOOP command
to check for new mail.  In this case, the IMAP server being used always
sends unsolicited replies containing message status flags of newly arrived
messages in INBOX, during NOOP processing.  This is not a required, but
an allowed behavior according to RFC 2060, page 48:

                                     ... Server implementations that
   offer multiple simultaneous access to the same mailbox SHOULD also
   send appropriate unilateral untagged FETCH and EXPUNGE responses if
   another agent changes the state of any message flags or expunges any
   messages.

as well as page 57:

7.4.2.  FETCH Response

   Contents:   message data

      The FETCH response returns data about a message to the client.
      The data are pairs of data item names and their values in
      parentheses.  This response occurs as the result of a FETCH or
      STORE command, as well as by unilateral server decision (e.g. flag
      updates).         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      ^^^^^^^^^

Now, let's go back to the strace above.  Here, the folder contained two
messages, and one new message just arrived.  Clicking on "Get Messages"
in Mozilla, resulted in mozilla sending the following:

11 noop
* 3 EXISTS
* 1 RECENT

The IMAP server also sends the flags of the new, third message, as follows:

* 3 FETCH (FLAGS (\\Recent))

Not all servers do that, as part of NOOP processing, they are not required
to do so.  It is acceptable for the IMAP client to subsequently send an
explicit request for message status flags, as well as any other needed
metadata.  But, in this case, the server sends the message flags to the
client, hoping that the client can use the information.  Finally, the
server acknowledges the NOOP command:

11 OK NOOP completed

In the previous example, the crucial step is the unsolicited FETCH
message from the server.  That's where things apparently go wrong somewhere
in Mozilla.  If the server does not send that unsolicited response, Mozilla
will not act the way it is about to act.  The next command from Mozilla
appears to be quiet innocent:

12 UID fetch 10388:* (FLAGS)

It obviously ignored the FLAGS update, and wants to explicitly ask for
message flags from the server.  That's OK.

* 3 FETCH (UID 10479 FLAGS (\\Recent))
12 OK FETCH completed.

And here's the next command from Mozilla, the real whopper:

13 UID fetch -1,10479 (UID RFC822.SIZE FLAGS ENVELOPE BODY.PEEK[HEADER.FIELDS
(Priority X-Priority References Newsgroups)])

Mozilla apparently wants to know the flags for message #-1.  Message sequence
numbers are defined by RFC 2060 as unsigned positive integers, and the
above IMAP command is syntactically invalid.  The negative sign is an
invalid lexical token in RFC 2060.  The IMAP server has every right to
acknowledge that command as follows:

13 NO Error in IMAP command received by server.

At this point, Mozilla mail/news is hopelessly confused.  The only way
to clear the condition is to kill Mozilla entirely (shutting down and
restarting mail/news doesn't work, the connection is still kept alive,
and Mozilla's internal data remains in inconsistent state).

This behavior is 100% repeatable as long as the IMAP server sends the
unsolicited FETCH reply before NOOP completes.  If that FETCH does not
get sent, Mozilla's fine.

I am the author of the GPL IMAP server in question.  This problem is not new
to Mozilla, it goes back at least to Netscape 4.0's mail/news clients,
and I've been receiving flames from Netscape Messenger's users for three
years now.  Nothing would please me more than to have this issue finally
nailed and resolved.  Thank you.


Reproducible: Always
Steps to Reproduce:
1. Log on to an IMAP server that sends unsolicited messages as part of NOOP
processing.

2. Open INBOX

3. Send a test message to the INBOX

4. Click "Get Messages" in Mozilla.



Actual Results:  The error from the IMAP server (see above), gets reported by
Mozilla, and basically things come to a halt there.



Expected Results:  New message to show up in the folder index.

The IMAP server in question: http://www.courier-mta.org/

Comment 1

[Antony Mawer]

I'm using the Courier IMAP server, and this bug basically renders Mail 
completely useless to me, because I can't check my mail properly. On some 
folders I just can't get the message listing at all. Maybe I'll have to 
recompile with the Netscape 4.x work-arounds enabled... I've been using Outlook 
Express for email until now, and I'd hoped when Mozilla came out, it'd no 
longer have the same bugs that it's predecessor had so I wouldn't have to worry 
about dodgy work-arounds. 

Oh, and this is on Win2000, I'm guessing this should probably be OS/Platform = 
All...

Comment 2

[Asa Dotzler [:asa]]

setting bug status to New

Comment 3

[David :Bienvenu]

I suspect this is a dup of a bug I have.

Comment 4

[David :Bienvenu]

changing priorities

Comment 5

[Antony Mawer]

Is there any progress on this bug, or is it in limbo?

It looks as though bugs 60360, 57844 and 63854 may be related to this bug.

Comment 6

[Sam Varshavchik]

The other three bugs are probably a different issue.  This bug strictly affects
checking for new messages only, and Netscape/Mozilla issues a pretty noticeable
error when that happens.  The scope of this bug is confined entirely to new mail
checking.

Comment 7

[David :Bienvenu]

I believe this should be fixed now. I don't think it's the unsolicitedness of
the response, but the fact that we're not getting the UID of the new message in
the response. Please retry this and re-open if it still happens.

Comment 8

[daveb]

Was the fix checked in after the 0.9.2 build? If not, it is still open. Looking
over the server response code, I didn't quickly find anything that jumped out as
causing the -1 message number to work it's way into the future fetch requests.

Comment 9

[David :Bienvenu]

yes, well after .9.2, and even partially after .9.3.

Comment 10

[Henrik Gemal]

the -1 issue in the fetch command is fixed

build 20011018