59704 - support for IMAP Referrals (RFC2221, RFC2193), when server advertises MAILBOX-REFERRALS we should use RLIST and RLSUB (will help with MS Exchange and others)

Status: NEW

(MailNews Core :: Networking: IMAP, enhancement)

Description

[David :Bienvenu]

Add support for IMAP referrals, including using APPEND when copy between folders
fails with a REFERRAL response.

Comment 1

[David :Bienvenu]

accepting - this will help with MS Exchange interoperability.

Comment 2

[Karen Huang]

Adding MS Exchange IMAP on the summary.

Comment 3

[David :Bienvenu]

IMAP referrals are not limited to MS Exchange. It's a general standard.

Comment 4

[David :Bienvenu]

p5

Comment 5

[David :Bienvenu]

correcting summary - IMAP referrals are not a proprietary MS Exchange feature.

Comment 6

[David :Bienvenu]

Here's the RFC: http://www.faqs.org/rfcs/rfc2193.html

In essence, if the server advertises the MAILBOX-REFERRALS, then the client is
supposed to use RLIST and RLSUB instead of LIST and LSUB. Any command that takes
a mailbox name can return NO and one or more IMAP URLS that are referrals to the
server and folder to use instead.

 "When a client processes an IMAP mailbox referral, it will open a new
   connection or use an existing connection to the remote server so that
   it is able to issue the commands necessary to process the remote
   mailbox"

So our imap parser would need to be extended to handle getting referral
responses, and acting on them. There are two problems here - one is that we need
to be able to run the new url just as if it was the original url, and the other
is that we need to be able to connect and login to the remote server and folder
w/o their being an account in the acct mgr for the remote server. And we need to
have the connection mgmt that's handled by nsImapIncomingServer to re-use
existing connections, etc. This will all be quite tricky to do cleanly. I don't
want there to be dozens of checks throughout the code that we're handling an
imap referral.

The first question is can we re-use the existing url, or do we need to create a
new url to run the referral? Logically, for the clients of the url, it should
look like we're still running the first url so that the client gets the correct
notifications, etc. The url doesn't know much about the protocol object, so it
is possible we could get/create an nsImapIncomingServer object for the referred
server, get/create a protocol object, and then associate the original url with
the protocol object, and run the url. We'd have to set the authentication info
from the referral on the protocol object, so it would know to use it. We'd need
to clean up the original protocol object appropriately, w/o setting the url state...

Anyway, this is a difficult but fun problem to solve.

Comment 7

[Alec Flett]

I now have to use an exchange server which requires IMAP referrals right at
login. Perhaps we can at least support just the login aspect to start? It makes
it very frustrating to configure a new account - you have to set it up to talk
to the parent server (i.e. ms-exchange.foo.com) then login, and read the error
message, and then edit the server properties to change it (i.e. to
some-referral.foo.com)

Comment 8

[David :Bienvenu]

Alec, no promises, but can you attach a protocol log of what you're server is
sending you? thx.

Comment 9

[Alec Flett]

Attachment: IMAP log

As promised.. (domain replaced with foo.com)

Comment 10

[David :Bienvenu]

logon referrals sound like a job for a special logon redirector but maybe it
could be done in-line in the imap code.

Comment 11

[David :Bienvenu]

*** Bug 300367 has been marked as a duplicate of this bug. ***

Comment 12

[David McBride]

Just to be clear, there are two different classes of IMAP referral -- Login Referrals (as defined by RFC 2221) and Mailbox Referrals (as defined by RFC 2193).

In addition to being useful for Exchange interoperability, widespread support for these standards would also be useful for other IMAP server aggregation schemes.

(At present, the only effective way to aggregate a set of IMAP servers is with a relaying proxy server like Perdition, which introduces obvious security and scalability issues.)

Votes +=1.

Comment 13

[Wayne Mery (:wsmwk)]

improved summary for searchability.

since current behavior violates RFC, would it be appropriate to change severity to something other than ENH?

Comment 14

[David :Bienvenu]

the current behavior doesn't violate the core IMAP RFC, afaik - we just don't support the REFERRAL extension RFC's.

Comment 18

[micah]

This may not be obvious, but the Referral RFCs, of which there are two, RFC 2193 and RFC2221 (login vs. mailbox referrals) are not specifically for supporting Microsoft products, such as Exchange. These RFCs would be a scaling win for mail server operators if clients supported the RFCs. 

Pine/Alpine support these RFCs, and the Outlook suite of products do as well. Its unfortunte that Thunderbird does not.

The dovecot IMAP server does support these RFCs, but without client support in Thunderbird it isn't very useful.

Comment 19

[Adrien de Croy]

+1 to 2221, I don't really mind about MAILBOX-REFERRALS, but LOGIN-REFERRAL should be fairly simple and very useful.

Comment 20

[Adrien de Croy]

Actually looks like Outlook, Outlook Express and Exchange do NOT support these.

http://support.microsoft.com/kb/217388
http://technet.microsoft.com/en-us/library/bb310763.aspx

Comment 21

[Wayne Mery (:wsmwk)]

rkent would you have an interest in this?

Comment 22

[Wayne Mery (:wsmwk)]

Comment on attachment 147913 [details]
IMAP log

>0[294518]: 1e36c28:ms-exchange.foo.com:NA:SetupWithUrl: clearing IMAP_CONNECTION_IS_OPEN
>3364[1cbdd60]: ImapThreadMainLoop entering [this=1e36c28]
>3364[1cbdd60]: 1e36c28:ms-exchange.foo.com:NA:ProcessCurrentURL: entering
>3364[1cbdd60]: 1e36c28:ms-exchange.foo.com:NA:ProcessCurrentURL:imap://[e-mail]:143/select%3E%5EINBOX:  = currentUrl
>3364[1cbdd60]: ReadNextLine [stream=1cf71f0 nb=98 needmore=0]
>3364[1cbdd60]: 1e36c28:ms-exchange.foo.com:NA:CreateNewLineFromSocket: * OK Microsoft Exchange IMAP4rev1 server version 5.5.2657.74 (ex-bridge-02.foo.com) ready

>3364[1cbdd60]: 1e36c28:ms-exchange.foo.com:NA:SendData: 1 capability

>3364[1cbdd60]: ReadNextLine [stream=1cf71f0 nb=98 needmore=0]
>3364[1cbdd60]: 1e36c28:ms-exchange.foo.com:NA:CreateNewLineFromSocket: * CAPABILITY IMAP4 IMAP4rev1 IDLE LITERAL+ LOGIN-REFERRALS MAILBOX-REFERRALS NAMESPACE AUTH=NTLM

>3364[1cbdd60]: ReadNextLine [stream=1cf71f0 nb=28 needmore=0]
>3364[1cbdd60]: 1e36c28:ms-exchange.foo.com:NA:CreateNewLineFromSocket: 1 OK CAPABILITY completed.

>3364[1cbdd60]: 1e36c28:ms-exchange.foo.com:NA:SendData: Logging suppressed for this command (it probably contained authentication information)
>3364[1cbdd60]: ReadNextLine [stream=1cf71f0 nb=82 needmore=0]
>3364[1cbdd60]: 1e36c28:ms-exchange.foo.com:NA:CreateNewLineFromSocket: 2 OK [REFERRAL imap://aflett;AUTH=*@s2009exm01.foo.com/] LOGIN completed.

>3364[1cbdd60]: 1e36c28:ms-exchange.foo.com:A:SendData: 3 namespace

>3364[1cbdd60]: ReadNextLine [stream=1cf71f0 nb=54 needmore=0]
>3364[1cbdd60]: 1e36c28:ms-exchange.foo.com:A:CreateNewLineFromSocket: * NAMESPACE (("" "/")) NIL (("Public Folders/" "/"))

>3364[1cbdd60]: ReadNextLine [stream=1cf71f0 nb=39 needmore=0]
>3364[1cbdd60]: 1e36c28:ms-exchange.foo.com:A:CreateNewLineFromSocket: 3 OK NAMESPACE completed successfully

>3364[1cbdd60]: 1e36c28:ms-exchange.foo.com:A:SendData: 4 lsub "" "*"

>3364[1cbdd60]: ReadNextLine [stream=1cf71f0 nb=119 needmore=0]
>3364[1cbdd60]: 1e36c28:ms-exchange.foo.com:A:CreateNewLineFromSocket: 4 NO [REFERRAL imap://aflett;AUTH=*@s2009exm01.foo.com/] The requested mailbox is not available on this server

>3364[1cbdd60]: 1e36c28:ms-exchange.foo.com:A:SendData: 5 lsub "" "Public Folders/*"

>3364[1cbdd60]: ReadNextLine [stream=1cf71f0 nb=119 needmore=0]
>3364[1cbdd60]: 1e36c28:ms-exchange.foo.com:A:CreateNewLineFromSocket: 5 NO [REFERRAL imap://aflett;AUTH=*@s2009exm01.foo.com/] The requested mailbox is not available on this server

>3364[1cbdd60]: 1e36c28:ms-exchange.foo.com:A:SendData: 6 list "" "INBOX"

>3364[1cbdd60]: ReadNextLine [stream=1cf71f0 nb=59 needmore=0]
>3364[1cbdd60]: 1e36c28:ms-exchange.foo.com:A:CreateNewLineFromSocket: 6 NO There is no replica for that mailbox on this server.

>3364[1cbdd60]: 1e36c28:ms-exchange.foo.com:A:SendData: 7 list "" "Trash"

>3364[1cbdd60]: ReadNextLine [stream=1cf71f0 nb=59 needmore=0]
>3364[1cbdd60]: 1e36c28:ms-exchange.foo.com:A:CreateNewLineFromSocket: 7 NO There is no replica for that mailbox on this server.

>3364[1cbdd60]: 1e36c28:ms-exchange.foo.com:A:SendData: 8 create "Trash"

>3364[1cbdd60]: ReadNextLine [stream=1cf71f0 nb=59 needmore=0]
>3364[1cbdd60]: 1e36c28:ms-exchange.foo.com:A:CreateNewLineFromSocket: 8 NO There is no replica for that mailbox on this server.

>3364[1cbdd60]: 1e36c28:ms-exchange.foo.com:A:SendData: 9 select "INBOX"

>3364[1cbdd60]: ReadNextLine [stream=1cf71f0 nb=59 needmore=0]
>3364[1cbdd60]: 1e36c28:ms-exchange.foo.com:A:CreateNewLineFromSocket: 9 NO There is no replica for that mailbox on this server.

>0[294518]: 1e36c28:ms-exchange.foo.com:A:SendData:TellThreadToDie: 10 logout

>0[294518]: 1e36c28:ms-exchange.foo.com:A:TellThreadToDie: close socket connection
>3364[1cbdd60]: ImapThreadMainLoop leaving [this=1e36c28]

Comment 23

[Kent James (:rkent)]

(In reply to Wayne Mery (:wsmwk) from comment #21)
> rkent would you have an interest in this?

Probably not. I've never attempted to work in the IMAP protocol area.

Comment 24

[David Lechner (:dlech)]

Also see Bug 495318. LIST-EXTENDED (RFC 5258) provides an alternate way to get the same information.

This bug might have my name on it!