564642 - add pref to disable the automatic CAPABILITY. [IMAP CAPABILITY command is not issued even though optional CAPABILITY response is not returned with OK response to LOGIN]

Status: NEW

(MailNews Core :: Networking: IMAP, enhancement)

Description

[d0m1n1k]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 (.NET CLR 3.5.30729)
Build Identifier: rerequest imap capabilities (to enable idle) after login 

a telnet session to see the different capability answers and that the server really supports idle (2 message announcements at the end)
maybe this affects more things in thunderbird, which rely on the capability answer?

* OK IMAP4 Ready lbmfim03.lix.bon.at 0001dd18
001 CAPABILITY
* CAPABILITY IMAP4 IMAP4REV1
001 OK CAPABILITY
002 LOGIN [user_removed] [password_removed]
002 OK You are so in
003 CAPABILITY
* CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID LOGINDISABLED ACL RIGHTS=kxte QUOTA MA
LBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND
BINARY SORT SORT=MODSEQ THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE CA
ENATE CONDSTORE SCAN IDLE LISTEXT LIST-SUBSCRIBED X-NETSCAPE URLAUTH
003 OK Completed
004 SELECT INBOX
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen NonJunk)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen NonJunk \*)]
* 1 EXISTS
* 0 RECENT
* OK [UNSEEN 1]
* OK [UIDVALIDITY 1257502569]
* OK [UIDNEXT 1008]
* OK [NOMODSEQ] Sorry, modsequences have not been enabled on this mailbox
* OK [URLMECH INTERNAL]
004 OK [READ-WRITE] Completed
005 IDLE
+ idling
* 2 EXISTS
* 1 RECENT
* 3 EXISTS
* 2 RECENT

Reproducible: Always

Steps to Reproduce:
1. check if the imap server supports idle
2. check the capability response before and after login; if the server tells you that he can idle BEFORE login, everything's ok?
3. activate the "use imap if available" option at the extended settings for an imap account
4. write a mail to the idle enabled account
5. wait ... (nothing happens)
Actual Results:  
Thunderbird didn't idle (because of the first capability answer) and checked the account periodically

Expected Results:  
thunderbird should check after login (move the check to this point?) and use idle if the server supports it

Comment 1

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)
Which version of Thunderbird?(paste build ID in Help/About Thunderbird, please)

> * OK IMAP4 Ready lbmfim03.lix.bon.at 0001dd18
> 001 CAPABILITY
> * CAPABILITY IMAP4 IMAP4REV1
> 001 OK CAPABILITY
> 002 LOGIN [user_removed] [password_removed]
> 002 OK You are so in
> 003 CAPABILITY
> * CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID LOGINDISABLED ACL RIGHTS=kxte QUOTA MA
> LBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND
> BINARY SORT SORT=MODSEQ THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE CA
> ENATE CONDSTORE SCAN IDLE LISTEXT LIST-SUBSCRIBED X-NETSCAPE URLAUTH
> 003 OK Completed

No CAPABILITY response code in the initial PREAUTH nor OK response to login.
Tb doesn't issue CAPABILITY command after login in this case?
Affected by bug 401293 and bug 470650?

Can you get IMAP log with timestamp?
> https://wiki.mozilla.org/MailNews:Logging

> 3. activate the "use imap if available" option at the extended settings for an
imap account

"use IDLE command if the server supports it"? "TLS, if avail" of Tb 2?
Does "activate" mean "changed from unchecked to checked"?
If yes, which folder? Inbox? If so, restart or user action on Inbox may be needed after the change, because IDLE is issued at end of action(process of IMA command and response) on a cached connection. It looks that no action on Inbox is invoked by you.

Comment 2

[d0m1n1k]

(In reply to comment #1)
> (In reply to comment #0)
> Which version of Thunderbird?(paste build ID in Help/About Thunderbird, please)
> Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.5pre) Gecko/20100430 Lightning/1.0b2pre Thunderbird/3.1b2

> No CAPABILITY response code in the initial PREAUTH nor OK response to login.
> Tb doesn't issue CAPABILITY command after login in this case?
> Affected by bug 401293 and bug 470650?
> 
> Can you get IMAP log with timestamp?
> > https://wiki.mozilla.org/MailNews:Logging
> 
> > 3. activate the "use imap if available" option at the extended settings for an
> imap account
> 
> "use IDLE command if the server supports it"? "TLS, if avail" of Tb 2?
> Does "activate" mean "changed from unchecked to checked"?
> If yes, which folder? Inbox? If so, restart or user action on Inbox may be
> needed after the change, because IDLE is issued at end of action(process of IMA
> command and response) on a cached connection. It looks that no action on Inbox
> is invoked by you.
i've tried with and without ssl/tls.

in fact, i had the idea that it could be somewhat in that direction and verified with wireshark.
so tb checked it, and then authenticated the user.
i've also switched the "use IDLE ..." option some times and restarted tb afterwards.

i'll attach a imap log in a few hours.

Comment 3

[d0m1n1k]

Attachment: the requested imap log with timestamp

the requested imap log (inc timestamps)

shows what wireshark already confirmed, no idle capability in pre-auth response, no capability request after auth, no idle command after loading a e-mail.

Comment 4

[WADA:World Anti-bad-Duping Agency]

> * OK IMAP4 Ready lbmfim01.lix.bon.at 0001e5ac
> 1 capability
> * CAPABILITY IMAP4 IMAP4REV1
> try to log in
> IMAP: trying auth method 0x4
> old-style auth
> Logging suppressed for this command
> 3 OK You are so in
> login succeeded
> 4 lsub "" "*"

Although no CAPABILITY response in OK, CAPABILITY is not issued after login.
Due to existence of " You are so in" in OK response?
  if "OK" only, issue CAPABILITY, else if CAPABILTY response exists, merge it,
  else, do nothing?

Confirming per log.

Comment 5

[WADA:World Anti-bad-Duping Agency]

CAPABILITY command is issed by Capability(), and is called at next.
  #1487 when preauthenticated
  #1629 if capability response is not received yet
  #1667 just after StartTLS
> http://mxr.mozilla.org/comm-central/source/mailnews/imap/src/nsImapProtocol.cpp#1487
> http://mxr.mozilla.org/comm-central/source/mailnews/imap/src> /nsImapProtocol.cpp#1629
> http://mxr.mozilla.org/comm-central/source/mailnews/imap/src/nsImapProtocol.cpp#1667

RFC 3501 CAPABILITY Response section.
> http://tools.ietf.org/html/rfc3501#section-7.2.1
> 7.2.1. CAPABILITY Response
      A server MAY send capabilities automatically, by using the
      CAPABILITY response code in the initial PREAUTH or OK responses,
      and by sending an updated CAPABILITY response code in the tagged
      OK response as part of a successful authentication.  It is
      unnecessary for a client to send a separate CAPABILITY command if
      it recognizes these automatic capabilities.

For updated capability, above is read as;
      A server MAY send capabilities automatically, (snip)
      and by sending an updated CAPABILITY response code in the tagged
      OK response as part of a successful authentication.
Tb's interpretation may be; 
  As server didn't send "CAPABILITY response code in the tagged
  OK response as part of a successful authentication",
  and as "server MAY send", 
  no update of capability after previous response to CAPABILITY command.
Your server's interpretation seems;
  Server doesn't need to send "CAPABILITY response code in the tagged
  OK response as part of a successful authentication",
  even when update of capability occurs after previouse response to
  CAPABILITY command from client.
  So client has to send CAPABILITY command after authentication.

I can't find statement of "client MUST send CAPABILITY after authentication".

Comment 6

[d0m1n1k]

(In reply to comment #5)
> > 7.2.1. CAPABILITY Response
>       A server MAY send capabilities automatically, by using the
>       CAPABILITY response code in the initial PREAUTH or OK responses,
>       and by sending an updated CAPABILITY response code in the tagged
>       OK response as part of a successful authentication.  It is
>       unnecessary for a client to send a separate CAPABILITY command if
>       it recognizes these automatic capabilities.
> 
> For updated capability, above is read as;
>       A server MAY send capabilities automatically, (snip)
>       and by sending an updated CAPABILITY response code in the tagged
>       OK response as part of a successful authentication.

i think that it could be also read as:
if the client didn't recognized the automatic capabilities, it should request the CAPABILITY in a seperate command.

isn't there a additional check if after OK follows a CAPABILITY string? (should the response should be that way?)
so that tb don't thinks "you are so in" is a capability response, with capabilitys you; are; ... it can't use?

Comment 7

[Nikolay Shopik]

RFC3501 section 7.1 says - Status responses MAY include an OPTIONAL "response code".

Comment 8

[Nikolay Shopik]

forgot to add according David comment#38 in bug 401293 - we now parse the capability response whether or not we issued a capability command.

Comment 9

[d0m1n1k]

well, it seems that a check somewhere next to http://mxr.mozilla.org/comm-central/source/mailnews/imap/src/nsImapProtocol.cpp#1457 fails, else it wouldn't try to fake a answer, when no capability string was in the response code.

after i added here http://mxr.mozilla.org/comm-central/source/mailnews/imap/src/nsImapProtocol.cpp#8340 another Capability(); command, i have idle, quota and all the other fancy stuff the server offers :)

(would be just a bit better if it would be that way out of the box; independent if the server adds uncommon strings in its responses as long as they don't start with CAPABILITY ;) )

Comment 10

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #6)
> i think that it could be also read as:
> if the client didn't recognized the automatic capabilities, it should request
> the CAPABILITY in a seperate command.

I can't say "should request CAPABILITY", but I believe "better to request CAPABILITY" for this bug's case, even if we can not find statement of "MUST/should request".

For "client didn't recognize the automatic capabilities".
(A) CAPABILITY response exists in OK response to authentication:
(A-1) client can detect and use optional CAPABILITY response:
      No need to issue separate CAPABILITY command.
(A-2) client can not detect nor use optional CAPABILITY response:
      Need to issue separate CAPABILITY, because CAPABILITY change can happen
      but server may not have capability of optional CAPABILITY response.
(B) CAPABILITY response doesn't exist in OK response to authentication:
(B-1) client can detect and use optional CAPABILITY response:
      Need to issue separate CAPABILITY command. By same reason as (A-2).
(B-2) client can not detect nor use optional CAPABILITY response:
      Need to issue separate CAPABILITY command. By same reason as (A-2).

Note.
Following is log obtained on 2008/3/01. Fix of Bug 401293 was done on 2008-11-19. So, behaviour of Tb in this bug's case is same since initial.
> :imap.gmail.com:NA:SetupWithUrl: clearing IMAP_CONNECTION_IS_OPEN
> :imap.gmail.com:NA:ProcessCurrentURL: entering
> :imap.gmail.com:NA:ProcessCurrentURL:imap://[e-mail]:993/create%3E/%5BGmail%5D/Sub_Folder:  = currentUrl
> : ReadNextLine [stream=3caba38 nb=68 needmore=0]
> :imap.gmail.com:NA:CreateNewLineFromSocket: * OK Gimap ready for requests from 222.9.106.185 m40if3379878waf.0
> :imap.gmail.com:NA:SendData: 1 capability
> : ReadNextLine [stream=3caba38 nb=75 needmore=0]
> :imap.gmail.com:NA:CreateNewLineFromSocket: * CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA XLIST CHILDREN XYZZY
> : ReadNextLine [stream=3caba38 nb=45 needmore=0]
> :imap.gmail.com:NA:CreateNewLineFromSocket: 1 OK Thats all she wrote! m40if3379878waf.0
> :imap.gmail.com:NA:SendData: Logging suppressed for this command (it probably contained authentication information)
> : ReadNextLine [stream=3caba38 nb=52 needmore=0]
> :imap.gmail.com:NA:CreateNewLineFromSocket: 2 OK [e-mail] authenticated (Success)
> :imap.gmail.com:A:SendData: 3 namespace

Comment 11

[Tim Altman]

I'm seeing this same issue (lack of working IDLE) on a server running Sun Java(tm) System Messaging Server 6.3-6.03.2.

Section 7.2.1 of RFC 3501 says, "It is unnecessary for a client to send a separate CAPABILITY command if it recognizes these automatic capabilities."  This implies that the client should send a separate capability command if no untagged capability response is included in the tagged OK response to successful logins.

Comment 12

[WADA:World Anti-bad-Duping Agency]

Requesting blocking 1.9.3(Tb 3.2), for current decision of "Mozilla foundation or Mozilla messaging company won't release Tb 3.2 until this bug will be fixed", not for blocking due to "Tb 3.2 can't be released unless this bug will be resolved", for ease of priority setting by Triage team.

Comment 13

[Mark Banner (:standard8)]

(In reply to comment #12)
> Requesting blocking 1.9.3(Tb 3.2)

WADA: If it hasn't got 'thunderbird' in the flag name, then it applies to core code, not Thunderbird code. We've got a bug filed for TB 3.2 flags, but it hasn't quite got here yet.

Comment 14

[Chris]

RFC 2060 (6.1.1) states:

This listing of capabilities is not dependent upon connection state or user. It is therefore not necessary to issue a CAPABILITY command more than once in a connection.

However, this RFC is obsoleted by RFC 3501, where no such thing is mentioned, and in fact that part of the paragraph is deleted, and the new example in 6.1.1 implies that the list of capabilities DOES in fact depend on server state.

I think someone has been reading the wrong RFC. Apparently it is becoming common in IMAP servers to not report all capabilities before the user is authenticated. Zimbra does this:

* OK IMAP4 Ready zimap.cwi.nl 0001d257
001 CAPABILITY
* CAPABILITY IMAP4 IMAP4REV1
001 OK CAPABILITY
002 LOGIN [snip] [snip]
002 OK You are so in
003 CAPABILITY
* CAPABILITY IMAP4rev1 ACL BINARY CATENATE CHILDREN CONDSTORE ENABLE ESEARCH ESORT I18NLEVEL=1 ID IDLE LIST-EXTENDED LIST-STATUS LITERAL+ LOGIN-REFERRALS MULTIAPPEND NAMESPACE QRESYNC QUOTA RIGHTS=ektx SASL-IR SEARCHRES SORT THREAD=ORDEREDSUBJECT UIDPLUS UNSELECT WITHIN
003 OK CAPABILITY completed
004 LOGOUT
* BYE zembox03.zaas.igi.nl IMAP4rev1 server terminating connection
004 OK LOGOUT completed

Comment 15

[Terry Child]

I'm in the position that at my place of employment the proxy servers are configured to report MORE capabilities that the actual servers. I've reported this to the server admins but apparently it can't be fixed in case it breaks some other clients. So using Thunderbird is a very unpleasant experience - and I may be forced in to using the Exchange Email client - or finding another IMAP client that queries for capabilities after login. Googling this - it doesn't come up that often - so I guess it's usually fixed by the server admins - which isn't going to happen any time soon in my case.

Comment 16

[Wayne Mery (:wsmwk)]

so the current interpretation is this would be an enhancement, iiuc.

Comment 17

[Lewis Rosenthal]

Hi, Wayne...

I'm not so sure that I would consider this an RFE vs a functional change. As security continues to tighten up in front of mail servers, I suspect we'll be seeing less and less openness (truthfulness) from the firewalls prior to authentication. So, going forward, I think we should consider the ramifications of changing our behavior (which is apparently defined only loosely in RFC 3501, which I just now re-read) to make a specific CAPABILITY request after successful AUTH. Again, I don't necessarily view this as an enhancement (or something so optional).

Just some random thoughts to throw into the mix before a final decision is made. Apologies for the bug spam, everyone.

Cheers

Comment 18

[David :Bienvenu]

I'm not sure the distinction is important - we need to do this, since servers are behaving this way. I can propose a patch...

Comment 20

[Nikolay Shopik]

Its nice we have movement here. But there is catch implemeting this patch we are intruducing additional round trip to server, lets not forget high latency connections. Back in day it me who reported bug 401293, which introduced handling capability in tagged reaponse, which saves us one round-trip. 

This is why rfc writen as such, because (proxy)server must make such decisions and send capabiltiy taged after auth. Otherwise as we can see such workarounds just bloat on client side and make it performance worse. Implementing pref for this doesnt quite solve problem, because nobody gonna use it(in case default on) and user expect app perform best as it can. It may solve problem somewhat if this is shiped with pref and default is off, but again will not make happy other users. So there no win-win solution i can see.
But I like reporter of bug 604182 solution of this.

Comment 21

[Lewis Rosenthal]

I like the proposal, Nikolay, but would prefer a more automated solution, per the reporter's "expected results" in bug 604182, i.e., "if the server did not include a CAPABILITY response code in the tagged OK response of a successful AUTHENTICATE command, then the client should notice this and issue a CAPABILITY command."

Having a pref to override this behavior to *always* send another CAPABILITY command would be a good idea, but your point about the excess traffic is well taken. Per David's comment #18, we need to do *something*.

Comment 22

[David :Bienvenu]

I don't see the point of ever sending an other capability command if the AUTH command returns an unsolicited CAPABILITY response. I can see having a pref that says "don't send an other capability command", though I'm not sure it's worth it.

Comment 23

[Lewis Rosenthal]

Hmmm... I see your point. I'm trying to think of a use case where an automatic CAPABILITY might not be enough. I guess if we apply the automatic logic at every AUTH (vs just the first time), then there would be no real need to force it. I do agree, then, that the ability to disable the automatic CAPABILITY command would be a logical pref.