555913 - Thunderbird IMAP "select command" fails at IMAP server, copy to Sent or Draft folder fails. Too many characters sent

Status: RESOLVED INCOMPLETE

(MailNews Core :: Networking: IMAP, defect)

Description

[boris]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2 (.NET CLR 3.5.30729)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.1.8) Gecko/20100227 Lightning/1.0b1 Thunderbird/3.0.3

The usage of Cyrus IMAP and MS-Windows-7 (64-bit) is severely impaired due to access problems. (applies to ALL flavors of Thunderbird)
The select commands seem to be truncated by the OS in a way that the IMAP-Server cannot recognize the command.
In particular operations are affected, such as copy to draft, sent, archive or remove attachment, etc.
Cyrus uses a directory structure like: INBOX, INBOX.Sent, Inbox.Drafts, ... which might be the reason for the different behaviour.

Other mail clients do not seem to be affected.

Reproducible: Always

Steps to Reproduce:
1. Install any Mozilla Thunderbird under a MS-Windows-7 (x64) OS
2. Configure a Cyrus IMAP Server (e.g. (Cyrus v2.2.12) with LMTPA)
3. Set the Sent, Drafts, Archive directories to sub directory paths on the IMAP account (note Cyrus wants to have them as subdirectories of the Inbox)
3b. Try to remove an attachment
Actual Results:  
The copies fail - Server responds with "Unrecognized command"
Moving a message manually to the folder works!

Expected Results:  
Sent mail should be automatically placed in sent mail folder, or in the original folder.
Drafts should be saved in the INBOX.Drafts folder
Attachments should be removed.

It is unclear whether 32-bit versions of Windows-7 are affected as well
Windows XP (32-bit) seems to be unaffected.

As a workaround the folders can be routed to local folders, making the basic IMAP-features obsolete.

Comment 1

[boris]

Funnily activating the Protocol features in Windows7 seems to solve the problem:
set NSPR_LOG_MODULES=IMAP:4,SMTP:4
set NSPR_LOG_FILE=%HOMEDRIVE%%HOMEPATH%\Desktop\tbird_log.txt

temporarily...

Comment 2

[boris]

It seems that the issues can be reproduced on multiple hardware platforms (tested on MS-Windows XP as well).
The issue might be related to Bug 97924, which describes an incompatibility of Thunderbird and the Cyrus IMAP server. Although both seem to act correctly regarding the protocols Cyrus uses "." (dot) as a separator for the folder hierarchy. Since Thunderbird accepts this, but is not aware of the problems involved if this separator is used in a folder name (which is allowed as well) the two collide and send requests for folders that Cyrus cannot resolve.

Without having the insight of Thunderbirds internal IMAP handling it seems that a fix could involve checking the forbidden characters (or at least having an option to do this manually) so that these are only used for purpose and are excluded within folder-names themselves.

Currently it is unclear what is really going on - there are some strange things involved: Sometimes the copy process (of new messages to .sent) works, sometimes it does not. Saving to the Drafts folder (.drafts) does not work and Thunderbird complains that it could not save it to the ->Sent<- folder (why .Sent and not .Drafts?)
Thunderbird also waits to check the mail server capabilities before it fails (only once after restart).
The advanced account settings of the Cyrus IMAP account are:
IMAP server directory: (blank)
Personal namespace: "INBOX."
Public (shared): ""
Other Users: "user."

Hence the separators are already integrated into the names, but obviously Thunderbird gets confused any way (is the dot used also for something else? ...

Comment 3

[boris]

The root of the problem seems to be an issue caused by the original hierarchical definition of the Cyrus namespace, that has been altered according to the RFCs in 2005 - quoting from: http://bugs.horde.org/ticket/?id=3004

As explained previously, the simple fact is users should *not*  be 
expected to understand the concept of namespaces.

Additionally, if you read the RFCs, it is *not* correct to put the 
default namespace under 'INBOX', even if the default namespace is 
'INBOX.'    This is because 'INBOX' must ALWAYS appear in the default 
namespace; obviously, INBOX can not be a subfolder of itself - it must 
be at the same level as the other mailboxes in the default namespace.   
People need to understand that the namespace name (e.g. 'INBOX', 
'#shared', etc.) are not mailbox names in and of themselves.  Rather, 
they are simply internal labels used to delimit between the various 
namespaces (labels != mailboxes)

Comment 4

[boris]

Further appearances of the same issue:
Debian Bug report logs 2004 - #274987: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=274987
TBird hangs sending mail to IMAP sent-mail 2005  - http://forums.mozillazine.org/viewtopic.php?t=218625
Problems saving sent messages to Sent folder 2010 - http://gs.mozillamessaging.com/mozilla_messaging/topics/problems_saving_sent_messages_to_sent_folder

This bug seems to be persistent and NO solution has been found, although it sometimes seems to work properly without any obvious changes to the system.

Comment 5

[boris]

"The current command did not succeed.  The mail server responded:Unrecognized command

."
Above is exactly the error message that is returned by TB when asking the Cyrus IMAP server. Testing it with a different (empty) folder also gives the same result, so there does NOT seem to be a racing condition or timeout involved.

Below is the message TB replies after start (sometimes):
"The current command did not succeed.  The mail server responded:Unexpected extra arguments to Select

."
And here comes the message it gives when a the save (as Draft to the server directory) is executed.
"The current command did not succeed.  The mail server responded:Unexpected extra arguments to Select

."
Note the carriage return and the dot...
To me it seems that TB tries to talk gibberish to the IMAP.

Comment 6

[boris]

After looking into the IMAP conversation between client and server:
1212[99e9040]: b6c1c00:mail.server.org:A:SendData: 7 select "INBOX" (CONDSTORE)
1212[99e9040]: ReadNextLine [stream=b6a52a8 nb=44 needmore=0]
1212[99e9040]: b6c1c00:mail.server.org:A:CreateNewLineFromSocket: 7 BAD Unexpected extra arguments to Select

And shutting down CONDSTORE for the server like mentioned elsewhere:
"mail.server.default.use_condstore to false, or
mail.server.serverxxx.use_condstore, where serverxx is the particular
imap server causing problems. "

- The first issue seems to be resolved:
This one is gone: TB replies after start (sometimes):
"The current command did not succeed.  The mail server responded:Unexpected
extra arguments to Select

."
- Now looks like this:
4312[a00f600]: a691400:mail.server.org:A:SendData: 7 select "INBOX"
4312[a00f600]: ReadNextLine [stream=a641c08 nb=85 needmore=0]

4312[a00f600]: a691400:mail.server.org:A:CreateNewLineFromSocket: * FLAGS (\Answered \Flagged \Draft \Deleted \Seen NonJunk Junk $Forwarded $MDNSent)
4312[a00f600]: ReadNextLine [stream=a641c08 nb=104 needmore=0]
...

BTW if you want to check the conversation of your client do the following:
Open a command shell under Windows:
[Type or copy this line to activate the LOG:]
set  NSPR_LOG_MODULES=imap:5 bzw imap:5,smtp:5,ImapAutoSync:5
[tell the client to write the log into a file (insert full path of log-file):]
set NSPR_LOG_FILE=[insert full path of log-file here]
[start thunderbird from THIS commandline]

Comment 7

[Ludovic Hirlimann [:Usul]]

Boris can you look into MailnewsCore/Imap and see if this is not a duplicate of one of the bugs present there ?

Comment 8

[boris]

The errors during copying the messages as sent mail seem to be caused by unmatched lengths of the messages when saving a Draft or Sent message automaticalle (note: it works well when copying). (Note this configuration is running with a large Cyrus-IMAP server installation with multiple Servers operating at different versions):

6924[6784ac0]: 8c64c00:mail.server.org:A:ProcessCurrentURL:imap://[e-mail]:143/appendmsgfromfile%3E.INBOX.Sent:  = currentUrl
6924[6784ac0]: 8c64c00:server.org:A:SendData: 9 append "INBOX.Sent" (\Seen) {692+}
6924[6784ac0]: 8c64c00:mail.server.org:A:SendData: Message-ID: <[e-mail]>
6924[6784ac0]: 8c64c00:mail.server.org:A:SendData: 
6924[6784ac0]: ReadNextLine [stream=a4d8848 nb=28 needmore=0]
6924[6784ac0]: 8c64c00:server.org:A:CreateNewLineFromSocket: 9 BAD Unrecognized command

Is it possible that Thunderbird counts LFs and CRs (e.g. LFCRLF or CRLF as single LF) wrong in this code?

Comment 9

[boris]

https://bugzilla.mozilla.org/show_bug.cgi?id=460078
Describes a bug also appearing in TB3.0.4 when Draft is supposed to be saved (it says it tries to copy into Sent folder) - this might or might not be related, but appears at the same time.

Comment 10

[boris]

(In reply to comment #7)
> Boris can you look into MailnewsCore/Imap and see if this is not a duplicate of
> one of the bugs present there ?

I have done that. Although there are quite a number of references to similar errors (hangs when copying to Sent folder e.g. https://bugzilla.mozilla.org/show_bug.cgi?id=546728) they either seem to be different or have no IMAP protocol attached.
I am not sure where the origin of the problem might be found. The server structure I have got the failure with is a multi server system that runs on different versions of Cyrus (for ~8000 users). The first bug (CONDSTORE) was labelled by the postmaster as a typical unforgiving behaviour of TB. Since the front-end server talks with condstore and the back-end server cannot. Hence, TB talks to the fe-Cyrus o ask for CONDSTORE and gets a yes, but when transferring the data later the older Cyrus fails (unknown command).

In the current case it seems that TB wants to write the message to the Sent or Drafts folder, but gets an unrecognized command failure , probably due to miscounting of the LFs it gets a longer block (https://bugzilla.mozilla.org/show_bug.cgi?id=555913#c8) than expected and says unrecognised command, which cancels the saving of the message and forces TB to halt or timeout the send to Sent folder process.

Comment 11

[Ludovic Hirlimann [:Usul]]

Removing myslef on all the bugs I'm cced on. Please NI me if you need something on MailNews Core bugs from me.

Comment 12

[Wayne Mery (:wsmwk)]

imap in the last 7 years has numerous patches. and Boris' email fails and we don't have clear STR here, so => incomplete