382117 - Messages with tags fail to copy to account on Exchange server due to "APPEND failed: unknown flag"

Status: RESOLVED WORKSFORME

(MailNews Core :: Networking: IMAP, defect)

Description

[[e-mail]]

User-Agent:       Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.8.0.10) Gecko/20070228 Camino/1.0.4
Build Identifier: version 2.0.0.0 (20070326)

I have mail accounts on 2 servers in diff. cities, both iMAP, call them A & B.  I back each up on the other.  Starting with TB 2.0 copying messages from A-->B fails.  However, copying B-->A works, copying A-->local folders & local folders-->B also works.  Definite cause: use of tags.  Messages arriving untagged always copy; messages with color-coded tags never do, nor do messages from which I've removed the tag.  New bug in 2.0; does not happen in 1.5.0.8 or earlier.

Reproducible: Always

Steps to Reproduce:
1. Set up folder "Server A Inbox Backup" on mail server B.
2. Select all messages in server A inbox. [also applies to individually selected messages]
3. From control-click popup contextual menu, select Copy To - Server B - Server A Inbox Backup.
4. Copying fails UNLESS message never has had tag assigned.
Actual Results:  
Copying fails with this error message (100% of time):  Alert: The current command did not succeed.  The mail server responded: Protocol error: "Specified set of flags is not valid..."
Experimentation shows: (a) this only happens to messages that have NEVER had been assigned a color-coded tag (selecting "Remove all tags" doesn't fix the failure to copy); (b) only happens going from server A ---> server B, and not vice versa.

Expected Results:  
Mail messages should have copied from folder on Server A to corresponding folder on Server B for use as backup.

I apologize, I'm not a programmer (first bug report I've ever filed here); but I'm an experienced user & I've done as much as I know how to do.  I've isolated the cause: the enhanced "tags" feature in TB 2.0.  I sort incoming mail by assigning color-coded tags; useful feature. I back up mail folders from each server to other so my mail is always available if one goes down; and the operator of server B does not back up e-mail accounts (I found out the hard way). A few messages I get aren't assigned a tag, & those do copy.

You probably need to know the mail server software (I asked the operators):
Server A:  Washington University IMAP daemon that comes with Pine; mail delivery agent is Postfix.
Server B:  Microsoft Exchange 2003.
Finally I still have TB 1.5.0.8 on my iBook; this bug definitely does not exist there (i.e., copying mail messages in both directions works).  Thank you.

Comment 1

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)
> Server A:  Washington University IMAP daemon
> Server B:  Microsoft Exchange 2003.
> copying messages from A-->B fails. copying B-->A works.
> (a) this only happens to messages that have NEVER had been assigned a
> color-coded tag(selecting "Remove all tags" doesn't fix the failure to copy); 
> (b) only happens going from server A ---> server B, and not vice versa.

Get imap:5 protocol log, and attach log file(mime-type=text/plain) to this bug thru link of "Add an attachment".
  http://kb.mozillazine.org/Session_logging_for_mail/news
  http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#imap

Comment 2

[WADA:World Anti-bad-Duping Agency]

To [e-mail](bug opener):
See Bug 353570 Comment #4 for difference between "Specified set of *flags* is not valid..." by Exchange server and "Messages with *tags* fail to copy" by your bug summary. Anyway, see protocol log first. 

Comment 3

[[e-mail]]

Attachment: Requested imap:5 protocol log file to document bug

See comment in this post.

Comment 4

[[e-mail]]

Re attachment (see Comment #3):  My son created the necessary log file (he's a programmer by profession; sorry for delay, but I had to wait for him to visit me).

In the attached log file, look for BAD Protocol Error - that's where the bug is
occurring (see my previous description).

Comment 5

[Mike]

I can verify this problem.  I started a thread on the MozillaZine support forums, but got no responses:

http://forums.mozillazine.org/viewtopic.php?t=573677&highlight=

I was going to open up a bug report but persistent searching through Bugzilla found this one.

Are there any updates?  If any further information is needed, let me know.

Comment 6

[WADA:World Anti-bad-Duping Agency]

Attachment: Part of log relates to the error

To [e-mail](bug opener):

Will problem occur in next test too? (indirect copy by Tb 2.0)
(1) Copy the mail to a folder in "Local Folders"
(1-1) Move a mail in the folder to other folder, then move back
(1-2) "Compact Folder" (force write of X-Mozilla-Keys:)
(1-3) View Message Source, and confirm that $Lable4 exists
(2) Copy the mail to the MS Exchange server

Comment 7

[WADA:World Anti-bad-Duping Agency]

Confirming, according to log.

Comment 8

[WADA:World Anti-bad-Duping Agency]

RFC 3501 says following. (http://www.faqs.org/rfcs/rfc3501.html)
According to RFC, I think at least response of "BAD(Protocol Error)" is invalid(RFC violation) when "unable to append a flag($Lable4, called keyword in many cases)".

> 6.3.11. APPEND Command
>(snip)
>   Result:     OK - append completed
>               NO - append error: can't append to that mailbox, error
>                    in flags or date/time or message text
>               BAD - command unknown or arguments invalid
>(snip)
>      If a flag parenthesized list is specified, the flags SHOULD be set
>      in the resulting message; otherwise, the flag list of the
>      resulting message is set to empty by default.  In either case, the
>      Recent flag is also set.

David Bienvenu, what is correct behavior in this case? (both IMAP server&Client) 

Comment 9

[David :Bienvenu]

we shouldn't be trying to set flags that the server has told us it can't set - I would say that NO would be the appropriate response, but the RFC leaves itself open to interpretation, since an invalid flag could be considered an invalid argument, I guess. 

But in any case, we should fix this in the client.

Comment 10

[popo]

Is this related to issue 808533 ?
https://bugzilla.mozilla.org/show_bug.cgi?id=808533

It's been going on for few years now, yet it seems like nobody @the TB team is trying to fix this.

Interoperability of IMAP clients is very important. Hope this could be resolved soon !

Comment 11

[[e-mail]]

(In reply to popo from comment #10)
> Is this related to issue 808533 ?
> https://bugzilla.mozilla.org/show_bug.cgi?id=808533
> 
> It's been going on for few years now, yet it seems like nobody @the TB team
> is trying to fix this.
> 
> Interoperability of IMAP clients is very important. Hope this could be
> resolved soon !

It must have been fixed. I reported the bug 5 years ago, and forgot about the issue, but haven't run into the problem for many versions of TBird now (I still use the same cross-server method of backing up my mail files described in the original bug report above, and it now works fine). I just assumed it had been fixed some years ago, either on purpose or inadvertently.

Comment 13

[David Lechner (:dlech)]

@godfreye and @popo: What server/version are you using for the destination server currently?

Comment 14

[popo]

Mine says: IMAP4rev1 2000.287

popo

Comment 15

[David Lechner (:dlech)]

(In reply to popo from comment #14)
> Mine says: IMAP4rev1 2000.287

So I can interpret this as Microsoft Exchange Server 2000?

Comment 16

[popo]

Why ? This IMAP server is running on an old SUN OS 5.8 server. I guess this is not M$.

Comment 17

[David Lechner (:dlech)]

(In reply to popo from comment #16)
> Why ?

I have a theory that I would like to confirm.

Servers issue a list of PERMANENTFLAGS like:
>  * OK [PERMANENTFLAGS (\11a shraga \* \Answered \Flagged \Deleted \Draft \Seen)] Permanent flags

When you copy/move a message, we get this command:
> 2 append "/home/mail/popo/Archives/Nov" (\Seen $label1) "14-Nov-2012 10:00:04 +0200" {1373}

My theory is that only for some servers, but not all, that if one of the labels in the append command is not one of the permanent flags, then the append command fails.

From the Comment 11 above, it sounds like something in Microsoft Exchange Server changed because nothing was ever fixed in Thunderbird.

Comment 18

[popo]

Attachment: Copy file when user flag is not defined on IMAP server

Comment 19

[popo]

Attachment: Manually creating the my_flag tag, using telnet

Comment 20

[popo]

Attachment: 3.log - copying after flag was set manually

Comment 21

[popo]

For some reason only my attachments were uploaded, but not my response to comment #17:

I did 2 tests, to check your theory:

1 (log file in comment 18):
Copy a message, tagged with a "my_flag" tag from exchange server to imap server.
The copy failed ("append failed...)

2 (log in comment 19):
I created manually the "my_flag" tag in the destination folder in the imap server, using telnet

3) (log file in comment 20):
Copied same message. This time the "my_flag" tag exists both in exchange, and in imap destination folder.
Copy filed with same "append failed..." error

Both times I got the same error, which probably means TB is not aware of the IMAP user flag at all.

popo

Comment 22

[David Lechner (:dlech)]

(In reply to comment 21)

I am not sure that anything can be done in Thunderbird to fix the problem for this server.

Attachment 682883 [details] shows "my_flag" in PERMANENTFLAGS, so my theory was wrong and following the suggestion in comment 9 does us no good.

It believe the best solution is to upgrade your IMAP server.

Comment 23

[Magnus Melin [:mkmelin]]

Close as INVALID then?

Comment 24

[popo]

David,
I thought that if the flag name and also \* appears in the PERMANENTFLAG section, then this means the server supports user made flags(tags). Isn't this the case here ?

-422581360[e4ff3270]: e49a4800:imap1:A:SendData: 12 select "/home/mail/popo/Archives/Nov"
-422581360[e4ff3270]: ReadNextLine [stream=e4915928 nb=12 needmore=0]
-422581360[e4ff3270]: e49a4800:imap1:A:CreateNewLineFromSocket: * 1 EXISTS
-422581360[e4ff3270]: ReadNextLine [stream=e4915928 nb=12 needmore=0]
-422581360[e4ff3270]: e49a4800:imap1:A:CreateNewLineFromSocket: * 0 RECENT
-422581360[e4ff3270]: ReadNextLine [stream=e4915928 nb=51 needmore=0]
-422581360[e4ff3270]: e49a4800:imap1:A:CreateNewLineFromSocket: * OK [UIDVALIDITY 1353245120] UID validity status
-422581360[e4ff3270]: ReadNextLine [stream=e4915928 nb=38 needmore=0]
-422581360[e4ff3270]: e49a4800:imap1:A:CreateNewLineFromSocket: * OK [UIDNEXT 11] Predicted next UID
-422581360[e4ff3270]: ReadNextLine [stream=e4915928 nb=60 needmore=0]
-422581360[e4ff3270]: e49a4800:imap1:A:CreateNewLineFromSocket: * FLAGS (my_flag \Answered \Flagged \Deleted \Draft \Seen)
-422581360[e4ff3270]: ReadNextLine [stream=e4915928 nb=93 needmore=0]
-422581360[e4ff3270]: e49a4800:imap1:A:CreateNewLineFromSocket: * OK [PERMANENTFLAGS (my_flag \* \Answered \Flagged \Deleted \Draft \Seen)] Permanent flags
-422581360[e4ff3270]: ReadNextLine [stream=e4915928 nb=37 needmore=0]
-422581360[e4ff3270]: e49a4800:imap1:A:CreateNewLineFromSocket: 12 OK [READ-WRITE] SELECT completed

Comment 25

[David Lechner (:dlech)]

popo,

I am afraid that the problem is just that your server is just really old. I found this from 2007: http://objectmix.com/imap/201915-mailutil-transfer-fails-unknown-flag.html

That server then was even a more recent version than yours is now. I do not know what else to say besides upgrade your server.

Comment 26

[popo]

OK
I tried a newer server (dovecot), and things are working OK. I think this issue can be closed.