295672 - moving message from local folders back to imap server says : Message contains invalid header

Status: RESOLVED DUPLICATE of bug 147191

(Thunderbird :: General, defect)

Description

[ivo zwonarz]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.8) Gecko/20050511 Firefox/1.0.4
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.8) Gecko/20050511 Firefox/1.0.4

1. Mail arrives in imap folder
2. With mouse: move it to a folder in Local Folders
3. When moving back to the imap server i get the error message:
The command did not succeed. The mail server responded: Message contains invalid
heeder.
P.S. 1 Undo move works (message goes back to the imap server)
P.S. 2 I see the following headers extra in my local folders:
X-Mozilla-Status: 0000
X-Mozilla-Status2: 00000000
>From - Fri May 27 07:52:29 2005
... following headers are already on the imap message
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
... rest of the headers omitted...

Reproducible: Always

Steps to Reproduce:
See message above
Actual Results:  
An error popup.

Expected Results:  
Move the message back to the server.

Comment 1

[Tanner Lovelace]

I can confirm this bug using Thunderbird 1.0.2 (20050317) on Windows 2000 trying to copy to a Cyrus 
IMAP server.  

I believe it may have something to do with the "From" field without a colon (but that's pure conjecture 
on my part).

Comment 2

[Gervase Markham [:gerv]]

This is an automated message, with ID "auto-resolve01".

This bug has had no comments for a long time. Statistically, we have found that
bug reports that have not been confirmed by a second user after three months are
highly unlikely to be the source of a fix to the code.

While your input is very important to us, our resources are limited and so we
are asking for your help in focussing our efforts. If you can still reproduce
this problem in the latest version of the product (see below for how to obtain a
copy) or, for feature requests, if it's not present in the latest version and
you still believe we should implement it, please visit the URL of this bug
(given at the top of this mail) and add a comment to that effect, giving more
reproduction information if you have it.

If it is not a problem any longer, you need take no action. If this bug is not
changed in any way in the next two weeks, it will be automatically resolved.
Thank you for your help in this matter.

The latest beta releases can be obtained from:
Firefox:     http://www.mozilla.org/projects/firefox/
Thunderbird: http://www.mozilla.org/products/thunderbird/releases/1.5beta1.html
Seamonkey:   http://www.mozilla.org/projects/seamonkey/

Comment 3

[Tanner Lovelace]

I tried to confirm this bug several months ago, but apparently don't have enough
power to do so.  How, then, may I confirm this bug?

Comment 4

[ivo zwonarz]

This works for Thunderbird 1.5 beta 1 now (not for Thunderbird 1.0.6)

Comment 5

[Kevin Brosnan]

No changes were made to the code base by this bug, worksforme.

Comment 6

[ivo zwonarz]

***********************************************************************************************************************************************
message now in inbox (imap)
***********************************************************************************************************************************************


From - Tue Nov 29 11:20:51 2005
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
Return-Path: <i.xxxxxxx@xxxxxxx>
Received: from server01.xxxxxxx ([unix socket])
	by server01.xxxxxxx (Cyrus v2.1.18-IPv6-Debian-2.1.18-1) with LMTP; Tue, 29 Nov 2005 11:20:49 +0100
X-Sieve: CMU Sieve 2.2
Received: from [10.0.1.24] (unknown [10.0.1.24])
	by server01.xxxxxxx (Postfix) with ESMTP id 973E542400E
	for <i.xxxxxxx@xxxxxxx>; Tue, 29 Nov 2005 11:20:49 +0100 (CET)
Message-ID: <438C2B80.1030805@xxxxxxx>
Date: Tue, 29 Nov 2005 11:20:48 +0100
From: xxxxxxx Ivo <i.xxxxxxx@xxxxxxx>
User-Agent: Thunderbird 1.5 (Windows/20051025)
MIME-Version: 1.0
To: xxxxxxx Ivo <i.xxxxxxx@xxxxxxx>
Subject: test
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

test



***********************************************************************************************************************************************
After Move to the local inbox folder we have following headers and move can even not be undone ore moved back: 
Massage contains invalid headers
***********************************************************************************************************************************************

From - Tue Nov 29 11:49:16 2005
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
>From - Tue Nov 29 11:20:51 2005
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
Return-Path: <i.xxxxxxx@xxxxxxx>
Received: from server01.xxxxxxx ([unix socket])
	by server01.xxxxxxx (Cyrus v2.1.18-IPv6-Debian-2.1.18-1) with LMTP; Tue, 29 Nov 2005 11:20:49 +0100
X-Sieve: CMU Sieve 2.2
Received: from [10.0.1.24] (unknown [10.0.1.24])
	by server01.xxxxxxx (Postfix) with ESMTP id 973E542400E
	for <i.xxxxxxx@xxxxxxx>; Tue, 29 Nov 2005 11:20:49 +0100 (CET)
Message-ID: <438C2B80.1030805@xxxxxxx>
Date: Tue, 29 Nov 2005 11:20:48 +0100
From: xxxxxxx Ivo <i.xxxxxxx@xxxxxxx>
User-Agent: Thunderbird 1.5 (Windows/20051025)
MIME-Version: 1.0
To: xxxxxxx Ivo <i.xxxxxxx@xxxxxxx>
Subject: test
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

test

Comment 7

[Maxime Delorme]

For release version 1.6a1 (20060203) this bugs exists
And then it crashes tb

Comment 8

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #6)

> ****************************
> message now in inbox (imap)
> ****************************
From - Tue Nov 29 11:20:51 2005
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000

> ************************************
> After Move to the local inbox folder
> ************************************
From - Tue Nov 29 11:49:16 2005    (<= Added by Tb after download)
X-Mozilla-Status: 0001             (<= Added by Tb after download)
X-Mozilla-Status2: 00000000        (<= Added by Tb after download) 
>From - Tue Nov 29 11:20:51 2005   (Sent by server, "From"  is escaped by ">")
X-Mozilla-Status: 0001             (Sent by server)
X-Mozilla-Status2: 00000000        (Sent by server)   

"Rejection by invalid header" is probably due to ">From - Tue Nov 29 11:20:51 2005" line, as some other comments say.

As you say next in comment #0,
> ... following headers are already on the imap message
> X-Mozilla-Status: 0001
> X-Mozilla-Status2: 00000000
these two lines(excess X-Mozilla-StatusX: headers) seems to be sent by original mail sender, intentionaly or due to bug.
 
Thunderbird escapes "From -" line because this line is same format as unix mbox mail separator and have to be escaped.
On upload, (A) since this ">From -" line is placed in mail header part, Thunderbird doesn't unescape it, (B) Thunderbird unescapes this ">From -" line.

What will happen when ">From -" is changed in local mail file?
ivo zwonarz, can you test next case?
 (1) Create a local mail folder(say XXX), then copy the mail in it twice.
 (2) Shutdown Thunderbird
 (3) Edit file of XXX by text editor
     (First mail)  Change ">From - ..." to "FromX - ..." (Still invalid header)
     (Second mail) Change ">From - ..." to "X-From: ..." (Faked header)
 (4) Delete file of XXX.msf
 (5) Restart Thunderbird
 (6) Try to upload these two mails to your IMAP server.

Why your IMAP server put "From - Tue Nov 29 11:20:51 2005" line (Unix mbox separator) in mail header part when mail download by Thunderbird?
Mail sender sent this separator line(already invald mail header) as first line data of mail to your IMAP server?
If this is true, why your IMAP server doesn't reject it when mail receive from mail sender?

By the way, you said "This works for Thunderbird 1.5 beta 1 now" in Comment #4 on 2005-09-28, then changed to FIXED.
It sounds that mail in comment #0 sucessufully moved back to IMAP server.
However, you said following in Comment #6 on 2005-11-29, then re-opened this bug with similar mail data to one in comment #0.
> move can even not be undone ore moved back: Massage contains invalid headers
Do you mean problem was fixed once really but regression occured?

Comment 9

[ivo zwonarz]

(In reply to comment #8)

These lines are on my source in Tbird (imap folder on server)
These 3 lines are not in the source on the server even not after opening the message with TBird.

From - Mon Feb 20 08:56:29 2006
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000

This is the start of the message. The rest remains the same.

> What will happen when ">From -" is changed in local mail file?
> ivo zwonarz, can you test next case?
>  (1) Create a local mail folder(say XXX), then copy the mail in it twice.
>  (2) Shutdown Thunderbird
>  (3) Edit file of XXX by text editor
>      (First mail)  Change ">From - ..." to "FromX - ..." (Still invalid header)
>      (Second mail) Change ">From - ..." to "X-From: ..." (Faked header)
>  (4) Delete file of XXX.msf
>  (5) Restart Thunderbird
>  (6) Try to upload these two mails to your IMAP server.

THe folder XX contains no messages when I open the folder in TBird.

> 
> Why your IMAP server put "From - Tue Nov 29 11:20:51 2005" line (Unix mbox
> separator) in mail header part when mail download by Thunderbird?
> Mail sender sent this separator line(already invald mail header) as first line
> data of mail to your IMAP server?
> If this is true, why your IMAP server doesn't reject it when mail receive from
> mail sender?
> 
As far as now my IMAP server is unaware of this 3 lines. It is Tbird that adds them.


> By the way, you said "This works for Thunderbird 1.5 beta 1 now" in Comment #4
> on 2005-09-28, then changed to FIXED.
> It sounds that mail in comment #0 sucessufully moved back to IMAP server.
> However, you said following in Comment #6 on 2005-11-29, then re-opened this
> bug with similar mail data to one in comment #0.
> > move can even not be undone ore moved back: Massage contains invalid headers
> Do you mean problem was fixed once really but regression occured?
> 

THe problem was probably never fixed, i must have made a mistake.

Comment 10

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #9)
> > Why your IMAP server put "From - Tue Nov 29 11:20:51 2005" line (Unix mbox
> > separator) in mail header part when mail download by Thunderbird?
> > Mail sender sent this separator line(already invald mail header) as first line
> > data of mail to your IMAP server?
> > If this is true, why your IMAP server doesn't reject it when mail receive from
> > mail sender?
> > 
> As far as now my IMAP server is unaware of this 3 lines. It is Tbird that adds
> them.

Are you saying "11:20:51 = 11:49:16"?

Not about first tree line in local folder.
4-th line of the mail in your local folder(=first line your IMAP server sent to Tb).
">From - Tue Nov 29 11:20:51 2005" is escaped format(">" is prepended) of "From - Tue Nov 29 11:20:51 2005" by Tb, because your mail server sent this line to Tb as first line of the mail.
First line in your local folder is "From - Tue Nov 29 11:49:16 2005", and is exact separater line in local mail folder which was added by Tb.

Comment 11

[dem107]

This is still a problem in Thunderbird 1.5.0.9.

Comment 12

[Tuukka Tolvanen (sp3000)]

Here's an interesting sampling of message sizes as reported in tbird's size column, for a bunch of messages moved to a local folder from an exchange server via imap with various tbird versions ~2-3 along that time -- of course the sizes may be entirely coincidental:

good: 3 4 7 76 439 465 547 547 571 594 622 650
bad: 539 572 596 599 611 616 622 628 632 645 646 737 1038 1442 2076 2504 3376

Comment 13

[Tuukka Tolvanen (sp3000)]

ok, size seems to not be an issue, rather the pattern of archiving large messages along with their threads that resulted in the local folder contents described above means that only larger messages would ever have been copied in individually. I filed bug 381472 with the relevant specific findings.

Comment 14

[WADA:World Anti-bad-Duping Agency]

To all problem reporters:
Is the problematic mail a mail downloaded for offline use? (See bug 381472) 

Comment 15

[Roland Neilands]

(In reply to comment #14)
> To all problem reporters:
> Is the problematic mail a mail downloaded for offline use? (See bug 381472) 
Not for me. I changed the destination folder to be downloaded for offline use as well and got the same result.

Comment 16

[dem107]

I have this problem as soon as I move it from an IMAP folder to a local one.  It doesn't matter if I mark it for offline use.

Comment 17

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #15)
> I changed the destination folder to be downloaded for offline use
> as well and got the same result.
(In reply to comment #16)
> I have this problem as soon as I move it from an IMAP folder to a local one. 
> It doesn't matter if I mark it for offline use.

Your case(not "offline use") sounds a variation of phenomenon of Bug 381472, although current bug summary refer to IMAP folder of "offline use" only.
 1. No extra headers when folder not set as "offline use".
 2. When copied to local, From/X-Mozilla-Status:/X-Mozilla-Status2: is written.
 3. When move back to IMAP folder, the "From" line is sent with ">" for escape.

To Roland Neilands and [e-mail] :
Can you try to recreate Bug 381472? Can you check whether your case is Bug 381472 or similar phenomenon?
Please add comment if new important test result is obtained.

Comment 18

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #9)
> These 3 lines are not in the source on the server even not after opening the
> message with TBird.

ivo zwonarz(bug opener):
I'm sorry but I couldn't understand correctly what it means, but I now understand what was happened correctly by Bug 381472. Watch Bug 381472, please.

Comment 19

[WADA:World Anti-bad-Duping Agency]

Bug 147191 was found to be a report for same case. See Bug 147191 Comment #17.

Comment 20

[WADA:World Anti-bad-Duping Agency]

Why & when ">From - ..." & duplicated X-Mozill-Status:/X-Mozilla-Status2: are
generated is being analyzed by Bug 381472.
And Bug 147191, Bug 158014, Bug 295672(this bug) are bugs for same phenomenon.
I think these three bugs are better to be consolidated to single bug.

Comment 21

[Chris]

The Problem still exists in version 2.0.0.6 (20070728). 

Comment 22

[ReimerStudt]

The problem still seems to exist. I have the problem with a mail from november 2009. Maybe, a solution would be to

Comment 23

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #22)
> The problem still seems to exist. I have the problem with a mail from november
> 2009. Maybe, a solution would be to

Once excess X-Mozilla-Status:/X-Mozilla-Status2: headers are generated in local mail folder by bug 381472, it'll never be removed automatically.
Copy the mail from IMAP server to local mail folder again using latest Thunderbird(3.0.0 or later. bug 381472 is alreay fixed). Does problem still occur?

Comment 24

[WADA:World Anti-bad-Duping Agency]

FYi.
Phenomenon of "excess header generated by Tb/excess From ... line in header upon upload from local to IMAP" which I can see is bug 426651 only.
  X-Account-Key:
  X-UIDL:
  X-Mozilla-Keys:
  "From ..." line which was created by other than Tb in Tb's mail store file(BerkleyStore)
                  and whose format is different from one generated by Tb.
All other excess Tb's X-...: header, excess ">From ..." line in header, excess "From ..." line in header which I can see by "upload from local to IMAP" is;
  Mail sender or server passed it to Tb, then it's uploaded to IMAP server by Tb.
i.e. This bug is already WORKSFORME or resolved by bug 381472.
Note:
  As for "From ... line in header which has different format from one generated by Tb",
  culprit of "From line in header sent from IMAP server to Tb" can be bug 426651 of Tb.
  However, "From ... in header or >From in header seen in this bug" has same format as
  format Tb uses. So, bug 426651 is irrelevant to this bug.

Comment 25

[Wayne Mery (:wsmwk)]

consolidating