346811 - Forwarding mail as attachment gets MIME encoding wrong

Status: VERIFIED INVALID

(Thunderbird :: General, defect)

Description

[Alan Kent]

User-Agent:       Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; InfoPath.1; .NET CLR 2.0.50727)
Build Identifier: version 1.5.0.4 (20060516)

Forwarding messages appears to get the MIME encoding wrong. I get the following lines in the middle of the encoded message. Some lines omitted. Look towards the end (for the "From [e-mail] ..." line with two blank lines in front).


From [e-mail]  Tue Aug  1 17:52:49 2006
Return-Path: <[e-mail]>
...
Message-ID: <[e-mail]>
Date: Tue, 01 Aug 2006 17:52:46 +1000
From: Alan Kent <[e-mail]>
User-Agent: Thunderbird 1.5.0.4 (Windows/20060516)
MIME-Version: 1.0
To: [e-mail]
Subject: [Fwd: 100TB Lab Status report for week ending 7/28/06]
Content-Type: multipart/mixed;
 boundary="------------090703010309020004030607"

This is a multi-part message in MIME format.
--------------090703010309020004030607
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit



--------------090703010309020004030607
Content-Type: message/rfc822;
 name="100TB Lab Status report for week ending 7/28/06"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filename="100TB Lab Status report for week ending 7/28/06"


From [e-mail]  Tue Aug  1 07:51:06 2006
Return-Path: <[e-mail]>
X-Original-To: [e-mail]
Delivered-To: [e-mail]
...


I have changed a username above for privacy reasons to [e-mail].  Note the line:

From [e-mail]  Tue Aug  1 07:51:06 2006

This is not a valid RFC822 header in my understanding.  Note also before this line there are two blank lines. My understanding is there should only be one blank line and that the "From " line is not a valid header. Its something mbox uses internally, its not valid RFC822.

The Java mail parsing library is picking up the From line onwards as the message body as if there are no header lines at all. Maybe the "From ..." line is confusing thunderbird so it thinks there are no headers (so the blank line is on purpose).

I am using Thunderbird with a Courier IMAP server - I do not know how to determine if that is returning wrong data (invalid messages) that Thunderbird is then relying on. That is, I do not know if Thunderbird is doing its best with bad input data.

However, MS Outlook seems to be handling the same IMAP server and forwarding exactly the same message correctly as an attachment. So I think its Thunderbird. (I should note that MS Outlook appears to try and be "clever" at times for different messages that I forward as attachments, so maybe Outlook is not a good comparison to use!!!)

Reproducible: Always

Steps to Reproduce:
1. Forward a mail message
2. Check raw message encoding, looking for double blank line between header and content and invalid "From " line (not "From:")


Actual Results:  
When parsing the email with the Java mail parsing library, the extra blank line between the header and content made Java think there were no headers - the headers got sucked into the message body instead.

Expected Results:  
Should not have been the "From ..." line (note: no colon after "From") and should have only been one blank line between header and content. When I manually do those changes, the Java mail parsing library works correctly.

Other systems may be more forgiving or I may be misunderstanding the MIME encoding rules. My understanding is there must be exactly one blank line between the headers and the body, and "From ..." is not a valid header. Mbox format uses it, but its not a valid RFC822 header.  Sorry if I am incorrect here.

I have verified it by examining messages saved from Java - which is a bit risky as maybe the Java library is messing the message up (but forwarded messages from other mail clients are working ok).

I have also verified it by going to the Maildir directory for our IMAP server and looking at the raw message there too. So I don't think its the Java library at fault.

Comment 1

[Mike Cowperthwaite]

I'm unable to reproduce this problem.  When I forward a message (as attachment) I'm not seeing the 'From ' mbox-separator nor an extra blank line in the forward's source.  In fact, if a 'From ' header were sent and the message incorporated into the user's mbox, it would appear as a separate message within the Inbox (or it would should the .MSF index get rebuilt).

Comment 2

[Alan Kent]

All I can say it is happens every time repeatably for me.

It could be that IMAP server I am using is not doing the right thing perhaps.  We are using Courier IMAP on Solaris.  When I look in the Maildir directory at a message, the message file contents starts with "From xxx@yyy" - note the lack of colon.  I am not an IMAP expert.  Perhaps the Courier IMAP server is returning an invalid message that Thunderbird is trying to attach.  The "From xxx@yyy" line (without a colon) is not a legal RFC822 header.  When Thunderbird sees the first line is not a valid RFC822 mail header, maybe it is assuming the full message contents is the body and there are zero mail headers.

Exchange may have a workaround to handle the "From xxx@yyy" first line - it seems to forward messages as I would expect using the same IMAP server.

Another experiment.  I just used "Copy To" to save a message from the IMAP server to a "local folder".  The local folder copy of the message (when saved using "Save As" with "xxx.eml") started with the following:

X-Mozilla-Status: 1001
X-Mozilla-Status2: 00000000
>From [e-mail]  Wed Sep 13 06:38:17 2006
Return-Path: <[e-mail]>
X-Original-To: [e-mail]

Notice a ">" was inserted before the "From" line.

When I view the message in Thunderbird, it displays fine.  It identifies the header lines as desired.  When I forward the local folder copy, it inserts the extra blank line as reported previously.  The first line of the attachment when displayed is ">From xxx@yyy".


When you tested it, did you try it with Courier IMAP?  Did you try it with a message starting with a "From xxx@yyy" line (not a "From: ..." line)?  I suspect the "From xxx@yyy" line (without the colon) is key to the problem.  Thunderbird seems to cope with it when displaying mail, but not when forwarding as an attachment.

I am happy to perform any tests to further demonstrate the problem if it helps.  But the behaviour is 100% repeatable - every mail I forward as an attachment from the Courier IMAP server comes out wrong without exception.



AHH! I just tried something. I went to the Courier IMAP server Maildir area and edited the message file to change the "From xxx@yyy" line by hand to "X-From: xxx@yyy" temporarily.  That is, the message file no longer starts with a "From xxx@yyy" line.  This message now forwards correctly in Thunderbird.  So it is definitely the "From xxx@yyy" line causing the problem.  I am not sure if this is Courier wrong or Thunderbird.

Alan

Comment 3

[David :Bienvenu]

Courier should never be returning a "From " line - that's not a valid rfc822 header, as you say. 

Comment 4

[Alan Kent]

Ahhh!  It turns out procmail is messing up the headers.  I might be using wrong command line options for procmail.  But its not a Thunderbird problem, so I am resolving this.
Thanks
Alan