366760 - Lightning failing to properly decode quoted-printable text/calendar parts of messages

Status: RESOLVED DUPLICATE of bug 351997

(Calendar :: Lightning Only, defect)

Description

[Jordan Kojouharov]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1
Build Identifier: Mozilla Thunderbird version 1.5.0.9 (20061207), Windows XP SP2

This was noticed when trying to handle .ics meeting invites sent by Microsoft Outlook. Thunderbird Lightning was incorrectly displaying the date of the meeting invite (Lightning version 0.3).

I do NOT believe this is a Lightning problem, as I can reproduce it with other types of files, other than .ics.

If Thunderbird receives an email (using IMAP connection to dovecot IMAP server, in my case) that has an attachment encoded using 'quoted-printable' encoding, AND that attachment contains lines longer than 76 characters - Thunderbird incorrectly handles the 'soft line break' characters used by quoted-printable when accessing the attachment - when saving the attachment, the 'soft line break' characters ('=') are NOT removed during the decoding process.

http://en.wikipedia.org/wiki/Quoted-printable



Reproducible: Always

Steps to Reproduce:
1. Receive (or create) a valid email with a valid quoted-printable encoded attachment, making sure it has lines longer than the 76 character limit set by quoted-printable. This will force the use of 'soft line break' characters at the end of the lines.

This raw attachment can be used as an example:

------_=_NextPart_001_01C735D0.96205123
Content-class: urn:content-classes:calendarmessage
Content-Type: text/calendar;
        method=REQUEST;
        name="meeting.ics"
Content-Transfer-Encoding: quoted-printable

DTSTART;TZID=3D"(GMT-08.00) Pacific Time (US & Canada)/Tijuana":20070111T=
1500
 00
SUMMARY:TEST
UID:040000008200E00074C5B7101A82E00800000000604038878D35C7010000000000000=
00
 010000000F9CB61847E1C154AB4ED9DD6DE3E9735
LOCATION:HERE
DTEND;TZID=3D"(GMT-08.00) Pacific Time (US & Canada)/Tijuana":20070111T15=
3000

------_=_NextPart_001_01C735D0.96205123--

2. Having set 'Display attachments inline', access the attachment using Thunderbird, and try to 'Open', 'Save As' or 'Detach' it using a right-click on the attachment itself.

Actual Results:  
The file viewed/saved/detached does NOT have the soft line break characters (the '=' signs above) properly decoded and removed in the saved attachment. Hence, it is not a valid attachment, the decoding from quoted-printable encoding has not worked correctly.


Expected Results:  
The soft line break characters at the end of every line longer than 76 characters should be removed, and the file properly decoded.

This occurs when the attachment is saved, detached, or opened. However, if the message is 'forwarded', including the attachment - the new email created has a perfectly valid attachment, encoded using '7bit' encoding. So Thunderbird must use a different approach to encoding/decoding when forwarding a message (including attachment), as opposed to when saving, detaching, or opening an attachment.

Again, this was noticed when trying to use Lightning to import calendar events sent by Outlook, which seems to encode using 'quoted-printable' by default. Other invites sent using other encoding methods do not experience this problem.

Current Workaround that I am using is to forward the message onto myself once received. The forwarding of the message, including it's attachment appears to properly deal with the 'quoted-printable' encoding, and the new message received is encoded using '7bit', which works perfectly when saved, detached, or when accessed by Lightning in order to be added as a calendar event.

Comment 1

[Mike Cowperthwaite]

I'm unclear on the testcase -- the longest lines in the data provided has the line-ending '=' in column 74.

Are you expecting the decoded version of line 1 to end with:
 ....20070111T1500
or
 ....20070111T150000
or
 ....20070111T1500 00
?

When I save that data, decoded, to disk, the file has seven lines, the longest of which is 75 characters long.  It looks to me like you're expecting a five-line file, with line 3 about 116 characters long.  If that's so, the data as shown is incorrectly encoded with quoted-printable.

Comment 2

[Jordan Kojouharov]

The '=' is the soft line break character for quoted-printable encoding, as far as I understand, and should not be at all visible in the decoded version of the attachment. From the wikipedia page above:

"Lines of quoted-printable encoded data must not be longer than 76 characters. To satisfy this requirement without altering the encoded text, soft line breaks may be added as desired. A soft line break consists of an "=" at the end of an encoded line, and does not cause a line break in the decoded text."

When I 'save as' the attachment above, i get the first line decoded as it is in the attachment - the soft line breaks ('=') do not get decoded correctly and are visible in the saved file.

When the message is forwarded to myself again, the decoding works fine, the attachment is encoded in 7bit, as opposed to quoted-printable and the file is as expected:

DTSTART;TZID="(GMT-08.00) Pacific Time (US & Canada)/Tijuana":20070111T1500
 00
SUMMARY:TEST
UID:040000008200E00074C5B7101A82E00800000000604038878D35C701000000000000000
 010000000F9CB61847E1C154AB4ED9DD6DE3E9735
LOCATION:HERE
DTEND;TZID="(GMT-08.00) Pacific Time (US & Canada)/Tijuana":20070111T153000

So to answer the question, the first line when decoded correctly should be:

DTSTART;TZID="(GMT-08.00) Pacific Time (US & Canada)/Tijuana":20070111T1500

(the soft line break (=) is removed, and the newline after it removed, ie. - the lines are merged to form the original >76 character line)

Please let me know if there is a better test I can help setup - probably the easiest is to send a simple attachment using Outlook ensuring that the attachment is encoded using quoted-printable, and then view/save that email using thunderbird. 

Comment 3

[Jordan Kojouharov]

Just adding - the attachment should be decoded as a 7-line file, only 3 of the lines are longer than 75 characters, so they end up having soft line breaks. Those are the ones that start with DTSTART, UID and DTEND above. The two lines that start with a space are not actually meant to be joined to the lines above - sorry about the confusion - a calendar file is probably not the best example. I'll see if I can whip up a simple text attachment, or something...

Comment 4

[Mike Cowperthwaite]

Well, I'm not able to reproduce with the text copied from what you pasted.  Better that you should attach the message with the failing attachment to this bug -- save the message as a .EML file, and use the Add Attachment link above.

Comment 5

[Jordan Kojouharov]

Attachment: Test meeting invite sent from Outlook that shows this problem.

This is the test case meeting invite that shows the problem. Some IPs and domain names were removed from the 'Received:' headers for privacy, otherwise the file is untouched. Saved as .EML as per Mike's suggestion.

Comment 6

[Mike Cowperthwaite]

In that sample, the calendar part isn't an attachment.  The message is a multipart/alternative with three parts: text/plain, text/html, text/calendar.
I see the HTML part because it's the latest part in the message that my version can show, since I don't have any calendar extension installed.  And because the text/calendar is an alternate part, it doesn't show as an attachment: I can't get access to it from the TB UI.

You claim that you can see this with other file types; I have to say I'm skeptical of this claim, because:  Unfortunately, quoted-printable decoding isn't handled down in MIME; every software component that can be handed body text needs to check for and decode quoted-printable.  While that could -- should -- be fixed by redesigning MIME, that's not going to happen soon.  

That's why I suspect the problem needs to be handled by Lightning.  But if you can provide a sample message showing this problem for a regular attachment, I'll be happy to confirm the bug.

Comment 7

[Jordan Kojouharov]

Mike,

I *think* I was able to reproduce this with a different type attachment, but in all my attempts to do it now - I have failed. I appreciate your time spent on this, and I am also starting to think that this is a Lightning problem, as opposed to Thunderbird itself.

Thunderbird with the Lightning plugin (0.3.1) shows the text/calendar part as a 'message.ics' attachment, which displays the faulty decoding behavior I explained above.

As Lightning is still in beta, I'm not sure if (or how) this bug needs to be moved there. Please act with it as you think is best, or close it - if it is still present in the first release of Lightning I'll be happy to log it again with Lightning.

Again, I appreciate the time you have spent on this.

Comment 8

[David :Bienvenu]

Is this fixed in 2.0? I think I fixed a core problem in the mime code the calendaring project folks wrote to deal with custom mime types...

Comment 9

[Christian Schmidt]

I looks as if this problem has been fixed between the releases of Thunderbird 2 Alpha 1 and Thunderbird 2 beta 2.