303540 - Forward doesn't work with some email messages when mailnews.send_default_charset and mailnews.view_default_charset both are set to utf-8

Status: RESOLVED FIXED

(MailNews Core :: Composition, defect)

Description

[Tiziano Tissino]

User-Agent:       Mozilla/5.0 (Windows; U; Win 9x 4.90; it-IT; rv:1.7.10) Gecko/20050717 Firefox/1.0.6
Build Identifier: Mozilla/5.0 (Windows; U; Win 9x 4.90; it-IT; rv:1.7.10) Gecko/20050717 Firefox/1.0.6

When trying to forward a message as in-line, the new message window appears with
no text, if the original message hasn't the Content-Type header.

Reproducible: Always

Steps to Reproduce:
1. In the main window, select a message that has no Content-Type headers
2. Menu Message / Forward As... / In line 

Actual Results:  
The body of the new message is empty, apart my signature.

Expected Results:  
The software should default to 'Content-Type: text/plain' if the 'Content-Type'
header is missing (as it already does when showing the original message).

Comment 1

[Mike Cowperthwaite]

This is working for me with TB 1.0+0803 and TB 1.0.6.  Which build are you 
seeing this problem with?  Can you supply a sample message?  (Save the message 
as a .EML file and attach to this bug using Create New Attachment, above.)

Comment 2

[Tiziano Tissino]

Attachment: an email message

It seems that the problem is not linked, as I thought, to the Content-Type
header. The  attached message has that header, but the program doesn't use its
body text to build the reply message.

I'm using Thunderbird v. 1.06, italian localization, on a Windows 98 machine.

Comment 3

[Mike Cowperthwaite]

I've integrated that test message into my mailboxes, and I have no problem 
forwarding it; using plain-text or HTML composition, when I forward inline, the 
new message contains the headers and body of the original.

TB 1.0.6 and TB 1.0+0805, Win2K.  I suppose it's possible the localization has 
broken something, but I'm not set up to test that.

Comment 4

[Sönke Tesch]

Cannot confirm that either, works fine with my German Thunderbird 1.0 (20041206).

Comment 5

[Mike Cowperthwaite]

Tiziano Tissino, do you have any extensions installed for TB?  If so, try 
running TB in Safe Mode and see if you get the same result.

Comment 6

[Tiziano Tissino]

Tryed using Tb in safe mode and get the same result :-(
version 1.0.6 (20050716), italian localization; no themes installed; extensions
installed: enigmail & mailredirect

Comment 7

[domi]

same with my own messages from customers (without attached file) :-(
with 1.5RC1  or 1.07  / under XPpro or 98se 
no special extension 
the same if inline or offline

the problem is ONLY with some msg having a multipart content
and using the advanced option "including the old msg in the main" 

if the msg contain several "Content-Type: multipart/related" segments
all will be lots and i will have en empty window

if the msg contain one "Content-Type: multipart/alternative"
i've got only a partiel forward with some sentences fropm the msg but not all of them

it s OK if choose replying and modify the adress
the "content-type: multipart" are lost from the header of the new msg
but all the sentences are there

it s OK if the old msg is sent as an attached file

Comment 8

[Rachel]

Not sure which bug to file this under, but I'll try several and hopefully will get some answer.  Using [ ] instead of < > just to make sure this displays properly.

Whenever I have an HTML email of this structure:
[html]
[head]
[title]...[/title]
[style type="text/css"]...[/style]
[/head]
[body]remainder of email here[/body]
[/html]
I can receive it fine.  However, if I forward it (Inline), my email is not formatted properly.  I looked at the source, and what Thunderbird does is process the first part of the message as normal (i.e. html, head, title, style & body tags), but directly after the body tags, adds 2 [br] tags, an HTML table consisting of the To,From,Subject, etc., then another 2 [br] tags, then the [title] and [style] tags again, then no body tag, but all the content of the [body] tag, and finally the ending [html] tag.  If I forward my message as an attachment it works just fine, as Thunderbird does not change the HTML of my message in any way.  Also, I tested my email in Outlook and it works fine there.    Is there any way you can change the way you display the email headers in a forwarded message so that the message source is not changed as such?  Oh yeah, and in my particular instance, I am using XHTML and NO tables, however, I have noticed similar behavior in any HTML email which has styles embedded in the [head] tag.

Comment 9

[Mike Cowperthwaite]

(In reply to comment #8)
> However, if I forward it (Inline), my email is not
> formatted properly.  

I'm not sure what the problem is: is your basic complaint the table of email headers at the beginning of the message (bug 62698 maybe?), or the superfluous contents from the <head> being put into the message body?  Or is the rest of the forwarded message somehow not being rendered correctly?

Comment 10

[christianoswal]

This bug does it occur if “Enable spell as you time” is disabled ?


(In reply to comment #9)
> (In reply to comment #8)
> > However, if I forward it (Inline), my email is not
> > formatted properly.  
> 
> I'm not sure what the problem is: is your basic complaint the table of email
> headers at the beginning of the message (bug 62698 maybe?), or the superfluous
> contents from the <head> being put into the message body?  Or is the rest of
> the forwarded message somehow not being rendered correctly?
> 

Comment 11

[Robert]

It seems this bug is still present in latest TB 2.0.0.14 and also in 3.0 alpha. This is quite annoying bug since all content is lost when you try to forward such mail.

Comment 12

[Robert]

This bug is still present in Thunderbird 3.0 beta 1. I can't stress enough how important is that this thing work properly since email content is lost if people doesn't notice this behavior.

Of course some conditions has to be met before this bug can be reproduced:

1. Options -> Composition -> General tab: Forward messages has to be set to "Inline" (default in TB 3!)
2. Options -> Display -> Formatting -> Fonts button: Character encodings has to be set to Unicode (UTF-8) for outgoing and incoming mail.
3. Mail message we want to forward is usually sent from MS Outlook, MS Exchange or some other MS mailing system. 

Ticking "Enable spell as you type" as suggested, doesn't have effect.

The bug is present on Windows and Linux operating systems.

Forwarded message is usually cut with the first non ASCII character. 

Workaround is still possible with replying to such message where you have to replace replied mail address with the actual recipient.

Comment 13

[Mark Banner (:standard8)]

I've just done a little test with my debug build. I imported the attached message into a folder (I think I could have just saved it as an eml though), and tried replied and forward in line, after having set my prefs as per comment 12.

From the forwarding case, it would appear these errors are preventing forwarding inline:

WARNING: Invalid UTF-8 string: file /Users/moztest/comm/main/src/mailnews/base/util/nsMsgI18N.cpp, line 152
###!!! ASSERTION: not a UTF8 string: 'Error', file ../../dist/include/string/nsUTF8Utils.h, line 130
###!!! ASSERTION: Input wasn't UTF8 or incorrect length was calculated: 'Error', file /Users/moztest/comm/main/src/mozilla/xpcom/string/src/nsReadableUtils.cpp, line 287

When replying, I'm getting:

###!!! ASSERTION: Illegal value (length > position): 'aLen > aPos', file /Users/moztest/comm/main/src/mozilla/intl/lwbrk/src/nsJISx4501LineBreaker.cpp, line 775

though the reply action does work.

So it appears we're assuming/trying to force UTF-8 in the forward case, even thought it isn't (the first function failing being nsMsgI18NConvertToUnicode).

Comment 14

[[e-mail]]

I looked at mime_parse_stream_complete in mimedrft.cpp; the body is in ISO-8859-1 but there is no charset override, header charset or mail charset and so the body does not get converted. (Ironically if there was a charset then the body would have been converted to UTF-16, then back to UTF-8 so it could be escaped and wrapped in a <PRE> block, then back to UTF-16...)

Should the charset default to ISO-8859-1 if no other charset can be found?

Comment 15

[Magnus Melin [:mkmelin]]

Why not UTF-8?

Comment 16

[[e-mail]]

Because the default character set for email is ISO-8859-1?

Comment 17

[Magnus Melin [:mkmelin]]

Sure, but what if the characters can't fit there?

Comment 18

[[e-mail]]

No, this is the source characterset...

Comment 19

[Robert]

Can this be fixed in 3.0?

The reason we switched to pure UTF-8 was because of HTML signatures which didn't work well with ISO-8859-2 and most of emails sent from other clients.

So we either have to switch back to plain text signatures or use "reply" workaround. I guess UTF-8 is in use so widely, that it should be fixed.

Comment 20

[Robert]

Any news about this bug? I have to make something in my company since people keep loosing their emails because of this.

Comment 21

[Magnus Melin [:mkmelin]]

I'll take a look at this.

Comment 22

[David :Bienvenu]

Thx, Magnus, putting in b3, but would be really nice to have for b2.

Comment 23

[Magnus Melin [:mkmelin]]

Attachment: proposed fix

Use CopyASCIItoUTF16 if we tried to convert to unicode but couldn't. Won't display correct text for all encoding of course, but since the message doesn't specify any... Since we tried to do the correct thing (or maybe that's debatable due to the override...) and failed, at least this displays something and often the correct thing.

Comment 24

[Mark Banner (:standard8)]

Comment on attachment 359938 [details] [diff] [review]
proposed fix

I'm not convinced this is right. I tested on my mac with the preferences set as per comment 12.

I loaded in the attached email as an eml file, and selected forward. With and without the patch, nothing was displayed (emacs confirms its in dos format which should be iso-8859-1).

Additionally I also had a couple of japanese messages in my folders, and the display when forwarding was actually worse with this patch than without.

Comment 25

[Magnus Melin [:mkmelin]]

I only tested the patch with the testcase as mbox, which works fine. There is some other issue with forwarding .eml files - mdd->options->default_charset isn't getting set so we end up having no bodyCharset at all. libmime fun...

Did the japanese messages have charsets specified? Did fwd work at all before this patch? I'm surprised it's worse since the patch would only change what happened if conversion failed, so we'd most often have just an empty body.

Comment 26

[Magnus Melin [:mkmelin]]

Comment on attachment 359938 [details] [diff] [review]
proposed fix

Reminder :)

Comment 27

[Mark Banner (:standard8)]

Comment on attachment 359938 [details] [diff] [review]
proposed fix

Sorry for the delay in getting back to this.

It appears my issues were to do with incorrectly encoded files that I was trying to poke into mbox. Fixing them now means that this patch works as expected. So r=me lets get this in now and watch out for any issues.

Comment 28

[Magnus Melin [:mkmelin]]

Filed bug 480957 for the .eml case.

changeset:   2108:8a16c5a6c298
http://hg.mozilla.org/comm-central/rev/8a16c5a6c298

->FIXED

Comment 29

[Robert]

Looks like it's working properly with latest nightly build :-)
I will test some more emails to make sure.

Thanks!