308231 - forward truncates text of some messages containing certain non ASCII character(0x93)

Status: RESOLVED WORKSFORME

(Thunderbird :: Message Compose Window, defect)

Description

[J Luh]

Forward message 

To reproduce:

1) Load attached file as a mail folder.
2) Attempt to forward the test message as inline.

Should create a message containing both lines of test message, but instead
creates a message containing only the first line of the test message.

I ran into this problem when trying to forward a travel confirmation message
sent by Travelocity. Sorry if this is a dupe; I looked for similar bugs but did
not find any.

Comment 1

[J Luh]

Attachment: testcase mailbox

Comment 2

[J Luh]

This is version 1.5 Beta 1 (20050908) on Mac OS X 10.4.2 (8C46).

Comment 3

[WADA:World Anti-bad-Duping Agency]

Following is binary compare result after replacing 0x93 & 0x0A(LF) in your test case by 0x20(Space) & 0x0D(CR).
 
> C:\TEST\bug-308231>fc /b original-data.TXT
          0x93_and_0x0A_is_replaced_by_0x20_and_0x0D_respectively.TXT
> Comparing files original-data.TXT and
> 0x93_and_0x0A_is_replaced_by_0x20_and_0x0D_respectively.TXT
> 0000001F: 0A 0D
> 00000036: 0A 0D
> 00000052: 0A 0D
> 00000067: 0A 0D
> 0000007E: 0A 0D
> 0000008F: 0A 0D
> 000000B3: 0A 0D
> 000000B4: 0A 0D
> 000000DB: 93 20
> 000000DC: 0A 0D
> 00000104: 0A 0D
> 00000105: 0A 0D

New-line in your test case was 0x0A(LF), and the last character of the first line, a character between "message." and LF(x0A), was 0x93.
And no Content-Type: header, no Message-ID: header.

(Q1) What "Character Coding" is selected when viewing the mail of the test case?
(Q2) What is your OS's locale/language setting?
     What character set is used by OS?
     What is your Tb's default character coding when mail viewing?
     What is your Tb's default character coding when mail composition?
(Q3) Who sent the mail with no Message-ID: header?
If no no Message-ID: header, mail server usually appends Message-ID: header.
Is it mail which was attached in a mail as message/rfc822 part?
If this is the case, see Bug 314349.

Comment 4

[J Luh]

As noted in the initial report, the original message was sent by Travelocity.com. The original message was sent without a Content-Type header.

Using Thunderbird 1.5rc2 on US English Mac OS X 10.4.3. I am using UTF-8 for message display and iso-8859-1 for message composition. If I switch to iso-8859-1 for message display, Forward works properly.

The absence of a Message-ID header is irrelevant; Message-ID is not a required header.

Comment 5

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #4)
> As noted in the initial report, the original message was sent by
> Travelocity.com.

My qestion was "What mailer(mail client,Mail User Agent) generated", instead of "What From:".
Sorry for my confusing question.
Seems to be application generated mail instead of mail created by usual mail client, since travel confirmation mail from Travelocity.com.

> I am using UTF-8 for message display
> If I switch to iso-8859-1 for message display, Forward works properly.

(Q4) Do you enable next option?
  - Always use this default character encoding when message are displayed
    (ignore character encoding specified by MIME header)
    Note: prefs.js entry for this option is mailnews.force_charset_override.
          "true" means the option is checked, and "false" means unchecked.
          Default is "false" when Seamonkey, but I don't know when Tb.
          Go Tools/Options/Advanced/Config editor. 
(Q5) (Q1) again.
     What "Character Coding" is selected automaticaly by Thunderbird
     when viewing the mail of the test case?

Comment 6

[WADA:World Anti-bad-Duping Agency]

Unicode Character U+0093 is '<control>'.
See http://www.fileformat.info/info/unicode/char/0093/index.htm.
If you forced UTF-8 by mailnews.force_charset_override=true and mailnews.view_default_charset=utf-8, phenomenon of this bug can easily be explained, and I think "This works as designed".

Comment 7

[J Luh]

Thunderbird surely does not work this way by design. When you forward a message, your outgoing message should contain the entire text of the message. Thunderbird renders both lines of the test message when displaying the message, but it only includes one line of the test message when you forward inline. Either the rendering code handles the data improperly or the message forwarding/composition code handles it improperly.

Comment 8

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #7)
> Thunderbird surely does not work this way by design.

Oh, you are absolutely right.
I've forgot incident of "all text even after '0x93' was displayed" even when viewing as UTF-8.
(I probably thought U+0093 'control' is a kind of "End of text"...)
Code converter from Unicode to OS's character coding seems to stop(or fail) when 0x93(U+0093 when Unicode) appears.
  - See Bug 120728 for problem in conversion from iso-2022-jp/euc-jp to
    Shift_JIS(Japanes MS Win's char coding) when forward/edit and long text.
  - See stil-open Bug 269812 for problem in conversion to OS's character coding
    when "Save as text" and long text.

Can setting mailnews.force_charset_override=false be a workaround?

RFC defines that default charset is us-ascii when no/malformed charset paramater or no Content-Type: header.
But RFC also says "us-ascii" is not always appropriate and recommends mail user agent to has more clever way, then Mozilla family has auto-detect or forcing pre-defined charset functionality in this case.
Since your case is "7bit ascii and 0x93 only text", auto-detect(if exists) possibly treats it as windows-1252 and handling of 0x93 is expected to be successful.

Or both of mailnews.force_charset_override=false and mailnews.view_default_charset=windows-1252 are required?
Or both of mailnews.force_charset_override=true and mailnews.view_default_charset=windows-1252 are required?

Comment 9

[WADA:World Anti-bad-Duping Agency]

Please ignore my question in previous comment. Sorry for spam.

Your case is "No Content-Type: header" and "mail body text", then mailnews.force_charset_override doesn't have relation to viewing mail, and mailnews.view_default_charset(=utf-8) is always applied to mail body, unless user changes view/character coding manualy.
(I confused with (A)No charset parameter & body case, (B)Malformed/invalid charset parameter & body case, (C)Wrong charset parameter & body case, (D)Non-encoded data cases of mail headers in different charset from body text when valid Content-Type: header/no Content-Type: header/no charset/malformed charset.)

Comment 10

[Mike Cowperthwaite]

I've attempted to reproduce this problem with the testcase, using TB 1.5 under Windows 2000.

When the message is loaded from a TB folder, forward-inline results in both lines of text -- but the oddball character is *not* included.  Possibly relevant: The Character Encoding submenu shows 'Western (ISO-8859-1)' which 
is my default.

But when I open the message using File|Open, the Character Encoding submenu 
has no checked item, and only the first line is copied into the forward.

Comment 11

[Mike Cowperthwaite]

xref bug 301291

Comment 12

[J Luh]

This seems to be working ok with 2.0b2 (20070116) on Mac OS X 10.4.8 (PPC).

Comment 13

[Mike Cowperthwaite]

Really.  I'm seeing the exact same results described in comment 10, with 
2b2-0207.

Comment 14

[J Luh]

(In reply to comment #13)
> Really.  I'm seeing the exact same results described in comment 10, with 
> 2b2-0207.
> 

Yes, sorry, this should not be closed

Comment 15

[Mike Cowperthwaite]

The issue of missing the second line of text is still true for the opened-from-file case, with 3a1 and 2.0.  In both cases, there is no encoding shown selected in the menu; and changing the encoding (per the below observations) doesn't change the behavior.

For the open-from-folder case, the behavior has changed slightly (I don't know when).  This behavior depends on which charset the message is viewed as.

In 2.0:
If the folder's default charset is either ISO-8859-1 or Win-1252, when the message is first displayed, the 0x93 is displayed as an unknown-character glyph.  Changing to the other charset, display changes to show the left-
double-quote (LDQ), and the LDQ continues to be displayed regardless of which of these two charsets is subsequently chosen.

If the message is viewed as ISO-8859-1, when you forward inline, the oddball character doesn't appear at all.

If the message is viewed as Win-1252, when you forward inline, the character is displayed as the LDQ.

In 3a1-0418:
The character is always displayed as LDQ, with either charset.

If the message is viewed as ISO-8859-1, when you forward inline, the oddball character is displayed as an 'unrecognized character' glyphs.  Interestingly, if you type the character in (Alt+0147) or paste it in, it displays as LDQ.

If the message is viewed as Win-1252, when you forward inline, the character is displayed as the LDQ (no change from 2.0).


(For what it's worth, the unknown-character glyph on message display, with 2.0, is different from the glyph seen in 3a1 on composition, even tho I'm using the same font for both cases.)

When forwarding inline, it seems that the charset for composition is always the same as the charset that the message originally appeared in.  I'm pretty sure there's a bug about wanting to override that with the default compose pref, but I can't find that right now.

Comment 16

[Wayne Mery (:wsmwk)]

testcase works for me