263797 - Can't forward a long Japanese plain text message in ISO-2022-JP correctly: body text is treated just as ASCII

Status: VERIFIED FIXED

(MailNews Core :: Internationalization, defect)

Description

[Tomoo Nomura]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8a4) Gecko/20040927
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8a4) Gecko/20040927

When replying to Japanese message in line, mozilla can't show the message
correctly. Is this the same as #66098 ?

Reproducible: Always
Steps to Reproduce:
1. unzip the attached file. 
2. copy it to mozilla Mail folder
3. launch mozilla
4. choose the message in test1 folder.
5. click "forward inline" then you will see non Japanese character, may be 8th
bit has been dropeped.
Actual Results:  
Akiko Matusima wrote:

>$B!c!c!c(BKissNet$B%K%e!<%9!!30It5!4X>pJs!!(B2004$B!!(BOct,07$B!d!d!d(B
>
>$B4X@>#I#T6&F1BN!J#K#I#S#S!K(B
>$B2q0w3F0L(B
>
>$B2<5-#27o$N>pJs$r$4;29M$^$G$K$*CN$i$;CW$7$^$9!#!c#K#I#S#S;vL36I!d(B
>$B(,"##I#N#D#E#X"#(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(,(B
>
>$B#1!%#C#R#M6(5D2q4X@>K\It!!Bh#22sL\%;%_%J!<(B
>$B#2!%Bh#22s(B $B%/%j%(%$%7%g%s!&%3%"ElBg:e!V$b$N$E$/$j3W?7=N!W(B

Expected Results:  
Akiko Matusima wrote:


>>＜＜＜KissNetニュース　外部機関情報　2004　Oct,07＞＞＞
>>
>>関西ＩＴ共同体（ＫＩＳＳ）
>>会員各位
>>
>>下記２件の情報をご参考までにお知らせ致します。＜ＫＩＳＳ事務局＞
>>━■ＩＮＤＥＸ■━━━━━━━━━━━━━━━━━━━━━━━━
>>
>>

Is this related to #263254 ?

Comment 1

[Tomoo Nomura]

Attachment: test1 forwared

This is tha same as #263254 Attachment No.1

Comment 2

[Mike Cowperthwaite]

> Is this related to #263254 ?

Maybe, but this one is easy to reproduce.
I get the same result whether my composition preference is set to ISO-2022-JP, 
or to ISO-8859-1 with the "Always use this encoding" preference checked or 
unchecked.  In each case, the actual encoding used is ISO-2022-JP: if pref is 
set to 8859, the titlebar shows "ISO-2022-JP" but the Options|Encoding flyout 
menu does not show any encoding checked; if the pref is set to 2022, the 
titlebar doesn't show any encoding (as expected), and the menu shows 2022 
checked.

However, I have another test message (spam) that matches the test message in 
these ways: text/plain, charset=ISO-2022-JP; Subject header is RFC2047-tagged as 
ISO-2022-JP; From: header is in ASCII -- and differs in that there is no To: 
header, and was sent by (reportedly) Eudora rather than Outlook Express.  That 
message is encoded correctly when forwarded, and also causes no problems in the 
Encoding menu not showing as checked regardless of the preference for 
composition charset.

Reproduced with
  Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8a5) Gecko/20041005
Does not happen with Mozilla 1.7.2 or TB 0.8.

Comment 3

[Mike Cowperthwaite]

xref bug 234958.

Comment 4

[Jungshik Shin]

At one point, I was able to reproduce this bug and suspected that one of my
patches resulted in this regression. However, it turned out that my build (with
which I reproduced the bug) was based on an outdated source tree (that didn't
include two latest patches to nsMsgCompose.cpp. I have separate source trees on
two computers and I tested it on the compute with the outdated tree). With the
latest build of my own, I can't reproduce this bug any more. 

Can you try the latest nightly and see if you can still reproduce this bug. 

Comment 5

[Tomoo Nomura]

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8a5) Gecko/20041013 still
reproduces the problem.

Comment 6

[Jungshik Shin]

Ok. Now I figured out why I wasn't able to reproduce this bug. I was forwarding
an email sent with Mozilla-mail which has

Content-Type: text/....; charset=ISO-2022-JP

while Nomura-san was forwarding (so was I at one point) an email sent with MS OE
which has 

Content-Type: text/....; 
    charset="iso-2022-jp"

This shouldn't make a difference, but somehow it does when we forward such a
message. I have a couple of suspicous spots I'm gonna take a look at.

Comment 7

[Jungshik Shin]

It turned out that it's not the way Content-Type header is written but U+2014 in
your test message that triggered this problem. Our ISO-2022-JP converter regards
U+2014 (EM dash) as covered by ISO-2022-JP. However, iconv in glibc thinks that
it's not covered by ISO-2022-JP.  It seems that we use the native converter
(iconv on Linux) in a place where we're not supposed to. However, I couldn't
figure out where. My brief search turned up nothing. 

In addition, Mike reported that the same thing happened on Windows, too, which
makes my hypothesis less likely to be the case.  

Comment 8

[WADA:World Anti-bad-Duping Agency]

Attachment: Drafts folder file (2 iso-2022-jp mails are contained)

This is file of Drafts folder and contains 2 draft mails I created.
 (1) Subject : Bug 263797 - short message
 (2) Subject : Bug 263797 - long message

Both are plain text mail of "text/plain; iso-2022-jp".
Only single Japanese character is used in these mails, repeated though.
Difference between 2 mail is line number of body text only.

Save this text file in your mail deirectry and restart Mozilla or Thunderbird.
Say file name of "TEST".
You will find mail folder of "TEST". 
Try "Forward" and "Edit as new".
Next problem occurs only on second mail (long mail).
 - ISO-2022-JP is displayed as "ASCII text" when "Forward" and "Edit as new".
Copy 2 mails to Drafts folder then try "Edit Draft".
Next problem occurs only on second mail (long mail).
 - ISO-2022-JP is displayed as "ASCII text" when "Edit Draft"

Comment 9

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #7)
> but U+2014 in your test message that triggered this problem.

Jshin, is it true?
I think this bug can be said as follows ;
 - If iso-2022-jp plain text mail's body size exceeds some limit
   (such as buffer size for code translation)
   no code translation is done on displaying mail,
   or code translation failure is ignored on displaying mail,
   when "Forward", "Edit as new" and "Edit Draft".

Comment 10

[WADA:World Anti-bad-Duping Agency]

There is different but similar problem - Bug 269812, "Save As Text" problem when
long non-ascii mail.
Root cause of it and this bug seems to be same.

Adding descrption in summary for ease of search.

Comment 11

[Jungshik Shin]

This doesn't happen with Japanese messages in Shift_JIS or EUC-JP, does it? 

Comment 12

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #11)
> This doesn't happen with Japanese messages in Shift_JIS or EUC-JP, does it?

As you say, this bug's problem didn't occur on "Edit Draft"/"Forward" when
Shift_JIS mail on Japanese MS Windows of standard locale, Japan
(Japanese,Shift_JIS).
(Note :  Bug 269812 occured even when Shift_JIS mail on MS Win-JP) 

Comment 13

[Tomoo Nomura]

Isn't it the same reason as Bug 134762 ?
I reported both of two.
I used to see the same symptom at very early version.

Comment 14

[Jungshik Shin]

Thanks. It helps ! This may depend on bug 138578. (see also bug 120278)

Comment 15

[WADA:World Anti-bad-Duping Agency]

"Forward"/"Edit as new"/"Edit Draft" worked very well for
ISO-2022-JP,plain/text, long(11KB and 72KB) mail with 2005022405-trunk on
Japanese Win-2K.
Jshin, thanks a lot.

Closing as FIXED (by fix for Bug 138578) and VERIFIED.