64119 - Import from Eudora-J: Ja chars in the mails of Out and Trash folder are not displayed correctly

Status: VERIFIED WORKSFORME

(MailNews Core :: Internationalization, defect)

Description

[ji]

This bug is seperated from Bug 58897.

After import finishes, there is Eudora account created. In (jushin), Out
(soushin) and Trash (Gomibako) folder in addtion to some default folders are
created under this account. For the mails in Out and Trash folder, the Ja 
characters in the subject are displayed as question marks both in the thread 
pane and msg view pane. The Ja chars in the mail body are displayed as dots.

Comment 1

[nhottanscp]

Reassign to [e-mail], cc to putterman.

Comment 2

[Katsuhiko Momoi]

Mass change of QA contact to ji for the bugs filed by her.

Comment 3

[nhottanscp]

Scott, could you reassign if [e-mail] is not available?

Comment 4

[scottputterman]

reassigning to cavin.

Comment 5

[Cavin Song]

So, the problem does not happen to In(box), right? Or does it apply to all
folders? Just want to make sure because the summary only mentions Out and Trash
folders.

Also, can someone send me some Eudora folders with Japanese chars so that I can
import the msgs and debug the problem right away? You can just send me
everything under 'Eudora' directory in a zip file. I want to avoid installing
Eudora-J because I just visited http://www.eudora.ne.jp/ and I could not find
the download button as I can't read Japanese (I ran NJWIN viewer to show
Japanese in the browser).

Comment 6

[Cavin Song]

OK, I can now read messages with Japanese chars on my NT machine. Somehow I was
asked to download and install 'MS Global IME for Japanese' from the browser when
I visited one of the Japanese page, and I was able to view Japanese chars after
the installation.

So my guess is that the message in question either has the wrong charset in the
Subject field or the encoding of the string is wrong. I have to see what the
message looks like first. Can anyone provide me with a message to reproduce the
problem?

Comment 7

[ji]

Attachment: A zip file of Eudora mail boxes.

Comment 8

[ji]

I just attached a zip file which contains a set of Eudora4.3-J mailboxes.
To reproduce the problem, you need a Japanese system or Japanese enabled English 
 W2k system:
Unzip the file and copy over the mailboxes to your Eudora directory, then run 
6.1 to import mails from Eudora.
The mailboxes include default In, Out and Trash mailbox and another user created 
mailbox. All these mailbox names are in Japanese, that's why you need a Japanese 
system or Japanese enabled English W2K system.
Only mails in In mailbox can be imported correctly from Eudora-J. The mails 
in other mailboxes, like Out, Trash and user created mailbox, can't.

Comment 9

[Cavin Song]

So is there any way to do this on an NT system or it has to be a w2k?  I went to
Regional Settings under Control Panel I didn't see Japanese on the list. How do
I install/add it to the list?

I also looked at messages in the test folders and the message subjects are not
encoded by Eudora at all. An example of this (encoded in base64) looks like the
following:

  Subject: =?iso-2022-jp?B?GyRCJFgkaCQmJDMkPRsoQg==?=

We do display correctly for the above subject data.  I'll have to see what the
import does when handling 8 bit data in the subject field.

Comment 10

[ji]

It has to be Japanese NT or a Japanese enabled English w2k system.
To make English W2K enabled for Japanese, please do the following steps:
1. Go to Control Panel -> Regions
2. On Language tab (not sure about the name of the tab, it's the 1st one)
you can see a language list, check on Japanese
3. On the same window, click on Input Locale,
 add Japanese, then click on Apply, it will ask you for the installation 
CD. Reboot machine, you'll have a Japanese enabled Win2K. The win2k UI 
stays in English.

Comment 11

[Cavin Song]

My development machine runs English NT, is there any easy way to make it a
Japanese NT?  If it's too muich trouble then I'll have to debug it on win2K.

Comment 12

[ji]

It looks like you can access the testing folders on English NT, although the 
folder names are probably not displayed correctly. Since this bug is about the 
message display and we don't have problems access folders, I think you can debug 
on English NT for this particular bug. Naoki, what do you think?

Comment 13

[nhottanscp]

I think the implementation is depending on OS file system charset (e.g. folder
name), so need JA system.

Comment 14

[Cavin Song]

Hm, looks like nhotta is right as we do perform system specific conversion in
nsImportService::SystemStringToUnicode() where it uses platform specific charset
to convert the input string to unicode. On my English NT box the charset is
'windows-1252' and the conversion is wrong because I can never see the Japanese
chars displayed (I can view a correctly encoded Japanese subject on my machine).
The import code does encode the subject header in the following format though:

  Subject: =?iso-2022-jp?B?LCwsLGZlZlhmZw==?=

Here is the data conversion flow that I found from reading the code:

                charset = system's         charset = UTF-8  
  JP chars --> ConvertSysToUnicode() --> ConvertFromUnicode() 
                       ^                         ^                      
                  GetHeaderValue()           SetSubject()        
                   Import code              ComposeField code

Then the final step is to encode the subject (after doing all the above
conversions) by calling nsMsgI18NEncodeMimePartIIStr() with charset
'iso-2202-jp'.  I'll need to try it on a Japanese charset to see if the (final
step of) encoding is right or not. 

Comment 15

[Cavin Song]

OK, finally get to work on it on a Japanese win2K machine and here is what I 
found.  All messages in the Trash and Out folders do not have the Content-Type: 
header which contains the charset info.  As such, the charset is defaulted 
"US-ASCII" when we encode the subject in function 
nsMsgI18NEncodeMimePartIIStr(). Since the charset is wrong we get the 
incorrect encoded subject header.  For testing purpose, I hard coded the charset  
to "Shift_JIS" (ie, platform's charset), import the same message again and it 
worked fine.  So I think the fix is to set the default message content charset 
to the platform's if it's missing from the message.

Another problem I found with import is that some of the Japanese chars (I guess 
those with high bit set) in the body are converted to dots and it's caused by 
the following function call:

 rv = nsMsgI18NConvertFromUnicode( theCharset, uniBody, body);

This needs to be fixed as well.  The call also causes a lot of assertion dialogs 
to be displayed.

Comment 16

[Cavin Song]

The body problem I found was caused by the charset problem too. So if you pass 
the system/platform charset (when the msg does not specify one) to 
nsMsgI18NConvertFromUnicode() everything works fine and the assertion dialogs go 
away too. I'll submit the patch right after this.

Comment 17

[Cavin Song]

Ccing sspitzer.

Comment 18

[Cavin Song]

Attachment: Fix the char set conversion problem.

Comment 19

[(not reading, please use [e-mail] instead)]

r=sspitzer

Comment 20

[Scott MacGregor]

sr=mscott

Comment 21

[Cavin Song]

Fix checked in.

Comment 22

[ji]

Checked with 08/10 trunk build. It seems that mails with attachments are not 
imported properly. After import, the attachment is displayed as a path. 
Screen shot to follow.

Comment 23

[ji]

Attachment: A scrrenshot.

Comment 24

[Cavin Song]

It could be a separate issue and I need to look into it more. ji, do the Eudora
mailboxes you submitted earlier have the attachments I can test?  If not, please
attach one to the bug. Thanks.

Comment 25

[ji]

Attachment: A zip file of a Eudora mailbox.

Comment 26

[ji]

The Eudora mailbox attached above contains two Ja mails. The first one is 
attached with a HTML file with filename in Japanese. The second one is attached 
with a txt file with Shift-JIS contents but the attachment filename is in 
English.

Comment 27

[ji]

Filed seperate bug 95613 for the attachment issue, since the attachment problem 
affects all the mailboxes including In mailbox.
Marked this as verified.

Comment 28

[ji]

This is broken on 10/02 0.9.4 build. The import wizard can finish w/o any error.
But Japanese mails in Out and Trash folders are not imported correctly. The mail
body is displayed garbled after import. Ja mails in In folder look okay. Reopened.

Comment 29

[ji]

Nominating for nsbeta1.

Comment 30

[ji]

Renominating for nsbeta1 since it causes data corruption.
apanese Eudora users will come across this problem when switching to our client.

Comment 31

[scottputterman]

marking nsbeta1- per mail triage.

Comment 32

[Frank Tang]

remove nsbeta1. please consider this as a adt2 nsbeta1+ for m1.2final release

Comment 33

[ji]

Naoki and I looked at this bug again yesterday. It seems that for the mails sent
and received on the same machine, charset info for mail body is removed after
import. It caused the display problem reported in this bug. But for the mails
sent and received on different machines, charset info is reserved.
Since usually users send and received mails using different accounts and on
different machines ，they won't see this problem.
Marked this one as wfm.  

Comment 34

[ji]

Marked it as verified.

Comment 35

[ji]

Above comments is about mails in Trash folder. For the mails in Out folder, it
looks like Eudora-J itself removes the mail body charset info whenever it copies
the mail to Out folder, a "limitation" of Eudora-J.