24399 - [Beta] Ja sender's name shows garbage in the msg view pane.

Status: VERIFIED FIXED

(MailNews Core :: Internationalization, defect, P3)

Description

[ji]

Build: Linux 2001011908

The sender's name in Japanese shows garbage in the msg view pane.
It displays all right on 4.x. I'll check today's Win build as soon as
we get a testable build.

Comment 1

[Katsuhiko Momoi]

Naoki is on vacation until next week.
Rich, this looks like your area. I'm also cc'ing mscott since this problem began hapenning
after 1/5/00.  I don't have an exact date but it was Ok with 1/5/00 build, then we have some untestable
daily builds, then with 1/10/00 build, you see that the display is broken.  This is a cross-platform problem.
ji, can you provide a mailbox file with the sender's name in Japanese and attach it to this bug?

Comment 2

[rhp (gone)]

Message display problems for the headers is all mscott's at this point. Scott,
let me know if you want me to look, but I know that you have dealt with a bunch
of this stuff already.

- rhp

Comment 3

[ji]

Attachment: Attached is a mail folder containing mails with Ja sender name.

Comment 4

[Scott MacGregor]

First off, I'm trying this test on windows and not on linux.

I don't have japanese fonts installed on my system, I would need these installed
in order to view the characters correctly in the message header pane right?

Also, when I try to view these messages, we can't render the headers in the
thread pane either. I see ... for the subject and sender's names.

Comment 5

[Katsuhiko Momoi]

Scott, I placed the Cyberbit font here:

ftp://ftp.netscape.com/pub/communicator/extras/fonts/windows/

Get the Cyberbit.zip file, unarchive and install it using the Control Panel's Font utility. This will get you
display for many languages.
Also there is already bug assigned to putterman about the non-display of sender's names in Japanese in
the thread pane. (Bug 8405.) This bug is about a display problem in the Message Viewing window
which was working OK until recently.

Comment 6

[Katsuhiko Momoi]

Bob, this is a recent regression. Are we going to try to get this done for M13?

Comment 7

[Scott MacGregor]

Thanks Kat...I'm installing the fonts now....

Comment 8

[Scott MacGregor]

I'm not sure why they aren't displaying.
The strings have been converted to unicode by the time they make their way into
the java script used to display the headers.

In javascript, I invoke our msg header address parser by passing in UTF-8 as the
character set of the string and JS automatically converts the unicode string
into a char * which gets passed to the header address parser (which is a C++
object).

so in JS, I call:
msgHeaderParser.ParseHeadersWithEnumerator("UTF-8", emailAddresses);
where emailAddresses is a unicode string. and JS automatically converts that
string to a char *.

I wonder if JS isn't converting it to a UTF-8 string...if it is converting it to
something else instead? Hmmm...

Comment 9

[Katsuhiko Momoi]

Scott, I see in one case, the character is supposed to be \u30bd, but I only see a Latin 1
character correpsonding to 0xBD, etc. In other words, I don't see Japanese but what amounts
to the trail byte equivalent in Latin 1 and nothing matching the lead byte.

Comment 10

[Scott MacGregor]

Kat, I have a fix for this in my tree. I can now display email addresses that
contain japanese characters. The change involved changing an interface used by
JS such that JS only saw unicode. The doing some utf8 to unicode conversions in c++.

If you guys feel this is an M13 stopper bug for I18N, I'll go get it code
reviewed and checked in.

Moderate risk of the fix --> low

I ran through the rest of the I18N smoketest files you've sent me over the past
few weeks and they all came through fine.

Comment 11

[Scott MacGregor]

So what do you guys want to do about this bug? If we want to get my fix checked
in for M13, we need to get it on the radar real soon I think so I can begin the
checkin petition process....

Comment 12

[ji]

We need this fix for M13. Have you tried your fix with the mail folder which
contains the mail with Ja sender name?

Comment 13

[Katsuhiko Momoi]

Scott, if you have the fix in hand and a safe one, please go ahead. Bear in mind that while this is a regression,
it might not meet the strict requirement as it does not impair critical functionality. It just makes us look very
bad to lose something we already had. Note that the subject header is showing OK in the message view.

Comment 14

[Scott MacGregor]

Yeah, I think I posted earlier in the bug report that it passed my I18n
smoketest which included the email folder kat sent me.

Comment 15

[Scott MacGregor]

Thanks Kat. I'll see if I can talk them into it. Moving to M13.

Comment 16

[Katsuhiko Momoi]

After discussing this among Intl QA members, we decided that this meets our M13 stopper requirement.
If a sender's name is in Japanese, there no longer is any way to see this name as the thread pane
display has been broken for some time and if the view in msg pane is broken, that leaves no way
around recovering the name.

Comment 17

[Scott MacGregor]

I've run into a hitch. I was doing some last minute regression testing and
noticed that my changes don't work on Linux! =(

instead of seeing the email address, I see "UTF-8" in place of the name.

Comment 18

[Scott MacGregor]

Ugghh I'm confused. On Linux, the following call on nsString2 returns "UTF-8".
  // need to convert unicode to UTF-8...
  // (line is a PRUnichar *)
  nsAutoString tempString (line);
  char * utf8String = tempString.ToNewUTF8String();
  printf("%s\n", utf8String);

regardless of the value of line, this code prints out the string: "UTF-8"!! so
all my addresses on linux now show up as UTF-8.

Does nsString2.ToNewUTF8String not work on Linux? is there a better more I18n
friendly way to convert PRUnichar * to utf8?

Comment 19

[Katsuhiko Momoi]

Added erik and ftang to the CC list as nhotta is not around this week.

Comment 20

[Scott MacGregor]

Thanks for adding erik and frank Kat. Guys, here's my problem.

In JS code used to display the addresses in the message pane, I have a unicode
string that corresponds to the addresses of a header (like the to line). I then
call in through an IDL interface (nsIMsgHeaderParser) to extract each individual
address.

The method I'm calling Foo takes a unicode string and then converts it to UTF-8
so I can call the old header parsing routine which expects a char * and a charset.

NS_IMETHODIMP Foo (PRUnichar * aLine)
{
   // need to convert unicode to UTF-8...
  nsAutoString tempString (line);
  char * utf8String = tempString.ToNewUTF8String();
}

On linux, utf8String has a literal value of "UTF-8". I've even added a printf in
that method to verify that indeed utf8String is UTF-8.

Am I converting my unicode string to UTF-8 incorrectly?

all of this code works fine on my windows box, the string is converted
correctly. It's my linux platform that has problems.

Comment 21

[Scott MacGregor]

Hi Frank and Erik, can one you guys help me out with the problem I described
above? Thanks!

Comment 22

[Frank Tang]

I am currently looking at it. Where is your code which call the ToNewUTF8String 
?

Comment 23

[Frank Tang]

The first problem I see is the IDL itself. It seems you are define those string 
as type "string" instead of "wstring". Are you sure JS will convert those string 
to Unicode to you correctly if you spec it as "string" ?

The look at  nsMsgHeaderParser::ParseHeaderAddresses 
248   char *utf8Str, *outStrings;
249   MimeCharsetConverterClass *converter = nsnull;
250   nsresult rv;
251 
252   if (nsnull == line || MIME_ConvertString(CHARSET(charset), "UTF-8", line, 
&utf8Str) != 0) {
253     utf8Str = nsnull;
254   }

The utf8Str in line 248 is not initialized and could point to random memory 
location, depend on the compiler and the run time memory. The Window and Linux 
may put different value here. Then... in line 252

252   if (nsnull == line || MIME_ConvertString(CHARSET(charset), "UTF-8", line, 
&utf8Str) != 0) {
253     utf8Str = nsnull;
254   }
so... if line is null the MIME_ConvertString will be called. MIME_ConvertString 
will call MIME_ConvertCharset and aMimeCharsetConverterClass. it will then 
return the value of aMimeCharsetConverterClass.Convert. This function will 
return a value of 0 or -1. It looks like 0 mean success and -1 mean unsuccess. 
If it return sucess then the line 253 "utf8Str=nsnull;" will be skip. 

I guess what happen is the line is null when you get into ParseHeaderAddresses. 
and utf8Str is point to the static "UTF-8", and MIME_ConvertString success. 

The other problem I spot is in
275   if (!converter)
276   {
277     rv = MIME_ConvertCharset(PR_FALSE, "UTF-8", CHARSET(charset), *names, 
278                             len_all, &outStrings, &outStrLen, NULL) ; 
279   }
280   if (NS_SUCCEEDED(rv))
281   {
282       PR_Free(*names);
283       *names = outStrings;
284     }
285   }

first , the return value of MIME_ConvertCharset is a PRInt32- either 0 or -1. 0 
mean success. Howerver, you test it against NS_SUCCEEDED() , which expect a 
nsresult. 

I suggest you first try change the following line
248   char *utf8Str, *outStrings;
to
char *utf8Str = nsnull;
char *outString = nsnull;

first. 

Comment 24

[Frank Tang]

Please ignore my early comment about string/wstring issue in IDL. Since this is 
a mime encoded header, I think using string instead of wstring is correct.

Comment 25

[Scott MacGregor]

Frank, I think your confused. None of my changes are checked in because of this
problem I discovered on linux. Part of my changes are indeed to change part of
this interface to a wstring instead of a string.

Please read the example I wrote in the bug report which is a better illustration
than what's in the code base. Because my changes aren't checked in.

Thanks!

Comment 26

[Scott MacGregor]

Okay I figured it out. I had a bad patch in the JS file when I sent it over to
my linux box. I just re-applied that patch and we are all good now.

I'll get this checked into the M13 branch today.

Comment 27

[Scott MacGregor]

This has been checked into the M13 Branch tonight. 

I won't be checking it into the tip until probably Sunday when I landed the rest 
of the changes in my tree from the past week. So don't worry if you still see 
the problem in the tip builds before Monday =).

Comment 28

[Scott MacGregor]

I checked in the same changes into the tip this evening  as  well. so it should 
be fixed in M13 and M14.

Comment 29

[ji]

Verified with 2000012401-M13 linux build. It is fixed.

Comment 30

[Katsuhiko Momoi]

This is also working on Win32 build from today: Build: 2000012400. 
(Mac is not testable for this part today.)
Based on these 2 reports, I'm marking it verified as fixed