Last Comment Bug 7844 - JPN Local Mail folder names do not show correctly
: JPN Local Mail folder names do not show correctly
Status: VERIFIED FIXED
[PDT+] need approval
:
Product: MailNews Core
Classification: Components
Component: Internationalization (show other bugs)
: Trunk
: x86 Windows NT
: P3 normal (vote)
: M14
Assigned To: scottputterman
: Katsuhiko Momoi
:
Mentors:
: 23623 (view as bug list)
Depends on: 6770
Blocks: 7228 11091
  Show dependency treegraph
 
Reported: 1999-06-09 11:10 PDT by Katsuhiko Momoi
Modified: 2008-07-31 01:22 PDT (History)
7 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments

Description Katsuhiko Momoi 1999-06-09 11:10:47 PDT
** Observed with 6/8/99 Win32 build **

It doesn't look like Japanese folder names are supported
currently in that they seem to be interpreted as Latin 1 or
UTF-8. Try something like: テスト

There is also a special problem with 0x5c JPN
names in that folders containing such names do not show up
at all.  Example is something like: 日本語の表示
Comment 1 nhottanscp 1999-06-09 11:18:59 PDT
This bug is for local folder names, I think.
Probably charset conversion is needed. I18n can provide infomation but I don't
know about the current design and implementation of local folder name.
Reassigning to putterman@netscape.com.
Comment 2 David :Bienvenu 1999-07-06 10:39:59 PDT
I don't think this is happening for m9, though I did change the GetName
nsMsgFolder routine to support wide chars, I don't think we ever put wide chars
in the name. please correct me if I'm wrong.
Comment 3 scottputterman 1999-07-06 10:46:59 PDT
I don't think I'm doing that.
Comment 4 scottputterman 1999-08-02 17:45:59 PDT
reassigning to M10
Comment 5 Phil Peterson 1999-08-27 11:16:59 PDT
Triaging to M11
Comment 6 scottputterman 1999-09-19 15:04:59 PDT
This will not be fixed for M11 and I question whether it will be fixed for M12.
I will mark it M12 to allow for those who might want it fixed to comment on
this.
Comment 7 msanz 1999-09-20 12:24:59 PDT
We need this fixed for beta; Japanese users won't be able to display these
folder names in their mail.
Comment 8 lchiang 1999-09-20 17:01:59 PDT
(target milestone is M11 or M12 - add to mail beta tracking bug)
Comment 9 bobj 1999-11-22 16:05:59 PST
Added [BETA] per 9/20 comment.
Comment 10 Katsuhiko Momoi 2000-01-11 00:13:59 PST
Hi, we need to get moving on this for Beta.

I think what we need to do on this.

1. Make sure that when we display folder names on a system,
   we assume that the file names are in the file-system charset
   of the operating system Mozilla is operating on.
2. Make sure that when we create a new local mailbox names,
   we convert whatever we receive in input to the OS's file-system
   charset.

The I18n group be able to provide info on how to query the system
for the file-system charset.

Bob/Naoki, we need to get this on our Beta radar.
Comment 11 Katsuhiko Momoi 2000-01-11 00:23:59 PST
*** Bug 23623 has been marked as a duplicate of this bug. ***
Comment 12 nhottanscp 2000-01-11 09:49:59 PST
Here is the code to get the platform file charset.

  #define NS_IMPL_IDS
  #include "nsIPlatformCharset.h"
  #undef NS_IMPL_IDS

  nsCOMPtr <nsIPlatformCharset> platformCharset;
  nsAutoString aPlatformCharset;
  rv = nsComponentManager::CreateInstance(NS_PLATFORMCHARSET_PROGID, nsnull,
                                          NS_GET_IID(nsIPlatformCharset),
                                          getter_AddRefs(platformCharset));
  if (NS_SUCCEEDED(rv))
  {
  rv = platformCharset->GetCharset(kPlatformCharsetSel_FileName,
                                   aPlatformCharset);
  }

For the charset conversion from unicode to the platform charset,
ConvertFromUnicode (see nsMsgI18N.h) can be used.
Comment 13 Phil Peterson 2000-01-27 10:38:36 PST
Is this the same as bug 7130?
Comment 14 Katsuhiko Momoi 2000-01-27 11:18:59 PST
No. This is for Local folder names. Bug 7130 is for IMAP
folder names. I believe the latter requires special conversion
as IMAP requires the folder names to use modified UTF-7.
Comment 15 nhottanscp 2000-01-27 11:22:51 PST
I think 7130 is about conversion between unicode (M-UTF-7 <-> UCS2).
This is about unicode and file system charset.
BTW, the sample code I put is already available as a function (nsMsgI18N.h).
Comment 16 leger 2000-01-28 17:48:05 PST
Putting on PDT+ radar for beta1.
Comment 17 scottputterman 2000-02-12 20:12:20 PST
OK I will try this out.  This isn't as bad as I had once imagined because as 
David said, we are now passing PRUnichar*'s around whereas I thought we were 
still passing char*.  Now I just need to make sure they get converted correctly.

At least the last time I looked, we were getting the folder name from the uri 
and we were getting the uri from the filename. Am I going to be able to test 
this out on my machine?
Comment 18 Katsuhiko Momoi 2000-02-12 23:31:42 PST
I have Japanese local mailboxes but these would not do 
for US Windows. I think for US Windows machines, Latin 1
accented folder names will have to do for debugging.
For Japanese and other languages, if you let us know 
that you checked in a code that should work, we'll check it
out for you. (You can change the system's default language
via Control Panel but this tends to be very messy and not
recommended.)

I just tried migration of JPN mail fodlers from
4.x and it is botching a file with 0x5c 3nd byte.
It chops off the name in half. I'll file another bug on
this.
Comment 19 scottputterman 2000-02-14 10:37:45 PST
sorry, I guess what I was asking before is that I can view Japanese Imap folder
names.  Is there any reason why I shouldn't be able to view Japanese local
folder names?
Comment 20 Katsuhiko Momoi 2000-02-14 10:42:55 PST
Yes, because the local folder names use the system charset, e.g.
Shift_JIS on Japanese Windows while IMAP folder names are
stored in modified UTF-7. We would need different
conversion, don't we?
Comment 21 scottputterman 2000-02-14 10:47:20 PST
ok.  I will try out what you suggested now using the code Naoki suggested.
Comment 22 scottputterman 2000-02-14 17:04:01 PST
I think one of the problems is that we build up the file names using
nsFileSpec's and I think there are only char * interfaces.  Won't this make this
not work?


Comment 23 bobj 2000-02-15 17:00:07 PST
nsFileSpec reads and writes files using the OS's file system charset
encoding.  These encodings should be represented using char *.
Does that answer your question?  nhotta knows the details...
Comment 24 scottputterman 2000-02-15 17:01:12 PST
Yeah it does.  I talked to Naoki about this earlier.  I tried something out and 
according to Kat it was better but not there yet.
Comment 25 scottputterman 2000-02-18 14:32:37 PST
changing estimated date.  I will need to debug on a Japanese machine.  The hope 
is that this will be easy to find.  Putting 2/25 down just in case it isn't.
Comment 26 nhottanscp 2000-02-18 15:09:45 PST
After I applied a patch from Scott, I was able to see a Japanese folder on my 
localized Japanese windows. But cannot see the messeages in that folder.
We can schedule and use my machine to debug.
Comment 27 Jean-Francois Ducarroz 2000-02-18 18:05:50 PST
Scott, I can help you on this one. As the MacOS charset is not ISO Latin 1, I 
should be able to reprocude this problem by just using some 8 bits characters on 
my "English" Mac.
Comment 28 nhottanscp 2000-02-18 21:02:17 PST
Agreed, that should be easier than debugging raw Shift_JIS data on my Japaense 
NT.
Comment 29 Jean-Francois Ducarroz 2000-02-21 11:39:03 PST
I've just created a local forder named "L'été" and everything is fine except that the filename used for the folder was 
using the wrong charset.
Comment 30 scottputterman 2000-02-22 09:11:18 PST
I wonder if the patch I gave Naoki would fix JFD's charset problem?

Anyway, I want to work on this today.  Could someone give me a Japanese folder I 
can ftp over to my mac?  Also, what should I be expecting to see in the 
debugger?
Comment 31 nhottanscp 2000-02-22 10:38:39 PST
I failed to upload Japanese named file to the server, neither mail attachment 
succeeded. Could you pick up a floppy in my cube?
Anyway, the folder contains two Japanese characters (0x82A0 and 0x82A2 in 
Shift_JIS), in unicode they are \u3042 and \u3044.
If you debug on non Japanese machine, you also need to hard code the charset 
name (Shift_JIS) in msgCompFileSystemCharset().

Comment 32 nhottanscp 2000-02-22 14:43:47 PST
The debugging tip I mentioned does not work on Windows unless using localized 
Japanese system. The file name turned to question marks so not able to convert 
even with hard coded charset name.
But it works on Macintosh. The file name was passed in native encode (Shift_JIS) 
but not turned to question marks. I used MacOS9 with Japanese option installed.
Comment 33 Phil Peterson 2000-02-23 18:20:46 PST
per PDT, this will become PDT- after 2/25
Comment 34 scottputterman 2000-02-24 17:34:35 PST
The following patch  makes it so I can see a folder with accents on my Mac 
without the recurring children that momoi was seeing (I saw this too so I know 
this fixes it).  I can also select a folder and see all of the headers.  
Unfortunately, I can't read any messages.  Naoki, could you test this out and 
verify that this works on a Japanese machine?

ndex: nsLocalMailFolder.cpp
===================================================================
RCS file: /cvsroot/mozilla/mailnews/local/src/nsLocalMailFolder.cpp,v
retrieving revision 1.185
diff -r1.185 nsLocalMailFolder.cpp
72a73
> #include "nsMsgI18N.h"
218a220,221
>     nsAutoString convertedFolderNameStr;
>     const nsString fileCharset = msgCompFileSystemCharset();
223a227
>         
225c229
<         currentFolderNameStr = folderName;
---
>         currentFolderNameStr = folderName;        
232a237,238
>         ConvertToUnicode(fileCharset, folderName, convertedFolderNameStr);
>         child->SetName(convertedFolderNameStr.GetUnicode());
Comment 35 nhottanscp 2000-02-24 18:17:49 PST
I applied the patch and ran on WinNT-J (Japanese localized NT).
A folder with Japanese name was displayed correctly.
But I was not be able to see the headers in thread pane. Probably that's because 
my tree is not up to date. I will try again tomorrow  with updated tree.
Comment 36 scottputterman 2000-02-24 18:22:08 PST
Yeah, that's important.  It needs the changes that bienvenu checked in to fix 
problems with uris and non-ascii characters.  The reason I can't display a 
message is that strtok is finding a '/' in one of my characters and messing up 
the path.  I think it needs to be escaped.
Comment 37 scottputterman 2000-02-24 19:14:28 PST
If I add the following code, I can display messages as well.  This is the same 
code bienvenu wrote to replace strtok in nsMsgFolder.  I wonder if this is 
affecting us in even more places.  Naoki,  when you get a new build, can you try 
this out as well.  I think this will fix this bug.

Index: nsLocalUtils.cpp
===================================================================
RCS file: /cvsroot/mozilla/mailnews/local/src/nsLocalUtils.cpp,v
retrieving revision 1.33
diff -r1.33 nsLocalUtils.cpp
189d188
<     while ((*curPos)=='/') curPos++;
194,214c193,219
<     
<     // for each token in between the /'s, put a .sbd on the end and
<     // append to the path
<     char *newStr=nsnull;
<     char *temp = PL_strdup(curPos);
<     char *token = nsCRT::strtok(temp, "/", &newStr);
<     while (token) {
<       nsCAutoString dir(token);
<       
<       // look for next token
<       token = nsCRT::strtok(newStr, "/", &newStr);
<       
<       // check if we're the last entry
<       if (token) {
<         NS_MsgHashIfNecessary(dir);
<         dir += sbdSep;            // no, we're not, so append .sbd
<         pathResult += (const char *) dir;
<       }
<       else {
<         NS_MsgHashIfNecessary(dir);
<         pathResult += (const char *) dir;
---
>         
>         nsCAutoString newPath("");
> 
>     nsCAutoString oldPath(curPos);
>     nsCAutoString pathPiece;
> 
>     PRInt32 startSlashPos = oldPath.FindChar('/');
>     PRInt32 endSlashPos = (startSlashPos >= 0) 
>         ? oldPath.FindChar('/', PR_FALSE, startSlashPos + 1) : 
oldPath.Length() - 1;
>     if (endSlashPos == -1)
>         endSlashPos = oldPath.Length();
>     // trick to make sure we only add the path to the first n-1 folders
>     PRBool haveFirst=PR_FALSE;
>     while (startSlashPos != -1) {
>       oldPath.Mid(pathPiece, startSlashPos + 1, endSlashPos - startSlashPos);
>       // skip leading '/' (and other // style things)
>       if (pathPiece.Length() > 0) {
> 
>         // add .sbd onto the previous path
>         if (haveFirst) {
>           newPath+=".sbd";
>           newPath += "/";
>         }
>         
>         NS_MsgHashIfNecessary(pathPiece);
>         newPath += pathPiece;
>         haveFirst=PR_TRUE;
215a221,230
>       // look for the next slash
>       startSlashPos = endSlashPos;
> 
>       endSlashPos = (startSlashPos >= 0) 
>             ? oldPath.FindChar('/', PR_FALSE, startSlashPos + 1) : 
oldPath.Length() - 1;
>       if (endSlashPos == -1)
>             endSlashPos = oldPath.Length();
> 
>       if (startSlashPos == endSlashPos)
>           break;
217c232,234
<     PL_strfree(temp);
---
>     
>     pathResult+=newPath.GetBuffer();
> 
Comment 38 nhottanscp 2000-02-25 09:47:40 PST
I applied both patches and I can see Japanese folder name and headers in thread 
pane also messages in the view pane. So it is working for Japanese (tested on 
localized Japanese Windows NT).

The only problem I saw was that a Japanese folder name which contains 0x5c 
(backslash) is not shown correctly (a character contains 0x5c is truncated).
No thread pane header displayed for that folder either. I think this special 
case can be filed as a separate bug.
Comment 39 scottputterman 2000-02-25 13:14:57 PST
changing status to need approval to checkin
Comment 40 bobj 2000-02-25 13:44:33 PST
I assume that bienvenu and nhotta have already reviewed this, correct?
Have ftang review this too.  He's been tracking down lots of nsString
misuages which might be causing the 0x5c problem.
After ftang's review, you can check-in on my approval.  (Run the pre-checkin
tests, yada yada.  You know the drill...)
Comment 41 Frank Tang 2000-02-25 13:45:23 PST
I cannot review the of mozilla/mailnews/local/src/nsLocalMailFolder.cpp becuase
I cannot see the context. Can you attach -c version here.
Also, you should take a look at the patch in 6770 and try the new
PRUnichar*/nsString flavor of GetLeafName()
Comment 42 Frank Tang 2000-02-25 13:47:13 PST
also, when you have time (not necessary before you check in.) apply the patch in

bug 28424 (remember to change the DEBUG_ftang to your name) and verify there are

no illegal nsString usage in your code.


Comment 43 nhottanscp 2000-02-25 13:54:00 PST
>Also, you should take a look at the patch in 6770 and try the new
> PRUnichar*/nsString flavor of GetLeafName()
I'll do this since it needs Japanese system in order to verify.
Comment 44 nhottanscp 2000-02-25 17:11:43 PST
I include the patch of 6770 and rebuild. I also change the patch of this bug to 
use the unicode based GetLeaf method.
I was able to see the folder name which contains 0x5c but cannot see the headers 
in thread pane. Looks like more place need change in order to resolve 0x5c bug.
Comment 45 nhottanscp 2000-02-25 18:21:12 PST
As I mentioned, there are more than one problem to fix 0x5c problem.
I got bobj's approval for the check in. Scott, please check in your patch.
Comment 46 Robert John Churchill 2000-02-25 18:23:04 PST
Note: 6770 has been fixed. Per ftang, the additional method is:  
    GetLeafName(nsString);
not
    GetLeafUniName(nsString);
Comment 47 scottputterman 2000-02-25 22:54:51 PST
I'll check this in tomorrow.  What do I have to change in my patch?  I don't 
have any way of trying out any changed code over the weekend, but if you tell me 
what the changes are, I can put them in.
Comment 48 nhottanscp 2000-02-25 23:16:53 PST
I talked to bobj and agreed that your patch can be checked in without the new 
GetLeafName (nsString version).

As far as I tried using the new GetLeafName, it solved the display problem of 
0x5c contained folder (not able to see headers in thread pane). So not useful 
until we fix other problems around 0x5c (which we don't know exactly where). 

Basically by using the nsString version of GetLeafName, you do not have to do 
the conversion in your code. But if we change to use it we need to test (I 
cannot test over the weekend). And as I mentioned, that change itself just fixes 
the folder name display, messages still cannot be read.
Comment 49 scottputterman 2000-02-26 13:52:03 PST
fix checked in.  Should we open up another bug for the 0x5c problem?
Comment 50 nhottanscp 2000-02-27 09:58:27 PST
There is going to be a local mail specific change needed for 0x5c problem (to 
use nsString GetLeafName). But I also see that I cannot create the name contains 
0x5c with IMAP and local. I will ask IQA to test and file bugs.
Comment 51 ji 2000-02-28 11:52:16 PST
Using today's build (2000-02-28), I see two problems:
1. Can't create a Japanese folder with IMAP.
2. With local folders, there are two problems with the new created japanese folder name:
   1). If the Ja folder name doesn't contain 0x5c chars, the Ja folder can be created, but
       the Ja folder name is displayed as dots.
   2). If the Ja folder name contains 0x5c chars, the specified new folder is not created.
       Instead, a folder with the same name as the local folder is created. For example,
       1. Highlight Local Folders and select File | New | Folder..
       2. Enter a japanese folder name containing 0x5c char, like hyouji.
       3. Click on OK button
       You'll see under Local Folders, another Local Folders is created, not the Japanese one you entered.

Do I need to file seperate bugs for these?
Comment 52 Katsuhiko Momoi 2000-02-28 11:59:08 PST
IMAP bug is a separate one.
If this cannot be done, then it is a regression. We probably should
open a new bug.
Comment 53 Katsuhiko Momoi 2000-02-28 12:02:54 PST
Sorry, the IMAP bug is already there. It is Bug 26745. It is still open.
Comment 54 nhottanscp 2000-02-28 12:04:44 PST
Yes, please file separate bugs.
Also, once the IMAP bug is fixed, please try a folder with 0x5c on IMAP 
(and file a bug if that exists).
Comment 55 David :Bienvenu 2000-02-28 12:55:31 PST
You're referring by "this" and "it" to #2, right? We don't need a new bug for
IMAP, at least I don't.
Comment 56 Katsuhiko Momoi 2000-02-28 14:09:38 PST
David. I just noted in Bug 26745 that we should check for 0x5c 
filename issue whenit is resolved.  That should be sufficient
as you say.
Comment 57 Katsuhiko Momoi 2000-02-28 22:10:54 PST
Bug 29543 is the general bug for creating JPN name folders.
Bug 29546 is a specific problem with creating a JPN folder
 name with 0x5c in it.

With these 2 bugs filed, this bug has been reduced to just the display
one for migrated folder names. 
We still can't see any contents of migrated folders with 0x5C 
in it, but I take it that this is taken care of in another bug.

For Windows 02-28-2000 M15 build, this bug then has been fixed.
All Japanese names show correctly.

Will Look at Mac and Linux next.
Comment 58 Katsuhiko Momoi 2000-03-06 15:56:12 PST
With Linux 3/6/2000 M15 build, I see Local mailbox names under
EUC locale. 

With Mac 3/5/2000 M15 build, under Japanese Mac OS 8.6, 
I can see Japanese folder names correctly. 

On Mac and Windows NT/2000, you really don't need to have
a Japanese OS to see Japanese file names. This latter 
issue is not addressed by the current fix. As I understand it,
we have to deal with this latter problem at a more general
level.

With these provisos, the current fix has been cnfirmed.
Marking it verified.

Note You need to log in before you can comment on or make changes to this bug.