Closed Bug 7844 Opened 21 years ago Closed 20 years ago

JPN Local Mail folder names do not show correctly

Categories

(MailNews Core :: Internationalization, defect, P3)

x86
Windows NT
defect

Tracking

(Not tracked)

VERIFIED FIXED

People

(Reporter: momoi, Assigned: scottputterman)

References

Details

(Whiteboard: [PDT+] need approval)

** Observed with 6/8/99 Win32 build **

It doesn't look like Japanese folder names are supported
currently in that they seem to be interpreted as Latin 1 or
UTF-8. Try something like: テスト

There is also a special problem with 0x5c JPN
names in that folders containing such names do not show up
at all.  Example is something like: 日本語の表示
Assignee: nhotta → putterman
This bug is for local folder names, I think.
Probably charset conversion is needed. I18n can provide infomation but I don't
know about the current design and implementation of local folder name.
Reassigning to putterman@netscape.com.
Blocks: 7228
Target Milestone: M8
Status: NEW → ASSIGNED
Target Milestone: M8 → M9
I don't think this is happening for m9, though I did change the GetName
nsMsgFolder routine to support wide chars, I don't think we ever put wide chars
in the name. please correct me if I'm wrong.
I don't think I'm doing that.
Target Milestone: M9 → M10
reassigning to M10
Target Milestone: M10 → M11
Triaging to M11
Target Milestone: M11 → M12
This will not be fixed for M11 and I question whether it will be fixed for M12.
I will mark it M12 to allow for those who might want it fixed to comment on
this.
We need this fixed for beta; Japanese users won't be able to display these
folder names in their mail.
Blocks: 11091
(target milestone is M11 or M12 - add to mail beta tracking bug)
Summary: JPN Mail folder names do not show correctly → [BETA] JPN Mail folder names do not show correctly
Added [BETA] per 9/20 comment.
Target Milestone: M12 → M14
Hi, we need to get moving on this for Beta.

I think what we need to do on this.

1. Make sure that when we display folder names on a system,
   we assume that the file names are in the file-system charset
   of the operating system Mozilla is operating on.
2. Make sure that when we create a new local mailbox names,
   we convert whatever we receive in input to the OS's file-system
   charset.

The I18n group be able to provide info on how to query the system
for the file-system charset.

Bob/Naoki, we need to get this on our Beta radar.
*** Bug 23623 has been marked as a duplicate of this bug. ***
Here is the code to get the platform file charset.

  #define NS_IMPL_IDS
  #include "nsIPlatformCharset.h"
  #undef NS_IMPL_IDS

  nsCOMPtr <nsIPlatformCharset> platformCharset;
  nsAutoString aPlatformCharset;
  rv = nsComponentManager::CreateInstance(NS_PLATFORMCHARSET_PROGID, nsnull,
                                          NS_GET_IID(nsIPlatformCharset),
                                          getter_AddRefs(platformCharset));
  if (NS_SUCCEEDED(rv))
  {
  rv = platformCharset->GetCharset(kPlatformCharsetSel_FileName,
                                   aPlatformCharset);
  }

For the charset conversion from unicode to the platform charset,
ConvertFromUnicode (see nsMsgI18N.h) can be used.
Keywords: beta1
Is this the same as bug 7130?
No. This is for Local folder names. Bug 7130 is for IMAP
folder names. I believe the latter requires special conversion
as IMAP requires the folder names to use modified UTF-7.
Summary: [BETA] JPN Mail folder names do not show correctly → [BETA] JPN Local Mail folder names do not show correctly
I think 7130 is about conversion between unicode (M-UTF-7 <-> UCS2).
This is about unicode and file system charset.
BTW, the sample code I put is already available as a function (nsMsgI18N.h).
Putting on PDT+ radar for beta1.
Whiteboard: [PDT+]
Summary: [BETA] JPN Local Mail folder names do not show correctly → JPN Local Mail folder names do not show correctly
Whiteboard: [PDT+] → [PDT+]No estimate at the moment based on any facts. I'd say sometime in the next 2 weeks. If someone else wants to take this because of this fact, be my guest. Otherwise I will look at this after I finish some of my other PDT+ bugs.
OK I will try this out.  This isn't as bad as I had once imagined because as 
David said, we are now passing PRUnichar*'s around whereas I thought we were 
still passing char*.  Now I just need to make sure they get converted correctly.

At least the last time I looked, we were getting the folder name from the uri 
and we were getting the uri from the filename. Am I going to be able to test 
this out on my machine?
I have Japanese local mailboxes but these would not do 
for US Windows. I think for US Windows machines, Latin 1
accented folder names will have to do for debugging.
For Japanese and other languages, if you let us know 
that you checked in a code that should work, we'll check it
out for you. (You can change the system's default language
via Control Panel but this tends to be very messy and not
recommended.)

I just tried migration of JPN mail fodlers from
4.x and it is botching a file with 0x5c 3nd byte.
It chops off the name in half. I'll file another bug on
this.
sorry, I guess what I was asking before is that I can view Japanese Imap folder
names.  Is there any reason why I shouldn't be able to view Japanese local
folder names?
Yes, because the local folder names use the system charset, e.g.
Shift_JIS on Japanese Windows while IMAP folder names are
stored in modified UTF-7. We would need different
conversion, don't we?
ok.  I will try out what you suggested now using the code Naoki suggested.
I think one of the problems is that we build up the file names using
nsFileSpec's and I think there are only char * interfaces.  Won't this make this
not work?


Whiteboard: [PDT+]No estimate at the moment based on any facts. I'd say sometime in the next 2 weeks. If someone else wants to take this because of this fact, be my guest. Otherwise I will look at this after I finish some of my other PDT+ bugs. → [PDT+] 2/18. I'm aiming for 2/15 but I think there might be a couple of problems
nsFileSpec reads and writes files using the OS's file system charset
encoding.  These encodings should be represented using char *.
Does that answer your question?  nhotta knows the details...
Yeah it does.  I talked to Naoki about this earlier.  I tried something out and 
according to Kat it was better but not there yet.
changing estimated date.  I will need to debug on a Japanese machine.  The hope 
is that this will be easy to find.  Putting 2/25 down just in case it isn't.
Whiteboard: [PDT+] 2/18. I'm aiming for 2/15 but I think there might be a couple of problems → [PDT+] 2/25.
After I applied a patch from Scott, I was able to see a Japanese folder on my 
localized Japanese windows. But cannot see the messeages in that folder.
We can schedule and use my machine to debug.
Scott, I can help you on this one. As the MacOS charset is not ISO Latin 1, I 
should be able to reprocude this problem by just using some 8 bits characters on 
my "English" Mac.
Agreed, that should be easier than debugging raw Shift_JIS data on my Japaense 
NT.
I've just created a local forder named "L'été" and everything is fine except that the filename used for the folder was 
using the wrong charset.
I wonder if the patch I gave Naoki would fix JFD's charset problem?

Anyway, I want to work on this today.  Could someone give me a Japanese folder I 
can ftp over to my mac?  Also, what should I be expecting to see in the 
debugger?
I failed to upload Japanese named file to the server, neither mail attachment 
succeeded. Could you pick up a floppy in my cube?
Anyway, the folder contains two Japanese characters (0x82A0 and 0x82A2 in 
Shift_JIS), in unicode they are \u3042 and \u3044.
If you debug on non Japanese machine, you also need to hard code the charset 
name (Shift_JIS) in msgCompFileSystemCharset().

The debugging tip I mentioned does not work on Windows unless using localized 
Japanese system. The file name turned to question marks so not able to convert 
even with hard coded charset name.
But it works on Macintosh. The file name was passed in native encode (Shift_JIS) 
but not turned to question marks. I used MacOS9 with Japanese option installed.
per PDT, this will become PDT- after 2/25
Whiteboard: [PDT+] 2/25. → [PDT+] 2/25. Will become PDT- after 2/25
The following patch  makes it so I can see a folder with accents on my Mac 
without the recurring children that momoi was seeing (I saw this too so I know 
this fixes it).  I can also select a folder and see all of the headers.  
Unfortunately, I can't read any messages.  Naoki, could you test this out and 
verify that this works on a Japanese machine?

ndex: nsLocalMailFolder.cpp
===================================================================
RCS file: /cvsroot/mozilla/mailnews/local/src/nsLocalMailFolder.cpp,v
retrieving revision 1.185
diff -r1.185 nsLocalMailFolder.cpp
72a73
> #include "nsMsgI18N.h"
218a220,221
>     nsAutoString convertedFolderNameStr;
>     const nsString fileCharset = msgCompFileSystemCharset();
223a227
>         
225c229
<         currentFolderNameStr = folderName;
---
>         currentFolderNameStr = folderName;        
232a237,238
>         ConvertToUnicode(fileCharset, folderName, convertedFolderNameStr);
>         child->SetName(convertedFolderNameStr.GetUnicode());
Whiteboard: [PDT+] 2/25. Will become PDT- after 2/25 → [PDT+] w/b minus on 02/25
I applied the patch and ran on WinNT-J (Japanese localized NT).
A folder with Japanese name was displayed correctly.
But I was not be able to see the headers in thread pane. Probably that's because 
my tree is not up to date. I will try again tomorrow  with updated tree.
Yeah, that's important.  It needs the changes that bienvenu checked in to fix 
problems with uris and non-ascii characters.  The reason I can't display a 
message is that strtok is finding a '/' in one of my characters and messing up 
the path.  I think it needs to be escaped.
If I add the following code, I can display messages as well.  This is the same 
code bienvenu wrote to replace strtok in nsMsgFolder.  I wonder if this is 
affecting us in even more places.  Naoki,  when you get a new build, can you try 
this out as well.  I think this will fix this bug.

Index: nsLocalUtils.cpp
===================================================================
RCS file: /cvsroot/mozilla/mailnews/local/src/nsLocalUtils.cpp,v
retrieving revision 1.33
diff -r1.33 nsLocalUtils.cpp
189d188
<     while ((*curPos)=='/') curPos++;
194,214c193,219
<     
<     // for each token in between the /'s, put a .sbd on the end and
<     // append to the path
<     char *newStr=nsnull;
<     char *temp = PL_strdup(curPos);
<     char *token = nsCRT::strtok(temp, "/", &newStr);
<     while (token) {
<       nsCAutoString dir(token);
<       
<       // look for next token
<       token = nsCRT::strtok(newStr, "/", &newStr);
<       
<       // check if we're the last entry
<       if (token) {
<         NS_MsgHashIfNecessary(dir);
<         dir += sbdSep;            // no, we're not, so append .sbd
<         pathResult += (const char *) dir;
<       }
<       else {
<         NS_MsgHashIfNecessary(dir);
<         pathResult += (const char *) dir;
---
>         
>         nsCAutoString newPath("");
> 
>     nsCAutoString oldPath(curPos);
>     nsCAutoString pathPiece;
> 
>     PRInt32 startSlashPos = oldPath.FindChar('/');
>     PRInt32 endSlashPos = (startSlashPos >= 0) 
>         ? oldPath.FindChar('/', PR_FALSE, startSlashPos + 1) : 
oldPath.Length() - 1;
>     if (endSlashPos == -1)
>         endSlashPos = oldPath.Length();
>     // trick to make sure we only add the path to the first n-1 folders
>     PRBool haveFirst=PR_FALSE;
>     while (startSlashPos != -1) {
>       oldPath.Mid(pathPiece, startSlashPos + 1, endSlashPos - startSlashPos);
>       // skip leading '/' (and other // style things)
>       if (pathPiece.Length() > 0) {
> 
>         // add .sbd onto the previous path
>         if (haveFirst) {
>           newPath+=".sbd";
>           newPath += "/";
>         }
>         
>         NS_MsgHashIfNecessary(pathPiece);
>         newPath += pathPiece;
>         haveFirst=PR_TRUE;
215a221,230
>       // look for the next slash
>       startSlashPos = endSlashPos;
> 
>       endSlashPos = (startSlashPos >= 0) 
>             ? oldPath.FindChar('/', PR_FALSE, startSlashPos + 1) : 
oldPath.Length() - 1;
>       if (endSlashPos == -1)
>             endSlashPos = oldPath.Length();
> 
>       if (startSlashPos == endSlashPos)
>           break;
217c232,234
<     PL_strfree(temp);
---
>     
>     pathResult+=newPath.GetBuffer();
> 
I applied both patches and I can see Japanese folder name and headers in thread 
pane also messages in the view pane. So it is working for Japanese (tested on 
localized Japanese Windows NT).

The only problem I saw was that a Japanese folder name which contains 0x5c 
(backslash) is not shown correctly (a character contains 0x5c is truncated).
No thread pane header displayed for that folder either. I think this special 
case can be filed as a separate bug.
changing status to need approval to checkin
Whiteboard: [PDT+] w/b minus on 02/25 → [PDT+] need approval
I assume that bienvenu and nhotta have already reviewed this, correct?
Have ftang review this too.  He's been tracking down lots of nsString
misuages which might be causing the 0x5c problem.
After ftang's review, you can check-in on my approval.  (Run the pre-checkin
tests, yada yada.  You know the drill...)
I cannot review the of mozilla/mailnews/local/src/nsLocalMailFolder.cpp becuase
I cannot see the context. Can you attach -c version here.
Also, you should take a look at the patch in 6770 and try the new
PRUnichar*/nsString flavor of GetLeafName()
also, when you have time (not necessary before you check in.) apply the patch in

bug 28424 (remember to change the DEBUG_ftang to your name) and verify there are

no illegal nsString usage in your code.


>Also, you should take a look at the patch in 6770 and try the new
> PRUnichar*/nsString flavor of GetLeafName()
I'll do this since it needs Japanese system in order to verify.
Depends on: 6770
I include the patch of 6770 and rebuild. I also change the patch of this bug to 
use the unicode based GetLeaf method.
I was able to see the folder name which contains 0x5c but cannot see the headers 
in thread pane. Looks like more place need change in order to resolve 0x5c bug.
As I mentioned, there are more than one problem to fix 0x5c problem.
I got bobj's approval for the check in. Scott, please check in your patch.
Note: 6770 has been fixed. Per ftang, the additional method is:  
    GetLeafName(nsString);
not
    GetLeafUniName(nsString);
I'll check this in tomorrow.  What do I have to change in my patch?  I don't 
have any way of trying out any changed code over the weekend, but if you tell me 
what the changes are, I can put them in.
I talked to bobj and agreed that your patch can be checked in without the new 
GetLeafName (nsString version).

As far as I tried using the new GetLeafName, it solved the display problem of 
0x5c contained folder (not able to see headers in thread pane). So not useful 
until we fix other problems around 0x5c (which we don't know exactly where). 

Basically by using the nsString version of GetLeafName, you do not have to do 
the conversion in your code. But if we change to use it we need to test (I 
cannot test over the weekend). And as I mentioned, that change itself just fixes 
the folder name display, messages still cannot be read.
fix checked in.  Should we open up another bug for the 0x5c problem?
Status: ASSIGNED → RESOLVED
Closed: 20 years ago
Resolution: --- → FIXED
There is going to be a local mail specific change needed for 0x5c problem (to 
use nsString GetLeafName). But I also see that I cannot create the name contains 
0x5c with IMAP and local. I will ask IQA to test and file bugs.
Using today's build (2000-02-28), I see two problems:
1. Can't create a Japanese folder with IMAP.
2. With local folders, there are two problems with the new created japanese folder name:
   1). If the Ja folder name doesn't contain 0x5c chars, the Ja folder can be created, but
       the Ja folder name is displayed as dots.
   2). If the Ja folder name contains 0x5c chars, the specified new folder is not created.
       Instead, a folder with the same name as the local folder is created. For example,
       1. Highlight Local Folders and select File | New | Folder..
       2. Enter a japanese folder name containing 0x5c char, like hyouji.
       3. Click on OK button
       You'll see under Local Folders, another Local Folders is created, not the Japanese one you entered.

Do I need to file seperate bugs for these?
IMAP bug is a separate one.
If this cannot be done, then it is a regression. We probably should
open a new bug.
Sorry, the IMAP bug is already there. It is Bug 26745. It is still open.
Yes, please file separate bugs.
Also, once the IMAP bug is fixed, please try a folder with 0x5c on IMAP 
(and file a bug if that exists).
You're referring by "this" and "it" to #2, right? We don't need a new bug for
IMAP, at least I don't.
David. I just noted in Bug 26745 that we should check for 0x5c 
filename issue whenit is resolved.  That should be sufficient
as you say.
Bug 29543 is the general bug for creating JPN name folders.
Bug 29546 is a specific problem with creating a JPN folder
 name with 0x5c in it.

With these 2 bugs filed, this bug has been reduced to just the display
one for migrated folder names. 
We still can't see any contents of migrated folders with 0x5C 
in it, but I take it that this is taken care of in another bug.

For Windows 02-28-2000 M15 build, this bug then has been fixed.
All Japanese names show correctly.

Will Look at Mac and Linux next.
With Linux 3/6/2000 M15 build, I see Local mailbox names under
EUC locale. 

With Mac 3/5/2000 M15 build, under Japanese Mac OS 8.6, 
I can see Japanese folder names correctly. 

On Mac and Windows NT/2000, you really don't need to have
a Japanese OS to see Japanese file names. This latter 
issue is not addressed by the current fix. As I understand it,
we have to deal with this latter problem at a more general
level.

With these provisos, the current fix has been cnfirmed.
Marking it verified.
Status: RESOLVED → VERIFIED
Product: MailNews → Core
Product: Core → MailNews Core
You need to log in before you can comment on or make changes to this bug.