JPN Local Mail folder names do not show correctly

VERIFIED FIXED in M14

Status

MailNews Core
Internationalization
P3
normal
VERIFIED FIXED
18 years ago
9 years ago

People

(Reporter: Katsuhiko Momoi, Assigned: scottputterman)

Tracking

Trunk
x86
Windows NT
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [PDT+] need approval)

(Reporter)

Description

18 years ago
** Observed with 6/8/99 Win32 build **

It doesn't look like Japanese folder names are supported
currently in that they seem to be interpreted as Latin 1 or
UTF-8. Try something like: テスト

There is also a special problem with 0x5c JPN
names in that folders containing such names do not show up
at all.  Example is something like: 日本語の表示

Updated

18 years ago
Assignee: nhotta → putterman

Comment 1

18 years ago
This bug is for local folder names, I think.
Probably charset conversion is needed. I18n can provide infomation but I don't
know about the current design and implementation of local folder name.
Reassigning to putterman@netscape.com.

Updated

18 years ago
Blocks: 7228
(Assignee)

Updated

18 years ago
Target Milestone: M8
(Assignee)

Updated

18 years ago
Status: NEW → ASSIGNED

Updated

18 years ago
Target Milestone: M8 → M9

Comment 2

18 years ago
I don't think this is happening for m9, though I did change the GetName
nsMsgFolder routine to support wide chars, I don't think we ever put wide chars
in the name. please correct me if I'm wrong.
(Assignee)

Comment 3

18 years ago
I don't think I'm doing that.
(Assignee)

Updated

18 years ago
Target Milestone: M9 → M10
(Assignee)

Comment 4

18 years ago
reassigning to M10

Updated

18 years ago
Target Milestone: M10 → M11

Comment 5

18 years ago
Triaging to M11
(Assignee)

Updated

18 years ago
Target Milestone: M11 → M12
(Assignee)

Comment 6

18 years ago
This will not be fixed for M11 and I question whether it will be fixed for M12.
I will mark it M12 to allow for those who might want it fixed to comment on
this.

Comment 7

18 years ago
We need this fixed for beta; Japanese users won't be able to display these
folder names in their mail.

Updated

18 years ago
Blocks: 11091

Comment 8

18 years ago
(target milestone is M11 or M12 - add to mail beta tracking bug)

Updated

18 years ago
Summary: JPN Mail folder names do not show correctly → [BETA] JPN Mail folder names do not show correctly

Comment 9

18 years ago
Added [BETA] per 9/20 comment.
(Assignee)

Updated

18 years ago
Target Milestone: M12 → M14
(Reporter)

Comment 10

18 years ago
Hi, we need to get moving on this for Beta.

I think what we need to do on this.

1. Make sure that when we display folder names on a system,
   we assume that the file names are in the file-system charset
   of the operating system Mozilla is operating on.
2. Make sure that when we create a new local mailbox names,
   we convert whatever we receive in input to the OS's file-system
   charset.

The I18n group be able to provide info on how to query the system
for the file-system charset.

Bob/Naoki, we need to get this on our Beta radar.
(Reporter)

Comment 11

18 years ago
*** Bug 23623 has been marked as a duplicate of this bug. ***

Comment 12

18 years ago
Here is the code to get the platform file charset.

  #define NS_IMPL_IDS
  #include "nsIPlatformCharset.h"
  #undef NS_IMPL_IDS

  nsCOMPtr <nsIPlatformCharset> platformCharset;
  nsAutoString aPlatformCharset;
  rv = nsComponentManager::CreateInstance(NS_PLATFORMCHARSET_PROGID, nsnull,
                                          NS_GET_IID(nsIPlatformCharset),
                                          getter_AddRefs(platformCharset));
  if (NS_SUCCEEDED(rv))
  {
  rv = platformCharset->GetCharset(kPlatformCharsetSel_FileName,
                                   aPlatformCharset);
  }

For the charset conversion from unicode to the platform charset,
ConvertFromUnicode (see nsMsgI18N.h) can be used.

Updated

18 years ago
Keywords: beta1

Comment 13

18 years ago
Is this the same as bug 7130?
(Reporter)

Comment 14

18 years ago
No. This is for Local folder names. Bug 7130 is for IMAP
folder names. I believe the latter requires special conversion
as IMAP requires the folder names to use modified UTF-7.
Summary: [BETA] JPN Mail folder names do not show correctly → [BETA] JPN Local Mail folder names do not show correctly

Comment 15

18 years ago
I think 7130 is about conversion between unicode (M-UTF-7 <-> UCS2).
This is about unicode and file system charset.
BTW, the sample code I put is already available as a function (nsMsgI18N.h).

Comment 16

18 years ago
Putting on PDT+ radar for beta1.
Whiteboard: [PDT+]

Updated

18 years ago
Summary: [BETA] JPN Local Mail folder names do not show correctly → JPN Local Mail folder names do not show correctly
(Assignee)

Updated

18 years ago
Whiteboard: [PDT+] → [PDT+]No estimate at the moment based on any facts. I'd say sometime in the next 2 weeks. If someone else wants to take this because of this fact, be my guest. Otherwise I will look at this after I finish some of my other PDT+ bugs.
(Assignee)

Comment 17

18 years ago
OK I will try this out.  This isn't as bad as I had once imagined because as 
David said, we are now passing PRUnichar*'s around whereas I thought we were 
still passing char*.  Now I just need to make sure they get converted correctly.

At least the last time I looked, we were getting the folder name from the uri 
and we were getting the uri from the filename. Am I going to be able to test 
this out on my machine?
(Reporter)

Comment 18

18 years ago
I have Japanese local mailboxes but these would not do 
for US Windows. I think for US Windows machines, Latin 1
accented folder names will have to do for debugging.
For Japanese and other languages, if you let us know 
that you checked in a code that should work, we'll check it
out for you. (You can change the system's default language
via Control Panel but this tends to be very messy and not
recommended.)

I just tried migration of JPN mail fodlers from
4.x and it is botching a file with 0x5c 3nd byte.
It chops off the name in half. I'll file another bug on
this.
(Assignee)

Comment 19

18 years ago
sorry, I guess what I was asking before is that I can view Japanese Imap folder
names.  Is there any reason why I shouldn't be able to view Japanese local
folder names?
(Reporter)

Comment 20

18 years ago
Yes, because the local folder names use the system charset, e.g.
Shift_JIS on Japanese Windows while IMAP folder names are
stored in modified UTF-7. We would need different
conversion, don't we?
(Assignee)

Comment 21

18 years ago
ok.  I will try out what you suggested now using the code Naoki suggested.
(Assignee)

Comment 22

18 years ago
I think one of the problems is that we build up the file names using
nsFileSpec's and I think there are only char * interfaces.  Won't this make this
not work?


Whiteboard: [PDT+]No estimate at the moment based on any facts. I'd say sometime in the next 2 weeks. If someone else wants to take this because of this fact, be my guest. Otherwise I will look at this after I finish some of my other PDT+ bugs. → [PDT+] 2/18. I'm aiming for 2/15 but I think there might be a couple of problems

Comment 23

18 years ago
nsFileSpec reads and writes files using the OS's file system charset
encoding.  These encodings should be represented using char *.
Does that answer your question?  nhotta knows the details...
(Assignee)

Comment 24

18 years ago
Yeah it does.  I talked to Naoki about this earlier.  I tried something out and 
according to Kat it was better but not there yet.
(Assignee)

Comment 25

18 years ago
changing estimated date.  I will need to debug on a Japanese machine.  The hope 
is that this will be easy to find.  Putting 2/25 down just in case it isn't.
Whiteboard: [PDT+] 2/18. I'm aiming for 2/15 but I think there might be a couple of problems → [PDT+] 2/25.

Comment 26

18 years ago
After I applied a patch from Scott, I was able to see a Japanese folder on my 
localized Japanese windows. But cannot see the messeages in that folder.
We can schedule and use my machine to debug.
Scott, I can help you on this one. As the MacOS charset is not ISO Latin 1, I 
should be able to reprocude this problem by just using some 8 bits characters on 
my "English" Mac.

Comment 28

18 years ago
Agreed, that should be easier than debugging raw Shift_JIS data on my Japaense 
NT.
I've just created a local forder named "L'été" and everything is fine except that the filename used for the folder was 
using the wrong charset.
(Assignee)

Comment 30

18 years ago
I wonder if the patch I gave Naoki would fix JFD's charset problem?

Anyway, I want to work on this today.  Could someone give me a Japanese folder I 
can ftp over to my mac?  Also, what should I be expecting to see in the 
debugger?

Comment 31

18 years ago
I failed to upload Japanese named file to the server, neither mail attachment 
succeeded. Could you pick up a floppy in my cube?
Anyway, the folder contains two Japanese characters (0x82A0 and 0x82A2 in 
Shift_JIS), in unicode they are \u3042 and \u3044.
If you debug on non Japanese machine, you also need to hard code the charset 
name (Shift_JIS) in msgCompFileSystemCharset().

Comment 32

18 years ago
The debugging tip I mentioned does not work on Windows unless using localized 
Japanese system. The file name turned to question marks so not able to convert 
even with hard coded charset name.
But it works on Macintosh. The file name was passed in native encode (Shift_JIS) 
but not turned to question marks. I used MacOS9 with Japanese option installed.

Comment 33

18 years ago
per PDT, this will become PDT- after 2/25
Whiteboard: [PDT+] 2/25. → [PDT+] 2/25. Will become PDT- after 2/25
(Assignee)

Comment 34

18 years ago
The following patch  makes it so I can see a folder with accents on my Mac 
without the recurring children that momoi was seeing (I saw this too so I know 
this fixes it).  I can also select a folder and see all of the headers.  
Unfortunately, I can't read any messages.  Naoki, could you test this out and 
verify that this works on a Japanese machine?

ndex: nsLocalMailFolder.cpp
===================================================================
RCS file: /cvsroot/mozilla/mailnews/local/src/nsLocalMailFolder.cpp,v
retrieving revision 1.185
diff -r1.185 nsLocalMailFolder.cpp
72a73
> #include "nsMsgI18N.h"
218a220,221
>     nsAutoString convertedFolderNameStr;
>     const nsString fileCharset = msgCompFileSystemCharset();
223a227
>         
225c229
<         currentFolderNameStr = folderName;
---
>         currentFolderNameStr = folderName;        
232a237,238
>         ConvertToUnicode(fileCharset, folderName, convertedFolderNameStr);
>         child->SetName(convertedFolderNameStr.GetUnicode());

Updated

18 years ago
Whiteboard: [PDT+] 2/25. Will become PDT- after 2/25 → [PDT+] w/b minus on 02/25

Comment 35

18 years ago
I applied the patch and ran on WinNT-J (Japanese localized NT).
A folder with Japanese name was displayed correctly.
But I was not be able to see the headers in thread pane. Probably that's because 
my tree is not up to date. I will try again tomorrow  with updated tree.
(Assignee)

Comment 36

18 years ago
Yeah, that's important.  It needs the changes that bienvenu checked in to fix 
problems with uris and non-ascii characters.  The reason I can't display a 
message is that strtok is finding a '/' in one of my characters and messing up 
the path.  I think it needs to be escaped.
(Assignee)

Comment 37

18 years ago
If I add the following code, I can display messages as well.  This is the same 
code bienvenu wrote to replace strtok in nsMsgFolder.  I wonder if this is 
affecting us in even more places.  Naoki,  when you get a new build, can you try 
this out as well.  I think this will fix this bug.

Index: nsLocalUtils.cpp
===================================================================
RCS file: /cvsroot/mozilla/mailnews/local/src/nsLocalUtils.cpp,v
retrieving revision 1.33
diff -r1.33 nsLocalUtils.cpp
189d188
<     while ((*curPos)=='/') curPos++;
194,214c193,219
<     
<     // for each token in between the /'s, put a .sbd on the end and
<     // append to the path
<     char *newStr=nsnull;
<     char *temp = PL_strdup(curPos);
<     char *token = nsCRT::strtok(temp, "/", &newStr);
<     while (token) {
<       nsCAutoString dir(token);
<       
<       // look for next token
<       token = nsCRT::strtok(newStr, "/", &newStr);
<       
<       // check if we're the last entry
<       if (token) {
<         NS_MsgHashIfNecessary(dir);
<         dir += sbdSep;            // no, we're not, so append .sbd
<         pathResult += (const char *) dir;
<       }
<       else {
<         NS_MsgHashIfNecessary(dir);
<         pathResult += (const char *) dir;
---
>         
>         nsCAutoString newPath("");
> 
>     nsCAutoString oldPath(curPos);
>     nsCAutoString pathPiece;
> 
>     PRInt32 startSlashPos = oldPath.FindChar('/');
>     PRInt32 endSlashPos = (startSlashPos >= 0) 
>         ? oldPath.FindChar('/', PR_FALSE, startSlashPos + 1) : 
oldPath.Length() - 1;
>     if (endSlashPos == -1)
>         endSlashPos = oldPath.Length();
>     // trick to make sure we only add the path to the first n-1 folders
>     PRBool haveFirst=PR_FALSE;
>     while (startSlashPos != -1) {
>       oldPath.Mid(pathPiece, startSlashPos + 1, endSlashPos - startSlashPos);
>       // skip leading '/' (and other // style things)
>       if (pathPiece.Length() > 0) {
> 
>         // add .sbd onto the previous path
>         if (haveFirst) {
>           newPath+=".sbd";
>           newPath += "/";
>         }
>         
>         NS_MsgHashIfNecessary(pathPiece);
>         newPath += pathPiece;
>         haveFirst=PR_TRUE;
215a221,230
>       // look for the next slash
>       startSlashPos = endSlashPos;
> 
>       endSlashPos = (startSlashPos >= 0) 
>             ? oldPath.FindChar('/', PR_FALSE, startSlashPos + 1) : 
oldPath.Length() - 1;
>       if (endSlashPos == -1)
>             endSlashPos = oldPath.Length();
> 
>       if (startSlashPos == endSlashPos)
>           break;
217c232,234
<     PL_strfree(temp);
---
>     
>     pathResult+=newPath.GetBuffer();
> 

Comment 38

18 years ago
I applied both patches and I can see Japanese folder name and headers in thread 
pane also messages in the view pane. So it is working for Japanese (tested on 
localized Japanese Windows NT).

The only problem I saw was that a Japanese folder name which contains 0x5c 
(backslash) is not shown correctly (a character contains 0x5c is truncated).
No thread pane header displayed for that folder either. I think this special 
case can be filed as a separate bug.
(Assignee)

Comment 39

18 years ago
changing status to need approval to checkin
Whiteboard: [PDT+] w/b minus on 02/25 → [PDT+] need approval

Comment 40

18 years ago
I assume that bienvenu and nhotta have already reviewed this, correct?
Have ftang review this too.  He's been tracking down lots of nsString
misuages which might be causing the 0x5c problem.
After ftang's review, you can check-in on my approval.  (Run the pre-checkin
tests, yada yada.  You know the drill...)

Comment 41

18 years ago
I cannot review the of mozilla/mailnews/local/src/nsLocalMailFolder.cpp becuase
I cannot see the context. Can you attach -c version here.
Also, you should take a look at the patch in 6770 and try the new
PRUnichar*/nsString flavor of GetLeafName()

Comment 42

18 years ago
also, when you have time (not necessary before you check in.) apply the patch in

bug 28424 (remember to change the DEBUG_ftang to your name) and verify there are

no illegal nsString usage in your code.


Comment 43

18 years ago
>Also, you should take a look at the patch in 6770 and try the new
> PRUnichar*/nsString flavor of GetLeafName()
I'll do this since it needs Japanese system in order to verify.

Updated

18 years ago
Depends on: 6770

Comment 44

18 years ago
I include the patch of 6770 and rebuild. I also change the patch of this bug to 
use the unicode based GetLeaf method.
I was able to see the folder name which contains 0x5c but cannot see the headers 
in thread pane. Looks like more place need change in order to resolve 0x5c bug.

Comment 45

18 years ago
As I mentioned, there are more than one problem to fix 0x5c problem.
I got bobj's approval for the check in. Scott, please check in your patch.
Note: 6770 has been fixed. Per ftang, the additional method is:  
    GetLeafName(nsString);
not
    GetLeafUniName(nsString);
(Assignee)

Comment 47

18 years ago
I'll check this in tomorrow.  What do I have to change in my patch?  I don't 
have any way of trying out any changed code over the weekend, but if you tell me 
what the changes are, I can put them in.

Comment 48

18 years ago
I talked to bobj and agreed that your patch can be checked in without the new 
GetLeafName (nsString version).

As far as I tried using the new GetLeafName, it solved the display problem of 
0x5c contained folder (not able to see headers in thread pane). So not useful 
until we fix other problems around 0x5c (which we don't know exactly where). 

Basically by using the nsString version of GetLeafName, you do not have to do 
the conversion in your code. But if we change to use it we need to test (I 
cannot test over the weekend). And as I mentioned, that change itself just fixes 
the folder name display, messages still cannot be read.
(Assignee)

Comment 49

18 years ago
fix checked in.  Should we open up another bug for the 0x5c problem?
Status: ASSIGNED → RESOLVED
Last Resolved: 18 years ago
Resolution: --- → FIXED

Comment 50

18 years ago
There is going to be a local mail specific change needed for 0x5c problem (to 
use nsString GetLeafName). But I also see that I cannot create the name contains 
0x5c with IMAP and local. I will ask IQA to test and file bugs.

Comment 51

18 years ago
Using today's build (2000-02-28), I see two problems:
1. Can't create a Japanese folder with IMAP.
2. With local folders, there are two problems with the new created japanese folder name:
   1). If the Ja folder name doesn't contain 0x5c chars, the Ja folder can be created, but
       the Ja folder name is displayed as dots.
   2). If the Ja folder name contains 0x5c chars, the specified new folder is not created.
       Instead, a folder with the same name as the local folder is created. For example,
       1. Highlight Local Folders and select File | New | Folder..
       2. Enter a japanese folder name containing 0x5c char, like hyouji.
       3. Click on OK button
       You'll see under Local Folders, another Local Folders is created, not the Japanese one you entered.

Do I need to file seperate bugs for these?
(Reporter)

Comment 52

18 years ago
IMAP bug is a separate one.
If this cannot be done, then it is a regression. We probably should
open a new bug.
(Reporter)

Comment 53

18 years ago
Sorry, the IMAP bug is already there. It is Bug 26745. It is still open.

Comment 54

18 years ago
Yes, please file separate bugs.
Also, once the IMAP bug is fixed, please try a folder with 0x5c on IMAP 
(and file a bug if that exists).

Comment 55

18 years ago
You're referring by "this" and "it" to #2, right? We don't need a new bug for
IMAP, at least I don't.
(Reporter)

Comment 56

18 years ago
David. I just noted in Bug 26745 that we should check for 0x5c 
filename issue whenit is resolved.  That should be sufficient
as you say.
(Reporter)

Comment 57

18 years ago
Bug 29543 is the general bug for creating JPN name folders.
Bug 29546 is a specific problem with creating a JPN folder
 name with 0x5c in it.

With these 2 bugs filed, this bug has been reduced to just the display
one for migrated folder names. 
We still can't see any contents of migrated folders with 0x5C 
in it, but I take it that this is taken care of in another bug.

For Windows 02-28-2000 M15 build, this bug then has been fixed.
All Japanese names show correctly.

Will Look at Mac and Linux next.
(Reporter)

Comment 58

18 years ago
With Linux 3/6/2000 M15 build, I see Local mailbox names under
EUC locale. 

With Mac 3/5/2000 M15 build, under Japanese Mac OS 8.6, 
I can see Japanese folder names correctly. 

On Mac and Windows NT/2000, you really don't need to have
a Japanese OS to see Japanese file names. This latter 
issue is not addressed by the current fix. As I understand it,
we have to deal with this latter problem at a more general
level.

With these provisos, the current fix has been cnfirmed.
Marking it verified.
Status: RESOLVED → VERIFIED
Product: MailNews → Core
Product: Core → MailNews Core
You need to log in before you can comment on or make changes to this bug.