26745 - IMAP: can't create non-ascii folder names

Status: VERIFIED FIXED

(MailNews Core :: Backend, defect, P3)

Description

[David :Bienvenu]

all the folder creation routines take single byte names, so double byte folders 
can't be created. Not sure if this is a beta blocker for i18n or not.

Comment 1

[David :Bienvenu]

accepting. Bob, please nominate as beta if you want. Whatever I nominate gets 
shot down.

Comment 2

[David :Bienvenu]

OK, I've checked in a fix (can't test it, though, since I don't know how to 
enter double byte folder names)

Comment 3

[bobj]

Thanks.  Changing QA assigned to momoi.

Comment 4

[Katsuhiko Momoi]

David, is this for IMAP folders only? or for both IMAP and local
folders? I had thought that it is for IMAP only but you
didn't qualify it that way.

Comment 5

[Katsuhiko Momoi]

CC'ing ji and marina. Need to check this out on Linux also.

Comment 6

[David :Bienvenu]

It's really for IMAP only, but I did a lot of the work that will be required to
do it for local folders. That's not done, however.

Comment 7

[Katsuhiko Momoi]

Thanks, I modified the summary line to make t easier to search for this later.

Comment 8

[Katsuhiko Momoi]

** Checked with 2/7/2000 Win32 build **

I tried creating IMAP folder names in Japanese, I got 2 types of
results:

1. Server failure -- invalid mailbox name response. Some Kanji names like "Tokyo" had this problem.
2. A new folder got created but the name is not Japanese at all and wrong: I tried "momo" in Hiragana,
   the folder was created but it showed ASCII  "D".

2. 

Comment 9

[David :Bienvenu]

I've found some more core msg code that assumes folder names are single byte - 
I'll fix those and check the changes in when the tree opens.

Comment 10

[David :Bienvenu]

checked in more fixes - if you could try it again with the 02/08/00 build, I'd 
appreciate it.

Comment 11

[David :Bienvenu]

I'll mark it fixed, though I don't know if it is.

Comment 12

[Katsuhiko Momoi]

With 2000020817 Win32 build, Japanese folder creation is not
working. I tried 3-4 different Japanese names under WinNT4-US
version using Global IME Japanese input editor and also 
under NT4-Japanese with native IME.  All 4 attempts
failed with "invalid mailbox name" msg.

Re-opening it for now...

Comment 13

[David :Bienvenu]

From Scott Putterman:

The problem was that the uri for this folder had a " " at the end.  So it was 
getting entered into the rdf resource hash table with a
space.  But, when we call the function parseURI, it calls nsStdURL::SetSpec.  
SetSpec erases any " " at the beginning or end. 
Because it's operating on the actual uri and the uri is the actual key into the 
resource hash table, every time it would try to find the
original uri in the hash table it couldn't because it had been modified to lose 
the " ".  So GetResource would keep creating the folder
over and over again which caused problems. 

Scott, if you check in this change, let me know so I can reassign this to you, 
or just mark it fixed./
The step I took to solve it is  in nsMsgFolder::parseURI instead of passing the 
folder's uri to SetSpec, I pass a copy.  This way the uri
is never modified.  When I did this, the Japanese folder showed up with the 
correct name and my problems went away.  The only
problem I see with this is that we use the result of SetSpec to get other 
information in parseURI and this won't necessarily be the same
as the information from the original uri since the uri passed to SetSpec could 
be modified. 

Comment 14

[David :Bienvenu]

Scott, did Andreas checkin his change to url parsing? Did that fix this for you?  
Perhaps someone in i18N could try this again? thanks.

Comment 15

[scottputterman]

Andreas just checked it in and it works for me.

Comment 16

[David :Bienvenu]

OK, I'm going to mark this fixed, then. PDT team, please note that I didn't 
actually change any source code for this, but it's going to show up as one of 
those evil evil non PDT+ bugs fixed anyway. Sorry 'bout that.

Comment 17

[Katsuhiko Momoi]

** Checked with 2/16/2000 Win32 build: 2000021615 **

I don't think this is fixed. I tried various Japanese
characters for creating a folder and none succeeded. 
Each I got a server error msg informing me that it is
"invalid mailbox name".

Re-opening ...

Comment 18

[David :Bienvenu]

weird, why would it work for Scott but not Momoi?

Comment 19

[scottputterman]

It definitely worked for me when it wasn't working before.  But, I only tried 
one folder and there might be other names that don't work.

Comment 20

[Katsuhiko Momoi]

I'll re-try this with today's build. The build I looked at last night
may have been "bad" in more than one way. We need re-confirmation
with another build.

Comment 21

[David :Bienvenu]

I doubt it has to do with the build - it's probably more likely some difference 
in folder names you're entering. Could you try turning on imap protocol logging 
to see what kind of folder name we're creating?

set NSPR_LOG_MODULES=IMAP:5
set NSPR_LOG_FILE=c:\tmp\mapio.txt

Comment 22

[Katsuhiko Momoi]

I'm writing down a reminder to look at 0x5c (2nd byte) Japanese names as it requires special
handling. This needs to be done for local folders because "\" indicates a path
separator on Windows. We should probably look at other similar path seprator or directory related 
single-byte code points for IMAP.

Comment 23

[David :Bienvenu]

I've found a bug in the charset converter that converts to imap utf7 (or my use 
of it) - it's not ending the escaping. So if I pass in хий I should get 
"&AMgAyQDK-" but instead I get "&AMgAyQDK" or "AMgAyQDK"

Here's how I'm using the charset converter. Please let me know if I'm doing 
something wrong.

char*
CreateUtf7ConvertedStringFromUnicode(const PRUnichar * aSourceString)
{
  nsresult res;
  char *dstPtr = nsnull;
  PRInt32 dstLength = 0;
  char *convertedString = NULL;
  
  NS_WITH_SERVICE(nsICharsetConverterManager, ccm, kCharsetConverterManagerCID, 
&res); 

  if(NS_SUCCEEDED(res) && (nsnull != ccm))
  {
    nsString aCharset("x-imap4-modified-utf7");
    PRInt32 unicharLength;

      // convert from 8 bit ascii string to modified utf7
      nsString unicodeStr(aSourceString);
      nsIUnicodeEncoder* encoder = nsnull;
      aCharset.SetString("x-imap4-modified-utf7");
      res = ccm->GetUnicodeEncoder(&aCharset, &encoder);
      if(NS_SUCCEEDED(res) && (nsnull != encoder)) 
      {
        res = encoder->GetMaxLength(unicodeStr.GetUnicode(), 
unicodeStr.Length(), &dstLength);
        // allocale an output buffer
        dstPtr = (char *) PR_CALLOC(dstLength + 1);
        unicharLength = unicodeStr.Length();
        if (dstPtr == nsnull) 
        {
          res = NS_ERROR_OUT_OF_MEMORY;
        }
        else 
        {
          res = encoder->Convert(unicodeStr.GetUnicode(), &unicharLength, 
dstPtr, &dstLength);
          dstPtr[dstLength] = 0;
        }
      }
	  // ack, this is silly - why pass through unicodeStr2 back to 
convertedString?
      NS_IF_RELEASE(encoder);
      nsString unicodeStr2(dstPtr);
      convertedString = (char *) PR_Malloc(dstLength + 1);
      if (convertedString)
        unicodeStr2.ToCString(convertedString, dstLength + 1, 0);
        }
    PR_FREEIF(dstPtr);
    return convertedString;
}

Comment 24

[nhottanscp]

For the return value, it can just return dstPtr instead of re-allocating the 
buffer. But that shouldn't change the result string.
cc to ftang and cata in case problem in the converter.

Comment 25

[David :Bienvenu]

yes, I knew I should have pointed out that I was doing that wrong, but I don't 
think that's what's broken.

Comment 26

[David :Bienvenu]

I've tried debugging this, but the code just seems wrong - I don't see 
FinishNoBuff getting called, which seems to be the code that would do the right 
thing.

Comment 27

[nhottanscp]

How about calling nsIUnicodeEncoder::Finish explicitly after the conversion? 
I think that will ensure the buffer to be flushed.

Comment 28

[David :Bienvenu]

thanks, but that didn't work - it just stuck the '-' at the start of the string. 
Maybe I'm supposed to call it with the end of the string, but this just seems 
weird. Shouldn't the converter be doing this for me? Also, sometimes it seems to 
skip the '&' at the beginning.

Comment 29

[nhottanscp]

>Shouldn't the converter be doing this for me?
That's because it needs to handle streams. The caller needs to call Finish() 
after Convert().
>I'm supposed to call it with the end of the string
Yes, the data to be appended to the data already converted by Convert().

Comment 30

[David :Bienvenu]

Wow, that's utterly brutal. Is there some other conversion mechanism I should 
use instead?

Comment 31

[nhottanscp]

About the buffer size question (in 26877), GetMaxLength() should return the size 
big enough to hold the data generated by Finish(). But I need to double check 
that with Cata.

Comment 32

[David :Bienvenu]

I have a simple workaround for this problem in the utf7 converter. Nominating as 
beta1. As I understand it, the utf7 converter just doesn't work the way I'm 
using it, so I have to finish the conversion myself.

Comment 33

[marina]

why the summary got changed to "can not create a non-ascii folder name"? It is 
absolutely possible to create 8-bit folder name in ISO-8859 on IMAP.

Comment 34

[David :Bienvenu]

because if the last character is non-ascii, I don't think you'll be able to 
create the folder.

Comment 35

[marina]

you're right....i didn't try this combination.

Comment 36

[David :Bienvenu]

neither did I :-(

Comment 37

[David :Bienvenu]

Attachment: proposed fix

Comment 38

[Phil Peterson]

[PDT+] W/b minus on 3/3

Comment 39

[David :Bienvenu]

fix checked in

Comment 40

[nhottanscp]

I used today's release win32 build and I was able to create a japanese folder 
which contains 0x5c. Looks like that is local folder only problem.

Comment 41

[Katsuhiko Momoi]

** Checked with 3/6/2000 Win32 build **

I can niw create JPN or Latin 1 folders with accents
with the above build on the IMAP.

Marking it verified as fixed.

Comment 42

[Katsuhiko Momoi]

ji, please check on Linux and report here.
cwang, please check on Mac and report here.

Comment 43

[ji]

Verified/fixed with linux 2000030709 build.