Persona is no longer an option for authentication on BMO. For more details see Persona Deprecated.
Last Comment Bug 7130 - IMAP: does not interpret double byte M-UTF-7 in folder names
: IMAP: does not interpret double byte M-UTF-7 in folder names
[PDT+] ETA 2/15
Product: MailNews Core
Classification: Components
Component: Internationalization (show other bugs)
: Trunk
: x86 Windows NT
: P3 normal (vote)
: M14
Assigned To: David :Bienvenu
: Katsuhiko Momoi
: 8504 (view as bug list)
Depends on: 6814
Blocks: 7228 11091
  Show dependency treegraph
Reported: 1999-05-26 12:38 PDT by John G. Myers
Modified: 2008-07-31 01:22 PDT (History)
6 users (show)
See Also:
Crash Signature:
QA Whiteboard:
Iteration: ---
Points: ---


Description John G. Myers 1999-05-26 12:38:04 PDT
Two of my IMAP folders, "r&AOk-pertoires" and "wallace&-gromit", are not being
displayed correctly in the folder pane.
Comment 1 nhottanscp 1999-05-26 12:49:59 PDT
Currently waiting for the converter (bug 6983).
Reassigning to
Adding, to cc.
Comment 2 David :Bienvenu 1999-05-26 13:10:59 PDT
Are you saying I should be writing a utf-7 converter? I must be missing
something. I know our code is #ifdeffed out, but I thought we were waiting for
something from the i18n group.
Comment 3 nhottanscp 1999-05-26 15:41:59 PDT
No, Cata is going to write the converter. When the bug 6983 is resolved, please
enable your code and resolve this bug. Sorry for the confusion.
Comment 4 John G. Myers 1999-06-09 12:49:59 PDT
M-UTF-7 converter is now bug 6814.
Comment 5 bobj 1999-06-10 23:49:59 PDT
The converter was checked in earlier this week (6814).
Is this just a matter of turning on/off the #ifdef?
If so, can we do this for M7?
Comment 6 David :Bienvenu 1999-06-10 23:52:59 PDT
No, I have to write code to invoke the converter, which is code that is not
written yet. I have other major features to do for m7.
Comment 7 Katsuhiko Momoi 1999-06-18 16:00:59 PDT
*** Bug 8504 has been marked as a duplicate of this bug. ***
Comment 8 David :Bienvenu 1999-06-23 11:11:59 PDT
Are these converters thread safe? Can I call them from a non-mozilla main
Comment 9 cata 1999-06-23 11:32:59 PDT
I don't really understand your questions: being able to call a converter from a
non-mozilla main thread does not necessarily mean that the converter is

A converter is not thread-safe: it has internal state and its methods are not
synchronized. But why do you need it to be thread-safe?! You can't have two
simultaneous users for it anyways. (internal state in some stateful decodings
like UTF-7...)

However, a converter can be called from any thread, as long as it is only one.
The rule should be: if you need to call some converter from another thread, just
instantiate a new component, a new converter of the same type.

As a side-point, a converter can be reused by resetting its state: call Reset().

Hope I helped. Please let me know if you have any other questions.
Comment 10 David :Bienvenu 1999-06-23 11:47:59 PDT
sorry, I really meant : is the converter code thread safe? It sounds like it is.
It's just that the imap code in the past always assumed that none of the code it
called was thread-safe, so it would proxy over to the mozilla thread to do
things like utf7 conversion.
Comment 11 David :Bienvenu 1999-06-24 15:32:59 PDT
fixed in nsImapProtocol.cpp to convert to unicode and then back to 8 bit ascii.
Comment 12 Katsuhiko Momoi 1999-06-27 01:37:59 PDT
** Checked with 6/25/99 Win32 M8 build **

I have 2 types of IMAP folder names -- Latin 1 8bit characters
and Japanese.

Latin 1 names are showing OK but Japanese names are not. Each
Japanese character is displayed as a dot instead. Since Modified UTF-7
converters are shown to be working now without errors, my
assumption would be that is't not the converter.
You mentioned something about using UTF-7 conversion. Is this
taking place in this fix. We do have 2 outstanding bugs for UTF-7
converters, Bug 6983 and Bug 8343.

For now, I'm ging to re-open this.
Comment 13 Katsuhiko Momoi 1999-06-27 01:48:59 PDT
I have a question about IMAP folder summary files and Windows
file system. If I make a Latin 1 8bit folder under US Windows
and then refer to the same IMAP folder on a dual boot machine
under Japanese Windows, Messenger will create another folder
name file of the same thing but without the accents. So if
the original Latin file name was "e'cho" (where ' indicates an acute
accent), then on the Japanese system, it will create a file
named "echo" without the accent. Thus I end up having
2 folders of the same thing. It seems like the Windows files system
cannot somehow identify the existing one on the hard disk  with what
it is finding on the server.

My question is: How can we avoid this? Is there a way to save IMAP folder
names locally in such a way to be able to read them correctly under
any Windows systems? Related question is what charset is the folder
name saved currently? Does it have dependency on the locale of the OS?
Comment 14 David :Bienvenu 1999-06-27 08:30:59 PDT
This is as I expected - we convert back to 8 bit ascii, as I mentioned above, so
non 8-bit ascii characters, e.g., Japanese, aren't going to work because (I
assume) the converter really does convert them to Unicode. Switching the imap
code to use unicode strings will be a lot of work, and will be done by m9 at the
earliest - I suspect it may have to wait for necko to finish landing because a
lot of the code involved is changing to support necko.

We do save the real, decoded imap folder name locally, in the .msf file, and
compare it against the server name. We're planning to save the m-utf-7 folder
name, as we discussed a couple months ago, and compare against this instead,
because it does not depend on locale.
Comment 15 leger 1999-06-28 12:21:59 PDT
Clearing Fixed resolution since this bug has been reopened.
Comment 16 David :Bienvenu 1999-07-06 09:01:59 PDT
changing summary to reflect remaining bug.
Comment 17 David :Bienvenu 1999-07-21 18:46:59 PDT
Not going to happen in m9
Comment 18 David :Bienvenu 1999-08-29 21:06:59 PDT
moving to m11 - I'm reworking our inter thread communication to use nsisupports
proxying, which will make it easier to do this.
Comment 19 lchiang 1999-09-20 15:28:59 PDT
(target milestone is M11 or M12 - add to mail beta tracking bug)
Comment 20 David :Bienvenu 1999-09-22 09:07:59 PDT
This isn't making B1 - we decided to work on performance instead.
Comment 21 David :Bienvenu 1999-12-21 17:40:59 PST
I think now that we should try to get this into B1. Anyone have any opinions?
Comment 22 bobj 1999-12-21 18:19:59 PST
Yes, this would be great to have in B1.
This will a lot help to get Japanese users to use B1.
Comment 23 Phil Peterson 2000-01-26 14:46:41 PST
Marking M14 to match last comments
Comment 24 Katsuhiko Momoi 2000-01-26 14:50:39 PST
Marking the keyword as Beta1.
Comment 25 David :Bienvenu 2000-01-26 21:40:33 PST
Phil, I thought we weren't going to make this a beta stopper.
Comment 26 leger 2000-01-27 16:44:09 PST
bobj, we need some input for pdt.  pdt+?  needed for beta1?
Comment 27 bobj 2000-01-27 18:44:15 PST
This is a Beta1 requirement.

If this is not fixed, then Japanese users with existing IMAP accounts
will not be able to correctly see any folders with Japanese names.
Also, if a user creates a Japanese folder name, then it will be created
using a corrupted string.  The folder will probably not be accessible
(maybe even not removable) after we fix the code to correctly use UTF-7.

    Possibly you can avoid converting the imap code to use pruUnichar
(UCS-2 Unicode).  If you convert to UTF-8 instead of 8-bit ASCII, the imap
code may not have to change since both are null-terminated byte streams.
When you talk to other code, you'll need to convert from UTF-8 to pruUnichar
instead of just casting up ot 16-bit quanities. This approach might be much
easier to implement...
Comment 28 Katsuhiko Momoi 2000-01-27 18:50:06 PST
Let me update the status with 1/27/2000 Win32 M14 build.

1. Display of existing JPN folder names --> displays only as dots.
2, Access to existing JPN folder names --> not successful. Cannot
   display msgs in it.
3. Creation of JPN folder names --> not accepted by the server 
   as we seem to be sending invalid characters and thus fails
   to create a folder.

Note: Latin 1 accented folders are accessible.
Comment 29 leger 2000-01-28 16:55:43 PST
Putting on PDT+ radar for beta1.
Comment 30 David :Bienvenu 2000-01-28 16:59:20 PST
I really don't know how long this is going to take. Converting the imap code to 
use unicode would be the wrong thing to do - the conversion has to happen inside 
nsImapMailFolder - the imap protocol doesn't accept unicode characters.
Comment 31 David :Bienvenu 2000-02-03 21:16:58 PST
OK, I've made a pass at setting the unicode name of a folder based on the utf-7 
name as returned by the imap server. I'd appreciate any feedback you can give on 
whether it helps. I haven't done anything about folder creation.
Comment 32 Katsuhiko Momoi 2000-02-03 22:00:33 PST
Is this in tomorrow's build? If so, I'll take a look.
Comment 33 David :Bienvenu 2000-02-03 22:02:20 PST
yes, it will be in tomorrow's build.
Comment 34 Katsuhiko Momoi 2000-02-04 12:41:56 PST
Looks good. Now I can see Japanese folder names as well as Latin 1
names. However, I cannot access these folders. I'm told that
the server responded that such a folder does not exist. 
You say that we cannot create such folders yet and so I assume 
that fixing that will take care of the viewing content problem.
Comment 35 David :Bienvenu 2000-02-04 12:43:21 PST
I'm glad you can see them,anyway. No, it sounds like some other problem. If you
could set up these folders in a test account I could get access to, then I could
try to debug it. Thanks for your help!
Comment 36 Katsuhiko Momoi 2000-02-04 13:02:02 PST
I've set up an internal test account with
2 Japanese-named folders and 1 Latin 1 name folder.
All three folders have 5 Intl Smoketest msgs.
At present you can only see the content of the Latin 1
folder named "Be^ta", not the content of the 2 Japanese
folders. Will send a login info by mail.
Comment 37 David :Bienvenu 2000-02-04 18:52:57 PST
OK, I can see the folder names and read the messages. Marking fixed. Thanks for 
all the help!
Comment 38 Katsuhiko Momoi 2000-02-04 18:59:33 PST
David, should we create a new bug for creating non-ASCII named folder on IMAP servers?
Comment 39 David :Bienvenu 2000-02-04 19:02:38 PST
I might have fixed that one too - try it again, if you would.
Comment 40 David :Bienvenu 2000-02-05 21:51:40 PST
actually, all the folder creation routines in the system take const char *, so 
this can't work. It's not just the imap routines, but I'll try to fix everybody 
to take PRUnichars - it'll take a while, though. I'll open a new bug for that.
Comment 41 Katsuhiko Momoi 2000-02-05 22:22:16 PST
Yes. I tried 2/5/2000 Win32 build and was able to 
display the msgs in the Japanese folders but not
create a new one -- an illegal character warning was issued.
I'll verify this one again on Monday's build.
Comment 42 Katsuhiko Momoi 2000-02-07 13:14:03 PST
**Checked with 2/7/2000 Win32 build **

Japanese folder names on IMAP servers display correctly.
Latin 1 names continue to be displayed correctly.
I'm going to mark this verified/fixed.

ji, please re-confirm with Linux build and if you find a problem there,
please re-open this bug.
Comment 43 ji 2000-02-07 13:28:08 PST
Verified the fix with linux 2000020708 build.

Note You need to log in before you can comment on or make changes to this bug.