Closed Bug 7130 Opened 25 years ago Closed 25 years ago

IMAP: does not interpret double byte M-UTF-7 in folder names

Categories

(MailNews Core :: Internationalization, defect, P3)

Product:
MailNews Core
Component:
Internationalization
Platform:
x86
Windows NT
Type:
defect

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: jgmyers, Assigned: Bienvenu)

References

Details

(Whiteboard: [PDT+] ETA 2/15) 

Two of my IMAP folders, "r&AOk-pertoires" and "wallace&-gromit", are not being
displayed correctly in the folder pane.
Assignee: nhotta → bienvenu
Currently waiting for the converter (bug 6983).
Reassigning to bienvenu@netscape.com.
Adding cata@netscape.com, bobj@netscape.com to cc.
Are you saying I should be writing a utf-7 converter? I must be missing
something. I know our code is #ifdeffed out, but I thought we were waiting for
something from the i18n group.
No, Cata is going to write the converter. When the bug 6983 is resolved, please
enable your code and resolve this bug. Sorry for the confusion.
Depends on: 6983
Depends on: 6814
No longer depends on: 6983
M-UTF-7 converter is now bug 6814.
Blocks: 7228
The converter was checked in earlier this week (6814).
Is this just a matter of turning on/off the #ifdef?
If so, can we do this for M7?
No, I have to write code to invoke the converter, which is code that is not
written yet. I have other major features to do for m7.
*** Bug 8504 has been marked as a duplicate of this bug. ***
Are these converters thread safe? Can I call them from a non-mozilla main
thread?
I don't really understand your questions: being able to call a converter from a
non-mozilla main thread does not necessarily mean that the converter is
thread-safe.

A converter is not thread-safe: it has internal state and its methods are not
synchronized. But why do you need it to be thread-safe?! You can't have two
simultaneous users for it anyways. (internal state in some stateful decodings
like UTF-7...)

However, a converter can be called from any thread, as long as it is only one.
The rule should be: if you need to call some converter from another thread, just
instantiate a new component, a new converter of the same type.

As a side-point, a converter can be reused by resetting its state: call Reset().

Hope I helped. Please let me know if you have any other questions.
sorry, I really meant : is the converter code thread safe? It sounds like it is.
It's just that the imap code in the past always assumed that none of the code it
called was thread-safe, so it would proxy over to the mozilla thread to do
things like utf7 conversion.
Status: ASSIGNED → RESOLVED
Closed: 25 years ago
Resolution: --- → FIXED
fixed in nsImapProtocol.cpp to convert to unicode and then back to 8 bit ascii.
Status: RESOLVED → REOPENED
** Checked with 6/25/99 Win32 M8 build **

I have 2 types of IMAP folder names -- Latin 1 8bit characters
and Japanese.

Latin 1 names are showing OK but Japanese names are not. Each
Japanese character is displayed as a dot instead. Since Modified UTF-7
converters are shown to be working now without errors, my
assumption would be that is't not the converter.
You mentioned something about using UTF-7 conversion. Is this
taking place in this fix. We do have 2 outstanding bugs for UTF-7
converters, Bug 6983 and Bug 8343.

For now, I'm ging to re-open this.
I have a question about IMAP folder summary files and Windows
file system. If I make a Latin 1 8bit folder under US Windows
and then refer to the same IMAP folder on a dual boot machine
under Japanese Windows, Messenger will create another folder
name file of the same thing but without the accents. So if
the original Latin file name was "e'cho" (where ' indicates an acute
accent), then on the Japanese system, it will create a file
named "echo" without the accent. Thus I end up having
2 folders of the same thing. It seems like the Windows files system
cannot somehow identify the existing one on the hard disk  with what
it is finding on the server.

My question is: How can we avoid this? Is there a way to save IMAP folder
names locally in such a way to be able to read them correctly under
any Windows systems? Related question is what charset is the folder
name saved currently? Does it have dependency on the locale of the OS?
Target Milestone: M8 → M9
This is as I expected - we convert back to 8 bit ascii, as I mentioned above, so
non 8-bit ascii characters, e.g., Japanese, aren't going to work because (I
assume) the converter really does convert them to Unicode. Switching the imap
code to use unicode strings will be a lot of work, and will be done by m9 at the
earliest - I suspect it may have to wait for necko to finish landing because a
lot of the code involved is changing to support necko.

We do save the real, decoded imap folder name locally, in the .msf file, and
compare it against the server name. We're planning to save the m-utf-7 folder
name, as we discussed a couple months ago, and compare against this instead,
because it does not depend on locale.
Resolution: FIXED → ---
Clearing Fixed resolution since this bug has been reopened.
Summary: IMAP: does not interpret M-UTF-7 in folder names → IMAP: does not interpret double byte M-UTF-7 in folder names
changing summary to reflect remaining bug.
Target Milestone: M9 → M10
Not going to happen in m9
Target Milestone: M10 → M11
moving to m11 - I'm reworking our inter thread communication to use nsisupports
proxying, which will make it easier to do this.
Blocks: 11091
(target milestone is M11 or M12 - add to mail beta tracking bug)
Target Milestone: M11 → M15
This isn't making B1 - we decided to work on performance instead.
I think now that we should try to get this into B1. Anyone have any opinions?
Yes, this would be great to have in B1.
This will a lot help to get Japanese users to use B1.
Marking M14 to match last comments
Target Milestone: M15 → M14
Marking the keyword as Beta1.
Keywords: beta1
Phil, I thought we weren't going to make this a beta stopper.
bobj, we need some input for pdt.  pdt+?  needed for beta1?
This is a Beta1 requirement.

If this is not fixed, then Japanese users with existing IMAP accounts
will not be able to correctly see any folders with Japanese names.
Also, if a user creates a Japanese folder name, then it will be created
using a corrupted string.  The folder will probably not be accessible
(maybe even not removable) after we fix the code to correctly use UTF-7.

David,
    Possibly you can avoid converting the imap code to use pruUnichar
(UCS-2 Unicode).  If you convert to UTF-8 instead of 8-bit ASCII, the imap
code may not have to change since both are null-terminated byte streams.
When you talk to other code, you'll need to convert from UTF-8 to pruUnichar
instead of just casting up ot 16-bit quanities. This approach might be much
easier to implement...
Let me update the status with 1/27/2000 Win32 M14 build.

1. Display of existing JPN folder names --> displays only as dots.
2, Access to existing JPN folder names --> not successful. Cannot
   display msgs in it.
3. Creation of JPN folder names --> not accepted by the server 
   as we seem to be sending invalid characters and thus fails
   to create a folder.

Note: Latin 1 accented folders are accessible.
Putting on PDT+ radar for beta1.
Whiteboard: [PDT+]
I really don't know how long this is going to take. Converting the imap code to 
use unicode would be the wrong thing to do - the conversion has to happen inside 
nsImapMailFolder - the imap protocol doesn't accept unicode characters.
Whiteboard: [PDT+] → [PDT+] ETA 2/15
OK, I've made a pass at setting the unicode name of a folder based on the utf-7 
name as returned by the imap server. I'd appreciate any feedback you can give on 
whether it helps. I haven't done anything about folder creation.
Is this in tomorrow's build? If so, I'll take a look.
yes, it will be in tomorrow's build.
Looks good. Now I can see Japanese folder names as well as Latin 1
names. However, I cannot access these folders. I'm told that
the server responded that such a folder does not exist. 
You say that we cannot create such folders yet and so I assume 
that fixing that will take care of the viewing content problem.
Right?
I'm glad you can see them,anyway. No, it sounds like some other problem. If you
could set up these folders in a test account I could get access to, then I could
try to debug it. Thanks for your help!

I've set up an internal test account with
2 Japanese-named folders and 1 Latin 1 name folder.
All three folders have 5 Intl Smoketest msgs.
At present you can only see the content of the Latin 1
folder named "Be^ta", not the content of the 2 Japanese
folders. Will send a login info by mail.
OK, I can see the folder names and read the messages. Marking fixed. Thanks for 
all the help!
Status: REOPENED → RESOLVED
Closed: 25 years ago → 25 years ago
Resolution: --- → FIXED
David, should we create a new bug for creating non-ASCII named folder on IMAP servers?
I might have fixed that one too - try it again, if you would.
actually, all the folder creation routines in the system take const char *, so 
this can't work. It's not just the imap routines, but I'll try to fix everybody 
to take PRUnichars - it'll take a while, though. I'll open a new bug for that.
Yes. I tried 2/5/2000 Win32 build and was able to 
display the msgs in the Japanese folders but not
create a new one -- an illegal character warning was issued.
I'll verify this one again on Monday's build.
**Checked with 2/7/2000 Win32 build **

Japanese folder names on IMAP servers display correctly.
Latin 1 names continue to be displayed correctly.
I'm going to mark this verified/fixed.

ji, please re-confirm with Linux build and if you find a problem there,
please re-open this bug.
Status: RESOLVED → VERIFIED
Verified the fix with linux 2000020708 build.
Product: MailNews → Core
Product: Core → MailNews Core
You need to log in before you can comment on or make changes to this bug.