166735 - Unicode file/io in Necko-nsIOService

Status: RESOLVED INVALID

(Core :: Internationalization, defect)

Description

[Roy Yokoyama]

Need to update the nsIOService to handle UCS2 FileName
nsIOService::GetURLSpecFromFile() 
and
InitFileFromURLSpec()

patch to follow.

Comment 1

[Roy Yokoyama]

Attachment: Call Unicode APIs instead of calling NativePath()

dougt: can you review?

Comment 2

[Darin Fisher]

please see bug 166792.  why do you want to do this only for XP_WIN?  also, are
you sure this is the right thing to do?  won't it break copy/paste for file://
URL strings?  won't you also have compatibility problems in file formats, etc.?
 i originally tried to do this when i worked on the nsIFile API changes, but
there just seem to be way too many problems with using UTF-8 instead of the
native charset for file:// URLs.

Comment 3

[Roy Yokoyama]

Darin: thanks for your comment.  This bug is one of changes we need 
for making mozilla an unicode application in Windows platform.
( unicode app means we call RegisterClassW(), DefWindowProcW(), 
  GetOpenFileNameW(), CallWIndowProcW(), DispatchMessageW(), etc)
You can find related bugs here : 58866, 9449, 104305, 162361, 162362

Those bugs are due to the fact that we are selectively using the locale 
base Windows system APIs.  There is no way to fix these bugs except
to register moz as unicode app and start calling W APIs.

>why do you want to do this only for XP_WIN?
- so that we dont' break everything at once and only Windows OS 
  for now.

>there just seem to be way too many problems with using UTF-8
-  yes, I will be modifying NSPR, XPCOM/IO, Widget and Necko. (and more??)
   However, with above changes, we _now_ can open/save doc in unicode filename,
   windows title shows correctly.

>won't it break copy/paste for file:// URL strings?
Not sure. I haven't test this case yet. (thus MOZ_UNICODE :) ) 
As long as we don't call xxNativeFoo() functions, we should be ok.  

Fran, DougT, Wan-Teh and myself had a couple of discussions
with my approach. Please advise us of any considerations.

Comment 4

[Darin Fisher]

as for the cut/copy/paste issue... if you copy a file:// URL encoded using UTF-8
into an application like Netscape 6.2, you'll be unable to load the file:// URL.
 so, you either have to make the cut/copy/paste code do the conversion, or you
have to live with this deficiency.  what operating system APIs require UTF-8
file:// URLs?

Comment 5

[Darin Fisher]

BTW: if you simply modify the native charset to be UTF-8 when encoding file
paths as narrow strings, then you'd get UTF-8 file:// URLs automatically.

Comment 6

[Roy Yokoyama]

>native charset to be UTF-8
We thought of the same initially; but 
- we want to distinguish the differences between
  the native charset and UTF-8 in Windows OS
  Some modules may _really_ want the native charset. 

Necko may be a special case where it requires to  
store URI as UTF-8 (i could be wrong though). 
However, we want to keep the existing interfaces of XPCOM/IO 
either in UCS2 or in the native charset; but not three.

+---------+-----------+---------+
| Necko   |   Widget  |  Others |
|         |           |         |
+---------+-----------+---------+
      ^                 ^
      |                 |
    (UCS2)        (Native char)   
      |                 |
      V                 V
+-------------------------------+
|     XPCOM/IO                  |
|    stores Paths as UTF8       |
| (may be changed to UCS2)      |
+-------------------------------+
      ^                 ^
      |                 |
     (UCS2)          (UCS2)   
      |                 |
      V                 |
+---------+             |
|  NSPR   |             |   
|         |             V
+-------------------------------+
|       OS System               |
|                               |
+-------------------------------+

In the near future, we want to change the XPCOM/IO::mPaths 
to be in UCS2.  

Comment 7

[Darin Fisher]

so, if file paths are not going to use UTF-8, then why should file:// URLs be
any different?  they are supposed to have the same format.  i just know you are
going to hit a lot of regressions if you try to change this, and i don't see the
reason for doing so.  what WIN32 wide-API expects an UTF-8 file:// URL?  in
other words, why does the format of the file:// URL need to change?

Comment 8

[Roy Yokoyama]

>why does the format of the file:// URL need to change?
We are not going to change the format of file:// URL. URL can be in UTF8.
Problem here is that because we are calling nsIFile::GetNatvePath() 
in nsIOService which corrupts path name if you open non-ASCII filename in Win-En.
We are trying to eliminate the calls to xxNativeFoo().

>what WIN32 wide-API expects an UTF-8 file:// URL?
None.  We use wide-API with UCS2.

Comment 9

[Roy Yokoyama]

Here is the list of bugs related to unicode file i/o:
58866
104305
151162
172337
173151
107941
154629
159773
169712
101573
104305
112031
112741

I haven't verified if all the above bugs get addressed
by my attached patch; but, it will be a good start. :)
We need to move away from FS file URLs.....

Comment 10

[Darin Fisher]

Comment on attachment 97877 [details] [diff] [review]
Call Unicode APIs instead of calling NativePath()

>Index: base/src/nsIOServiceWin.cpp

>+    nsCAutoString ePath;
>+    ePath.Assign(NS_ConvertUCS2toUTF8(ucsPath).get()); 

efficiency nit:

      NS_ConvertUCS2toUTF8 ePath(ucsPath);

remember, this patch will make mozilla use a file:// URL format that
is incompatible with older applications.  we need to weigh this fact
against the benefit of using unicode encoding.	an alternative would
be to hide file:// URLs from the user, which is what IE appears to do.

Comment 11

[Roy Yokoyama]

Attachment: incorporating nit

darin: thanks.	Would you review the patch?

>an alternative would be to hide file:// URLs
One more question, would you know where I can find the
code where i can strip the 'file://' from URL before
displaying to the user?

Comment 12

[Roy Yokoyama]

strip 'file://' and unescape the URL before displaying to the user,
of-course :)

Comment 13

[Darin Fisher]

i'm not sure where that is handled.  it may not be centralized.

Comment 14

[Darin Fisher]

Comment on attachment 103918 [details] [diff] [review]
incorporating nit

>Index: base/src/Makefile.in

>+# For Unicode mozilla
>+ifdef MOZ_UNICODE
>+DEFINES += -DMOZ_UNICODE 
>+endif

shouldn't you really be utilizing the file mozilla-config.h that is generated
after running configure or something like that instead of tweaking individual
makefiles like this?

also, i know i asked this question before, but why don't you just convert
NSPR narrow API to use UTF-8?  then wouldn't all of this be unnecessary?
you could then make nsIFile::GetNativePath (and friends) return UTF-8.

what am i missing?  why wouldn't this work?

e.g., necko just uses whatever nsIFile thinks the native charset is.  it
doesn't make any assumptions about the charset.

Comment 15

[Roy Yokoyama]

right, Darin.
We wouldn't need to change Necko.  Sorry about this.
I can make this work with changes in nsIFile only.
Marking invalid.