95481 - upgrade Mac CFM file system code to use Unicode when HFS+ exists

Status: RESOLVED WONTFIX

(Core :: Internationalization, defect)

Description

[Frank Tang]

read 
http://gemma.apple.com/techpubs/macosx/Carbon/Files/FileManager/File_Manager/
Functions/FunctionGroups.html

It looks like at least the following APIs support Unciode/CFString
Creating a File System Reference (FSRef)
 FSMakeFSRefUnicode     	Constructs an FSRef for a file or directory, given a 
parent directory and a Unicode name. 
 PBMakeFSRefUnicodeSync	 	Constructs an FSRef for a file or directory, 
given a parent directory and a Unicode name. 
 PBMakeFSRefUnicodeAsync	Constructs an FSRef for a file or directory, given a 
parent directory and a Unicode name. 

Creating Directories
 FSCreateDirectoryUnicode	Creates a new directory (folder) with a Unicode name. 
 PBCreateDirectoryUnicodeSync	Creates a new directory (folder) with a Unicode 
name. 
 PBCreateDirectoryUnicodeAsync	  Creates a new directory (folder) with a Unicode 
name. 

Creating Files
 FSCreateFileUnicode	Creates a new file with a Unicode name. 
 PBCreateFileUnicodeSync	Creates a new file with a Unicode name. 
 PBCreateFileUnicodeAsync	Creates a new file with a Unicode name. 

Determining the Unicode Names of the Data and Resource Forks
 FSGetDataForkName	Returns a Unicode string constant for the name of the data 
fork.
 FSGetResourceForkName	Returns a Unicode string constant for the name of the 
resource for

 Moving and Renaming Files or Directories
 FSRenameUnicode	Renames a file or folder.
 PBRenameUnicodeSync	Renames a file or folder.
 PBRenameUnicodeAsync	Renames a file or folder.

Comment 1

[Frank Tang]

see also 95478 about file picker

Comment 2

[Frank Tang]

All the Unicode related funciont is using FSRef instead of FSSpec. This could be 
hard since we have FSSpec in mnay many places-
including plugin, nspr, liveconnect, java, mrj, xpcom/io, etc.

Comment 3

[Simon Fraser [no longer active]]

It's also hard because an FSRef must point to an existing file system entity; you 
can't have one point to a file that is yet to be created, like you can with an 
FSSpec.

Comment 4

[Conrad Carlen (not reading bugmail)]

If you look at the Availability info for these Unicode routines, they are
available in InterfaceLib 9.0 and later. In other words, this does not require
OS X an should be done for any system on which the Gestalt selector
gestaltFSAttr returns a response with the gestaltHasHFSPlusAPIs bit set.

Can I have this bug?

Comment 5

[Frank Tang]

sfraser: about your comment 
"FSRef must point to an existing file system entity".

see http://developer.apple.com/technotes/tn/tn2022.html
TN2022 The Death of typeFSSpec: moving along to typeFileURL

"In a nutshell, typeFileURL is a core-foundation URL en"coded to a stream of
byte s in UTF8 format. This is the suggested data type to use when your
application would like to create a reference to 
a file that has yet to be created. Furthermore, there are a number of other good
reasons to use this type; depending on your processing requirements, you may
wish to use this data type in a number of different circumstances. Here are some
properties and features of the typeFileURL data format:

    * typeAlias and typeFSRef require deterministic reference and cannot refer
to files that have yet to be created. typeFileURL uses the weak "by name" style
reference provided by URLs, and as such is quite capable of providing
pre-determined references to files.

"


wtc- please read
http://gemma.apple.com/techpubs/macosx/SystemOverview/SystemOverview/FileSystem/File_Encodings_and_Fonts.html

"In addition, all code that calls BSD system routines should ensure that the
const *char parameters of these routines are in UTF-8 encoding. All BSD system
functions expect their string parameters to be in UTF-8 encoding and nothing else.
"

and "QA1028 ICLaunchURL, "file:///" URLs and Mac OS X"
http://developer.apple.com/qa/qa2001/qa1028.html
"    * Text Encodings -- In Mac OS 9 the URL components are assumed to be
percent-encoded PStrings, as returned by PBGetCatInfo. This is inadequate for
Mac OS X, where the File Manager fully supports Unicode file names. In Mac OS X
the URL components are defined as percent-encoded UTF-8.
...
One possible workaround is to change your code to generate the "file:///" URL in
the right format. You can do this with a sequence of File Manager, CFURL, and
CFString calls.

An easier workaround is to avoid ICLaunchURL entirely and change your code to
call the Launch Services routine LSOpenCFURLRef. This code is very simple.

   1. If you're starting with an FSRef, just skip to step 2. If you're starting
with an FSSpec, convert it to an FSRef using FSpMakeFSRef.
   2. Create a CFURL based on the FSRef using CFURLCreateFromFSRef.
   3. Pass that CFURL to LSOpenCFURLRef.
   4. Release your reference to the CFURL.
"

Comment 6

[Frank Tang]

see 95560 about return "UTF-8" as filename charset in nsIPlatformCharset on
MacOS X. This will make the current file: url handling code assume "UTF-8" as
the file name encoding. However, this may introduce some problem with some Mac
file operation. 

Comment 7

[Frank Tang]

reassign to ccarlen since he ask for it.

Comment 8

[Conrad Carlen (not reading bugmail)]

taking

Comment 9

[Conrad Carlen (not reading bugmail)]

*** Bug 87831 has been marked as a duplicate of this bug. ***

Comment 10

[Rui Xu]

Add myself to CC for reviewing bug 87831 later.

Comment 11

[Conrad Carlen (not reading bugmail)]

-> 0.9.8

Comment 12

[nhottanscp]

I am attending Apple's Carbon kiitchen.
There are clients of nsLocalFile which use methods with FSSpec. How about adding methods with FSRef, so 
clients do not have to use FSSpec? There are conversion API which convert between FSSpec and FSRef. I 
added methods which takes FSRef and just convert to FSSpec which is used for nsLocaleFileMac internal 
implementation.
Conrad, what do you think?

Comment 13

[nhottanscp]

Attachment: Adding methods with FSRef.

Comment 14

[Simon Fraser [no longer active]]

Note that an FSRef cannot refer to a file that does not yet exist, whereas an 
FSSpec can.

Comment 15

[nhottanscp]

Simon, so is nsLocalFile used for that situtation, I mean for a file does not
exist? How about CFURL, can that be used for non existing files?
I think we cannot stick with FSSpec (e.g., the user can assign non default
language file names in Finder in OSX).

Comment 16

[Conrad Carlen (not reading bugmail)]

> so is nsLocalFile used for that situtation, I mean for a file does not
exist?

Yes. Frequently.

> How about CFURL, can that be used for non existing files?

That may be the way to go. Another is to use an FSRef + appended unicode path.
Some portion of a file path (at least the volume) must exist. A non-existent
file path can be specified with the combination of those 2.

It will either be done with CFURL or FSRef + unicode appended path, depending on
which gives better performance. But, either way, FSSpec will be going away.

Comment 17

[nhottanscp]

Sorry, I still don't get it about the file not exist situation. Could you give
me examples? If that happens frequently, why not Apple supports it in FSRef?

Comment 18

[Conrad Carlen (not reading bugmail)]

See comment #5.

In mozilla, it happens all over the place. Typical scenario is:
(1) get a directory from directory service
(2) append a node or two (neither of them exist)
(3) if it doesn't exist, call create

After (2), the nsIFile specifies the file system object even though it doesn't
exist.

Comment 19

[Conrad Carlen (not reading bugmail)]

*** Bug 106271 has been marked as a duplicate of this bug. ***

Comment 20

[Frank Tang]

*** Bug 104935 has been marked as a duplicate of this bug. ***

Comment 21

[Conrad Carlen (not reading bugmail)]

Mass move to 0.9.9

Comment 22

[nhottanscp]

Conrad, any decision about CFURL or FSRef which one to use for nsLocalFileMac?

Comment 23

[Conrad Carlen (not reading bugmail)]

  There should be an InitWithCFURL() on nsILocalFileMac (maybe nsILocalFileMacX
which would be a subclass). This seems useful for specifying files which may or
may not exist.
  Internally, the implementation will probably use an FSRef + mAppendedPath just
as the current impl uses FSSpec + mAppendedPath. In the HFSPlus version of the
object, mAppendedPath will be a unicode string instead of a char string.
  That's what I'm thinking now - it'll become more clear after actually working
on it.

Comment 24

[Conrad Carlen (not reading bugmail)]

nominating nsbeta1

Comment 25

[Conrad Carlen (not reading bugmail)]

*** Bug 89064 has been marked as a duplicate of this bug. ***

Comment 26

[Steve Dagley]

Just my usual reminder that FSRefs can only point to exisitng file system
objects making them  less than exact replacements for FSSpecs.

Comment 27

[Steve Dagley]

Adding this as blocker for 102998 (since 89064 got closed as a duplicate of this)

Comment 28

[Conrad Carlen (not reading bugmail)]

> Just my usual reminder that FSRefs can only point to exisitng file system
objects making them  less than exact replacements for FSSpecs.

You don't have to remind me of that ;-) Got it under control. Read some of the
previous comments.

Comment 29

[Steve Dagley]

Sorry, comment #23 scared me :-)

Comment 30

[Conrad Carlen (not reading bugmail)]

Changing summary since OSX is not required to take advantage of this.

Comment 31

[J Luh]

Will this do anything for bug 125621 (non-ASCII path interpreted incorrectly by
input type=file)?

Comment 32

[Conrad Carlen (not reading bugmail)]

*** Bug 133611 has been marked as a duplicate of this bug. ***

Comment 33

[Frank Tang]

I think it is too risky and too late to do this one for beta1. I suggest we
change this bug from nsbeta1+ to nsbeta1- and try this again later. 

Comment 34

[J Luh]

There are a number of problems with Japanese and non-ASCII file names on Mac OS X:
bug 107181, bug 125614, bug 125621, bug 125640, bug 137846, bug 138008, bug 138350

I don't know how many of them pertain to this issue (if any) but if many of them
do, that may make it more worthwhile to try to get this change into Mozilla 1.0.

Comment 35

[Paul Baker]

Just to give everyone an example of how badly this needs to be fixed for 1.0, I
was not able to download Mozilla 1.0 RC1 for Windows off of mozilla.org using
the Mozilla 1.0 RC1 build for Mac OS X, because the file name is too long. It's
too long because Mozilla on OS X doesn't understand the fact that OS X actually
supports file names longer than 31 characters. This really makes Mozilla look
pretty lame on OS X. Can you hear IE uses now? "What? I can't download files
with names longer than 31 characters? And they call themselves standards complient."

Comment 36

[Conrad Carlen (not reading bugmail)]

I agree that this needs to be fixed ASAP. Believe me, if I had my choice of what
to work on today, it would be this. Unfortunately, there are some higher
priority bugs I have to deal with first.

> I was not able to download Mozilla 1.0 RC1 for Windows off of mozilla.org using
the Mozilla 1.0 RC1 build for Mac OS X

Hmm. That used to work - See bug 56295 which is resolved. In the meanwhile, the
file downloading code was heavily re-worked and that may have circumvented the
fix from 56295.

Comment 37

[Nick Kocharhook]

Believe you me, I want this bug fixed as much as anyone.

That being said, the latest version of IE in OS X does exactly the same thing:
downloads are limited to 31 characters.

So the worst that can be said is that Moz is "average" instead of "ahead of the
pack" in this area.

Comment 38

[Chris Lewicki]

To continue the chatter on this:

> That being said, the latest version of IE in OS X does exactly the same thing:
> downloads are limited to 31 characters.

> So the worst that can be said is that Moz is "average" instead of "ahead of the
> pack" in this area.

No!  It is behind the pack, as opposed to IE in OS X, Mozilla won't even
download files longer than 31 characters.  IE truncates the name at least, and
saves it to file.  Mozilla gives you the impression that its saving, but the
file vanishes into the ether.

Comment 39

[Conrad Carlen (not reading bugmail)]

*** Bug 142043 has been marked as a duplicate of this bug. ***

Comment 40

[benc]

Chris:
I think parts of that download problem were recently addressed in bug 139360.
You should check the fix date, and report remaining problems there.

Comment 41

[Frank Tang]

also, please see 
http://developer.apple.com/technotes/tn/tn1150.html

look at HFSUniStr255 

also
Text Encodings

Current Mac OS programming interfaces pass filenames as Pascal strings (either
as a StringPtr or as a Str63 embedded in an FSSpec). The characters in those
strings are not Unicode; the encoding varies depending on how the system
software was localized and what language kits are installed. Identical sequences
of bytes can represent vastly different Unicode character sequences. Similarly,
many Unicode characters belong to more than one Mac OS text encoding.

HFS Plus includes two features specifically designed to help Mac OS handle the
conversion between Mac OS-encoded Pascal strings and Unicode. The first feature
is the textEncoding field of the file and folder catalog records. This field is
defined as a hint to be used when converting the record's Unicode name back to a
Mac OS- encoded Pascal string.

....

struct HFSPlusCatalogKey {
    UInt16              keyLength;
    HFSCatalogNodeID    parentID;
    HFSUniStr255        nodeName;
};

...
struct HFSPlusCatalogFolder {
....
    UInt32              textEncoding;
....
};

...
struct HFSPlusCatalogThread {
....
    HFSUniStr255        nodeName;
};

Comment 42

[Conrad Carlen (not reading bugmail)]

Frank, an HFS+ file implementation is already checked in on the Chimera branch.
It does no conversion at all between Mac OS-encoded Pascal strings and Unicode.
The *Unicode* methods on nsIFile go straight to HFS+ APIs as Unicode. The
*Native* methods use UTF-8. So, the only conversion is between Unicode & UTF-8.
UTF-8 was chosen as native because all of the BSD-level routines on OSX take
*only* UTF-8. So, a path from GetNativePath() can be passed to fopen() and the
right thing happens. Also, the conversion between Unicode & UTF-8 is quick and
non-lossy.

That said, there is a problem in getting this code to work in a CFM (Code
Warrior)  environment because MSL's stdio routines take only HFS, narrow
charset, colon-delimited paths (bogus) and because nsIFileSpec for Mac CFM is
FSSpec-based.

Comment 43

[Simon Fraser [no longer active]]

> That said, there is a problem in getting this code to work in a CFM (Code
> Warrior)  environment because MSL's stdio routines take only HFS, narrow
> charset, colon-delimited paths (bogus) and because nsIFileSpec for Mac CFM is
> FSSpec-based.

Does that change with Pro 8?

Comment 44

[nhottanscp]

*** Bug 137846 has been marked as a duplicate of this bug. ***

Comment 45

[Matthias Versen [:Matti]]

*** Bug 159637 has been marked as a duplicate of this bug. ***

Comment 46

[Greg K.]

How about an updated target for this?

Comment 47

[Conrad Carlen (not reading bugmail)]

-> 1.3

BTW, the HFS+ file impl is now in use by the Mach-0 build. Fixing summary to
reflect that this is a CFM-only problem.

Comment 48

[Frankie]

BTW, how well does the Mach-O build work? I've been using OSX latest-trunk
builds for the past year, and I've been pretty happy with them. Can I just drop
replace the CFM app with Mach-O and everything will keep working?

Comment 49

[Andreas Becker]

Changing QA contact to bobj for now. Bob, please re-assign further as you see is
appropriate.

Comment 50

[Frankie]

Now that CFM has been retired in favor of Mach-O, has this bug become more or
less irrelevant? Some of the bug dependencies probably should be changed.

Comment 51

[Frankie]

Sorry for the spam. Fizzilla CFM build is dead; should this bug go with it?

Comment 52

[Steve Dagley]

To paraphrase Dr. McCoy, CFM is dead Jim.

Comment 53

[Paul Baker]

Does that mean 1.3b will only have a macho build?

Comment 54

[Steve Dagley]

To answer the 1.3b question, yes, it's Mach-O only for Mac

Comment 55

[Frankie]

*** Bug 197003 has been marked as a duplicate of this bug. ***