408138 - File URL incorrect when file path is not in filesystem encoding

Status: RESOLVED FIXED

(Core :: Networking: File, defect)

Description

[Mook (songbird dead account)]

Attachment: testcase (for xpcshell)

Tested on trunk, Linux/OSX (cannot get Windows/NTFS to give me a bad file name)

The OS is set to use UTF8 for file names, as default for any recent distro.

STR: Run attached xpcshell test case

Expected results: No lines between "Enumeating..." and "done"

Actual results: At least one entry (the file created near line 8) is listed

The problem is that, although the filesystem supposedly is using UTF8, it can actually still hold files that are not.  In this case, the file name is "\202.txt" (octal escaped; it's not a valid UTF8 encoding).

The spec ends up as file:///tmp/%C2%82.txt ("%C2%82" is the escaped UTF8 encoding of \202); when attempting to read that file, it does not exist.

However, file:///tmp/%82.txt does exist (if that is manually entered, it can be opened).

Workaround:
Do the Wrong Thing, and construct the spec as "file://" + nsLocalFile.persistentDescriptor (and be sad.  and be scared of getting broken whenever somebody feels like it)

Comment 1

[Michal Novotny [:michal]]

Attachment: proposed fix for unix

Problem is that native charset is set to UTF8 and filename isn't valid UTF8 string. Conversion UTF8 -> UTF16 fails when URL is build from nsLocalFile and latin1 -> UTF16 conversion is performed at http://mxr.mozilla.org/mozilla/source/xpcom/io/nsNativeCharsetUtils.cpp#747.

0x82 (latin1) == 0x82 0x00 (unicode) == 0xc2 0x82 (UTF8)

My proposed solution uses native path in net_GetURLSpecFromFile() instead of UTF16 path and tries if it can be converted from UTF16 to the same native path back. If conversion is correct then it is safe to use UTF8 path name. Otherwise native path is used. I'm not sure if this problem can occur also when native charset is other than UTF8. If not then it can be simplified and conversion check could be omitted.

nsURLHelperOS2.cpp can be fixed in the same way, but I can't test it so I didn't change it.

But I have no idea how to fix this problem on OSX. The lossy conversion is done inside nsLocalFile implementation. nsLocalFile::GetNativePath() doesn't return native path but UTF8 path and it is already crippled. I think that this is a bug in OSX implementation of nsLocalFile.

Comment 2

[Christian :Biesinger (don't email me, ping me on IRC)]

Comment on attachment 328175 [details] [diff] [review]
proposed fix for unix

+    // Convert to unicode and back to check correct coversion to native charset

coversion -> conversion

+        ePath = NS_ConvertUTF16toUTF8(path);

CopyUTF16toUTF8(path, ePath);

Comment 3

[Michal Novotny [:michal]]

Attachment: patch changed according to comment #2 [Checkin: Comment 8]

Comment 4

[Boris Zbarsky [:bzbarsky]]

Comment on attachment 331472 [details] [diff] [review]
patch changed according to comment #2
[Checkin: Comment 8]

sr=bzbarsky, but please make sure to file the OSX bug, and hunt down someone to fix it!

Comment 5

[Christian :Biesinger (don't email me, ping me on IRC)]

Is it possible on OS X to have non-UTF-8 paths? At least http://en.wikipedia.org/wiki/HFS+ stores filenames in Unicode.

Comment 6

[Michal Novotny [:michal]]

There is written in the description that the testcase was tested and failed on linux and OSX.

Comment 7

[Michal Novotny [:michal]]

Can somebody test this bug under OS/2? OS/2 code is almost identical to Unix code, so I can create the patch for OS/2 too, but I'm unable to test it.

Comment 8

[Serge Gautherie (:sgautherie)]

Comment on attachment 331472 [details] [diff] [review]
patch changed according to comment #2
[Checkin: Comment 8]

http://hg.mozilla.org/mozilla-central/rev/a81c47c501a8

Comment 9

[Serge Gautherie (:sgautherie)]

R.Fixed.

Maybe file a "followup" OS/2 bug too ?

Comment 10

[Boris Zbarsky [:bzbarsky]]

Did the OSX bug ever get filed?

Comment 11

[Michal Novotny [:michal]]

(In reply to comment #10)
> Did the OSX bug ever get filed?

Yes, #457702

Comment 12

[Michal Novotny [:michal]]

(In reply to comment #11)
> (In reply to comment #10)
> > Did the OSX bug ever get filed?
> 
> Yes, #457702

Sorry, that was #452516