Last Comment Bug 413171 - NSS does not allow specifying a profile path containing characters outside of the system native codepage
: NSS does not allow specifying a profile path containing characters outside of...
Status: RESOLVED DUPLICATE of bug 336829
: helpwanted
Product: NSS
Classification: Components
Component: Libraries (show other bugs)
: 3.0
: x86 Windows XP
: -- normal with 1 vote (vote)
: ---
Assigned To: nobody
:
Mentors:
: 440686 (view as bug list)
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2008-01-19 14:43 PST by Ted Mielczarek [:ted.mielczarek]
Modified: 2011-12-21 10:23 PST (History)
14 users (show)
benjamin: blocking1.9.1-
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
Quick and dirty patch (1.71 KB, patch)
2008-07-11 03:18 PDT, Wladimir Palant
benjamin: review+
Details | Diff | Review
Quick and dirty PSM patch, updated (checked in) (1.25 KB, patch)
2008-12-15 02:43 PST, Wladimir Palant
trev.moz: review+
Details | Diff | Review

Description Ted Mielczarek [:ted.mielczarek] 2008-01-19 14:43:09 PST
I have a profile in a Windows XP VM that has chinese characters in the username (en-US Windows).  With 2008-01-19-04, I get an error message on startup:
"Could not initialize the application's security component. The most likely cause is problems with files in your application's profile directory. Please check that this directory has no read/write restrictions and your hard disk is not full or close to full. It is recommended that you exit the application and fix the problem. If you continue to use this session, you might see incorrect application behaviour when accessing security features."

I never used to get this.
Comment 1 Ted Mielczarek [:ted.mielczarek] 2008-01-20 06:18:29 PST
See also bug 396052, (was blocking-1.9+) and bug 396199
Comment 2 Kai Engert (:kaie) 2008-01-21 08:56:33 PST
Ted gave me some more details that might help to reproduce this bug.

I'm not able to work on this right away, so I'm dumping the information into this bug. I'd greatly appreciate any help to debug where exactly NSS is failing.

Ted: "I think PSM or NSS is doing something wrong on windows during startup. probably using GetNativePath."

Is there a way to type chinese chars on my system?

additional info: "i created a user on my windows xp VM with a chinese username. 
 i just copy/paste them from wikipedia. http://en.wikipedia.org/wiki/Hanzi . if you have XP, you can enable the "select a user" screen and just click on it."
Comment 3 Ted Mielczarek [:ted.mielczarek] 2008-01-21 17:54:22 PST
I can debug this if you can point me at a function to step through.
Comment 4 Benjamin Smedberg [:bsmedberg] 2008-01-21 18:13:43 PST
I'll happily blame http://mxr.mozilla.org/mozilla/source/security/manager/ssl/src/nsNSSComponent.cpp#1519

But the problem is really that there isn't a wide-character version of NSS_InitReadWrite: http://mxr.mozilla.org/mozilla/source/security/nss/lib/nss/nss.h#110
Comment 5 Ted Mielczarek [:ted.mielczarek] 2008-01-21 18:16:49 PST
Bummer.  Could NSS be changed to expect a UTF-8 string there?  Not declaring what charset you're accepting with a char* is sort of bogus.
Comment 6 Benjamin Smedberg [:bsmedberg] 2008-01-21 18:19:19 PST
I doubt it, NSS has very strict backwards-compatibility requirements. You'd have better luck adding NSS_InitW and NSS_InitReadWriteW variants, I think.
Comment 7 Kai Engert (:kaie) 2008-01-22 01:23:35 PST
This bug claims that NSS_InitReadWrite does not accept UTF-8 for configdir.

=> NSS for triaging
Comment 8 Benjamin Smedberg [:bsmedberg] 2008-01-22 05:53:52 PST
Well... it shouldn't accept UTF8 IMO... the previous behavior and normal behavior for APIs that take char* is to use the system native codepage, which is never UTF8. Of course Windows sucks and you can't represent the full unicode repertoire with any of its native codepages.
Comment 9 Nelson Bolyard (seldom reads bugmail) 2008-01-22 10:35:50 PST
I think it's going to be difficult to reproduce this or debug it without 
having an exact copy of the directory pathname string passed to NSS_Init*.  
Could you add a comment with a dump in hexadecimal of the string with which 
the failure occurs? 

Also, what character sets is your windows box configured to use?  
What "code page" is it configured to use?

IIRC, the only requirement that NSS puts on the strings passed to NSS_Init* 
for directory names is that they NOT be "wide" character sets, such as UTF-16 
(a.k.a. UCS-2).  Any single-byte or "multi-byte" (variable width) character 
set, in which a single NUL byte terminates a string, is acceptable to NSS, 
assuming it is also acceptable to the underlying OS.  

In short, the caller of NSS_Init* must pass a string that is acceptable to 
the OS in the call to the OS's open function.  The required character set is whatever the OS expects for the open and stat calls.

ISTR there's some code that converts forward slash to backslash for path 
names when loading DLLs.  I wonder if that's matching on one byte of a 
multi-byte character.  Having a copy of the original directory name string 
would confirm or refute that hypothesis.

Also, ISTR there is some code in NSPR that calls Windows functions to convert 
between wide and narrow character strings.  I think the character set is 
implicitly chosen as whatever narrow character set ("code page") for which 
the OS has been configured.  I suspect this bug is going to end up being 
converted into an NSPR bug before it's done.
Comment 10 Ted Mielczarek [:ted.mielczarek] 2008-01-22 10:40:28 PST
Nelson: the problem here is that you can have Unicode characters in a file path that are not representable in a Windows native encoding.  My VM is probably using Windows-1252, since it's just en-US, and the path contains Chinese characters, which have no representation in that encoding.  Using GetNativePath() at all with a path like that will not work.
Comment 11 Benjamin Smedberg [:bsmedberg] 2008-01-22 10:42:26 PST
The problem is that there is *no* char* string which can represent the actual unicode directory in question.

e.g. create a directory named C:\ขฃ (Thai characters) in any version of windows. There is no char* which can represent this directory, and you cannot open a file in that directory using a call to open(). You have to use WCHAR* and _wopen to open a file in this directory.
Comment 12 Nelson Bolyard (seldom reads bugmail) 2008-01-22 13:10:34 PST
(In reply to comment #10)
NSS contains no calls to GetNativePath. Neither does NSPR. However PSM does.

(In reply to comment #11)
> The problem is that there is *no* char* string which can represent the actual
> unicode directory in question.

Well, of course, anything that can be represented in unicode can be represented in UTF-8 in a char * string.  

Perhaps the problem is that Windows cannot be persuaded to interpret the path 
name in a FileOpenA call as a UTF8 string?
Comment 13 Benjamin Smedberg [:bsmedberg] 2008-01-22 13:12:26 PST
Correct. The Windows "A" APIs are incapable of opening a file named with Thai characters.

PSM could certainly pass a UTF8 char* to NSS_Init, but I didn't think that would match your needs for backwards compatibility.
Comment 14 Nelson Bolyard (seldom reads bugmail) 2008-01-22 17:59:01 PST
So, I think our choices are roughly among these:

a) redefine the existing NSS_Init* functions, declaring that the char* string 
parameters that they take for file/directory pathnames and prefixes are UTF-8
by definition.  This probably is bad for backwards compatibility, at least 
on Windows.

b) redefine the existing NSS_Init* functions, declaring that IFF the string 
argument values contain non-ASCII characters and also are valid UTF-8 strings,
then we will translate them into UCS2 and use wide-character functions for 
opening/stating files, etc., at least on Windows.  

I think most non-ASCII strings containing characters from narrow character 
sets other than UTF8 are NOT also valid UTF-8 strings, So the test for 
"IS this a valid UTF8 string" (which NSS already has) would be a pretty 
accurate way to differentiate between UTF8 strings and strings in other 
non-ASCII character sets.  If that is correct, this choice wouldn't be an incompatibility for most users.  But there may exist some non-ASCII strings that can be interpreted as valid UTF-8 AND as valid strings in some other character set that is a native character set in windows.  
If so, that could be a problem.

c) We could define a new set of NSS_Init* functions that are defined to 
take only UTF8 strings for path names, etc., as opposed to the present ones
that seem to take strings in the OS's present narrow character set ("code page").  I think this choice is the most likely to succeed of any of these.

d) We could define a new set of NSS_Init* functions that take wide character
strings instead of narrow character strings.  Frankly I'm afraid this would 
be a nightmare to do in NSS.  

I need to learn about what non-Windows OSes allow in pathnames.  
Do Linux, MacOS and most Unixen have wide-character forms of open?  
Do they accept UTF8 file names in open?  
Do they accept other non-ASCII non-UTF8 character strings in open?

The NSS team needs to discuss this in our next meeting.  
Help me gauge the urgency of this.
I gather that this is not perceived as a show-stopper for FF3, right?
Comment 15 Ted Mielczarek [:ted.mielczarek] 2008-01-22 18:02:54 PST
Mac and Linux generally use UTF-8 natively, as I understand it, so they don't have this problem.  From within PSM, we could do c) or d) very easily.

This is probably not a showstopper for Firefox 3, but we've spent a bit of time making sure that this case works, and it'd be a shame to regress it at the last minute.
Comment 16 Benjamin Smedberg [:bsmedberg] 2008-01-22 18:09:15 PST
> Do Linux, MacOS and most Unixen have wide-character forms of open?  

No.

> Do they accept UTF8 file names in open?  

Yes, *if* you have LANG set to UTF8, which is the default on modern linuxes/macos (mine is en_US.UTF-8)

> Do they accept other non-ASCII non-UTF8 character strings in open?

The LANG environment variable determines the codepage used for the filesystem. So if you had LANG set to en-US.EUC-JP then the char* passed to open() and other system calls would be interpreted as EUC-JP instead of UTF8.

I don't think this can be a FF3 stop-ship blocker, but we've really been trying to be fully unicode-correct for FF3.
Comment 17 Nelson Bolyard (seldom reads bugmail) 2008-01-22 18:26:55 PST
One element of this bug that seems confusing.  In comment 0 and in comment 15 Ted seems to suggest that this is a regression in some recent version of the 
browser.  But the situation Benjamin describes, if relevant, would suggest 
that this has NEVER worked with NSS.  

So, I'd like to narrow this bug down, and decide:
Is this a problem that has always existed in NSS? or 
Is this a regression in some recent version of the browser?

Can we get a regression window?  Or at least some browser versions with which
the problem is not seen, as well as some version(s) with which it is seen?
Is it only a trunk issue?

Let me repeat that I'd really like to get a hex dump of the string(s) passed
to NSS_Init* by PSM when this failure occurs.
Comment 18 Ted Mielczarek [:ted.mielczarek] 2008-01-23 11:39:15 PST
I think given the date of that code (from blame), this is probably not a regression.  I was probably mistaken in my original assessment.  We fixed a number of issues related to proper Unicode support in file paths, see the bugs mentioned in comment 1.  I thought I had tested after those landed, but I must have been wrong.  Regardless, I still think that being fully Unicode aware on Windows is important.
Comment 19 Ted Mielczarek [:ted.mielczarek] 2008-06-20 06:50:08 PDT
*** Bug 440686 has been marked as a duplicate of this bug. ***
Comment 20 Dženan Zukić 2008-06-20 07:18:43 PDT
Well, FF3 shipped and this bug is stopping me from using it. My home path is "C:\Documents and Settings\Dženan Zukić". "ć" is the only non-en_US character. If you want, you can copy it from this comment.
Comment 21 Nelson Bolyard (seldom reads bugmail) 2008-06-20 10:13:37 PDT
(In reply to comment #20)
> Well, FF3 shipped and this bug is stopping me from using it. My home path is
> "C:\Documents and Settings\Dženan Zukić". "ć" is the only non-en_US
> character. 

I'll wager that "ž" is also not an ASCII character.  :)

Dženan, are you also a FF2 user?  
Does FF2 have the same problem?  (I expect that it does)
Comment 22 Ted Mielczarek [:ted.mielczarek] 2008-06-20 10:19:31 PDT
Two points:

1) This should only be a problem if you have characters in your username that are not representable in your system's native codepage. Non-ASCII is fine, as long as it's in your codepage. If you're using Windows-1251 and trying to use non-ASCII, then you have a problem here.
2) Firefox 2 is likely to have even worse problems, as we fixed a number of Unicode path handling bugs in the startup/profile system for Firefox 3.
Comment 23 Dženan Zukić 2008-06-20 12:43:44 PDT
ž is part of code page 1252 (the default=en-US), code point 158. However, ć is not (therefore, it is not representable).
I use firefox2 at university (where this problem is occurring), I had it before and now I went back on it again (because I cannot use Firefox 3 obviously).
FF2, along many other applications, had create another folder profile "C:\Documents and Settings\Dženan Zukic", with "c" instead of "ć", and consistently uses it, so there is no problem.
I would say that some parts of FF3 use Unicode paths properly, and some other parts don't, which results in FF3 not able to start up at all.
Comment 24 Dženan Zukić 2008-06-20 14:34:08 PDT
Firefox 2 does not have a problem with profile being named "Dženan Zukić", becuase FF2 consistently uses folder "Dženan Zukic". It is wrong, and would make quite a mess if there were some other user named "Dženan Zukic", besides one name "Dženan Zukić". However, that is not the case and FF2 works as it should (although it stores it data in wrong folder - possible security issue with permissions).
Comment 25 Wladimir Palant 2008-07-11 01:44:01 PDT
(In reply to comment #13)
> Correct. The Windows "A" APIs are incapable of opening a file named with Thai
> characters.

What if the file path is first passed through GetShortPathName() before it goes to NSS? I don't see any documented guarantee that the result of GetShortPathName is ASCII but it is still better than nothing.
Comment 26 Wladimir Palant 2008-07-11 03:18:54 PDT
Created attachment 329051 [details] [diff] [review]
Quick and dirty patch

This patch takes care of the issue for me - I can use XULRunner despite the profile path containing Cyrillic characters. It also cures the crash (bug 440766).
Comment 27 Ted Mielczarek [:ted.mielczarek] 2008-07-11 04:55:45 PDT
Comment on attachment 329051 [details] [diff] [review]
Quick and dirty patch

Kai, bsmedberg, could you take a look at this?
Comment 28 Wladimir Palant 2008-07-11 05:06:14 PDT
I was told that there is a registry setting to disable 8.3 names in NTFS - new files created won't have short names then, and GetShortPathName() will fail for them. But since that settings isn't the default (and is unlikely to ever become the default for reasons of backwards compatibility) it is a minor issue as opposed to the major issue that we have right now.
Comment 29 Wan-Teh Chang 2008-07-11 09:48:16 PDT
Comment on attachment 329051 [details] [diff] [review]
Quick and dirty patch

Another approach is to try the original pathname first.
If it fails, then try the short (canonical) pathname as
a fallback.
Comment 30 Kai Engert (:kaie) 2008-09-05 19:51:01 PDT
Comment on attachment 329051 [details] [diff] [review]
Quick and dirty patch

Please brainstorm about Wan-Teh's proposal and if you believe his proposal is better, please attach a new patch.

If you're certain this patch is ok, moa=kaie
Comment 31 Wladimir Palant 2008-09-06 01:32:32 PDT
Wan-Teh's proposal:

1. General assumption is that using the original path is better than using the canonical path. So far I don't see any evidence confirming that however, canonical path is always less problematic than the original path.
2. "Try and check if it fails" assumes that we know where it will fail. However, the code path failing now might be changed later - and we will happily accept the original path that will fail further along the road.

Which means that so far I don't see any reason to take this route (please prove me wrong). Note that we are already using my patch for TomTom HOME, without any issues reported so far.
Comment 32 Wladimir Palant 2008-09-06 01:33:57 PDT
PS: I have no idea what "moa" means...
Comment 33 Nelson Bolyard (seldom reads bugmail) 2008-09-06 12:08:01 PDT
Wan-Teh, perhaps you should elaborate slightlyon your proposal. 
I have understood it to be a proposal for PSM to call the NSS initialization
function potentially twice, calling it first as it does today, and if that
fails, to then try again using the path string obtained from 
GetShortPathName().  Is that a correct understanding of the proposal?

Wladimir,  TomTom HOME uses NSS ?
(I don't know what moa means either.)
Comment 34 Wan-Teh Chang 2008-09-06 13:17:49 PDT
Yes, that's my proposal.  Here is the precise version:

  Try calling NSS_Init* with the original pathname first.
  If NSS_Init* fails, then try calling NSS_Init* with the
  short (canonical) pathname as a fallback.

moa stands for module owner approval.
Comment 35 Wladimir Palant 2008-09-07 01:36:20 PDT
There are three calls - NSS_InitReadWrite, NSS_Init and NSS_NoDB_Init (the second being the fallback for first and third fallback for second). Since write permissions are independent of whether we can use the original path, we would have to add new fallbacks to each of those calls, so it would now be six initialization attempts. But I still don't see why we should make that effort.

TomTom HOME uses XULRunner which in turn uses NSS - and we applied my patch a while ago, due to numerous complains from our Chinese department.
Comment 36 Nelson Bolyard (seldom reads bugmail) 2008-09-07 15:01:50 PDT
There are actually (at least) 5 flavors of NSS initialization functions.
In addition to the 3 you mentioned, there are also NSS_Initialize and 
NSS_InitWithMerge.  Of those, NSS_Initialize is the most general.  It can
do anything that the others can do.  You can think of the others as 
simplified versions of NSS_Initialize that are used in common cases.
Comment 37 Wladimir Palant 2008-09-07 23:55:00 PDT
Nelson, I only listed the initialization functions actually being called in the code. We would need fallbacks for each of those calls - changing NSS_Initialize won't work because at that point we don't have an nsILocalFile any more, so we cannot get the canonical path.
Comment 38 Wan-Teh Chang 2008-09-08 10:25:49 PDT
The purpose of calling NSS_InitReadWrite with the original pathname
first is to minimize the change from the current code.  We only need
to retry NSS_InitReadWrite with the short (canonical) pathname if
the error code is SEC_ERROR_BAD_DATABASE.  You can add this second
NSS_InitReadWrite call between the NSS_Init and NSS_NoDB_Init calls.
(NSS_NoDB_Init ignores its argument.)  Alternatively, you can add a
helper function that does the NSS_InitReadWrite and NSS_Init sequence,
and you call this helper function once with the original pathname
and once with the short (canonical) pathname.  Again, NSS_NoDB_Init
only needs to be called once, with "" or NULL as the argument.

If you are sure that it is safe to just use the short (canonical)
pathname, using your patch has the advantage that the new code is
always used -- any bug in using the short (canonical) pathname
will be immediately noticed.  Since I am not familiar with short
pathnames, I propose a more conservative approach.
Comment 39 Wladimir Palant 2008-12-15 02:43:32 PST
Created attachment 353007 [details] [diff] [review]
Quick and dirty PSM patch, updated (checked in)

Attaching updated version of the patch (diffed against 1.9.1b2 and proper format), taking review over from previous patch.
Comment 40 Nelson Bolyard (seldom reads bugmail) 2008-12-15 12:06:56 PST
This bug documents a deficiency in NSS's API, and it should remain an NSS bug.
But the patch being pursued in this bug in not an NSS patch.  It is a PSM 
patch.  It would be best if a new bug were to be filed against PSM 
(product:core, component:security/PSM) about this, and have this patch be
attached to that bug, and reviewed there, etc.  Because when this patch is
eventually committed, it will *probably) solve the problem being experienced
by PSM, but will not solve the problem in NSS.

Also, in NSS, we don't allow the practice of "carrying forward" an r+ from
one patch to another.  Each patch gets separately reviewed.  

So, please make a PSM bug.  Thanks.
Comment 41 Benjamin Smedberg [:bsmedberg] 2008-12-16 12:04:35 PST
Created bug 469856 for PSM.
Comment 42 Dženan Zukić 2009-09-16 02:38:01 PDT
There has been almost a year since last comment, is there any progress on this?
Comment 43 Nelson Bolyard (seldom reads bugmail) 2009-09-16 13:14:35 PDT
Dženan, the progress on bug 469856 seems to have removed all pressure to fix
this bug.  Will the solution to bug 469856 not solve the problem for you?
Comment 44 Wladimir Palant 2009-09-16 22:58:23 PDT
Nelson, the solution to bug 469856 is the patch in this bug...
Comment 45 Nelson Bolyard (seldom reads bugmail) 2009-09-16 23:11:46 PDT
... which is a PSM patch, not an NSS patch.  NSS people cannot review/approve
PSM patches and PSM people cannot review/approve NSS patches.  

Try this.  Take the patch that is currently attached to this bug and attach 
it to bug 469856, and ask some PSM people to review and approve it.
Comment 46 Nelson Bolyard (seldom reads bugmail) 2009-10-01 13:48:10 PDT
Comment on attachment 353007 [details] [diff] [review]
Quick and dirty PSM patch, updated (checked in)

This PSM patch was committed.  See bug 469856.
Comment 47 Nelson Bolyard (seldom reads bugmail) 2010-03-27 22:00:10 PDT
I believe this bug, and bug 336829 are duplicates of each other.
Comment 48 Ted Mielczarek [:ted.mielczarek] 2010-03-28 09:26:15 PDT
Sounds right. I'll dupe this bug over since it's got a lot of irrelevant content.

*** This bug has been marked as a duplicate of bug 336829 ***
Comment 49 L A Walsh 2011-12-20 15:24:10 PST
The solution to this ****appears*** simple.


Maybe someone could explain why this wouldn't work.

Currently, mozilla/xpcom/io/nslocalfileWin.cpp
has :
2794 
2795 NS_IMETHODIMP
2796 nsLocalFile::GetNativePath(nsACString &_retval)
2797 {
2798     //NS_WARNING("This API is lossy. Use GetPath !");
2799     nsAutoString tmp;
2800     nsresult rv = GetPath(tmp);
2801     if (NS_SUCCEEDED(rv))
2802         rv = NS_CopyUnicodeToNative(tmp, _retval);
2803 
2804     return rv;
2805 }
2806 

Instead, I propose:

2794 
2795 NS_IMETHODIMP
2796 nsLocalFile::GetNativePath(nsACString &_retval)
2797 {
2798     //NS_WARNING("This API is lossy. Use GetPath !");
2799     nsAutoString tmp;
2800     nsresult rv = GetPath(tmp);
2801     if (NS_SUCCEEDED(rv))
2802 +       rv = NS_ConvertUTF16toUTF8(tmp).get());
2802 -       rv = NS_CopyUnicodeToNative(tmp, _retval);
2803 
2804     return rv;
2805 }
2806 
----
Then no information will be lost, and wherever you are passing the path to back to the OS, you have to be using CopyNative to Unicode (as the OS doesn't accept local charsets, natively), so change that call into the corresponding ConvertUTF8toUTF16 call...

Problem transparently solved with with no backward incompat changes other than things that used to not work, now, will.
(comment appended to the bug this was dupped to as well as here...)
Comment 50 Benjamin Smedberg [:bsmedberg] 2011-12-21 10:23:21 PST
The OS does accept local charsets through all of the "A" functions as well as stdio functions like "open". The "native path" from nsILocalFile represents the path that you would pass to "open", which is unfortunately lossy but there's no helping Windows suckage.

Note You need to log in before you can comment on or make changes to this bug.