Closed Bug 127282 Opened 24 years ago Closed 24 years ago

Mozilla does not handle non-ASCII file names well in <img > element

Categories

(Core :: Internationalization, defect)

Product:
Core
Component:
Internationalization
Platform:
x86
Windows 2000
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
VERIFIED FIXED
Milestone:
mozilla1.0

People

(Reporter: momoi, Assigned: nhottanscp)

References

(
URL
)

Details

(Keywords: intl, topembed+)

Attachments

(3 files, 2 obsolete files)

An actual web site which uses Japanese chars as flags below the domain name.
1.29 KB, text/html
Details
Changed GetDocumentCharacterSet to be const.
1.90 KB, patch
nhottanscp
: review+
nhottanscp
: superreview+
asa
: approval+
Details | Diff | Splinter Review
patch for 0.9.9
3.36 KB, patch
Details | Diff | Splinter Review
** Observed with 2002-02-21 Win32 trunk build ** The above test page contains 5 images which are loaded through an <img> element.The page is marked as "Shift_JIS" via a document meta equivalent charset declaration. Here's what 5 of them look like (view this under Japanese (Shift_JIS) encoding): 1. <img src="Taz.jpg" width="150" height="223" alt="taz"> 2. <img src="たず.jpg" width="150" height="223" alt="たず"> 3. <img src="たず.jpg" width="150" height="223" alt="Taz"> 4. <img src="たず.jpg" width="150" height="223"> 5. <img src="%82%BD%82%B8.jpg" width="150" height="223"> When viewed with the current Mozilla, only the images 1 and 5 load. Images 2-3 use a non-ASCII file name. In this case, all have the same name encoded in Shift_JIS. I don't believe Mozilla is currently handling this type of name in the document encoding. Just like the non-ASCII URLs, we are probably handling this in UTF-8. When you view this page with IE 5.5, you get 2 different results: 1. With "Tools | Internet Options | Advanced | Browsing | Always send URLs as UTF-8" ON, you get the same result as Mozilla. 2. With "Tools | Internet Options | Advanced | Browsing | Always send URLs as UTF-8 OFF, IE 5.5 loads all 5 images. I believe IE is shipping with the above option OFF to Asia and ON to the rest of the world. We should think of a similar change.
Keywords: intl
QA Contact: ruixu → ylong
Additional Info: The 0.9.4 branch (NS 6.2 & 6.2.1) does not load the 3 images, either.
This sounds unrelated to <img>. The problem is handling illegal URLs. Only a subset of ASCII codepoints are legal in the current URI spec. From http://www.w3.org/International/O-URL-and-ident: The official URI syntax, RFC 2396, currently only allows as subset of ASCII, about 60 characters. It also defines a way to encode arbitrary bytes into URI characters: a % followed by two hexadecimal digits (%HH-escaping). However, for historical reasons, it does not define how arbitrary characters are encoded into bytes before using %HH-escaping. RFC: ftp://ftp.isi.edu/in-notes/rfc2396.txt As noted in Kat's previous comment, IE has an option, "Always send URLs as UTF-8". It seems that this bug is a request to add a similar feature. What is the behavior when this IE option is disabled? Does it encode the href URLs in the encoding of the current page? What happens in the location bar? I tried copying the URL into the location bar of IE6 on US WinXP with the UTF-8 option disabled and it did not work. Related bugs: Bug 93419 Non-ASCII in URL at http://www.elcoteq.fi/ Bug 124042 support internationalized resource identifiers (UTF-8 URI)
Changed QA contact to myself.
QA Contact: ylong → teruko
If we want to copy what IE does, something like the following seems to be appropriate: 1. When the UTF-8 option is ON. Send all non-ASCII URLs as %-escaped UTF-8. 2. When the the UTF-8 option is OFF, a. Send a document-encoded URLs in %-escaped document encoding. b. Send a direct input non-ASCII URL into the location bar using the %-escaped system default encoding. Here's the results of one additional testing when the UTF-8 URL option is OFF. Using Windows 2000 US version, In a document with a Shift_JIS meta equivalent charset tag: 1. <a href="http://home.att.ne.jp/wood/cool/たず.jpg">test</a> This link works and takes us to the image under both US-English and Japanese system/locale default settings. 2. Typing "http://home.att.ne.jp/wood/cool/たず.jpg" into the Navigator URL field. Works under Japanese system/locale default setting but does not work under US-English system/locale default setting. So in summary, IE 5.5 seems to use document encoding when URLs are embedded but system default in the navigator location field. How about an embedded URL which points to a server outside its own? I looked at a local HTML file which had a link to an external URL, i.e. the case in 1 above. It displayed the destination image OK. From this it's safe to say that the logic involved is not that complex -- use the document encoding regardless of where the URL points to.
I will not say that the above proposal will work for all the cases but it should work for a large majority of cases when the UTF-8 option is turned OFF. But that would be better than what Mozilla is doing now. Note that IE 5.5 or 6.0 will not work for all the cases, either, since an external URL may not match the system encoding (Location Bar) nor the document encoding (an embedded URL). But Mozilla will be on the par with IE.
Sounds reasonable. A few more questions we need to resolve: - Where do we add the pref in the UI? (in Navigator|Languages panel?) - What are the default value for the various localized browsers? - When changing the pref, will Mozilla require a restart like IE? - How is a query component affected? (I believe it is not converted to UTF-8.) - Are non-HTTP URLs treated the same? The IE description, implies IE treats all URLs the same: "Always send URLS as UTF-8 (requires restart)" (Side issue: What happens with query coponents in mailto URLs? Do we MIME real names, subjects, etc.?)
adding topemed+ since this will be needed for embeding customers at the earliest opportunity. Any ideas on the target milestone or time frame for landing?
Keywords: topembed+
Non ASCII and unescaped URI behavior is mentioned in the HTML spec. http://www.w3.org/TR/html4/appendix/notes.html#h-B.2.1 It recommends UTF-8 then the document charset. For HREF, mozilla implemented that way. For IMG, UTF-8 is used and no fallback by the document charset (retry is not easy because the IMG case is asynch unlike the HREF case).
nhotta: how much amount of work is required for this bug to get fixed? weeks or days?
I think that depends on what we do. Just to switch from UTF-8 to a document charset would not be hard.
and once bug 124042 is fixed, all you will have to do is pass the document charset to NewURI.
I guess my real question was how hard is it to make the document charset as a fallback charset when UTF-8 fails (as recommended by HTML spec) darin/nhotta?
i don't know, but i was under the impression that if an URL appears w/ non-ASCII characters that they should be interpreted according to the document charset. that is, if the document charset is ISO-8859-1, then non-ASCII characters in the URL should be URL-escaped ISO-8859-1 and not UTF-8. is this incorrect? this is how 4x and IE work isn't it?
I agree. I think the currently existing documents (and HTML tools) use a document charset for URI (either escaped or unescaped, even the unescaped one is illegal). Darin, with the change of bug 124042, we do escape the unescaped URI before sending the request?
> if the document charset is ISO-8859-1, then non-ASCII characters in the > URL should be URL-escaped ISO-8859-1 and not UTF-8. is this incorrect? this is > how 4x and IE work isn't it? IE does not seem to work this way when UTF-8 option is checked. My experiment shows that it works like that only when the option is OFF.
FYI, on Simplified Chinese XP, UTF-8 option is checked on IE6.0 by default.
perhaps we should have a similar preference. if everyone passes the document charset to necko (via NewURI... see patch for bug 124042), then necko could watch the UTF-8 pref and encode accordingly.
From comment #8 Non ASCII and unescaped URI behavior is mentioned in the HTML spec. http://www.w3.org/TR/html4/appendix/notes.html#h-B.2.1 It recommends UTF-8 then the document charset. For HREF, mozilla implemented that way. For IMG, UTF-8 is used and no fallback by the document charset (retry is not easy because the IMG case is asynch unlike the HREF case). After the asynch call, we display the broken image gif. At that point, can we add code to retry non-ASCII IMG URLs %-encoded in the document charset?
yes we could do that. it wouldn't be that hard. my question: what about all the callers of NS_MakeAbsoluteURIWithCharset. those prefer the document charset over UTF-8. is that wrong? how do we know that a particular URL should be encoded in the document charset vs. UTF-8? also, isn't this fallback strategy expensive on sites that prefer URLs in the document charset? imagine a document w/ many unicode image URLs.
I think nhotta and darin are far more knowledgeble about this issue than I am. Either of you want to take an ownership of this?
the solution's going to depend on bug 124042. i can take ownership of this bug if nhotta doesn't want it, but i am pretty busy with other things right now.
Depends on: 124042
I can take this. The immediate issue can be resolved by using a document charset. With bug 124042, it should be simple to make the change, although I have to look where to change. About the fallback recommended in the HTML spec, I think we can skip the fallback mechanism and use a document charset when that is expensive. URI for the case like image loading actually cannot be covered only by the HTML spec and we need a standard for non ASCII URI. Here is a link to the draft (note that it is already expired), http://www.w3.org/International/2000/03/draft-masinter-url-i18n-05.txt
Assignee: yokoyama → nhotta
>The immediate issue can be resolved by using a document charset. This behavior is same as 4.x. Now I checked my setting of "Internet Options" in Control Panel and discovered that "Always send URLs as UTF-8" is set. I wonder are there any existing servers relying on that.
see bug 101230, which really should be marked as a dup of this one.
Since "Internet Options" is in the Windows' Control Panel, could we use the setting for Mozilla (instead of defining yet another nsPref setting)?
that would be something profile migration could do (i suppose)... we shouldn't try to make mozilla use IE preferences for a variety of reasons... one being that mozilla is a XP browser and it makes the project easier to manage if we use a XP solution for preferences.
Can we add a boolean pref for "Always send URLs as UTF-8"? If that is set then we can ignore a document charset. Can that be done in nsIURI? By the pref we can prepare for the servers which might expect UTF-8 URI for unescaped URI (but we are not sure if any server exist). I think the server which expects UTF-8 URI can understands UTF-8 and the server's charset, which is a charset used for the server's file system (e.g. ISO-8859-1, Shift_JIS). And the document which contains unescaped URI are likely to be using the same charset as the server's (i.e. a document charset of the document and the server's file system charset matches). So I think practically we have no problem of sending URI in a document charset.
Status: NEW → ASSIGNED
Target Milestone: --- → mozilla1.0
nhotta: exactly... but instead of honoring the pref in the URL impl, we'll need to honor on a per protocol handler basis. for example, we may only want to honor this for network URLs. URLs that correspond to local files may want to continue using the document charset, or whatever. the point of my patch in bug 124042 is that protocol handlers can decide what to do with the document charset. it is therefore critical that callers of NewURI always pass in a document charset. it'll be some work after my patch lands to fix up callers of NewURI so that they always pass in the document charset (if available).
What will we use for the default value of this pref? IE used to default to UTF-8 URLs for non-Asian versions of IE and disable UTF-8 URLs for Asian version of IE. If we assume that the IURI proposal will become standard and standardize UTF-8 URLs, should we fallback to UTF-8 in the doc charset case? That way "conforming" URLs would not break (even if they pay a performance penalty). Of course this is making an assumption about the IURI direction...
won't websites expecting IURI's (per the draft spec) use a document charset of UTF-8? or is this unlikely due to UTF-8 being more "bloaty" than other charsets (e.g., ISO-8859-1)?
I actually don't know actual servers expecting UTF-8. The server has to depend on the the "send as UTF-8" feature (i.e. client has to be IE on Windows). There may be some internal sites where servers and clients are all Windows machines.
Naoki and I talked about this in the hallway. We're guessing at what web servers do these days... Web servers probably use the file systems strings for the URLs and probably only %-decode the URL before doing the file look-up. There are few OSes which make it easy to create UTF-8 filenames today. And if you managed to create UTF-8 filepaths, most other tools and apps would probably render these filepaths as garbage characters. So we guess that it is unlikely there are many UTF-8 URIs today... (Ideally web servers would use UTF-8 as a pivot encoding and would have some mechanism to convert from the UTF-8 URI encoding to the file system encoding for the give URI. But we guess they don't do that today.) Bloat is not the issue. Most Latin documents contain mostly characters in the ASCII range which are 1-byte in UTF-8 and a fraction of others (e.g., accented characters). So the bloat is very small. See http://www-106.ibm.com/developerworks/library/utfencodingforms/#h2 On average, English takes slightly over one unit per code point. Most Latin-script languages take about 1.1 bytes. Greek, Russian, Arabic and Hebrew take about 1.7 bytes, and most others (including Japanese, Chinese, Korean and Hindi) take about 3 bytes. Characters in surrogate space take 4 bytes, but as a proportion of all world text they will always be very rare.
This HTML file is an existing web page at: http://products2.taxan.co.jp/index.htm in Japan. It uses this page to load two frame documents. The top frame is loaded with the following URL encoded in Shift_JIS: <FRAME SRC="http://products2.taxan.co.jp/FMPro?-DB=taxan-item.fmj&-Format=index-b2.htm&ニュース掲載フラグ=OK&-sortfield=登録日付&-sortorder=descend&-sortfield=商品型番&-sortorder=ascend&-max=10&-find" NAME=b MARGINWIDTH=0 MARGINHEIGHT=0 FRAMEBORDER=0> Note the use of flag values in Japanese. Testing shows that IE 5.5 with UTF-8 option ON loads this page OK (i.e. performs fallback in this case) and Comm 4.x does OK. Mozilla fails to load the top frame and gets an error page.
Momoi san, could you explain how is your example related to the <IMG> URI problem?
My example is not related to <IMG SRC ...> directly. However, the solution to the problem raiased here has been discussing more overall framework of UTF-8 and the document encoding as a fallback. I cited the above example to show that IE 5.5 seems to be falling back to document encoding in this case under the UTF-8 option. What is still mysterious to me is that the example for this bug does not work the same way with IE 5.5. Note that some images fail to load under the UTF-8 option ON. Why would it not fall back in this case?
Thank you for the explanation. > What is still mysterious to me is that the example for this > bug does not work the same way with IE 5.5. Note that some images > fail to load under the UTF-8 option ON. Why would it not fall back > in this case? > I think it depends on the server. If the server does not understand UTF-8 (which is usually the case, I assume) then it cannot fallback because the fallback can only be done by converting the URI from UTF-8 to a document charset.
>I think it depends on the server. If the server does not understand UTF-8 (which > is usually the case, I assume) then it cannot fallback because the fallback can > only be done by converting the URI from UTF-8 to a document charset. Then, how do you explain the fact that Mozilla fails on this same page while IE 5.5 and Comm 4.x do not. The only plausible explanation is that the client side is trying a fallback (IE 5.5) or it always sends the document encoding (Comm 4.x).
That is because Mozilla always use UTF-8 for IMG (see my comment #8). We can change it to a document charset (the 4.x behavior). Or we can do it by the pref (the IE behavior).
With regard to comment #38, the behvaior I am talking about in comment #37 refers to not to the image case but the URL case in my comment #33. Server-side handling of UTF-8 URLs cannot explian my comment in #33.
The link http://products2.taxan.co.jp/index.htm which was mentioned in comment #33, that seems to work with IE regardless of the "UTF-8" setting.
nhotta: momoi mentioned that that link works in IE because IE is doing fallback to the document charset when UTF-8 is enabled and otherwise tries the document charset first. IOW... that site expects URLs to be sent in the charset of the document.
So sending in the document charset works as far as we know. I plan to do at least for that change in moz1.0. Do we want the "send UTF-8" option for moz1.0? So far, we haven't found sites which only work with UTF-8 URI.
nhotta: yeah, i agree that we should make sure that we are at least sending URLs in the document charset. the "try UTF-8 first and the fallback to the document charset" approach can probably wait until post-1.0
Now bug 124042 is fixed in the trunk. But how can I set a document charset to nsIURI (it's readonly)? readonly attribute ACString originCharset; I also cannot find NS_NewURI which takes charset. Darin, is there a way to set a charset by the caller?
NS_NewURI has a charset parameter. this corresponds to nsIIOService::newURI and nsIProtocolHandler::newURI which also have charset parameters.
>NS_NewURI has a charset parameter. Right, I was looking at a wrong file (dist/inlcude/necko/nsNetUtil.h).
For the <IMG> problem, we can change the following code to pass a document charset to NS_NewURI instead of null. I can get a document charset from the document. But we are already asking the base URL to the doucment by GetBaseURL. It would be easier if the base URL is returned with the document charset already set. And the base URL is created outside of nsIDocument. I cannot set the charset in nsDocument after it's created. But the document charset is usually available after getting the content by using the URI (e.g. charset is passed by HTTP header or specified in the docoument by META). So, I like to have a way to set the document charset after the URI is create. Darin, can originCharset change not readonly? http://lxr.mozilla.org/seamonkey/source/layout/html/base/src/nsImageFrame.cpp#1950 1942 void 1943 nsImageFrame::GetRealURI(const nsAReadableString& aSpec, nsIURI **aURI) 1944 { 1945 nsCOMPtr<nsIURI> baseURI; 1946 GetBaseURI(getter_AddRefs(baseURI)); 1947 NS_NewURI(aURI, aSpec, nsnull, baseURI); 1948 } 1949 1950 void 1951 nsImageFrame::GetBaseURI(nsIURI **aURI) 1952 { 1953 NS_PRECONDITION(nsnull != aURI, "null OUT parameter pointer"); 1954 1955 nsresult rv; 1956 nsCOMPtr<nsIURI> baseURI; 1957 nsCOMPtr<nsIHTMLContent> htmlContent(do_QueryInterface(mContent, &rv)); 1958 if (NS_SUCCEEDED(rv)) { 1959 htmlContent->GetBaseURL(*getter_AddRefs(baseURI)); 1960 } 1961 else { 1962 nsCOMPtr<nsIDocument> doc; 1963 if (mContent) { 1964 rv = mContent->GetDocument(*getter_AddRefs(doc)); 1965 if (doc) { 1966 doc->GetBaseURL(*getter_AddRefs(baseURI)); 1967 } 1968 } 1969 } 1970 *aURI = baseURI; 1971 NS_IF_ADDREF(*aURI); 1972 }
nhotta: i don't believe it should be necessary to have to change the charset of an URI after it is created. besides doing so would possibly change the URL string as it appears on the network. it just doesn't make sense to change the charset of an URI. btw: if a baseURI already has the correct charset associated w/ it, then the charset for URLs constructed relative to that baseURI will inherit the charset from the baseURI. of course, this doesn't work very well for URLs entered in the URL bar. hmm... is this why you want to be able to set the origin charset of an URI?
The problem I have is that the baseURI does not have the correct charset (it's always "UTF-8"). That is why I tried to reset the charset in nsDocument where the correct charset is available. Sorry, the description of my last comment was confusing.
understood... obviously that will always be the case when the URL is entered from the URL bar, but is not necessarily true when the URL comes from a link click, since there is a document charset when a link click happens. therefore, you'll need to get the charset from the document whenever a link is clicked. it's just not possible to change the charset of a document URI.
I filed bug 129726 for the "Always send URL as UTF-8" option.
Comment on attachment 73262 [details] [diff] [review] For image src, set a charset from the document and use it for NS_NewURI. sr=darin, why are charsets stored as UNICODE? aren't they defined to be ASCII?
Attachment #73262 - Flags: superreview+
>aren't they defined to be ASCII? I think so. But nsDocument uses Unicode for other ASCII only data too. 626 nsString mContentLanguage; 627 nsString mContentType;
that's unfortunate and should probably change, but that probably isn't going to happen anytime soon :(
Attachment #73262 - Attachment is obsolete: true
I added a support for Frame SRC. That fixes the problem of http://products2.taxan.co.jp/index.htm . Darin, would you review the patch?
Comment on attachment 73307 [details] [diff] [review] Frame SRC support, combined with the last patch. sr=darin
Attachment #73307 - Flags: superreview+
Comment on attachment 73307 [details] [diff] [review] Frame SRC support, combined with the last patch. r=attinasi, but I think that GetDocumentCharacterSet can probably be made const
Attachment #73307 - Flags: review+
right, i forgot that layout/content uses const methods. please do make that change to be consistent w/ layout/content style.
Attachment #73307 - Attachment is obsolete: true
Comment on attachment 73553 [details] [diff] [review] Changed GetDocumentCharacterSet to be const. carry the reviews
Attachment #73553 - Flags: superreview+
Attachment #73553 - Flags: review+
Comment on attachment 73553 [details] [diff] [review] Changed GetDocumentCharacterSet to be const. a=asa (on behalf of drivers) for checkin to the 1.0 trunk
Attachment #73553 - Flags: approval+
checked in to the trunk
why this bug is not mark as closed/fixed after we check in ?
fixed
Status: ASSIGNED → RESOLVED
Closed: 24 years ago
Resolution: --- → FIXED
nhotta, Since it's already checked into trunk, marked as fixed, and has a topembed+ approval, could you please make a corresponding change to 9.9 branch at your earliest convenience ? We need this ASAP for our embedding client. Thank you so much. HK ( Hansoo Kim )
hansoo: you should send email to drivers@mozilla.org with that request. explain why it is critical for your product, etc.
> hansoo: you should send email to drivers@mozilla.org with that request. > explain why it is critical for your product, etc. Thank you, darin. I will... Can anyone add KW edt0.9.9 for requesting the fix in 9.9 branch in the meantime ? Thanks.
Keywords: edt0.9.9
is there a patch for 0.9.9 branch? How risky would it be to land it?
Attached patch patch for 0.9.9 — — Splinter Review
I think this is low risk. This affects when URI contains non ASCII characters. It sets a charset of the document instead of null. If the charset is set to null then UTF-8 converter is used, with the change a converter of the specified charset is used instead.
I verified this in 3-27-08 Win32 trunk build. I will mark as verified until it is fixed in 0.9.9 branch.
Hi... I was just going over the background of non-ASCII URL bugs in general to understand them better. While I was reading comments from this bug, some "what seems like unresolved" issue raised by Kat san in comment #33 came to my mind. > > The top frame is loaded with the following URL encoded in Shift_JIS: > > <FRAME > SRC="http://products2.taxan.co.jp/FMPro?-DB=taxan-item.fmj&-Format=index- > b2.htm&ニュース掲載フラグ=OK&-sortfield=登録日付&-sortorder=descend&-> > sortfield=商品型番&-sortorder=ascend&-max=10&-find" > NAME=b MARGINWIDTH=0 MARGINHEIGHT=0 FRAMEBORDER=0> > > Note the use of flag values in Japanese. > Testing shows that IE 5.5 with UTF-8 option ON loads this page OK (i.e. > performs fallback in this case) and Comm 4.x does OK. > Mozilla fails to load the top frame and gets an error page. He was wondering why IE fallbacks to document charset even when UTF-8 was ON, while the image src case it did not. ( in comments #35 ) I think it has to do with where non-ASCII characters appear in URL. See the following comment from Comment #83 From Erik van der Poel in bug http://bugzilla.mozilla.org/show_bug.cgi?id=10373 > However, blindly converting *every* part of a URL to UTF-8 has bad > consequences in today's Web, as Microsoft discovered. They do not convert > the "query" part (the part after the question mark) to UTF-8. Also, they > have a preference for the part before the question mark. The "preference" in the last sentence, I think, refers to IE's UTF-8 option. And according to him, After "?" mark, we should not encode to UTF-8 even with that preference is ON, which I think makes sense since those are just arguments to CGI scripts, unless a web server converts those into the native encoding b/f sending them to CGI programs. So Kat san's example, since upto "?" mark, URL does not contain non-ASCII characters, it does not get encoded to UTF-8. And according to Erik's comments, whatever after "?" mark is intact anyway for sending to the server. So I think it's actually not a fallback performed by IE on the client side at all and that's why that example did not get affected by UTF-8 option's setting at all. Now my question is that "are we doing the same thing in Gecko ?" I have not spent much time with a patch for http://bugzilla.mozilla.org/show_bug.cgi?id=124042 ( My god, it's such a HUGE patch !!! ) so I can not tell it for sure. And I am not even sure the possible weird results if we are not doing the same thing. ( Can not think of any case which might bite us back if not doing the same thing as IE. ) Can anyone provide comments on this ? Thanks.
Hm.... What if the CGI programs expects arguments to be in certain encoding other that the server's system encoding ? ( 3rd party program ? ) Just a thought... Thanks.
It would be better to identify the query part at parse time and escape it as is, in addition to the patch of this bug which sets a charset of the document to nsIURI. If the document contains an unescaped non ASCII URI which is different from the document charset then the data might be skipped if that's not convertible to Unicode. But that would be a broken HTML, I mean, putting a raw 8 bit characters which is different from the document's charset.
Verified in 1.0 branch build.
Status: RESOLVED → VERIFIED
Blocks: 93419
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: