179392 - nsITextToSubURI.ConvertAndEscape missing terminator with ISO-2022-JP

Status: VERIFIED FIXED

(Core :: Internationalization, defect)

Description

[Shoji Matsumoto]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; ja-JP; rv:1.1) Gecko/20021030 for VineLinux 0vl6
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; ja-JP; rv:1.1) Gecko/20021030 for VineLinux 0vl6

JavaScript escape() in HTML page is compatible with NC4.x (bug 22594),
but it is unusable in XUL js when encode with a charset.



Reproducible: Always

Steps to Reproduce:
var str = "\u3042";
1. escape(str);
2. escape(scriptableUnicodeConverter.fromUnicode(str)) with charset="EUC-JP"
   (I think it is invalid because the argument of escape() shuld be UCS-2)1.
Actual Results:  
1. %E3%81%82          : it is escaped with UTF-8 (valid)
2. %C3%A3%C2%81%C2%82 : it is invalid, naturally

Expected Results:  
ex) NC4.x(JP) escapes \u3042 to
- %C2%A4 from charset=EUC-JP pages
- %82%A0 from charset=Shift_JIS pages
- %1B%24B%24%22%1B%28B from charset=ISO-2022-JP pages

And many many sites (except for IIS) expects escaped queries in this way.
But I found no such methods. I can not generate URIs include escaped Japanese
(and Chinese, Korean) with any encodings expcept UTF-8.

I made some escape() valiants which are usable in XUL js.

escapeJS12(str, encoding)    --- compatible with JavaScript 1.2, 1.3, 1.4
escapeRFC1738(str, encoding) --- RFC1738 escape method
escapeRFC2718(str)           --- RFC2718 + RFC1738 (UTF-8 escaped with RFC1738)
escapeECMA262(str)           --- ECMA-262 3rd escape()

str should be UCS-2.

tested on http://shom.homelinux.org/~shom/mozilla/testcases/escapetest/

Comment 1

[Shoji Matsumoto]

Attachment: escapes.js

fromUnicodeForEscapes(str, encoding)
escapeRFC1738(str, encoding)
escapeRFC2718(str)
escapeECMA262 (str)
escapeJS12(str, enc)

Comment 2

[Boris Zbarsky [:bzbarsky]]

You can already do this from chrome JS.  See
http://lxr.mozilla.org/seamonkey/source/intl/uconv/idl/nsITextToSubURI.idl#51

Comment 3

[Shoji Matsumoto]

Oooops..... My outlook was narrow....
It is usable except ISO-2022-JP terminator bug(related bug 114923).

exepcted:
escapeJS12(<ISO-2022-JP>):-%1B%24B%24%22%24%24%24%26%24%28%1B%28B

actual:
escapeUI  (<ISO-2022-JP>):-%1B%24B%24%22%24%24%24%26%24%28

in ISO-2022-JP, the string should be terminated in ASCII mode "<ESC>(B"

Comment 4

[Shoji Matsumoto]

modify summary

Comment 5

[Boris Zbarsky [:bzbarsky]]

That would be an Intl bug.

Comment 6

[nhottanscp]

reassign to nhotta

Comment 7

[nhottanscp]

Attachment: Call nsIUnicharEncoder::Finish to ensure the converted string is terminated.

Comment 8

[Shanjian Li]

Comment on attachment 105913 [details] [diff] [review]
Call nsIUnicharEncoder::Finish to ensure the converted string is terminated.

r=shanjian,

Comment 9

[Boris Zbarsky [:bzbarsky]]

Comment on attachment 105913 [details] [diff] [review]
Call nsIUnicharEncoder::Finish to ensure the converted string is terminated.

sr=bzbarsky, but I various other places in the tree (eg nsFormSubmission) that
are not calling Finish().  Please check all nsIUnicodeEncoder users to make
sure they are calling Finish() properly when they should be.... (and file bugs
on the ones that are not)

Comment 10

[nhottanscp]

checked in to the trunk

bug 179955 was filed for the remaining parts which do not call
nsIUnicharEncoder::Finish().
nsFormSubmission.cpp does not use nsIUnicharEncoder.

Comment 11

[Yuying Long]

Verified fixed in 11-20 trunk build / WinXP, there is no more terminator missed
with escaped iso-2022-jp string.