26938 - nsconv misconverts "\" in.properties files to "\\"

Status: VERIFIED FIXED

(Core :: Internationalization, defect, P3)

Description

[Katsuhiko Momoi]

** Observed with nsconv with 2/7/2000 Win32 build **

This problem was reported by Kazu Yamamoto (JLP member).

If you have the following kind of line in .properties file,

\t\r\n\f

which is not unusual in either in Shift_JIS file or US-ASCII
encoding for this type of file, and if you use "nsconv" to do 
conversion of the following type,

nsconv -f US-ASCII -t x-u-escaped file1.properties file2.properties
nsconv -f Shift_JIS -t x-u-escaped file1.properties file2.properties

In either case, the "\t\r\n\f" gets turned into "\\t\\r\\n\\f".
This does not seem to be a correct result. 

Reversing the conversion, i.e. from x-u-escaped to Shift_JIS on 
file2.properties does not eliminate the extra "\". 

Kazu reports that 'native2ascii' does not have this kind of problem.
This would be a big problem when localizers use nsconv for 
.properties file conversion since they convert back from x-u-escaped
files to SJIS and do revisions and convert back to x-uescaped, and
they repeat this process "\" will multiply.

Comment 1

[Frank Tang]

ok, what should we do now. I propose we do the following
from x-u-escaped to others-
\n \t \' \\ \" -> \n \t \' \\ \"
from others to x-u-escaped-
\n \t \' \\ \" -> \n \t \' \\ \"

Comment 2

[Frank Tang]

easy fix

Comment 3

[Frank Tang]

This is not easy as we thought.
If we convert "\" to "\" from unicode to x-u-escaped, then "\uxxxx" in unicode 
will be converted to "\uxxxx" in x-u-escaped. However, when it convert back to 
unicode, it will be convert to one character U+xxxx . If we convert "\" to "\\" 
from unicode to x-u-escaped, then "\uxxxx" in unicode will be converted to 
"\\uxxxx" which will do the right thing. however, that mean we need to convert 
"\\" to "\" from x-u-escaped to unicode. 

Comment 4

[bobj]

What is the Java spec for handling '\'.  Is it considered regular 7-bin ASCII
or a meta character?  What does native2ascii do?

Comment 5

[tao]

Please refer to this excerpt from Java Language Spec:

"Within the element string (but not the key), the ASCII escape sequences 
\t, \n, \r, \\, \", \', \ (a backslash and a space), and \uxxxx are 
recognized and converted to single characters. Moreover, if the last 
character on the line is \, then the next line is treated as a
continuation of the current line; the \ and line terminator are simply
discarded, and any leading whitespace characters on the continuation line 
are also discarded and are not part of the element string."

See bug 21418.

Comment 6

[Frank Tang]

I will back out the bad fix for 21418 so \\ will convert to \ in unicode. This 
should make srue the nsconv round trip conversion. 

Comment 7

[Frank Tang]

move to M16

Comment 8

[Frank Tang]

*** Bug 28406 has been marked as a duplicate of this bug. ***

Comment 9

[Frank Tang]

reassign to bobj

Comment 10

[leger]

Need more info on how critical this is to beta2.

Comment 11

[tao]

The prompting strings in the dialog UI will not be formatted/indented properly
if this bug is not fixed.

Comment 12

[tao]

I own 21418 now so I am taking this one over, too.

Comment 13

[leger]

Clearing [NEED INFO] to get on re-eval query for PDT.

Comment 14

[leger]

Putting on [nsbeta2+] radar for beta2 fix.

Comment 15

[Mike]

M16 has been out for a while now, these bugs target milestones need to be 
updated.

Comment 16

[tao]

Hi, Frank:



DO you have cycle on this? I'm swamped with PR2 tasks.



THanks

Comment 17

[Frank Tang]

do we really need this for nsbeta2 ? Do we have work around ?

Comment 18

[Katsuhiko Momoi]

This bug came from Yamamoto-san originally but since then they
started using the Mozilla Translator tool. The problem still
exists but I believe that Mozilla Translator tool can bypass this
issue entirely.
Kazu? Do you have an opinion on this?

Comment 19

[Frank Tang]

I believe we can live without this for nsbeta2. Remove [nsbeta2+] and send mail 
to different parties. 

Comment 20

[Kazu Yamamoto]

Now, I use the Mozilla Translator to localize messages.
So I don't need to fix this bug soon.

Comment 21

[Jay Patel [:jay]]

Putting on [nsbeta2-] radar. Not critical to beta2. 

Comment 22

[Frank Tang]

I have put fix for 26938 at http://warp/u/ftang/tmp/fix26938.txt . Notice, we 
have to check in http://warp/u/ftang/tmp/fix21418.txt before taking this fix. 
Otherwise, the property file will be break.

Comment 23

[Frank Tang]

reverse the dependency. The fix 21418 block 26938

Comment 24

[Frank Tang]

nsbeta3 and nsbeta3+

Comment 25

[Frank Tang]

fix and check in

Comment 26

[Teruko Kobayashi]

Verified.