Open Bug 345951 Opened 18 years ago Updated 2 years ago

Test Unicode normalization

Categories

(Core :: Internationalization, defect)

defect

Tracking

()

People

(Reporter: smontagu, Unassigned)

References

Details

Attachments

(3 files)

Spun off from bug 210502.

 ------- Comment #26 From Erik van der Poel  2005-04-14 13:03 PDT

Also, Unicode has a normalization test file:

http://www.unicode.org/Public/UNIDATA/NormalizationTest.txt

Has anyone run this test for Mozilla?

 ------- Comment #27 From Jungshik Shin  2005-04-14 18:16 PDT

Not me. Perhaps, we have to write a test program based on the file unless it's
already been written. 

 ------ Comment #30 From Simon Montagu  2005-04-14 22:19 PDT

There is a very short NFD test in intl/unicharutil/tests/UnicharSelfTest.cpp. We
could try importing the Unicode test file into there to do a more comprehensive
test.

 ------- Comment #32 From Erik van der Poel  2005-04-15 07:19 PDT

Simon, the normalization test file is 2 MB. I wouldn't check it into the tree.
Maybe you didn't mean to do that. (But it would be nice to check in a program
that performs the tests.)
Attached patch PatchSplinter Review
If I was smart I would probably parse the text file directly from C++ code, but disk space is cheaper than gray matter, so I am parsing in perl and creating a header file which can then be compiled into the C++ test program.

For the record, here is the output in current trunk:

NormalizationTest: test nsIUnicodeNormalizer. UCD version: 4.1.0
Test Part0: Specific cases
 23 cases passed, 0 failed

Test Part1: Character by character test
 1114111 cases passed, 0 failed

Test Part2: Canonical Order Test
 768 cases passed, 0 failed

Test Part3: PRI #29 Test
 12 cases passed, 143 failed

Test finished
Attachment #230992 - Flags: review?(jshin1987)
Comment on attachment 230992 [details] [diff] [review]
Patch

>Index: intl/unicharutil/tests/NormalizationTest.cpp
>+ * The Original Code is mozilla.org code.
>+ *
>+ * The Initial Developer of the Original Code is
>+ * Netscape Communications Corporation.
>+ * Portions created by the Initial Developer are Copyright (C) 1998
>+ * the Initial Developer. All Rights Reserved.
>+ *
>+ * Contributor(s):
>+ *   Simon Montagu <smontagu@smontagu.org>

nit: 
You're the initial developer, aren't you? And, it'd be 'Copyright (C) 2006'. 
Other files need have the same issue.

>+typedef struct {

another nit:
You can just use 'struct testcaseLine { .... }', can't you?

>+  wchar_t* c1;

Hmmm... wchar_t is assumed to be 16-bit unsigned (PRUnichar). That's the case
on Windows and Linux (with g++ when '-fshort-wchar' is used, which we do), but
I'm not very comfortable with that. 'Alternatives' include :

 1) calculate the maximum length of test strings (in Perl script) and use 'PRUnichar[5][MAXLEN+1]' . 
 2) generate 17,603 * 5 static PRUnichar arrays (line1_c1, line1_c2, ..., line1_c5, ..., line17603_c5) and make 'PRUnichar *c1, ..*c5'  point to  line1_c1, ..., line1_c5 (and so forth)
 3) Leave it as it is (who cares... this is just a test program !) and add a warning that this test only works where
|sizeof(wchar_t) == 2| (and perhaps a check for |sizeof(wchar) == 2| at the beginning
Attachment #230992 - Flags: review?(jshin1987)
Attachment #231300 - Flags: superreview?(jag)
Attachment #231300 - Flags: review?(jshin1987)
Comment on attachment 231300 [details] [diff] [review]
Addressed jshin's comments, and some pre-sr comments from jag

g++ -fshort-wchar works just fine on Mac.
Attachment #231300 - Flags: superreview?(jag) → superreview+
Comment on attachment 231300 [details] [diff] [review]
Addressed jshin's comments, and some pre-sr comments from jag

r=jshin
Attachment #231300 - Flags: review?(jshin1987) → review+
FYI, I tried to build it and run it on my intel Mac with 10.4.8.

I fetched http://www.unicode.org/Public/UNIDATA/NormalizationTest.txt and re-built in the intl/unicharutil/tests directory.

When I executed, I got the following:

$ cd ../../../dist/bin
$ ./run-mozilla.sh ../../intl/unicharutil/tests/NormalizationTest
NormalizationTest: test nsIUnicodeNormalizer. UCD version: 5.0.0
GetService failed
$ echo $?
1
$
$ ./run-mozilla.sh ../../intl/unicharutil/tests/UnicharSelfTest
==============================
Start nsICaseConversion Test 
==============================
Test 1 - GetService():
        1st GetService failed
        2nd GetService failed
==============================
Finish nsICaseConversion Test 
==============================
==============================
Start nsIEntityConverter Test 
==============================
        Failed!! return value != NS_OK
==============================
Start nsISaveAsCharset Test 
==============================
a b ? ? c 
        Failed!! return value != NS_OK
ISO-8859-1 attr_plainTextDefault entityNone
./run-mozilla.sh: line 451:  9571 Bus error               "$prog" ${1+"$@"}
%
Do other tests work? e.g. 
 ./run-mozilla.sh ../../xpcom/tests/TestServMgr
(which should display more information if it fails)
That one fails too. Just says "test 1 failed". I attached the log from running all the xpcom tests.
QA Contact: amyy → i18n
Blocks: 488628

The bug assignee didn't login in Bugzilla in the last 7 months, so the assignee is being reset.

Assignee: smontagu → nobody
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: