18338 - Locale-specific sort not working in subject and sender field for Japanese

Status: VERIFIED WONTFIX

(MailNews Core :: Internationalization, defect, P3)

Description

[ji]

Build: 1999110908
os: RH6.0

Unzip the file in the above URL, you'll see two mail folders we used
for mail sorting testing.
Put those mail folders under you POP mail directory.

1. bring up mozilla mail in C locale, go to sorttest folder
which we have been using for Latin-1 testing. Click on Subject title bar,
you'll see the subjects are sorted like
...
Hector
HTML header test
Héctor
....

2. Set locale to fr and bring up mozilla mail, go to sorttest folder.
This order is right in C locale. Click on Subject title bar,
you'll see the subjects sorting order is as same as observed in 1.
The correct order in fr locale should be like:
...
Hector
Héctor
HTML header test
....

Comment 1

[ji]

The sender field is not sorted correctly either.

Comment 2

[ji]

The sorting for the subject field is not working either for Japanese locales.
The Japanese sender's name is not displaying correctly in the thread pane.
Due to this problem, the sender's sorting can't be tested for M11.

There are inconsistency between Japanese locale names. (please see bug 18362)
But in any Japanese locales, the sorting order is not expected.

Steps of reproduce:
After unzip the file shown in the above URL, you'll get sortjpn mail folder.
put the folder under pop mail directory. repeat the operations described in
step 1.

Comment 3

[nhottanscp]

We basically calls C library function strxfrm() with setting a locale
setlocale().
http://lxr.mozilla.org/seamonkey/source/intl/locale/src/unix/nsCollationUnix.cpp
It will help if we can try some other unix machines to see the result of fr_FR.

Comment 4

[ji]

Correction:
Upon Naoki's explanation, the sorting order observed in ja_JP.EUC locale on Linux is correct.
So below is the matrix of the results:

Browser    OS           locale      sorting
5.0       Linux         ja_JP.EUC    OK
5.0       Linux         ja           Wrong
5.0       Linux         fr           Wrong
5.0       Linux         fr_FR        Wrong
4.7       Solaris 2.6   ja           OK
4.7       Solaris 2.6   japanese     OK

Comment 5

[ji]

Since the sorting order for ja_JP.EUC is correct, change the summary accordingly

Comment 6

[ji]

The sorting order of Latin-1 data is not correct either on
Solaris 4.7 using fr locale.
I'll file another bug for 4.7.

Comment 7

[ji]

Checked it again with locale changed to French at login window.
And bring up mozilla in fr_FR locale. I saw the sorting order like:
...
Héctor
Hector
HTML header test
...

It looks better.  But I have a question left:
Why the sorting order of Japanese data is correct when I login with English
and switch to ja_JP.EUC later. And I didn't get the correct result
when I did sameting to French locale.
By the way, there is no Japanese language option at the login window
even I have Japanese packages installed.

Comment 8

[Katsuhiko Momoi]

French locale and some other European locales are supported officially by RH 6. The Japanese locale
we are using is not. This accounts for the differences observed with CDE logins and other options
you can set via System utility.

Comment 9

[nhottanscp]

I put a debug printf in the build. You should see "nsCollation::Initialize
mLocale=xx" (xx is a locale name used for the sorting) in terminal output.
Please check tomorrow's build with "fr" and see what kind of locale is actually
used.

Comment 10

[ji]

Below is more info with Naoki's debug printf.
It looks like the program picks up the locale set at the login window.
If you change the locale later after login, no matter which locale you changed
to, the program will pick up C locale, But "ja" locale is an exception. The
program can pick up "ja" locale if you switch to it before the browser is
started. This explains why the sorting order for Latin-1 data is correct when I
set French locale at the login window. And it also explains why the sorting
order is inconstent between "ja" and "ja_JP.EUC" locale. That's because both
of these Japanese locales are not listed on the login window, but "ja" can be
recorgnized later while "ja_JP.EUC" can't.

For Japanese locale, the sorting order under ja_JP.EUC locale is inconsistent.
Sometimes it is sorted by jis code and sometimes it is sorted in different way.

Comment 11

[tao]

HI, Bob:

This is the locale-specific sorting bug I mentioned last night.

Comment 12

[nhottanscp]

I checked in a fix setlocale argument bug (NULL -> "").
Now, it should pick up current locale value from environment.

Comment 13

[ji]

I checked today's build (1999111809), the fix is not in.
It's still looking at C locale under ja_JP.EUC
Naoki, which build should this fix be in?

Comment 14

[nhottanscp]

Yes, the fix is in, it works with fr_FR, so the sorting of French can be tested.
If it's not working with ja_JP.EUC, the bug can be reopened and we can focus on
that problem.

Comment 15

[ji]

With build 1999111808, the fix works for Latin 1 locales.
For ja_JP.EUC, it's not consistent. I tried about 8 times and only once ja_JP.EUC is picked up.
But even it picked up ja_JP.EUC locale, the sorting order of Japanese subjects is not correct.
"ja" locale can always be picked up before and after the fix checked in,
but the sorting order under "ja" is not correct either.

Comment 16

[leger]

Clearing FIXED resolution due to reopen.

Comment 17

[nhottanscp]

I have a fix reviewed by tao.
There is a problem in the mapping functions between POSIX and XP locale (it
doesn't work correctly with any locales which have .xxx like ja_JP.EUC).

Comment 18

[nhottanscp]

I checked in the fix (to be available in tomorrow's build).

Comment 19

[ji]

With 1999112408-M12 build, ja_JP.EUC locale can be recorgnized as the other locales.
But the sorting order under ja_JP.EUC is still not correct.

Comment 20

[nhottanscp]

There is another bug in unix locale service. Locale name to charset name
mapping is not working correctly. I am going to investigate.

Comment 21

[tao]

*** Bug 18362 has been marked as a duplicate of this bug. ***

Comment 22

[tao]

*** Bug 18362 has been marked as a duplicate of this bug. ***

Comment 23

[nhottanscp]

Attachment: EUC-JP text file contains the collation key generation test program and the result

Comment 24

[nhottanscp]

I checked in the fix for locale to charset name mapping problem.
But the result of the Japanese sorting is still not correct. I think the problem
is not in mozilla code but the collation key generation implementation (i.e.
strxfrm).
I attached the result of Japanese collation key generation. The collation keys
in Solaris follows JIS order while Redhat 6.0 generates keys seems to have no
ordering.
Please check tomorrow's Solaris build to check Japanese sort order.

Comment 25

[ji]

Unfortunately Solaris mozilla build doesn't launch, and we don't have
commercial builds for Solaris. Can't use Solaris builds for reference at
this point.

Comment 26

[nhottanscp]

Currently, we are waiting for the availability of non Linux Unix builds to
verify the problem is Linux specific.
Once we make sure that, we can do Japanese collation order as JIS code point
which is done for Macintosh and generates a reasonable result.
I am moving the target M15.

Comment 27

[bobj]

Do any of the standard Unix commands (e.g., ls) work on RH 6.0?
If so, we should take a look at the source for those commands to see what they
are doing differently.

Comment 28

[ji]

"sort" command in ja locale gives correct sorting order.

Comment 29

[nhottanscp]

Reassign to tao for the investigation of "sort" command.

Comment 30

[bobj]

Tao, Since you're more familiar with Unix code than nhotta, we thought you
might be able to help nhotta quickly look at the RH 6.0 sort code and help
him figure out why nhotta's implementation of the Unix sorting APIs is not
working.  Talk to nhotta.  Thx.

Comment 31

[tao]

Attachment: gnu sort.c

Comment 32

[tao]

Hi, Naoki:


Here is the source code of sort.c. Feel free to let me know if you need further
help or the original package that containall the files to compile sort.c.
Bounce it back to you.


Thanks,

Tao

Comment 33

[nhottanscp]

Thanks tao for the information.
We still need to test non Linux Unix platform. Then may need to change our
implementation depends on the result.

Comment 34

[Katsuhiko Momoi]

QA contact --> ji

Comment 35

[nhottanscp]

I read the code provided by tao. It uses strcoll() to compare strings.
I wrote a simple sort program using strcoll() but the results are not good.
I will attach the code and the result.

Comment 36

[nhottanscp]

Attachment: simple sort program using strcoll()

Comment 37

[nhottanscp]

Attachment: sort result (text in EUC-JP)

Comment 38

[nhottanscp]

Any chance that this can be tested on other unix builds?

Comment 39

[ji]

Can't get mozilla build running on Soalris 2.6. Will talk to Tajima at Sun about this.

Comment 40

[ji]

Checked both beta1 build and 2000042408 commercial build on Japanese Red Hat 6.1.
None of them has right sort order on the subject column of a mail folder.

Comment 41

[nhottanscp]

With Japanese Red Hat 6.2, we did not get the correct sort result using the test 
program (03/22/00 13:54). The same program gave us the correct result using AIX 
with Japanese locale. So it's likely the problem is in LC_COLLATE support in 
Linux.
I checked in an option to use charset code point for collation keys 
(intl.collationKeyAsCodePoint, default is "false"), so it can be enabled for a 
localized version.
I want this bug to be reassign to a unix engineer for more investigation. 
Reassign to bobj.

Comment 42

[bobj]

Reassigned to nhotta who has been in contact with redhat folks on this issue.

Comment 43

[nhottanscp]

I contacted Redhat again this week, so far no reply.

Comment 44

[nhottanscp]

This is not a client by according to our investigation so far.
No plan to change client for this.
The work around is to set a pref "intl.collationKeyAsCodePoint" to true then the
sort will be done by EUC-JP code point for Linux JA locale.
Please reopen if this can be reproduced by other Unix.

Comment 45

[tao]

Is this a general problem that could be avoided by customizing the pref for Ja
Linux build?

Comment 46

[nhottanscp]

Yes, it is possible to set that pref to true for JA linux build.

Comment 47

[bobj]

Added "ja" keyword
I tried to add "jartm" keyword, but it is not recognized.

Comment 48

[ji]

jartm can only be added at bugscape.

Comment 49

[nhottanscp]

Please file a L10N bug to bugscape if we want to set the pref to Linux JA build.

Comment 50

[tao]

And setting this pref will not cause regression in other area? I am trying to
determine if we should ask Allan (l10n) to do this for JA RTM Linux release. Thx.

Comment 51

[nhottanscp]

The code path is slitely different so the testing should be done with this pref
on using Japanese Linux.

Comment 52

[ji]

Tested 2000102612 branch build with the pref on my Ja RH6.2.
The sorting for mail is working.

Comment 53

[ji]

Bugscape bug 3077 has been filed for the l10n build.

Comment 54

[msanz]

I have requested the keyword, but Asa hasn't replied yet. Asa, can you confirm
that it can be added?

Comment 55

[Asa Dotzler [:asa]]

The keywords can be added to Bugzilla.  I'd like to change the keywords from the
format "jaRTM" to "nsRTMja" or "nsrtmja" (to have the same case as the existing
Bugzilla/scape "rtm" keyword).  It would make sense to also change the existing
L10N RTM keywords in Bugscape for consistency.  While it's not directly related
to your needs I'd also like to rename the current rtm keywords in Bugilla and
Bugscape to nsrtm.  This way all of these vendor specific keywords would be
grouped together in the keyword list.

Comment 56

[ji]

As a workaround, pref of "intl.collationOption" in the file of 
.../locale/en-US/navigator/navigator.properties can be used in Japanese linux 
build.