Locale-specific sort not working in subject and sender field for Japanese

VERIFIED WONTFIX

Status

P3
normal
VERIFIED WONTFIX
20 years ago
8 years ago

People

(Reporter: ji, Assigned: nhottanscp)

Tracking

({platform-parity})

Trunk
Future
Other
Linux
platform-parity

Firefox Tracking Flags

(Not tracked)

Details

(URL)

Attachments

(4 attachments)

(Reporter)

Description

20 years ago
Build: 1999110908
os: RH6.0

Unzip the file in the above URL, you'll see two mail folders we used
for mail sorting testing.
Put those mail folders under you POP mail directory.

1. bring up mozilla mail in C locale, go to sorttest folder
which we have been using for Latin-1 testing. Click on Subject title bar,
you'll see the subjects are sorted like
...
Hector
HTML header test
Héctor
....

2. Set locale to fr and bring up mozilla mail, go to sorttest folder.
This order is right in C locale. Click on Subject title bar,
you'll see the subjects sorting order is as same as observed in 1.
The correct order in fr locale should be like:
...
Hector
Héctor
HTML header test
....
(Reporter)

Updated

20 years ago
Summary: Locale-specific sort not working for Latin-1 data in subject field → Locale-specific sort not working for Latin-1 data in subject and sender field
(Reporter)

Comment 1

20 years ago
The sender field is not sorted correctly either.
(Reporter)

Updated

20 years ago
Summary: Locale-specific sort not working for Latin-1 data in subject and sender field → Locale-specific sort not working in subject and sender field
(Reporter)

Comment 2

20 years ago
The sorting for the subject field is not working either for Japanese locales.
The Japanese sender's name is not displaying correctly in the thread pane.
Due to this problem, the sender's sorting can't be tested for M11.

There are inconsistency between Japanese locale names. (please see bug 18362)
But in any Japanese locales, the sorting order is not expected.

Steps of reproduce:
After unzip the file shown in the above URL, you'll get sortjpn mail folder.
put the folder under pop mail directory. repeat the operations described in
step 1.
(Assignee)

Updated

20 years ago
Status: NEW → ASSIGNED
(Assignee)

Comment 3

20 years ago
We basically calls C library function strxfrm() with setting a locale
setlocale().
http://lxr.mozilla.org/seamonkey/source/intl/locale/src/unix/nsCollationUnix.cpp
It will help if we can try some other unix machines to see the result of fr_FR.
(Reporter)

Comment 4

20 years ago
Correction:
Upon Naoki's explanation, the sorting order observed in ja_JP.EUC locale on Linux is correct.
So below is the matrix of the results:

Browser    OS           locale      sorting
5.0       Linux         ja_JP.EUC    OK
5.0       Linux         ja           Wrong
5.0       Linux         fr           Wrong
5.0       Linux         fr_FR        Wrong
4.7       Solaris 2.6   ja           OK
4.7       Solaris 2.6   japanese     OK
(Reporter)

Updated

20 years ago
Summary: Locale-specific sort not working in subject and sender field → Locale-specific sort not working in subject and sender field for Latin-1 data
(Reporter)

Comment 5

20 years ago
Since the sorting order for ja_JP.EUC is correct, change the summary accordingly
(Reporter)

Comment 6

20 years ago
The sorting order of Latin-1 data is not correct either on
Solaris 4.7 using fr locale.
I'll file another bug for 4.7.
(Assignee)

Updated

20 years ago
Target Milestone: M12
(Reporter)

Comment 7

20 years ago
Checked it again with locale changed to French at login window.
And bring up mozilla in fr_FR locale. I saw the sorting order like:
...
Héctor
Hector
HTML header test
...

It looks better.  But I have a question left:
Why the sorting order of Japanese data is correct when I login with English
and switch to ja_JP.EUC later. And I didn't get the correct result
when I did sameting to French locale.
By the way, there is no Japanese language option at the login window
even I have Japanese packages installed.

Comment 8

20 years ago
French locale and some other European locales are supported officially by RH 6. The Japanese locale
we are using is not. This accounts for the differences observed with CDE logins and other options
you can set via System utility.
(Assignee)

Comment 9

20 years ago
I put a debug printf in the build. You should see "nsCollation::Initialize
mLocale=xx" (xx is a locale name used for the sorting) in terminal output.
Please check tomorrow's build with "fr" and see what kind of locale is actually
used.
(Reporter)

Comment 10

20 years ago
Below is more info with Naoki's debug printf.
It looks like the program picks up the locale set at the login window.
If you change the locale later after login, no matter which locale you changed
to, the program will pick up C locale, But "ja" locale is an exception. The
program can pick up "ja" locale if you switch to it before the browser is
started. This explains why the sorting order for Latin-1 data is correct when I
set French locale at the login window. And it also explains why the sorting
order is inconstent between "ja" and "ja_JP.EUC" locale. That's because both
of these Japanese locales are not listed on the login window, but "ja" can be
recorgnized later while "ja_JP.EUC" can't.

For Japanese locale, the sorting order under ja_JP.EUC locale is inconsistent.
Sometimes it is sorted by jis code and sometimes it is sorted in different way.

Comment 11

20 years ago
HI, Bob:

This is the locale-specific sorting bug I mentioned last night.
(Assignee)

Updated

20 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 20 years ago
Resolution: --- → FIXED
(Assignee)

Comment 12

20 years ago
I checked in a fix setlocale argument bug (NULL -> "").
Now, it should pick up current locale value from environment.
(Reporter)

Comment 13

20 years ago
I checked today's build (1999111809), the fix is not in.
It's still looking at C locale under ja_JP.EUC
Naoki, which build should this fix be in?
(Assignee)

Comment 14

20 years ago
Yes, the fix is in, it works with fr_FR, so the sorting of French can be tested.
If it's not working with ja_JP.EUC, the bug can be reopened and we can focus on
that problem.
(Reporter)

Updated

20 years ago
Status: RESOLVED → REOPENED
Summary: Locale-specific sort not working in subject and sender field for Latin-1 data → Locale-specific sort not working in subject and sender field for Japanese
(Reporter)

Comment 15

20 years ago
With build 1999111808, the fix works for Latin 1 locales.
For ja_JP.EUC, it's not consistent. I tried about 8 times and only once ja_JP.EUC is picked up.
But even it picked up ja_JP.EUC locale, the sorting order of Japanese subjects is not correct.
"ja" locale can always be picked up before and after the fix checked in,
but the sorting order under "ja" is not correct either.

Updated

20 years ago
Resolution: FIXED → ---

Comment 16

20 years ago
Clearing FIXED resolution due to reopen.
(Assignee)

Updated

20 years ago
Status: REOPENED → ASSIGNED
(Assignee)

Comment 17

20 years ago
I have a fix reviewed by tao.
There is a problem in the mapping functions between POSIX and XP locale (it
doesn't work correctly with any locales which have .xxx like ja_JP.EUC).
(Assignee)

Updated

20 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 20 years ago20 years ago
Resolution: --- → FIXED
(Assignee)

Comment 18

20 years ago
I checked in the fix (to be available in tomorrow's build).
(Reporter)

Updated

20 years ago
Status: RESOLVED → REOPENED
(Reporter)

Comment 19

20 years ago
With 1999112408-M12 build, ja_JP.EUC locale can be recorgnized as the other locales.
But the sorting order under ja_JP.EUC is still not correct.
(Reporter)

Updated

20 years ago
Resolution: FIXED → ---
(Assignee)

Updated

20 years ago
Status: REOPENED → ASSIGNED
(Assignee)

Comment 20

20 years ago
There is another bug in unix locale service. Locale name to charset name
mapping is not working correctly. I am going to investigate.

Updated

20 years ago
Blocks: 20203

Comment 21

20 years ago
*** Bug 18362 has been marked as a duplicate of this bug. ***

Comment 22

20 years ago
*** Bug 18362 has been marked as a duplicate of this bug. ***
(Assignee)

Comment 23

20 years ago
Created attachment 3095 [details]
EUC-JP text file contains the collation key generation test program and the result
(Assignee)

Comment 24

20 years ago
I checked in the fix for locale to charset name mapping problem.
But the result of the Japanese sorting is still not correct. I think the problem
is not in mozilla code but the collation key generation implementation (i.e.
strxfrm).
I attached the result of Japanese collation key generation. The collation keys
in Solaris follows JIS order while Redhat 6.0 generates keys seems to have no
ordering.
Please check tomorrow's Solaris build to check Japanese sort order.
(Assignee)

Updated

20 years ago
Target Milestone: M12 → M13
(Reporter)

Comment 25

20 years ago
Unfortunately Solaris mozilla build doesn't launch, and we don't have
commercial builds for Solaris. Can't use Solaris builds for reference at
this point.
(Assignee)

Updated

19 years ago
Target Milestone: M13 → M15
(Assignee)

Comment 26

19 years ago
Currently, we are waiting for the availability of non Linux Unix builds to
verify the problem is Linux specific.
Once we make sure that, we can do Japanese collation order as JIS code point
which is done for Macintosh and generates a reasonable result.
I am moving the target M15.

Comment 27

19 years ago
Do any of the standard Unix commands (e.g., ls) work on RH 6.0?
If so, we should take a look at the source for those commands to see what they
are doing differently.
(Reporter)

Comment 28

19 years ago
"sort" command in ja locale gives correct sorting order.
(Assignee)

Updated

19 years ago
Assignee: nhotta → tao
Status: ASSIGNED → NEW
Summary: Locale-specific sort not working in subject and sender field for Japanese → [FEATURE] Locale-specific sort not working in subject and sender field for Japanese
(Assignee)

Comment 29

19 years ago
Reassign to tao for the investigation of "sort" command.

Comment 30

19 years ago
Tao, Since you're more familiar with Unix code than nhotta, we thought you
might be able to help nhotta quickly look at the RH 6.0 sort code and help
him figure out why nhotta's implementation of the Unix sorting APIs is not
working.  Talk to nhotta.  Thx.

Comment 31

19 years ago
Created attachment 4184 [details]
gnu sort.c

Updated

19 years ago
Assignee: tao → nhotta

Comment 32

19 years ago
Hi, Naoki:


Here is the source code of sort.c. Feel free to let me know if you need further
help or the original package that containall the files to compile sort.c.
Bounce it back to you.


Thanks,

Tao
(Assignee)

Updated

19 years ago
Status: NEW → ASSIGNED
(Assignee)

Comment 33

19 years ago
Thanks tao for the information.
We still need to test non Linux Unix platform. Then may need to change our
implementation depends on the result.
(Assignee)

Updated

19 years ago
Summary: [FEATURE] Locale-specific sort not working in subject and sender field for Japanese → [FEATURE][PP] Locale-specific sort not working in subject and sender field for Japanese

Updated

19 years ago
Keywords: pp

Updated

19 years ago
Summary: [FEATURE][PP] Locale-specific sort not working in subject and sender field for Japanese → [FEATURE] Locale-specific sort not working in subject and sender field for Japanese

Comment 34

19 years ago
QA contact --> ji
QA Contact: momoi → ji
(Assignee)

Comment 35

19 years ago
I read the code provided by tao. It uses strcoll() to compare strings.
I wrote a simple sort program using strcoll() but the results are not good.
I will attach the code and the result.
(Assignee)

Comment 36

19 years ago
Created attachment 6809 [details]
simple sort program using strcoll()
(Assignee)

Comment 37

19 years ago
Created attachment 6810 [details]
sort result (text in EUC-JP)
(Assignee)

Comment 38

19 years ago
Any chance that this can be tested on other unix builds?
Target Milestone: M15 → M17
(Reporter)

Comment 39

19 years ago
Can't get mozilla build running on Soalris 2.6. Will talk to Tajima at Sun about this.
(Reporter)

Comment 40

19 years ago
Checked both beta1 build and 2000042408 commercial build on Japanese Red Hat 6.1.
None of them has right sort order on the subject column of a mail folder.
(Assignee)

Comment 41

19 years ago
With Japanese Red Hat 6.2, we did not get the correct sort result using the test 
program (03/22/00 13:54). The same program gave us the correct result using AIX 
with Japanese locale. So it's likely the problem is in LC_COLLATE support in 
Linux.
I checked in an option to use charset code point for collation keys 
(intl.collationKeyAsCodePoint, default is "false"), so it can be enabled for a 
localized version.
I want this bug to be reassign to a unix engineer for more investigation. 
Reassign to bobj.
Assignee: nhotta → bobj
Status: ASSIGNED → NEW

Updated

19 years ago
Status: NEW → ASSIGNED
Summary: [FEATURE] Locale-specific sort not working in subject and sender field for Japanese → Locale-specific sort not working in subject and sender field for Japanese

Comment 42

19 years ago
Reassigned to nhotta who has been in contact with redhat folks on this issue.
Assignee: bobj → nhotta
Status: ASSIGNED → NEW
(Assignee)

Comment 43

19 years ago
I contacted Redhat again this week, so far no reply.
Status: NEW → ASSIGNED

Updated

19 years ago
No longer blocks: 20203
(Assignee)

Comment 44

19 years ago
This is not a client by according to our investigation so far.
No plan to change client for this.
The work around is to set a pref "intl.collationKeyAsCodePoint" to true then the
sort will be done by EUC-JP code point for Linux JA locale.
Please reopen if this can be reproduced by other Unix.
Status: ASSIGNED → RESOLVED
Last Resolved: 20 years ago19 years ago
Resolution: --- → WONTFIX
Target Milestone: M17 → Future

Comment 45

19 years ago
Is this a general problem that could be avoided by customizing the pref for Ja
Linux build?
(Assignee)

Comment 46

19 years ago
Yes, it is possible to set that pref to true for JA linux build.

Comment 47

19 years ago
Added "ja" keyword
I tried to add "jartm" keyword, but it is not recognized.
Keywords: ja
(Reporter)

Comment 48

19 years ago
jartm can only be added at bugscape.
(Assignee)

Comment 49

19 years ago
Please file a L10N bug to bugscape if we want to set the pref to Linux JA build.

Comment 50

19 years ago
And setting this pref will not cause regression in other area? I am trying to
determine if we should ask Allan (l10n) to do this for JA RTM Linux release. Thx.
(Assignee)

Comment 51

19 years ago
The code path is slitely different so the testing should be done with this pref
on using Japanese Linux.
(Reporter)

Comment 52

19 years ago
Tested 2000102612 branch build with the pref on my Ja RH6.2.
The sorting for mail is working.
(Reporter)

Comment 53

19 years ago
Bugscape bug 3077 has been filed for the l10n build.

Comment 54

19 years ago
I have requested the keyword, but Asa hasn't replied yet. Asa, can you confirm
that it can be added?
The keywords can be added to Bugzilla.  I'd like to change the keywords from the
format "jaRTM" to "nsRTMja" or "nsrtmja" (to have the same case as the existing
Bugzilla/scape "rtm" keyword).  It would make sense to also change the existing
L10N RTM keywords in Bugscape for consistency.  While it's not directly related
to your needs I'd also like to rename the current rtm keywords in Bugilla and
Bugscape to nsrtm.  This way all of these vendor specific keywords would be
grouped together in the keyword list.
(Reporter)

Comment 56

18 years ago
As a workaround, pref of "intl.collationOption" in the file of 
.../locale/en-US/navigator/navigator.properties can be used in Japanese linux 
build.
Status: RESOLVED → VERIFIED
Product: MailNews → Core
Product: Core → MailNews Core
You need to log in before you can comment on or make changes to this bug.