Last Comment Bug 8455 - Mail:sort in the local folder is done incorrectly
: Mail:sort in the local folder is done incorrectly
Product: MailNews Core
Classification: Components
Component: Internationalization (show other bugs)
: Trunk
: x86 Windows NT
P3 normal (vote)
: M10
Assigned To: David :Bienvenu
: Katsuhiko Momoi
Depends on:
  Show dependency treegraph
Reported: 1999-06-17 16:08 PDT by marina
Modified: 2008-07-31 01:22 PDT (History)
4 users (show)
See Also:
Crash Signature:
QA Whiteboard:
Iteration: ---
Points: ---

a zip file of a mailbox for sorting test (1.02 KB, application/octet-stream)
1999-06-29 16:46 PDT, nhottanscp
no flags Details
i18n message sort data (1.05 KB, application/octet-stream)
1999-08-16 15:02 PDT, nhottanscp
no flags Details

Description User image marina 1999-06-17 16:08:45 PDT
Steps to reproduce:
get your Messenger window;
click on the Inbox of the local mail folder;
click on the Subject to get it sorted (one click ascending, another click
//note: all high ascii are added at the end
for ex :
Yahoofr  5/8/99  read
ç'est la vie    6/1/99  read
Comment 1 User image nhottanscp 1999-06-21 10:57:59 PDT
Marina, could you zip your inbox and attach?
There is a field "Attachments: Create a new attachment (proposed patch,
testcase, etc.)", you can click and attach a file.
Comment 2 User image nhottanscp 1999-06-28 11:43:59 PDT
Comment 3 User image nhottanscp 1999-06-29 16:46:59 PDT
Created attachment 615 [details]
a zip file of a mailbox for sorting test
Comment 4 User image nhottanscp 1999-06-29 16:54:59 PDT
The problem is because collation key is generated from MIME encoded string. It
needs to be decoded before the key generation.
This doesn't seem to be a problem newly created after the sort was once broken
by rdf.
I checked with Marina but she has not seen it (i18n sort) working before.
Reassigning to
Comment 5 User image David :Bienvenu 1999-06-29 17:01:59 PDT
I'm sorry; I didn't understand the last comment. Are you saying I need to decode
the mime encoded string and then generate a collation key from the decoded
string? I think I just plugged in the patch someone sent me so I'm a little out
of my depth here.
Comment 6 User image nhottanscp 1999-06-29 17:06:59 PDT
Yes, it needs to be decoded. In 4.5, i18n provided a function to MIME decode
also create a collation key. But 5.0, they are separate interfaces.
Comment 7 User image marina 1999-06-29 17:19:59 PDT
I looked into sort today and after clicking sort by Subject my system runs out
of virtual memory and all processes after it taking several minutes to be
Comment 8 User image David :Bienvenu 1999-07-01 11:57:59 PDT
Not going to happen in m8
Comment 9 User image David :Bienvenu 1999-07-26 17:35:59 PDT
Should be fixed when the mailnews Necko branch lands. I'm now doing the decode
Comment 10 User image marina 1999-07-28 13:59:59 PDT
David, is it the right time to verify the fix (because i don't see it working
yet)or wait for Necko/M9 build?
Comment 11 User image David :Bienvenu 1999-07-28 14:02:59 PDT
Yes, wait for the m9/Necko build. We haven't landed necko yet, though it's
supposed to happen today.
Comment 12 User image Katsuhiko Momoi 1999-08-06 19:02:59 PDT
** Checked with 8/6/99 Win32 build **

Now that we decode MIME-encoded headers first before sorting
them using locale-specific APIs, we see dramatic improvement in
sorting results.

I looked at 2 platforms with both IMPAP and POP3 : WinNT-US and
WinNT-Japanese. Here are the results:

1. On NT-US, Latin 1 and ASCII headers are sorted in such
   a way as to put the base character like "a" right before
   an accented "a" rather than placing accented ones at the end.
   If non-Latin characters exist in the same folder, e.g.
   Japanese headers, they are sorted according to Uniocde point

2. On Japanese NT4, the sort is done according to JISX208 mostly
   but Kana characters are done according to pronunciation.
   This is also a desired result.

Locale-sensitive sort was already working but now the input into
the sort routine is also correct as the strings are first
Based on these results, I'm going to mark this fix verified.
Comment 13 User image nhottanscp 1999-08-16 14:38:59 PDT
It is working for subjects but not addresses.
For example, the below address is sorted at the top (seems like not MIME
From: =?iso-8859-1?Q?N=E1oki?= Hotta <>

IQA, please file a separate bug or reopen this, thanks.
Comment 14 User image Katsuhiko Momoi 1999-08-16 14:53:59 PDT
Naoki, I think we should deal with this additional issue when Bug 8405 is
fixed. At present, I only see Japanese characters as "dots" in the Sender field
display of the thread pane. It is very difficult to test the Sender field sort without the
proper display.
I thought this bug was about whether not the MIME-decode itself was being done.
And for that, my testing indicates that it is indeed occurring.
Comment 15 User image nhottanscp 1999-08-16 15:02:59 PDT
Created attachment 1263 [details]
i18n message sort data
Comment 16 User image nhottanscp 1999-08-16 15:04:59 PDT
Using my local build (pulled 8/16) and the attached data, Náoki Hotta is sorted
before Bob Jung.
Comment 17 User image Katsuhiko Momoi 1999-08-16 15:21:59 PDT
I'm a bit confused about this.
In Bug 8405, nhotta mentions that the MIME-decode is done in the Sender
field. But in this bug, MIME-decode is not done.
Is this a matter of timing as to where the sorting key is created
prior to MIME-decoding or post-decoding?
Since Latin 1 data nhotta provided provides justification,
I'll re-open this for the Sender field sort.
Comment 18 User image nhottanscp 1999-08-16 15:36:59 PDT
>Is this a matter of timing as to where the sorting key is created
>prior to MIME-decoding or post-decoding?
MIME decode for display and sort key creation may be done separately. They were
done separately in 4.x.
Comment 19 User image David :Bienvenu 1999-08-16 16:10:59 PDT
I don't believe I can fix this with the current api's that exist. Sorting needs
to be by the pretty name (not the e-mail address). To get the pretty name, I
need to use nsIMsgHeaderParser::ExtractHeaderAddressName, which takes a char *
string, and returns a char *string, after converting to utf-8, I believe. So I
can't decode before calling that routine, and it's too late to decode after. Any
suggestions? I don't believe there's anything I can do here.
Comment 20 User image nhottanscp 1999-08-16 16:21:59 PDT
nsIMsgHeaderParser::ExtractHeaderAddressName uses utf-8 for internal processing
I think it is possible to call the decoder,
nsMimeConverter::DecodeMimePartIIStr(const char *header,
                                     char       *charset,
                                     char **decodedString)
then call nsIMsgHeaderParser::ExtractHeaderAddressName with returned charset and
decoded string.
Comment 21 User image David :Bienvenu 1999-08-16 16:24:59 PDT
If I understand you correctly, this would be a change to libmime, so I'm
reassigning to Rich, on the off-chance he can get to it before sabbatical.
Comment 22 User image nhottanscp 1999-08-16 16:55:59 PDT
>If I understand you correctly
I don't thik so... I said the caller (not libmime) could decode before getting
the pretty name.
Comment 23 User image lchiang 1999-08-16 22:41:59 PDT
Should this be target milestone M9?  I realize Rich is going on sabbatical
Comment 24 User image rhp (gone) 1999-08-17 08:36:59 PDT
From Naoki's comments, it looks like the caller needs to change the way in
which DecodeMimePartIIStr() is called. David, if I'm mistaken, let me know.

- rhp
Comment 25 User image David :Bienvenu 1999-08-17 08:40:59 PDT
Not m9, for sure!
Comment 26 User image David :Bienvenu 1999-08-17 09:41:59 PDT
OK, I have a fix for this - it's quite involved, unfortunately, but I'll check
it in tonight if the tree opens...
Comment 27 User image David :Bienvenu 1999-08-17 21:59:59 PDT
fixed more or less as nhotta suggested, though I had to jump through a lot of
hoops to get the data in the right form.
Comment 28 User image Katsuhiko Momoi 1999-08-26 01:34:59 PDT
MIME-decoding does not seem to be taking place for the final
M9 build (8/24/99) but seems to be working for the latest M10
build. I assume that this did not get into M9, and will verify
this later for M10.
Comment 29 User image David :Bienvenu 1999-08-26 08:24:59 PDT
If the target fix milestone says m10, that means it's fixed in m10 - our QA
group is very picky about that. So it's important to pay attention to the target
fix version when verifying a "resolved fixed" bug. It will save you time.
Comment 30 User image Katsuhiko Momoi 1999-09-17 22:52:59 PDT
** Checked with 9/17/99 Win32 build **

Locale-sensitive sorting is working on Win-NT4 US and WinNT-J
well for Latin 1 and Japanese Subject and Sender fields with the
above build. On Win98 and Win95, locale-sensitive part is not
working yet but we get Unicode sort which is decent for Japanese.

The Sender field sort is working OK now for Latin 1 showing
that the MIME-decoder is engaged before sort keys are generated.
For Japanese names in the sender field, a display bug prevents us
from verifying this feature. This should be looked at when Bug 8405
is fixed.

Note You need to log in before you can comment on or make changes to this bug.