8455 - Mail:sort in the local folder is done incorrectly

Status: VERIFIED FIXED

(MailNews Core :: Internationalization, defect, P3)

Description

[marina]

Steps to reproduce:
get your Messenger window;
click on the Inbox of the local mail folder;
click on the Subject to get it sorted (one click ascending, another click
descending)
//note: all high ascii are added at the end
for ex :
Yahoofr        [e-mail]  5/8/99  read
ç'est la vie   [e-mail]    6/1/99  read

Comment 1

[nhottanscp]

Marina, could you zip your inbox and attach?
There is a field "Attachments: Create a new attachment (proposed patch,
testcase, etc.)", you can click and attach a file.

Comment 2

[nhottanscp]

M8

Comment 3

[nhottanscp]

Attachment: a zip file of a mailbox for sorting test

Comment 4

[nhottanscp]

The problem is because collation key is generated from MIME encoded string. It
needs to be decoded before the key generation.
This doesn't seem to be a problem newly created after the sort was once broken
by rdf.
I checked with Marina but she has not seen it (i18n sort) working before.
Reassigning to [e-mail].

Comment 5

[David :Bienvenu]

I'm sorry; I didn't understand the last comment. Are you saying I need to decode
the mime encoded string and then generate a collation key from the decoded
string? I think I just plugged in the patch someone sent me so I'm a little out
of my depth here.

Comment 6

[nhottanscp]

Yes, it needs to be decoded. In 4.5, i18n provided a function to MIME decode
also create a collation key. But 5.0, they are separate interfaces.

Comment 7

[marina]

I looked into sort today and after clicking sort by Subject my system runs out
of virtual memory and all processes after it taking several minutes to be
completed.

Comment 8

[David :Bienvenu]

Not going to happen in m8

Comment 9

[David :Bienvenu]

Should be fixed when the mailnews Necko branch lands. I'm now doing the decode
first.

Comment 10

[marina]

David, is it the right time to verify the fix (because i don't see it working
yet)or wait for Necko/M9 build?

Comment 11

[David :Bienvenu]

Yes, wait for the m9/Necko build. We haven't landed necko yet, though it's
supposed to happen today.

Comment 12

[Katsuhiko Momoi]

** Checked with 8/6/99 Win32 build **

Now that we decode MIME-encoded headers first before sorting
them using locale-specific APIs, we see dramatic improvement in
sorting results.

I looked at 2 platforms with both IMPAP and POP3 : WinNT-US and
WinNT-Japanese. Here are the results:

1. On NT-US, Latin 1 and ASCII headers are sorted in such
   a way as to put the base character like "a" right before
   an accented "a" rather than placing accented ones at the end.
   If non-Latin characters exist in the same folder, e.g.
   Japanese headers, they are sorted according to Uniocde point
   order.

2. On Japanese NT4, the sort is done according to JISX208 mostly
   but Kana characters are done according to pronunciation.
   This is also a desired result.

Locale-sensitive sort was already working but now the input into
the sort routine is also correct as the strings are first
MIME-decoded.
Based on these results, I'm going to mark this fix verified.

Comment 13

[nhottanscp]

It is working for subjects but not addresses.
For example, the below address is sorted at the top (seems like not MIME
decoded).
From: =?iso-8859-1?Q?N=E1oki?= Hotta <[e-mail]>

IQA, please file a separate bug or reopen this, thanks.

Comment 14

[Katsuhiko Momoi]

Naoki, I think we should deal with this additional issue when Bug 8405 is
fixed. At present, I only see Japanese characters as "dots" in the Sender field
display of the thread pane. It is very difficult to test the Sender field sort without the
proper display.
I thought this bug was about whether not the MIME-decode itself was being done.
And for that, my testing indicates that it is indeed occurring.

Comment 15

[nhottanscp]

Attachment: i18n message sort data

Comment 16

[nhottanscp]

Using my local build (pulled 8/16) and the attached data, Náoki Hotta is sorted
before Bob Jung.

Comment 17

[Katsuhiko Momoi]

I'm a bit confused about this.
In Bug 8405, nhotta mentions that the MIME-decode is done in the Sender
field. But in this bug, MIME-decode is not done.
Is this a matter of timing as to where the sorting key is created
prior to MIME-decoding or post-decoding?
Since Latin 1 data nhotta provided provides justification,
I'll re-open this for the Sender field sort.

Comment 18

[nhottanscp]

>Is this a matter of timing as to where the sorting key is created
>prior to MIME-decoding or post-decoding?
MIME decode for display and sort key creation may be done separately. They were
done separately in 4.x.

Comment 19

[David :Bienvenu]

I don't believe I can fix this with the current api's that exist. Sorting needs
to be by the pretty name (not the e-mail address). To get the pretty name, I
need to use nsIMsgHeaderParser::ExtractHeaderAddressName, which takes a char *
string, and returns a char *string, after converting to utf-8, I believe. So I
can't decode before calling that routine, and it's too late to decode after. Any
suggestions? I don't believe there's anything I can do here.

Comment 20

[nhottanscp]

nsIMsgHeaderParser::ExtractHeaderAddressName uses utf-8 for internal processing
only.
I think it is possible to call the decoder,
nsMimeConverter::DecodeMimePartIIStr(const char *header,
                                     char       *charset,
                                     char **decodedString)
then call nsIMsgHeaderParser::ExtractHeaderAddressName with returned charset and
decoded string.

Comment 21

[David :Bienvenu]

If I understand you correctly, this would be a change to libmime, so I'm
reassigning to Rich, on the off-chance he can get to it before sabbatical.

Comment 22

[nhottanscp]

>If I understand you correctly
I don't thik so... I said the caller (not libmime) could decode before getting
the pretty name.

Comment 23

[lchiang]

Should this be target milestone M9?  I realize Rich is going on sabbatical
soon...

Comment 24

[rhp (gone)]

From Naoki's comments, it looks like the caller needs to change the way in
which DecodeMimePartIIStr() is called. David, if I'm mistaken, let me know.

- rhp

Comment 25

[David :Bienvenu]

Not m9, for sure!

Comment 26

[David :Bienvenu]

OK, I have a fix for this - it's quite involved, unfortunately, but I'll check
it in tonight if the tree opens...

Comment 27

[David :Bienvenu]

fixed more or less as nhotta suggested, though I had to jump through a lot of
hoops to get the data in the right form.

Comment 28

[Katsuhiko Momoi]

MIME-decoding does not seem to be taking place for the final
M9 build (8/24/99) but seems to be working for the latest M10
build. I assume that this did not get into M9, and will verify
this later for M10.

Comment 29

[David :Bienvenu]

If the target fix milestone says m10, that means it's fixed in m10 - our QA
group is very picky about that. So it's important to pay attention to the target
fix version when verifying a "resolved fixed" bug. It will save you time.

Comment 30

[Katsuhiko Momoi]

** Checked with 9/17/99 Win32 build **

Locale-sensitive sorting is working on Win-NT4 US and WinNT-J
well for Latin 1 and Japanese Subject and Sender fields with the
above build. On Win98 and Win95, locale-sensitive part is not
working yet but we get Unicode sort which is decent for Japanese.

The Sender field sort is working OK now for Latin 1 showing
that the MIME-decoder is engaged before sort keys are generated.
For Japanese names in the sender field, a display bug prevents us
from verifying this feature. This should be looked at when Bug 8405
is fixed.