Mail:sort in the local folder is done incorrectly

VERIFIED FIXED in M10

Status

MailNews Core
Internationalization
P3
normal
VERIFIED FIXED
18 years ago
9 years ago

People

(Reporter: marina, Assigned: Bienvenu)

Tracking

Trunk
x86
Windows NT

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

(Reporter)

Description

18 years ago
Steps to reproduce:
get your Messenger window;
click on the Inbox of the local mail folder;
click on the Subject to get it sorted (one click ascending, another click
descending)
//note: all high ascii are added at the end
for ex :
Yahoofr        iqatest1@netscape.com  5/8/99  read
ç'est la vie   marina@netscape.com    6/1/99  read

Updated

18 years ago
Status: NEW → ASSIGNED

Comment 1

18 years ago
Marina, could you zip your inbox and attach?
There is a field "Attachments: Create a new attachment (proposed patch,
testcase, etc.)", you can click and attach a file.

Updated

18 years ago
Target Milestone: M8

Comment 2

18 years ago
M8

Comment 3

18 years ago
Created attachment 615 [details]
a zip file of a mailbox for sorting test

Updated

18 years ago
Assignee: nhotta → bienvenu
Status: ASSIGNED → NEW

Comment 4

18 years ago
The problem is because collation key is generated from MIME encoded string. It
needs to be decoded before the key generation.
This doesn't seem to be a problem newly created after the sort was once broken
by rdf.
I checked with Marina but she has not seen it (i18n sort) working before.
Reassigning to bienvenu@netscape.com.
(Assignee)

Comment 5

18 years ago
I'm sorry; I didn't understand the last comment. Are you saying I need to decode
the mime encoded string and then generate a collation key from the decoded
string? I think I just plugged in the patch someone sent me so I'm a little out
of my depth here.

Comment 6

18 years ago
Yes, it needs to be decoded. In 4.5, i18n provided a function to MIME decode
also create a collation key. But 5.0, they are separate interfaces.
(Reporter)

Comment 7

18 years ago
I looked into sort today and after clicking sort by Subject my system runs out
of virtual memory and all processes after it taking several minutes to be
completed.
(Assignee)

Updated

18 years ago
Target Milestone: M8 → M9
(Assignee)

Comment 8

18 years ago
Not going to happen in m8
(Assignee)

Updated

18 years ago
Status: NEW → ASSIGNED
(Assignee)

Updated

18 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 18 years ago
Resolution: --- → FIXED
(Assignee)

Comment 9

18 years ago
Should be fixed when the mailnews Necko branch lands. I'm now doing the decode
first.
(Reporter)

Comment 10

18 years ago
David, is it the right time to verify the fix (because i don't see it working
yet)or wait for Necko/M9 build?
(Assignee)

Comment 11

18 years ago
Yes, wait for the m9/Necko build. We haven't landed necko yet, though it's
supposed to happen today.

Updated

18 years ago
Status: RESOLVED → VERIFIED

Comment 12

18 years ago
** Checked with 8/6/99 Win32 build **

Now that we decode MIME-encoded headers first before sorting
them using locale-specific APIs, we see dramatic improvement in
sorting results.

I looked at 2 platforms with both IMPAP and POP3 : WinNT-US and
WinNT-Japanese. Here are the results:

1. On NT-US, Latin 1 and ASCII headers are sorted in such
   a way as to put the base character like "a" right before
   an accented "a" rather than placing accented ones at the end.
   If non-Latin characters exist in the same folder, e.g.
   Japanese headers, they are sorted according to Uniocde point
   order.

2. On Japanese NT4, the sort is done according to JISX208 mostly
   but Kana characters are done according to pronunciation.
   This is also a desired result.

Locale-sensitive sort was already working but now the input into
the sort routine is also correct as the strings are first
MIME-decoded.
Based on these results, I'm going to mark this fix verified.

Comment 13

18 years ago
It is working for subjects but not addresses.
For example, the below address is sorted at the top (seems like not MIME
decoded).
From: =?iso-8859-1?Q?N=E1oki?= Hotta <naoki2@polyglot.mcom.com>

IQA, please file a separate bug or reopen this, thanks.

Comment 14

18 years ago
Naoki, I think we should deal with this additional issue when Bug 8405 is
fixed. At present, I only see Japanese characters as "dots" in the Sender field
display of the thread pane. It is very difficult to test the Sender field sort without the
proper display.
I thought this bug was about whether not the MIME-decode itself was being done.
And for that, my testing indicates that it is indeed occurring.

Comment 15

18 years ago
Created attachment 1263 [details]
i18n message sort data

Comment 16

18 years ago
Using my local build (pulled 8/16) and the attached data, Náoki Hotta is sorted
before Bob Jung.

Updated

18 years ago
Status: VERIFIED → REOPENED

Updated

18 years ago
Resolution: FIXED → ---

Comment 17

18 years ago
I'm a bit confused about this.
In Bug 8405, nhotta mentions that the MIME-decode is done in the Sender
field. But in this bug, MIME-decode is not done.
Is this a matter of timing as to where the sorting key is created
prior to MIME-decoding or post-decoding?
Since Latin 1 data nhotta provided provides justification,
I'll re-open this for the Sender field sort.

Comment 18

18 years ago
>Is this a matter of timing as to where the sorting key is created
>prior to MIME-decoding or post-decoding?
MIME decode for display and sort key creation may be done separately. They were
done separately in 4.x.
(Assignee)

Comment 19

18 years ago
I don't believe I can fix this with the current api's that exist. Sorting needs
to be by the pretty name (not the e-mail address). To get the pretty name, I
need to use nsIMsgHeaderParser::ExtractHeaderAddressName, which takes a char *
string, and returns a char *string, after converting to utf-8, I believe. So I
can't decode before calling that routine, and it's too late to decode after. Any
suggestions? I don't believe there's anything I can do here.

Comment 20

18 years ago
nsIMsgHeaderParser::ExtractHeaderAddressName uses utf-8 for internal processing
only.
I think it is possible to call the decoder,
nsMimeConverter::DecodeMimePartIIStr(const char *header,
                                     char       *charset,
                                     char **decodedString)
then call nsIMsgHeaderParser::ExtractHeaderAddressName with returned charset and
decoded string.
(Assignee)

Updated

18 years ago
Assignee: bienvenu → rhp
Status: REOPENED → NEW
(Assignee)

Comment 21

18 years ago
If I understand you correctly, this would be a change to libmime, so I'm
reassigning to Rich, on the off-chance he can get to it before sabbatical.

Comment 22

18 years ago
>If I understand you correctly
I don't thik so... I said the caller (not libmime) could decode before getting
the pretty name.

Comment 23

18 years ago
Should this be target milestone M9?  I realize Rich is going on sabbatical
soon...

Updated

18 years ago
Assignee: rhp → bienvenu

Comment 24

18 years ago
From Naoki's comments, it looks like the caller needs to change the way in
which DecodeMimePartIIStr() is called. David, if I'm mistaken, let me know.

- rhp
(Assignee)

Updated

18 years ago
Target Milestone: M9 → M10
(Assignee)

Comment 25

18 years ago
Not m9, for sure!
(Assignee)

Updated

18 years ago
Status: NEW → ASSIGNED
(Assignee)

Comment 26

18 years ago
OK, I have a fix for this - it's quite involved, unfortunately, but I'll check
it in tonight if the tree opens...
(Assignee)

Updated

18 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 18 years ago18 years ago
Resolution: --- → FIXED
(Assignee)

Comment 27

18 years ago
fixed more or less as nhotta suggested, though I had to jump through a lot of
hoops to get the data in the right form.

Comment 28

18 years ago
MIME-decoding does not seem to be taking place for the final
M9 build (8/24/99) but seems to be working for the latest M10
build. I assume that this did not get into M9, and will verify
this later for M10.
(Assignee)

Comment 29

18 years ago
If the target fix milestone says m10, that means it's fixed in m10 - our QA
group is very picky about that. So it's important to pay attention to the target
fix version when verifying a "resolved fixed" bug. It will save you time.

Updated

18 years ago
Status: RESOLVED → VERIFIED

Comment 30

18 years ago
** Checked with 9/17/99 Win32 build **

Locale-sensitive sorting is working on Win-NT4 US and WinNT-J
well for Latin 1 and Japanese Subject and Sender fields with the
above build. On Win98 and Win95, locale-sensitive part is not
working yet but we get Unicode sort which is decent for Japanese.

The Sender field sort is working OK now for Latin 1 showing
that the MIME-decoder is engaged before sort keys are generated.
For Japanese names in the sender field, a display bug prevents us
from verifying this feature. This should be looked at when Bug 8405
is fixed.
Product: MailNews → Core
Product: Core → MailNews Core
You need to log in before you can comment on or make changes to this bug.