Steps to reproduce: get your Messenger window; click on the Inbox of the local mail folder; click on the Subject to get it sorted (one click ascending, another click descending) //note: all high ascii are added at the end for ex : Yahoofr email@example.com 5/8/99 read ç'est la vie firstname.lastname@example.org 6/1/99 read
Marina, could you zip your inbox and attach? There is a field "Attachments: Create a new attachment (proposed patch, testcase, etc.)", you can click and attach a file.
The problem is because collation key is generated from MIME encoded string. It needs to be decoded before the key generation. This doesn't seem to be a problem newly created after the sort was once broken by rdf. I checked with Marina but she has not seen it (i18n sort) working before. Reassigning to email@example.com.
I'm sorry; I didn't understand the last comment. Are you saying I need to decode the mime encoded string and then generate a collation key from the decoded string? I think I just plugged in the patch someone sent me so I'm a little out of my depth here.
Yes, it needs to be decoded. In 4.5, i18n provided a function to MIME decode also create a collation key. But 5.0, they are separate interfaces.
I looked into sort today and after clicking sort by Subject my system runs out of virtual memory and all processes after it taking several minutes to be completed.
Not going to happen in m8
Should be fixed when the mailnews Necko branch lands. I'm now doing the decode first.
David, is it the right time to verify the fix (because i don't see it working yet)or wait for Necko/M9 build?
Yes, wait for the m9/Necko build. We haven't landed necko yet, though it's supposed to happen today.
** Checked with 8/6/99 Win32 build ** Now that we decode MIME-encoded headers first before sorting them using locale-specific APIs, we see dramatic improvement in sorting results. I looked at 2 platforms with both IMPAP and POP3 : WinNT-US and WinNT-Japanese. Here are the results: 1. On NT-US, Latin 1 and ASCII headers are sorted in such a way as to put the base character like "a" right before an accented "a" rather than placing accented ones at the end. If non-Latin characters exist in the same folder, e.g. Japanese headers, they are sorted according to Uniocde point order. 2. On Japanese NT4, the sort is done according to JISX208 mostly but Kana characters are done according to pronunciation. This is also a desired result. Locale-sensitive sort was already working but now the input into the sort routine is also correct as the strings are first MIME-decoded. Based on these results, I'm going to mark this fix verified.
It is working for subjects but not addresses. For example, the below address is sorted at the top (seems like not MIME decoded). From: =?iso-8859-1?Q?N=E1oki?= Hotta <firstname.lastname@example.org> IQA, please file a separate bug or reopen this, thanks.
Naoki, I think we should deal with this additional issue when Bug 8405 is fixed. At present, I only see Japanese characters as "dots" in the Sender field display of the thread pane. It is very difficult to test the Sender field sort without the proper display. I thought this bug was about whether not the MIME-decode itself was being done. And for that, my testing indicates that it is indeed occurring.
Using my local build (pulled 8/16) and the attached data, Náoki Hotta is sorted before Bob Jung.
I'm a bit confused about this. In Bug 8405, nhotta mentions that the MIME-decode is done in the Sender field. But in this bug, MIME-decode is not done. Is this a matter of timing as to where the sorting key is created prior to MIME-decoding or post-decoding? Since Latin 1 data nhotta provided provides justification, I'll re-open this for the Sender field sort.
>Is this a matter of timing as to where the sorting key is created >prior to MIME-decoding or post-decoding? MIME decode for display and sort key creation may be done separately. They were done separately in 4.x.
I don't believe I can fix this with the current api's that exist. Sorting needs to be by the pretty name (not the e-mail address). To get the pretty name, I need to use nsIMsgHeaderParser::ExtractHeaderAddressName, which takes a char * string, and returns a char *string, after converting to utf-8, I believe. So I can't decode before calling that routine, and it's too late to decode after. Any suggestions? I don't believe there's anything I can do here.
nsIMsgHeaderParser::ExtractHeaderAddressName uses utf-8 for internal processing only. I think it is possible to call the decoder, nsMimeConverter::DecodeMimePartIIStr(const char *header, char *charset, char **decodedString) then call nsIMsgHeaderParser::ExtractHeaderAddressName with returned charset and decoded string.
If I understand you correctly, this would be a change to libmime, so I'm reassigning to Rich, on the off-chance he can get to it before sabbatical.
>If I understand you correctly I don't thik so... I said the caller (not libmime) could decode before getting the pretty name.
Should this be target milestone M9? I realize Rich is going on sabbatical soon...
From Naoki's comments, it looks like the caller needs to change the way in which DecodeMimePartIIStr() is called. David, if I'm mistaken, let me know. - rhp
Not m9, for sure!
OK, I have a fix for this - it's quite involved, unfortunately, but I'll check it in tonight if the tree opens...
fixed more or less as nhotta suggested, though I had to jump through a lot of hoops to get the data in the right form.
MIME-decoding does not seem to be taking place for the final M9 build (8/24/99) but seems to be working for the latest M10 build. I assume that this did not get into M9, and will verify this later for M10.
If the target fix milestone says m10, that means it's fixed in m10 - our QA group is very picky about that. So it's important to pay attention to the target fix version when verifying a "resolved fixed" bug. It will save you time.
** Checked with 9/17/99 Win32 build ** Locale-sensitive sorting is working on Win-NT4 US and WinNT-J well for Latin 1 and Japanese Subject and Sender fields with the above build. On Win98 and Win95, locale-sensitive part is not working yet but we get Unicode sort which is decent for Japanese. The Sender field sort is working OK now for Latin 1 showing that the MIME-decoder is engaged before sort keys are generated. For Japanese names in the sender field, a display bug prevents us from verifying this feature. This should be looked at when Bug 8405 is fixed.