115071 - IMAP folders no longer in alphabetical order

Status: VERIFIED FIXED

(SeaMonkey :: MailNews: Message Display, defect)

Description

[Mike Pinkerton (not reading bugmail)]

In 6.2, my IMAP folders in the folder pane are in alphabetical order, top to
bottom (with Inbox the exception, always at the top).

In the 12/12/01 build, the folders are in a seemingly random order. A 'w' folder
is at the top, with a 'b' folder a few below it. My inbox is smack in the middle
of the folder list.

Is there some new pref, or am i missing something here? When I go back to 6.2
the folders are in the correct order again, so it doesn't seem to be making any
changes on the server side...

Comment 1

[laurel]

I'm seeing this, too, with my IMAP folders using dec13 commercial trunk, mac OS X.

Comment 2

[scottputterman]

reassigning to naving and cc'ing naoki.  I know that some folder sorting code
went in.  Perhaps it doesn't work on OSX

Comment 3

[Navin Gupta]

need mac OS X build environment. 

Comment 4

[nhottanscp]

A different collator is used for MacOS X. Let me take a look at this.

Comment 5

[nhottanscp]

A collation key is a binary data, so a key might contain zeros.
Currently, nsXULSortService and nsXULOutlinerBuilder, assume that a key is null
terminated unicode string (see bug 114070).

I dumped the generated keys of MacOSX collator and some of them actually
contains 0x0000. I think the keys are truncated that causes the incorrect sort
order.

Comment 6

[nhottanscp]

Bug 115926 was filed for collation key null termination issue.

Comment 7

[scottputterman]

*** Bug 115537 has been marked as a duplicate of this bug. ***

Comment 8

[Asa Dotzler [:asa]]

related to bug 112301?

Comment 9

[Chris P. Ross]

I don't think so.  bug 112301 is principally that the sort orders are different
for different parts of the application for the same folders.  And it's OS
inspecific.  This bug is the wierdo sort order for the display in the main
MailNews folder window, and is as far as I know MacOS X specific...

Comment 10

[Frank Tang]

>MacOS X specific...
I check in the new MacOS collation code around that time. and It WILL generat 00
in the collation key (collation key should be opaque 8 bits array, not string by
null termination) . and we learned there are code ASSUME the collation key is
null terminate string. 

Comment 11

[nhottanscp]

0x00 is okay but 0x0000 can also be a part of a collation key which does not
work where the key is assumed as a null terminated PRUnichar*. The issue is
taken care by bug 115926.

Comment 12

[Chris P. Ross]

Umm, does this bug really depend on bug #116328 ?  Now that 115926 is fixed,
I would think we could get this one wrapped up too.  Does anything else need
to happen, or should the fix for #115926 just wrap this one?
It doesn't appear clear to me that #116328 should block this.  It sounds
more like a "something should work differently", not "this is broken, and
making other things broken".

Comment 13

[nhottanscp]

bug 115926 was needed because the current problem is caused by a key not treated
as binary. I agree bug 116328 is needed and it has to support a binary key.

Comment 14

[Chris P. Ross]

We require collalation-based sorting just to get this incorrect behaviour
(which used to be correct) on MacOS X fixed?  Ick...  Okay, I've started
monitoring that bug as well.  Oh, to have my folders display properly again...

Comment 15

[Chris P. Ross]

What?!?  C'mon.  This was an introduced bug (mid December) that makes the
MailNews system very annoying to use for us MacOS X people.  This is a bug
that hurts usability.  Why did it get pushed to 0.9.9?!?  This should definately
be a candidate for getting fixed before 0.9.8 is cut...  Please.  Even if
it doesn't get fixed in the "right" way immediately, I'd really hate to
see another "release" with this bug in it.  It's a pain in the ass to use
when it's fouled up in this way...

Comment 16

[nhottanscp]

So "Inbox" is not sorted at the top. I found a problem which at least affects
the sorting order of special folders. I think that can be fixed by changing the
collation option.
http://developer.apple.com/techpubs/macosx/Carbon/text/UnicodeUtilities/Unicode_Utilities_Ref/Enumerations/UCCollateOptions.html

Comment 17

[nhottanscp]

Attachment: Add flags to specify that the numbers not be affected by locale, this is needed because specifal folder sorting is done by prepending numbers.

Comment 18

[nhottanscp]

I dumped collation keys with and without the options. 
Without the options, the keys are starting as 0x0000 which immediatly causes the
truncation problem. While the problem is delayed with setting the digit override
option.
So the patch cannot really solve the problem, although we might want to do that
for 0.9.8 branch only.

With digit override option (i.e. with the patch):
string=0INBOX strlen=6 keylen=140 actuallen=25
002000000030079d003007fd003006e300300815003008d90000000100000020000000200000002000000020000000200000000000000008000000080000000800000008000000080000000000010030000000490000004e000000420000004f00000058
string=4Sent strlen=5 keylen=120 actuallen=21
002000040030086d0030072f003007fd0030088900000001000000200000002000000020000000200000000000000008000000020000000200000002000000000000003400000053000000650000006e00000074
string=5Trash strlen=6 keylen=140 actuallen=25
002000050030088900300845003006cf0030086d0030078300000001000000200000002000000020000000200000002000000000000000080000000200000002000000020000000200000000000000350000005400000072000000610000007300000068

No digit override option:
string=0INBOX strlen=6 keylen=140 actuallen=27
000006c50030079d003007fd003006e300300815003008d900000001000000200000002000000020000000200000002000000020000000000000000200000008000000080000000800000008000000080000000000000030000000490000004e000000420000004f00000058
string=4Sent strlen=5 keylen=120 actuallen=23
000006c90030086d0030072f003007fd00300889000000010000002000000020000000200000002000000020000000000000000200000008000000020000000200000002000000000000003400000053000000650000006e00000074
string=5Trash strlen=6 keylen=140 actuallen=27
000006ca0030088900300845003006cf0030086d00300783000000010000002000000020000000200000002000000020000000200000000000000002000000080000000200000002000000020000000200000000000000350000005400000072000000610000007300000068

Comment 19

[nhottanscp]

nsbeta1+ per i18n triage

Comment 20

[Simon Fraser [no longer active]]

What's the status on this? This is a dogfood bug for me. Why is it 'helpwanted'?

Comment 21

[Navin Gupta]

Now that we have a patch for 116328, I believe if we treat keys as
PRUint8*, it should work, right? nhotta. 

Also we will have to createNode for sort as RDFBlob.

I will work on a patch. 

 

Comment 22

[nhottanscp]

Naving, that's right, the current problem is because the key is truncated.
I reassign to you.

Comment 23

[Navin Gupta]

I have a patch and it works but I'm waiting for some issues in bug 116329 to be
resolved. 

Comment 24

[Navin Gupta]

Attachment: proposed fix

ducarroz/sfraser, If you could try out this patch with fixes for 116329 and
 122314, I'd appreciate, thx.

Comment 25

[Simon Fraser [no longer active]]

Comments after a quick glance at the patch:

+nsMsgFolder::CreateCollationKey(nsAutoString &aSource,  PRUint8 **aKey, PRUint32 
*aLength)

Don't use nsAutoString as a param. Use nsString&, const if possible.

Comment 26

[Navin Gupta]

Attachment: proposed fix

The fix is basically to create sort nodes/arcs as nsIRDFBlob and then 
sorting in folder-pane, menus, offline-folder selector works fine. 
This is because XULOutlinerBuilder and XULSortService QI for RDFBlob
for sorting based on collation keys. 

thanks sfraser for testing fix on mac os x 

nhotta, can you review.

Comment 27

[nhottanscp]

Looks fine for the collation part, please ask someone else to review the RDF
related code (nsMsgFolderDataSource, nsMsgRDFUtils).

small things,
+    folder->GetSortKey(&sortKey, &sortKeyLength);
+    createBlobNode(sortKey, sortKeyLength, target, getRDFService());
+    PR_FREEIF(sortKey);
please check the error code of the GetSortKey.

+  rv = kCollationKeyGenerator->CreateRawSortKey(kCollationCaseInSensitive,
aSource, *aKey, aLength);
+  NS_ENSURE_SUCCESS(rv, rv);
   return rv;
NS_ENSURE_SUCCESS is not necessary since it returns in the next line.

Comment 28

[Scott MacGregor]

Comment on attachment 67347 [details] [diff] [review]
proposed fix

I didn't see a check on aLength in CreateCollationKey
to make sure it is greater than 0. 

Instead of defining: 
[ptr] native octet_ptr(PRUint8);

can we instead use an array of short ints? Or is a unsigned short a 16 bit
integer? 

something like:
[array, size_is(length
)] out	short key

Comment 29

[Scott MacGregor]

Comment on attachment 67347 [details] [diff] [review]
proposed fix

naving & i talked about some of my questions during lunch. 

sr=mscott pending the check to make sure the length is greater than 0

Comment 30

[Navin Gupta]

fixed

Comment 31

[Mike Pinkerton (not reading bugmail)]

yay!!!!!!!!

Comment 32

[J Luh]

Any chance this can make it into Mozilla 0.9.8?

Comment 33

[Steve Dagley]

*** Bug 122885 has been marked as a duplicate of this bug. ***

Comment 34

[Not interested in Mozilla any more]

*** Bug 125001 has been marked as a duplicate of this bug. ***

Comment 35

[Karen Huang]

Verified on Mac OSX 05-23-05-1.0.0
Verified on Linux 05-20-09-1.0.0
Verified on Windows 05-22-08-1.0.0

Verify that IMAP folders are in alphabetical order.

Comment 36

[Waheed Islam]

*** Bug 119218 has been marked as a duplicate of this bug. ***

Comment 37

[math_modeler]

Using Mozilla 1.4 with Windows 98, I'm finding that some of my IMAP folders are
not listed in alphabetical order as expected (see hierarchy below).

Inbox
--> Drafts
--> sent-mail
--> --> other folders listed alphabetically
--> Trash
--> in-mail
--> --> other folders listed alphabetically
--> Templates
--> Trash2

However, the folders are displayed properly when I configure Netscape 4.79 to
read from the same account.