595484 - incorrect threading in a saved search virtual folder combining Inbox & Sent

Status: NEW

(Thunderbird :: Folder and Message Lists, defect)

Description

[al_9x]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.9) Gecko/20100824 Firefox/3.6.9
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.9) Gecko/20100907 Lightning/1.0b2 Thunderbird/3.1.3

I have a saved search virtual folder which shows all messages from Inbox & Sent on an IMAP account.  It's threaded and sorted by received.  There are numerous threading problem.  Many messages that should be threaded, are not.  Many of the ones that are threaded are in a wrong sequence.  For example:

out1
..in1
....out2
......in2
........out3
..........in3

turns into:

out1
..in1
....in2
....out3
......in3
....out2

In some of these wrong sequence cases if I resort the folder, correct order is reestablished, but switching to a different folder and back again messes it up.

Reproducible: Always

Comment 1

[Ludovic Hirlimann [:Usul]]

did you read https://wiki.mozilla.org/MailNews:Message_Threading#Preferences_Controlling_Threading ? Does this help you get better results ?

Comment 2

[al_9x]

(In reply to comment #1)
> did you read
> https://wiki.mozilla.org/MailNews:Message_Threading#Preferences_Controlling_Threading
> ? Does this help you get better results ?

strict_threading=false threads more messages, but the hierarchy/sequence is messed up.

correct_threading=false doesn't seem to have a discernible effect

The specific problem I described, that corrects itself after a re-sort and reappears when the the folder is reopened, remains regardless of the settings

Comment 3

[Ludovic Hirlimann [:Usul]]

Is it better in -safe-mode (http://support.mozillamessaging.com/en-US/kb/Safe+Mode) ?

Comment 4

[al_9x]

(In reply to comment #3)
> Is it better in -safe-mode
> (http://support.mozillamessaging.com/en-US/kb/Safe+Mode) ?

No.

Comment 5

[David :Bienvenu]

I've found one reproducible case of bad threading in cross-folder views that results in a thread getting broken up - if a reply comes before its parent (based on order received), and has no other reference headers, then the reply will become its own thread, and the parent will end up as the parent in the reply's thread.

We could fix this by fixing the threading of the parent - if it looks like it belongs in two threads, then it should be added to the thread it belongs to by references, and the thread that contains its early arriving children should get folded into that thread as well. I'll try to code that up.

Comment 6

[David :Bienvenu]

Attachment: improve xf threading - wip

This improves cross-folder threading in a couple cases - where the parent arrives after the child, and when we have duplicate messages in the search results. There's still an issue when a common ancestor is missing, *and* some of the messages are missing references that should be there, but things are quite a bit better with this patch. I need to write an xpcshell test for at least the first problem.

Comment 7

[Mitra Ardron]

I don't know if its the same problem but I'm seeing some seriously messed up threading, similar to that by al_9x.

In my case the messages are almost certainly not in the right order in the folder but things are threaded not just in a wrong sequence, but with messages they have no relationship to at all.

Comment 8

[Mitra Ardron]

I should say - I've just switched to iMap - and most of these messages were previously downloaded with POP.

Comment 9

[Jens Müller (:tessarakt)]

(In reply to comment #6)
> Created attachment 511757 [details] [diff] [review] [review]
> improve xf threading - wip
> 
> This improves cross-folder threading in a couple cases - where the parent
> arrives after the child, 

Threading where the parent (thread root) "arrives" after the child (in this case: was moved to the respective folder after the child) ... I just had a similar issue, but without any cross-folder stuff involved ...

Comment 10

[David :Bienvenu]

this patch actually works well. I just got stuck trying to write tests for it.

Comment 11

[Mitra Ardron]

David, I'm on the nightlies - 5.0b2pre currently (TB no longer gives more detail than that in the About screen :-(  

Is this patch in there? i.e. something I can test on the inbox that I have that still has problems .

Comment 12

[al_9x]

> In some of these wrong sequence cases if I resort the folder, correct order
> is reestablished, but switching to a different folder and back again messes
> it up.

This needs to be underscored.  A significant % of the threading problems are resolved by resorting the folder (two clicks on the received column header)

but if you leave the folder and come back, threading is back to where it was.

the other thing that I would like to get clarification on, is inter-thread sorting (by date):

Currently it looks like threads are intended to be sorted by the date of the first message, at least that's what the resorting operation seems to do.  However I would argue that thread sorting based on the date of the last message makes more sense, just as with forum threads.  Is there any way to configure inter-thread date sorting by last message?  If not I'll file a bug for that. 

But accepting that inter-thread sorting is supposed to be by first message, I should point out that this also is screwed up in seemingly random ways, and like much of threading, is resolved by resorting the folder, and screwed up again when reopening the folder.

So the first thing to fix, which should be easy (because the more or less correct code is already there), is to guarantee the same threading and sorting when you open the folder as after a fresh re-sort.  I think that will solve the bulk of these threading/sorting problems.

Comment 13

[Mitra Ardron]

To expand on what is said above - the threading problems I see are NOT fixed by clicking on the header of the Received column twice.

Also ... Unlike a_9x I'm seeing messages sorted by date of most recent message (correct)

Comment 14

[al_9x]

(In reply to comment #13)
> Unlike a_9x I'm seeing messages sorted by date of most recent
> message (correct)

Are you sorting by date or by received?

I have enabled the extraction and storage of the real received header:
user_pref("mailnews.customDBHeaders", "received");

What I am seeing is:

When I show the date column (no received) and sort by it, threads are always sorted by date of the last message.

But, when only the received column is shown, it gets weird:

When the folder is reopened (switch away and back to it) the sorting is screwed up (neither by first nor last message)

Re-sorting once (two clicks on received) sorts the threads by first message, re-sorting again, sort by last messages, and every re-sort toggles the by first/last order.

@David :Bienvenu - is this toggling from "by last" to "by first" and back, a feature?  I guess it would be if the folder actually restored the proper order when reopened.

Comment 15

[al_9x]

Bug 671583

Comment 16

[Mitra Ardron]

I'm sorting by date - but I tried two clicks (not a double click) on both Date and Received, neither cleared up the incorrect threading - its not the ordering that is wrong in the bug I reported, its which messages get associated with each other (i.e. when the little spiral column - I'm not sure the correct term - is active),

It sounds to me that we might have different bugs.

Comment 17

[al_9x]

(In reply to comment #16)
> I'm sorting by date

The sorting problem (Bug 671583) is with received.

> It sounds to me that we might have different bugs.

the description contains an example of the kind of bad threading this bug is about

Comment 18

[al_9x]

I am going to create a separate, more focused bug for the threading issues that are addressed by re-sorting, since that's not about threading per se but remembering threading across folder reopens, and is therefore easier to fix.

Comment 19

[al_9x]

Bug 671936 - threading gets worse when a search folder combining IMAP Inbox and Sent is reopened

Comment 20

[David :Bienvenu]

protz, I think this was the patch I mentioned on irc.

Comment 21

[Wayne Mery (:wsmwk)]

It might be nice to have this WIP patch finished ... and then see what is still left to be fixed in https://bugzilla.mozilla.org/buglist.cgi?f10=CP&f1=OP&o3=substring&list_id=6051254&short_desc=thread&f8=short_desc&f0=OP&o9=allwordssubstr&resolution=---&classification=Client%20Software&classification=Components&o2=anywordssubstr&f9=short_desc&j7=OR&f4=CP&query_format=advanced&j1=OR&f3=keywords&f2=short_desc&f11=CP&short_desc_type=allwordssubstr&f5=CP&f6=OP&v2=search%20cross%20xf&f7=OP&product=MailNews%20Core&product=Thunderbird&o8=substring 
(not saying of course that most or even any of them would be affected - just speculating, but none of them have any form of patches yet)

Comment 22

[:aceman]

While I could maybe refresh the patch (if no logic changes are needed) but surely could not do the test. Can it land without it?