315803 - threading broken (messages orphaned, threads merged)

Status: REOPENED

(Thunderbird :: Folder and Message Lists, defect)

Description

[Mikel Ward]

By default, messages are grouped according to subject, even if they are unrelated.

I find this terribly annoying, so I added
    user_pref("mail.thread_by_subject", false);

to my user.js.

Some messages are still put in the wrong thread, probably because of their subject, even when the In-Reply-To and References headers indicate they belong elsewhere.  This is with sorting by Date, Ascending, Threaded, which is the default except with Threading enabled.

-----
Vedr. Re: Tabs Menu (10/27) (Thread 1, Message 3)
...
Tabs Menu (11/10) (Thread 2, Message 1)
Tabs Menu (09/27) (Thread 1, Message 1)
+ Re: Tabs Menu (10/27) (Thread 1, Message 2)
+ Re: Tabs Menu (11/10) (Thread 2, Message 2)
  + Re: Tabs Menu (11/10) (Thread 2, Message 3)
    + Re: Tabs Menu (11/10) (Thread 2, Message 4)
-----

In this instance, there were two threads with the same subject.  The first was between endbracket.net and mail.ukl.yahoo.com, the other was between endbracket.net and gmail.com.

-----
<[e-mail]> (reply to <[e-mail]>)
...
<[e-mail]>
<[e-mail]>
+ <[e-mail]> (reply to <[e-mail]>)
+ <[e-mail]> (reply to <[e-mail]>)
  + <[e-mail]> (reply to <[e-mail]>)
  + <[e-mail]> (reply to <[e-mail]>)
-----

Notice that the oldest message, <[e-mail]>, the root of the first thread, should appear first (higher up the list when sorting in ascending order), but is instead wrongly made the root of the newer thread.  Everything from there is all just crazy with orphaned messages and messages from both threads merged together.

It should instead look like this:
-----
Tabs Menu (09/27) (Thread 1, Message 1)
+ Re: Tabs Menu (10/27) (Thread 1, Message 2)
  + Vedr. Re: Tabs Menu (10/27) (Thread 1, Message 3)
...
Tabs Menu (11/10) (Thread 2, Message 1)
+ Re: Tabs Menu (11/10) (Thread 2, Message 2)
  + Re: Tabs Menu (11/10) (Thread 2, Message 3)
    + Re: Tabs Menu (11/10) (Thread 2, Message 4)
-----
-----
<[e-mail]>
+ <[e-mail]> (reply to <[e-mail]>)
  + <[e-mail]> (reply to <[e-mail]>)
...
<[e-mail]>
+ <[e-mail]> (reply to <[e-mail]>)
  + <[e-mail]> (reply to <[e-mail]>)
    + <[e-mail]> (reply to <[e-mail]>)
-----

It's pretty important that threading works with
    user_pref("mail.thread_by_subject", false);

enabled.  The default to group only by subject is not sufficient for many users.

Comment 1

[Mikel Ward]

Attachment: screenshot of threads

This is a screenshot of the thread pane.

I am using search for messages with Subject or Sender containing "tabs" to help show only those messages affected in the screenshot.  The thread order is still the same.

The top (orphaned) message and the bottom six are the ones of interest.

Comment 2

[David :Bienvenu]

you realize we don't rethread when you just change that pref, right? You need to delete the .msf file to cause re-threading. If that still causes a problem, can you copy the messages to a local folder, and if opening that folder still shows the problem, can you attach or e-mail me the local folder? thx.

Comment 3

[Mikel Ward]

Yes, I had deleted all the files in ImapMail after changing user.js and before starting Thunderbird.

I'm now trying to reproduce the problem on my home computer, which does not have the threading customization (i.e. thread_by_subject remains TRUE).  I noticed that this mode also has problems with the threads, and I guess the problems might be related.

The behavior with thread_by_subject=TRUE is:
- Thread 1 appears before Thread 2, as it should
- The message in Thread 1 whose title begins with "Vedr. " is not properly threaded, despite it having what looks like a correct In-Reply-To header
- The first message in Thread 2 is orphaned, with the rest of the thread descending from the second message in the thread

It also seems that searching for only messages containing "Tabs" flattens the threads, whereas my other profile with thread_by_subject=FALSE allowed for a threaded view, even when searching.

I'm guessing that the problem in thread 1 is that any prefixes such as "Vedr." are not recognized, despite the threading headers providing the necessary information to deduce that the messages are related.

The message that is orphaned in thread 2 was generated by Apple Mail 2.623, was sent via GMail, and is encoded in US-ASCII, and uses DomainKeys.  My reply to it was generated by Mozilla Thunderbird 1.5 20051025, is encoded in ISO 8859-1, and includes both In-Reply-To and References headers that refer to the same Message-Id.

Comment 4

[Mikel Ward]

Oh dear.  This profile had (in prefs.js, rather than user.js):
    user_pref("mail.strict_threading", true);

but not (in either preference file):
    user_pref("mail.thread_by_subject", false);

How many different threading preferences are there!?  (In addition to these two, there was also once a preference called something like mail.thread_without_re, but I understand this is now unused.)  What should I be using to achieve true threading?

Comment 5

[Mikel Ward]

After establishing the configuration on this profile the same as I originally reported (mail.thread_by_subject=false, mail.strict_threading=true), I got yet a different set of results.  The results are the same whether on the IMAP server or in a new local folder.

The first message in the second thread with the same title is now somehow a child of the first thread!

Vedr. Re: Tabs Menu (Thread 1, Message 3)
...
Tabs Menu (11/10) (Thread 1, Message 1)
+ Re: Tabs Menu (Thread 1, Message 2)
+ Tabs Menu (Thread 2, Message 1)
  + Re: Tabs Menu (Thread 2, Message 2)
    + Re: Tabs Menu (Thread 2, Message 3)
      + Re: Tabs Menu (Thread 2, Message 4)

Could the results be dependent on the message list the IMAP server sends to Thunderbird, or something along the same lines such as the order in which the messages are retrieved?

Comment 6

[Mikel Ward]

Attachment: screenshot of threads after deleting cached folders (.msf files)

This is a screenshot reflecting my most recent comments after deleting the *.msf files under ImapMail and reapplying the sort preferences (Date, Ascending, Threaded).

My first screenshot was also using the same threading preferences and after deleting the *.msf files, yet it's threaded differently this time!

Comment 7

[David :Bienvenu]

A copy of the local folder is what would really be helpful. Screen shots are not so helpful. You can remove the message bodies since threading is only based on headers.

There are two threading prefs:
    "mail.thread_without_re" - puts two messages with the same subject in the same thread, even if the second subject does not begin with "re:"
    mail.strict_threading" - only thread by reference headers, ignore subject.

"mail.thread_by_subject",  is a fig newton of your imagination and has no effect, afaik.

Comment 8

[Mikel Ward]

Thanks for clearing up the preference name.  I had read about thread_by_subject a few places such as http://kb.mozillazine.org/Stop_threading_by_subject (which actually mentions thread_without_re and thread_by_subject).

Your comment 5 in bug 164115 lead me to believe mail.thread_without_re toggled threading by subject, but now the preference name makes sense and clarifies that this is certainly not the preference I am interested in.  I would be interested in working on the UI for the appropriate preference if I can get it working!

The threading is still definitely broken with mail.strict_threading=true.

If anything, I was more concerned about revealing people's addresses, I don't think the message bodies are anything interesting or confidential.  I'll attach it here.

Comment 9

[Mikel Ward]

Attachment: local mail folder containing only the affected threads

Comment 10

[Braden]

In my user.js, I have:

user_pref("mail.strict_threading", true);
user_pref("mail.thread_without_re", false);

And with that, I can't reproduce the problem here. I have no shortage of messages with identical subjects; they all seem to get threaded separately (as appropriate).

I am tempted to go ahead and mark this WORKSFORME; however, there are some things here I'd like to understand.

Mikel, were you setting thread_without_re to false? If you do that in addition to setting strict_threading to true, do things start working as they should?

Based on the description of strict_threading in Comment #7, I would expect setting it to true to moot the value of thread_without_re. However, if changing thread_without_re still has some effect in that case, that may be a bug.

Comment 11

[Wayne Mery (:wsmwk)]

better with 3.0?  (it should be)
  ftp://ftp.mozilla.org/pub/thunderbird/nightly/latest-comm-1.9.1/
  backup your profile before using

Comment 12

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

No response to last comment -> Incomplete. Please reopen if necessary, giving reasons why.

Comment 13

[Mikel Ward]

Wayne, Gary, I no longer use this setup, but I attached my mailbox here a while back, so you should be able to test with that.

Comment 14

[Wayne Mery (:wsmwk)]

Attachment: threads 3.0b4pre 20090722 with strict_threading