473439 - View->sort by->threaded not threading by subject, only by reference

Status: VERIFIED DUPLICATE of bug 449821

(MailNews Core :: Database, defect)

Description

[Tony Mechelynck [:tonymec]]

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b3pre) Gecko/20090113 SeaMonkey/2.0a3pre - Build ID: 20090113000455

Starting today, "compacting" the folders removes all threading of BMO bugmail. None of the following has any effect:
- Toggle mail.strict_threading then "File => Compact folders"
- Toggle mail.correct_threading then "File => Compact folders"
- Toggle mail.thread_without_re then "File => Compact folders"
- Any of the above, using "(right-click folder) => Properties => Rebuild summary file" instead of compacting
- "View => Sort by => Unthreaded" then toggle "View => Sort by => Ascending/Descending" then "View => Sort by => Threaded".

This bug didn't happen in yesterday's nightly: bugmail messages used to thread together if either they had the same subject (both bug number and Summary), or the first one was the bug creation and they had the same bug number (regardless of Summary changes).

See also:
- Thunderbird bug 449821
- RFE bug 164115 which seems to request the opposite of what I want (but maybe I misunderstood).

Comment 1

[Ashley Bischoff (blog at handcoding.com)]

Confirming with: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b3pre) Gecko/20090113 Shredder/3.0b2pre.

OS -> All

Comment 2

[Tony Mechelynck [:tonymec]]

Workaround:
1. View => Sort by => Subject
2. View => Sort by => Grouped by Sort (set)
     (threading reappears)
3. View => Sort by => Grouped by Sort (reset)
     (threading remains)
4. View => Sort by => (whatever) (in my case, "Order Received")
5. Wait long enough for the threads to get sorted (it's slow).

In reply to comment #1:
Since you see it on Thunderbird and I do on SeaMonkey, it must be a "MailNews Core" bug. However I can't find an appropriate Component there.

Comment 3

[Tony Mechelynck [:tonymec]]

Actually, the above workaround doesn't work: threading marks (twisties) appear, but clicking them doesn't expand anything (the twisty toggles but messages other than the first in the thread don't appear).

Comment 4

[Tony Mechelynck [:tonymec]]

After going back to yesterday's nightly,

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b3pre) Gecko/20090112 SeaMonkey/2.0a3pre - Build ID: 20090112000551

just "(right-click folder) => Properties => Rebuild Summary File" makes the problem disappear.

Moving to "MailNews Core::Database" since closer look at the components' description seems to indicate that that's where mail summary file problems common to SeaMonkey and Thunderbird belong.

Comment 5

[Ashley Bischoff (blog at handcoding.com)]

Near as I can tell, I believe these are the MailNews Core bugs that were checked in on 2009-01-12 (and which would have ended up in the 2009-01-13 nightly builds):

* Bug 324147 - Retention policy should not delete flagged/starred messages
* Bug 433619 - Reminder flag for compact folder popup ineffective when clicking "NO"
* Bug 468915 - comm-central build fails if built in srcdir

For what it's worth, I made use of this b.m.o query to find those:
https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced&short_desc_type=allwordssubstr&short_desc=&product=MailNews+Core&long_desc_type=substring&long_desc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&status_whiteboard_type=allwordssubstr&status_whiteboard=&keywords_type=allwords&keywords=&bug_status=RESOLVED&bug_status=VERIFIED&resolution=FIXED&emailtype1=exact&email1=&emailtype2=exact&email2=&bugidtype=include&bug_id=&votes=&chfieldfrom=2009-01-12&chfieldto=now&chfieldvalue=&cmdtype=doit&order=Reuse+same+sort+as+last+time&field0-0-0=noop&type0-0-0=noop&value0-0-0=

(That corresponds to a query of: Product = Mailnews Core, Status = RESOLVED|VERIFIED, Resolution = FIXED, Only bugs changed between= "2009-01-12" and "now". I then went through each bug in the list to check whether it was marked as FIXED on 2009-01-12.)

Naturally, I realize that a Bonsai query probably would have been more efficient, but unfortunately I'm a fish out of water when it comes to Bonsai.

Tony: Do you think any of those bugs look promising? (Or, just to be sure, do you see any flaws in my Bugzilla query or the process by which I found those bugs?)

PS I can also confirm that after going back to the 20090112 Shredder/3.0b2pre build and rebuilding the index for that folder, my threading came back to normal.

Comment 7

[Ashley Bischoff (blog at handcoding.com)]

I also just looked through that buglist (from comment #5) for bugs that may have been fixed in the early-morning hours of 2009-01-13 (in case those checkins may have also made it into the 2009-01-13 builds).

Not that it looks that promising, but that did yield this additional bug:

* Bug 370306 -  Move Address Book's local autocomplete implementations to be based on toolkit's [Marked FIXED at 2009-01-13 03:35:45 PST]

Comment 8

[Ashley Bischoff (blog at handcoding.com)]

Updating summary since this apparently affects more than just bugmail (per dupe bug 473589).

Old summary:
"Cannot thread bugmail anymore"

New summary:
"View->sort by->threaded not threading by subject, only by reference"

Comment 9

[Phil Ringnalda (:philor)]

Regression hunting: you're doing it the hard way.

tinderbox.mozilla.org will show you what revision was picked up by a particular nightly, and when it started. http://hg.mozilla.org/comm-central/pushloghtml will tell you when patches were pushed. Between the SeaMonkey Linux nightlies from comment 0, that would be http://hg.mozilla.org/comm-central/pushloghtml?startdate=2009-01-12+00%3A00%3A00&enddate=2009-01-13+00%3A00%3A00 including the first push for bug 451995 which as you can see in http://hg.mozilla.org/comm-central/rev/20af028c35be set mail.thread_without_re to false, despite not talking about threading in the bug or including that in the patch in the bug, which should lead you to think that bienvenu accidentally let something from another patch leak into the archives push.

Comment 10

[Tony Mechelynck [:tonymec]]

(In reply to comment #5)
[...]
> Tony: Do you think any of those bugs look promising? (Or, just to be sure, do
> you see any flaws in my Bugzilla query or the process by which I found those
> bugs?)
[...]

I wouldn't know. As you so colorfully say, "I'm a fish out of water" when it comes to Mozilla code and to which fix might be most likely to have produced which unforeseen problem. As for bug querying, I would have queried for any landings between the "good" build in comment #4, 2009-01-12 00:05:51, and the earlier of the "bad" ones in comment #0 and comment #1, probably 2009-01-13 00:04:55 since the other one has only a datestamp, no timestamp, which means (since these SeaMonkey builds were both produced very near midnight PST) that the culprit most probably happened on the 12th. But I'm a Mozilla "user", not a "coder", so I would defer to Philor (comment #9) who sounds like he knows what he's talking about.

Comment 11

[Ashley Bischoff (blog at handcoding.com)]

I think I've figured some things out here. 

* The good news: I believe that you can threading back the old way, by subject, if that's the way you like it.

* The bad news: This doesn't happen by default; there're a few prefs you need to adjust to get that behavior back.

---------------
Some background
---------------

After reading Phil's comment more closely (comment #9) and looking over the changeset that he mentioned there (http://hg.mozilla.org/comm-central/rev/20af028c35be), it struck me that the changeset that David checked in for bug 451995 appears to have (unintentionally) set "mail.thread_without_re" to false by default.

What "mail.thread_without_re" does is to allow for subject-based threading (even if a particular subject doesn't include "re:", as is the case with bugmail). So, once this was turned off, subject-based threading pretty much stopped working.

-----------------------------
Getting the old behavior back
-----------------------------

The good news -- and tried and confirmed this myself -- is that you can turn this back on and you'll get the old behaviour again. There're a couple steps to that, though:

1. Go into Thunderbird or Seamonkey's config editor. (In Shredder, this is under Preferences -> Advanced -> General -> Config Editor.)

2. Search for "thread" and you should see several threading-related prefs.

3. This page (https://wiki.mozilla.org/MailNews:Message_Threading) goes over what all those do, but if you literally want the old behavior back, you can just set "mail.thread_without_re" back to "true" again. (On the other hand, "mail.correct_threading" appears to be more flexible -- and it properly threads bugmail too -- so, personally, I've enabled that pref instead.)

4. Then, from what I can tell, I believe you then need to restart Shredder/Seamonkey to get threading-related prefs to take effect.

5. After restarting Shredder/Seamonkey, you'll need to rebuild the index on your bugmail folder. (In Shredder, this can be done by right-clicking on your bugmail folder, choosing "Properties", and from the dialog box that comes up, choosing "Rebuild Index".)

6. Lastly, because "Rebuild Index" resets your sort-by options, you'll need to go into View -> Sort by -> Threaded to re-enable threading for that folder.

--------------
Our next step?
--------------

As mentioned, that page from the Mozilla Wiki goes over the various threading options and it looks like "mail.correct_threading" is smart enough that, in my opinion, I think it could be enabled by default. What makes it especially convenient for bugmail is that it intelligently threads messages even if their common ancestor has since been deleted. (For instance, if you have a group of messages that all have a common ancestor of "Message 42" -- but Message 42 has since been deleted -- this threading option will still know to group those messages together.)

As far as more near-term steps, though, I'm not sure which of these would be most suitable:

* Do we REOPEN bug 451995 (since the checkin for that inadvertently set "mail.thread_without_re" to false)? 
* If we really want the old behavior back, do we file another bug to have "mail.thread_without_re" enabled by default?
* Or do we instead file a bug to get "mail.correct_threading" enabled by default?
* Or maybe we morph this bug with the goal of taking care of one of those goals?
* (Some other option?)

Comment 12

[Tony Mechelynck [:tonymec]]

In reply to comment #11:
An alternative is to set the prefs like you want them by means of <yourprofile>/user.js which has the same format as the prefs.js file where Mozilla apps store the nondefault pref settings at every closedown. (user.js does not exist by default.)
- The advantage of using user.js is that your chosen setting will be enforced at every startup, even if the default changes to it and away from it over successive versions (which would reset it to the new default if you were using about:config and pref.js)
- The inconvenient is that, if at some future time you change your mind about that pref's value, you will have to update user.js again.

> 5. After restarting Shredder/Seamonkey, you'll need to rebuild the index on
> your bugmail folder. (In Shredder, this can be done by right-clicking on your
> bugmail folder, choosing "Properties", and from the dialog box that comes up,
> choosing "Rebuild Index".)

The exact same method applies also to SeaMonkey (at least in the trunk builds which I'm using).

Alex, your post sounds encouraging. I'm going to set both mail.correct_threading and mail.thread_without_re in user.js then try the latest SeaMonkey trunk nightly. If it works, this bug will stop being a dogfood-stopper for me.

Comment 13

[Tony Mechelynck [:tonymec]]

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b3pre) Gecko/20090116 SeaMonkey/2.0a3pre - Build ID: 20090116000450

(In reply to comment #12)
[...]
> Alex, your post sounds encouraging. I'm going to set both
> mail.correct_threading and mail.thread_without_re in user.js then try the
> latest SeaMonkey trunk nightly. If it works, this bug will stop being a
> dogfood-stopper for me.

Well, it does; but between the 2009-01-12 nightly and this one all my mail passwords were lost, which I suspect is the result of a different bugfix, about percent-escapes in mail usernames. Happily my old Thunderbird2 profile still had them, so forgetting was not a problem.

So we have a workaround: In SeaMonkey's about:config page, or in Thunderbird's Config Editor, set at least one of mail.correct_threading and/or mail.thread_without_re to true (toggle by double-clicking) then restart the program, and then rebuild the indices for any mail folders where threading had got lost, by "Right-click => Properties => Rebuild Index".

What I didn't realize (and forgot to check) in my comment #0 is that those threading preferences require a restart to take effect. (Most prefs don't.)

Comment 14

[Tony Mechelynck [:tonymec]]

P.S. Is it correct_threading or strict_threading? (mail.strict_threading exists by default.)

Comment 15

[Ashley Bischoff (blog at handcoding.com)]

(In reply to comment #14)
> P.S. Is it correct_threading or strict_threading? (mail.strict_threading exists
> by default.)

At least on the Shredder nightly builds, both mail.strict_threading and mail.correct_threading exist by default (and both are also "false" by default).

According to that wiki page describing the prefs (https://wiki.mozilla.org/MailNews:Message_Threading#Preferences_Controlling_Threading), "mail.correct_threading" was "implemented by bug 181446, only available in 3.0 releases and later." And, while that would seem to imply [Shredder] 3.0, I did also notice that bug 181446 is for the "MailNews Core" product, so one would think that mail.correct_threading would exist for SeaMonkey too (?).

As far as the differences between them, here's gist of it:

* mail.thread_without_re : Thread by subject even when there is no "Re:" in the subject.

* mail.strict_threading : Disable subject-based threading; If enabled alongside mail.correct_threading, it'll still thread based on message IDs, but won't fall back to subject-based threading (as I understand it).

* mail.correct_threading : Thread things correctly using heuristics and other automagic. This first tries to use message IDs (which works with bugmail) and then falls back to using subject lines (as long as those subject lines have a "re:).

So, to answer your question, if your mail client supports mail.correct_threading, that's probably the one you want.

Comment 16

[Tony Mechelynck [:tonymec]]

In reply to comment #15: Ah, thanks.
I think mail.correct_threading is supported: "right-click => Reset" changes it to false, not to "".

Comment 19

[Rain [:rain1] (inactive)]

Pretty big regression, asking for blocking-tb3 to keep it on the radar for b2.

Comment 20

[Tony Mechelynck [:tonymec]]

(In reply to comment #19)
> Pretty big regression, asking for blocking-tb3 to keep it on the radar for b2.

This bug is common to Thunderbird and SeaMonkey. KaiRo, do you think the corresponding SeaMonkey flags should be raised too, or can we be content to "ride along" with the Thunderbird guys?

Comment 21

[Robert Kaiser]

I think riding along with the Thunderbird flags for such core issues is probably the best idea.

Comment 22

[Magnus Melin [:mkmelin]]

Just setting mail.thread_without_re to false without fixing 449821 will just have a lot of testers think threading is broken. I think we should set it back to true until bugmail threads correctly, it's such a pain to read it without the pref.

Comment 23

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Attachment: patch that resets thread_without_re to true

Comment 24

[David :Bienvenu]

Comment on attachment 357609 [details] [diff] [review]
patch that resets thread_without_re to true

sure, thx, Gary.

Comment 25

[Magnus Melin [:mkmelin]]

Comment on attachment 357609 [details] [diff] [review]
patch that resets thread_without_re to true

Thx, r=mkmelin

Comment 26

[Mark Banner (:standard8)]

Temporarily taking off checkin-needed due to current drivers discussions.

Comment 27

[David Ascher (:davida)]

There is some concern that this fix will work for bugmail, but cause false positive threading for emails that have subjects like "hi" or "(no subject)".  I'm hoping that we can come up with a hybrid approach, so that we can have fewer problems across the overall population of mail.

Comment 28

[David :Bienvenu]

we could turn on mail.correct_threading instead. The ramifications are that opening large folders will be somewhat slower and consume more memory. As davida has suggested, we could have an alternate heuristic like only threading by subject without re for longer subjects, so that things like "hi" or "test" don't thread together, but less common subjects do. 

Turning on correct_threading will help with bugzilla threading but it won't help with the general case of getting two messages with the same subject "don't forget Tom's birthday".

Comment 29

[Ashley Bischoff (blog at handcoding.com)]

(In reply to comment #28)
> Turning on correct_threading will help with bugzilla threading but it won't
> help with the general case of getting two messages with the same subject "don't
> forget Tom's birthday".

Feel free to correct me if I'm wrong about this, but if we turn on correct_threading but leave thread_without_re in its current state (false), would that not be able to solve most cases of two messages with the same subject "don't forget Tom's birthday" (since neither of those have "re")?

(Of course, we'd still end up threading stuff like "re: don't forget Tom's birthday" / "re: don't forget Tom's birthday", but perhaps a few false positives are okay?)

Or, if it was decided that avoiding false positives was more important than avoiding false negatives, another option would be something like this (via bug 449821 comment #16):

    strict_threading=true correct_threading=true thread_without_re=false

(And, indeed, that would disable subject-based threading entirely but still leave proper threading intact.)

Comment 30

[Michael Baffoni]

I'm not sure if I'm missing something here, but in the case of non-trivial subjects (BTW, don't forget <blank> ;-) ) that match, wouldn't it be a correct assumption that they _should_ be threaded?  In my mind, this is really no different than having different child threads of a parent thread which has been deleted (without the explicit relationship references of course) since the intention is that they are common subjects grouped into a common, related area, even if they eventually diverge.  So just because someone didn't reply to an actual email but had the same (non-trivial) subject anyway, logically they should be grouped/threaded as intended.  I find this immensely useful when receiving logs via email, or other auto-generated information where the subjects are identical, and as expected, are related. IMHO.

Comment 31

[Valery Kondakoff]

Excuse me if I'm wrong and this is not a right place to post such a requests, but, please, fix the following issue: when maillist adds some letters in the beginning to every message subject, the messages in the same thread are counted as separate threads. Here is a screenshot: http://http.rugby-forum.ru/temp/thread.jpg

Comment 32

[Valery Kondakoff]

here is the right URL: http://www.rugby-forum.ru/temp/thread.jpg

Comment 33

[David :Bienvenu]

(In reply to comment #30)
> I'm not sure if I'm missing something here, but in the case of non-trivial
> subjects (BTW, don't forget <blank> ;-) ) that match, wouldn't it be a correct
> assumption that they _should_ be threaded?

Yes, that's what I think. And correct_threading w/o thread_without_re doesn't give us that.

Comment 34

[David :Bienvenu]

duping to - Bug 449821

Comment 35

[Tony Mechelynck [:tonymec]]

(In reply to comment #34)
> duping to - Bug 449821
> 
> *** This bug has been marked as a duplicate of bug 449821 ***

Bug 449821 dealt with my original problem (bugmail doesn't thread), so -- okay, David, so be it.

If anyone wants to REOPEN, please give a convincing reason why.