Closed Bug 473439 Opened 11 years ago Closed 11 years ago

View->sort by->threaded not threading by subject, only by reference

Categories

(MailNews Core :: Database, defect)

defect
Not set

Tracking

(Not tracked)

VERIFIED DUPLICATE of bug 449821
Thunderbird 3.0b2

People

(Reporter: tonymec, Assigned: gkw)

References

Details

(Keywords: regression, Whiteboard: [workaround see comments #11 to #13][no l10n impact])

Attachments

(1 file)

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b3pre) Gecko/20090113 SeaMonkey/2.0a3pre - Build ID: 20090113000455

Starting today, "compacting" the folders removes all threading of BMO bugmail. None of the following has any effect:
- Toggle mail.strict_threading then "File => Compact folders"
- Toggle mail.correct_threading then "File => Compact folders"
- Toggle mail.thread_without_re then "File => Compact folders"
- Any of the above, using "(right-click folder) => Properties => Rebuild summary file" instead of compacting
- "View => Sort by => Unthreaded" then toggle "View => Sort by => Ascending/Descending" then "View => Sort by => Threaded".

This bug didn't happen in yesterday's nightly: bugmail messages used to thread together if either they had the same subject (both bug number and Summary), or the first one was the bug creation and they had the same bug number (regardless of Summary changes).

See also:
- Thunderbird bug 449821
- RFE bug 164115 which seems to request the opposite of what I want (but maybe I misunderstood).
Confirming with: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b3pre) Gecko/20090113 Shredder/3.0b2pre.

OS -> All
OS: Linux → All
Workaround:
1. View => Sort by => Subject
2. View => Sort by => Grouped by Sort (set)
     (threading reappears)
3. View => Sort by => Grouped by Sort (reset)
     (threading remains)
4. View => Sort by => (whatever) (in my case, "Order Received")
5. Wait long enough for the threads to get sorted (it's slow).

In reply to comment #1:
Since you see it on Thunderbird and I do on SeaMonkey, it must be a "MailNews Core" bug. However I can't find an appropriate Component there.
Actually, the above workaround doesn't work: threading marks (twisties) appear, but clicking them doesn't expand anything (the twisty toggles but messages other than the first in the thread don't appear).
After going back to yesterday's nightly,

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b3pre) Gecko/20090112 SeaMonkey/2.0a3pre - Build ID: 20090112000551

just "(right-click folder) => Properties => Rebuild Summary File" makes the problem disappear.

Moving to "MailNews Core::Database" since closer look at the components' description seems to indicate that that's where mail summary file problems common to SeaMonkey and Thunderbird belong.
Component: MailNews: Message Display → Database
Product: SeaMonkey → MailNews Core
QA Contact: message-display → database
Near as I can tell, I believe these are the MailNews Core bugs that were checked in on 2009-01-12 (and which would have ended up in the 2009-01-13 nightly builds):

* Bug 324147 - Retention policy should not delete flagged/starred messages
* Bug 433619 - Reminder flag for compact folder popup ineffective when clicking "NO"
* Bug 468915 - comm-central build fails if built in srcdir

For what it's worth, I made use of this b.m.o query to find those:
https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced&short_desc_type=allwordssubstr&short_desc=&product=MailNews+Core&long_desc_type=substring&long_desc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&status_whiteboard_type=allwordssubstr&status_whiteboard=&keywords_type=allwords&keywords=&bug_status=RESOLVED&bug_status=VERIFIED&resolution=FIXED&emailtype1=exact&email1=&emailtype2=exact&email2=&bugidtype=include&bug_id=&votes=&chfieldfrom=2009-01-12&chfieldto=now&chfieldvalue=&cmdtype=doit&order=Reuse+same+sort+as+last+time&field0-0-0=noop&type0-0-0=noop&value0-0-0=

(That corresponds to a query of: Product = Mailnews Core, Status = RESOLVED|VERIFIED, Resolution = FIXED, Only bugs changed between= "2009-01-12" and "now". I then went through each bug in the list to check whether it was marked as FIXED on 2009-01-12.)

Naturally, I realize that a Bonsai query probably would have been more efficient, but unfortunately I'm a fish out of water when it comes to Bonsai.

Tony: Do you think any of those bugs look promising? (Or, just to be sure, do you see any flaws in my Bugzilla query or the process by which I found those bugs?)

PS I can also confirm that after going back to the 20090112 Shredder/3.0b2pre build and rebuilding the index for that folder, my threading came back to normal.
Duplicate of this bug: 473589
I also just looked through that buglist (from comment #5) for bugs that may have been fixed in the early-morning hours of 2009-01-13 (in case those checkins may have also made it into the 2009-01-13 builds).

Not that it looks that promising, but that did yield this additional bug:

* Bug 370306 -  Move Address Book's local autocomplete implementations to be based on toolkit's [Marked FIXED at 2009-01-13 03:35:45 PST]
Updating summary since this apparently affects more than just bugmail (per dupe bug 473589).

Old summary:
"Cannot thread bugmail anymore"

New summary:
"View->sort by->threaded not threading by subject, only by reference"
Summary: Cannot thread bugmail anymore → View->sort by->threaded not threading by subject, only by reference
Regression hunting: you're doing it the hard way.

tinderbox.mozilla.org will show you what revision was picked up by a particular nightly, and when it started. http://hg.mozilla.org/comm-central/pushloghtml will tell you when patches were pushed. Between the SeaMonkey Linux nightlies from comment 0, that would be http://hg.mozilla.org/comm-central/pushloghtml?startdate=2009-01-12+00%3A00%3A00&enddate=2009-01-13+00%3A00%3A00 including the first push for bug 451995 which as you can see in http://hg.mozilla.org/comm-central/rev/20af028c35be set mail.thread_without_re to false, despite not talking about threading in the bug or including that in the patch in the bug, which should lead you to think that bienvenu accidentally let something from another patch leak into the archives push.
Severity: normal → major
Hardware: x86 → All
Blocks: 451995
No longer blocks: 236849
(In reply to comment #5)
[...]
> Tony: Do you think any of those bugs look promising? (Or, just to be sure, do
> you see any flaws in my Bugzilla query or the process by which I found those
> bugs?)
[...]

I wouldn't know. As you so colorfully say, "I'm a fish out of water" when it comes to Mozilla code and to which fix might be most likely to have produced which unforeseen problem. As for bug querying, I would have queried for any landings between the "good" build in comment #4, 2009-01-12 00:05:51, and the earlier of the "bad" ones in comment #0 and comment #1, probably 2009-01-13 00:04:55 since the other one has only a datestamp, no timestamp, which means (since these SeaMonkey builds were both produced very near midnight PST) that the culprit most probably happened on the 12th. But I'm a Mozilla "user", not a "coder", so I would defer to Philor (comment #9) who sounds like he knows what he's talking about.
I think I've figured some things out here. 

* The good news: I believe that you can threading back the old way, by subject, if that's the way you like it.

* The bad news: This doesn't happen by default; there're a few prefs you need to adjust to get that behavior back.

---------------
Some background
---------------

After reading Phil's comment more closely (comment #9) and looking over the changeset that he mentioned there (http://hg.mozilla.org/comm-central/rev/20af028c35be), it struck me that the changeset that David checked in for bug 451995 appears to have (unintentionally) set "mail.thread_without_re" to false by default.

What "mail.thread_without_re" does is to allow for subject-based threading (even if a particular subject doesn't include "re:", as is the case with bugmail). So, once this was turned off, subject-based threading pretty much stopped working.

-----------------------------
Getting the old behavior back
-----------------------------

The good news -- and tried and confirmed this myself -- is that you can turn this back on and you'll get the old behaviour again. There're a couple steps to that, though:

1. Go into Thunderbird or Seamonkey's config editor. (In Shredder, this is under Preferences -> Advanced -> General -> Config Editor.)

2. Search for "thread" and you should see several threading-related prefs.

3. This page (https://wiki.mozilla.org/MailNews:Message_Threading) goes over what all those do, but if you literally want the old behavior back, you can just set "mail.thread_without_re" back to "true" again. (On the other hand, "mail.correct_threading" appears to be more flexible -- and it properly threads bugmail too -- so, personally, I've enabled that pref instead.)

4. Then, from what I can tell, I believe you then need to restart Shredder/Seamonkey to get threading-related prefs to take effect.

5. After restarting Shredder/Seamonkey, you'll need to rebuild the index on your bugmail folder. (In Shredder, this can be done by right-clicking on your bugmail folder, choosing "Properties", and from the dialog box that comes up, choosing "Rebuild Index".)

6. Lastly, because "Rebuild Index" resets your sort-by options, you'll need to go into View -> Sort by -> Threaded to re-enable threading for that folder.

--------------
Our next step?
--------------

As mentioned, that page from the Mozilla Wiki goes over the various threading options and it looks like "mail.correct_threading" is smart enough that, in my opinion, I think it could be enabled by default. What makes it especially convenient for bugmail is that it intelligently threads messages even if their common ancestor has since been deleted. (For instance, if you have a group of messages that all have a common ancestor of "Message 42" -- but Message 42 has since been deleted -- this threading option will still know to group those messages together.)

As far as more near-term steps, though, I'm not sure which of these would be most suitable:

* Do we REOPEN bug 451995 (since the checkin for that inadvertently set "mail.thread_without_re" to false)? 
* If we really want the old behavior back, do we file another bug to have "mail.thread_without_re" enabled by default?
* Or do we instead file a bug to get "mail.correct_threading" enabled by default?
* Or maybe we morph this bug with the goal of taking care of one of those goals?
* (Some other option?)
In reply to comment #11:
An alternative is to set the prefs like you want them by means of <yourprofile>/user.js which has the same format as the prefs.js file where Mozilla apps store the nondefault pref settings at every closedown. (user.js does not exist by default.)
- The advantage of using user.js is that your chosen setting will be enforced at every startup, even if the default changes to it and away from it over successive versions (which would reset it to the new default if you were using about:config and pref.js)
- The inconvenient is that, if at some future time you change your mind about that pref's value, you will have to update user.js again.

> 5. After restarting Shredder/Seamonkey, you'll need to rebuild the index on
> your bugmail folder. (In Shredder, this can be done by right-clicking on your
> bugmail folder, choosing "Properties", and from the dialog box that comes up,
> choosing "Rebuild Index".)

The exact same method applies also to SeaMonkey (at least in the trunk builds which I'm using).

Alex, your post sounds encouraging. I'm going to set both mail.correct_threading and mail.thread_without_re in user.js then try the latest SeaMonkey trunk nightly. If it works, this bug will stop being a dogfood-stopper for me.
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b3pre) Gecko/20090116 SeaMonkey/2.0a3pre - Build ID: 20090116000450

(In reply to comment #12)
[...]
> Alex, your post sounds encouraging. I'm going to set both
> mail.correct_threading and mail.thread_without_re in user.js then try the
> latest SeaMonkey trunk nightly. If it works, this bug will stop being a
> dogfood-stopper for me.

Well, it does; but between the 2009-01-12 nightly and this one all my mail passwords were lost, which I suspect is the result of a different bugfix, about percent-escapes in mail usernames. Happily my old Thunderbird2 profile still had them, so forgetting was not a problem.

So we have a workaround: In SeaMonkey's about:config page, or in Thunderbird's Config Editor, set at least one of mail.correct_threading and/or mail.thread_without_re to true (toggle by double-clicking) then restart the program, and then rebuild the indices for any mail folders where threading had got lost, by "Right-click => Properties => Rebuild Index".

What I didn't realize (and forgot to check) in my comment #0 is that those threading preferences require a restart to take effect. (Most prefs don't.)
Severity: major → normal
Whiteboard: [workaround see comments #11 to #13]
P.S. Is it correct_threading or strict_threading? (mail.strict_threading exists by default.)
(In reply to comment #14)
> P.S. Is it correct_threading or strict_threading? (mail.strict_threading exists
> by default.)

At least on the Shredder nightly builds, both mail.strict_threading and mail.correct_threading exist by default (and both are also "false" by default).

According to that wiki page describing the prefs (https://wiki.mozilla.org/MailNews:Message_Threading#Preferences_Controlling_Threading), "mail.correct_threading" was "implemented by bug 181446, only available in 3.0 releases and later." And, while that would seem to imply [Shredder] 3.0, I did also notice that bug 181446 is for the "MailNews Core" product, so one would think that mail.correct_threading would exist for SeaMonkey too (?).

As far as the differences between them, here's gist of it:

* mail.thread_without_re : Thread by subject even when there is no "Re:" in the subject.

* mail.strict_threading : Disable subject-based threading; If enabled alongside mail.correct_threading, it'll still thread based on message IDs, but won't fall back to subject-based threading (as I understand it).

* mail.correct_threading : Thread things correctly using heuristics and other automagic. This first tries to use message IDs (which works with bugmail) and then falls back to using subject lines (as long as those subject lines have a "re:).

So, to answer your question, if your mail client supports mail.correct_threading, that's probably the one you want.
In reply to comment #15: Ah, thanks.
I think mail.correct_threading is supported: "right-click => Reset" changes it to false, not to "".
Duplicate of this bug: 474198
Duplicate of this bug: 474185
Pretty big regression, asking for blocking-tb3 to keep it on the radar for b2.
Flags: wanted-thunderbird3?
Flags: blocking-thunderbird3?
Target Milestone: --- → Thunderbird 3.0b2
(In reply to comment #19)
> Pretty big regression, asking for blocking-tb3 to keep it on the radar for b2.

This bug is common to Thunderbird and SeaMonkey. KaiRo, do you think the corresponding SeaMonkey flags should be raised too, or can we be content to "ride along" with the Thunderbird guys?
I think riding along with the Thunderbird flags for such core issues is probably the best idea.
Just setting mail.thread_without_re to false without fixing 449821 will just have a lot of testers think threading is broken. I think we should set it back to true until bugmail threads correctly, it's such a pain to read it without the pref.
Flags: wanted-thunderbird3?
Flags: blocking-thunderbird3?
Flags: blocking-thunderbird3+
Attachment #357609 - Flags: review?(mkmelin+mozilla)
Attachment #357609 - Flags: superreview?(bienvenu)
Assignee: nobody → nth10sd
Comment on attachment 357609 [details] [diff] [review]
patch that resets thread_without_re to true

sure, thx, Gary.
Attachment #357609 - Flags: superreview?(bienvenu) → superreview+
Attachment #357609 - Flags: review?(mkmelin+mozilla) → review+
Comment on attachment 357609 [details] [diff] [review]
patch that resets thread_without_re to true

Thx, r=mkmelin
Status: NEW → ASSIGNED
Keywords: checkin-needed
Temporarily taking off checkin-needed due to current drivers discussions.
Keywords: checkin-needed
There is some concern that this fix will work for bugmail, but cause false positive threading for emails that have subjects like "hi" or "(no subject)".  I'm hoping that we can come up with a hybrid approach, so that we can have fewer problems across the overall population of mail.
we could turn on mail.correct_threading instead. The ramifications are that opening large folders will be somewhat slower and consume more memory. As davida has suggested, we could have an alternate heuristic like only threading by subject without re for longer subjects, so that things like "hi" or "test" don't thread together, but less common subjects do. 

Turning on correct_threading will help with bugzilla threading but it won't help with the general case of getting two messages with the same subject "don't forget Tom's birthday".
(In reply to comment #28)
> Turning on correct_threading will help with bugzilla threading but it won't
> help with the general case of getting two messages with the same subject "don't
> forget Tom's birthday".

Feel free to correct me if I'm wrong about this, but if we turn on correct_threading but leave thread_without_re in its current state (false), would that not be able to solve most cases of two messages with the same subject "don't forget Tom's birthday" (since neither of those have "re")?

(Of course, we'd still end up threading stuff like "re: don't forget Tom's birthday" / "re: don't forget Tom's birthday", but perhaps a few false positives are okay?)

Or, if it was decided that avoiding false positives was more important than avoiding false negatives, another option would be something like this (via bug 449821 comment #16):

    strict_threading=true correct_threading=true thread_without_re=false

(And, indeed, that would disable subject-based threading entirely but still leave proper threading intact.)
I'm not sure if I'm missing something here, but in the case of non-trivial subjects (BTW, don't forget <blank> ;-) ) that match, wouldn't it be a correct assumption that they _should_ be threaded?  In my mind, this is really no different than having different child threads of a parent thread which has been deleted (without the explicit relationship references of course) since the intention is that they are common subjects grouped into a common, related area, even if they eventually diverge.  So just because someone didn't reply to an actual email but had the same (non-trivial) subject anyway, logically they should be grouped/threaded as intended.  I find this immensely useful when receiving logs via email, or other auto-generated information where the subjects are identical, and as expected, are related. IMHO.
Excuse me if I'm wrong and this is not a right place to post such a requests, but, please, fix the following issue: when maillist adds some letters in the beginning to every message subject, the messages in the same thread are counted as separate threads. Here is a screenshot: http://http.rugby-forum.ru/temp/thread.jpg
(In reply to comment #30)
> I'm not sure if I'm missing something here, but in the case of non-trivial
> subjects (BTW, don't forget <blank> ;-) ) that match, wouldn't it be a correct
> assumption that they _should_ be threaded?

Yes, that's what I think. And correct_threading w/o thread_without_re doesn't give us that.
Whiteboard: [workaround see comments #11 to #13] → [workaround see comments #11 to #13][no l10n impact]
duping to - Bug 449821
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 449821
(In reply to comment #34)
> duping to - Bug 449821
> 
> *** This bug has been marked as a duplicate of bug 449821 ***

Bug 449821 dealt with my original problem (bugmail doesn't thread), so -- okay, David, so be it.

If anyone wants to REOPEN, please give a convincing reason why.
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.