383985 - Sort order of threads in message list (sort threads by contained msg with oldest vs. newest date header) should not be controlled by "Sort by: Date" vs "Sort by: Order Received"

Status: NEW

(MailNews Core :: Backend, defect)

Description

[Nelson Bolyard (seldom reads bugmail)]

If you view the messages in a folder or newsgroup in a threaded manner,
so that the messages appear in a tree by thread, the order in which the
threads appear in the message list pane is either 
a) in order by the date of the first message in the thread, or 
b) in order by the most recently received message in the thread.

This order choice is controlled by whether the display is sorted by "Date" 
or by "Order Received".  To my way of thinking about those terms, this 
seems either (a) wrong, or (b) backwards.

Presently, when the messages are sorted by Date, the threads appear in 
the order of the most recently received message in the thread.  When sorted
in ascending order, the thread that most recently received a message will 
appear last.  

If you want the threads to appear in order according to the age of the 
thread (that is, in order by the date of the first/oldest message in the 
thread), you must choose to sort the threads by "Order Received".  

But the difference between "Date" and "Order Received" is supposed to be
(IINM) the difference between the date in the "Date:" header and the date
in the "Received" header, which really has nothing to do with the order
of threads.  If one is sorting threads to occur in the order of the first 
message in the thread, that could be done using either the "Date:" header 
date, or using the "Received" header date.  So, choosing "Date" or 
"Order Received" does not seem to imply the order in which the threads
themselves are sorted.

But if we're going to continue to use "Date" vs "Order received" to decide
whether the thread with the oldest message comes first, or if the thread
with the most recent message comes last, then to my way of thinking, having 
the threads ordered by the most recent message received seems like "order received".

Comment 1

[WADA:World Anti-bad-Duping Agency]

Same enhancement request as Bug 57898?

Meaning of "Order Received" column value looks to be different between local mail folder and newsgroup.
  A) When local mail folder, offset of a mail in mail folder file
  B) When newsgroup, sequence number of news article
This is probably because same field in ".msf" is used for offset(local folder) and sequence number(newsgroup).
So slightly different discussion will be required for local mails and newsgroup.

> But the difference between "Date" and "Order Received" is supposed to be(IINM) 
> the difference between the date in the "Date:" header and the date in the
> "Received" header

Partially incorrect.
When local mail folder, "Order Recieved" column value looks to be offset in mail folder file. (This can be seen by moving some mails, and move back in different order, and compact folder.)  

For "Date" column(not "Date:").
mailnews.use_received_date=false/true is already introduced in trunk(future Tb 3), and value of "Date" column(please note that not "Date:" column) becomes time stamp in last Received: header when mailnews.use_received_date=true. 
See Bug 341548 for this.
If you can accept loss of "sort by time stamp of Date:" in return for gain of "sort by time stamp of Receieved:", this will possibly improve threading because non-so-accurate time stamp at mail sender's PC is hidden by usually accurate your mail server's time stamp.
However, it looks that "order" used when threading is offset value when local mail folder. So, if you move mails from downloaded folder to other folder in random order, offset becomes completely different one from initial "Order Received". In this case, I think enhancement of Bug 57898 is mandatory.

By the way, My guess on difference of "Order Received" column between local mail folder and news group is as follows.
NNTP was supported after POP3, and field for offset was used for sequence number of newsgroup. And offset can be used as sequence number(leaps though) when single account/single folder era, then "Order Received" column name was assigned.

Comment 2

[WADA:World Anti-bad-Duping Agency]

Sorry but I was also partially wrong. I found Bug 262316 fixed on 2007-03-30.
> Bug 262316 – Sort by date does not sort messages in thread by date

According to the bug, order by "Order Receieved" when Threading is simply a initial order when Thread View is enabled. And when Thread View is active, Current *SPEC* seems to be that newly chosen "View/Sort By/Date" *SHOULD* be reflected to order in a Thread, but was not done for long time.

Patch is landed on trunk only. Can you check with latest-trunk nightly?

Comment 3

[Nelson Bolyard (seldom reads bugmail)]

In reply to comment 2,
bug 262316 is about the sort order of messages in a single thread.
This bug is about the order of the threads themselves.

Comment 4

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #3)
> This bug is about the order of the threads themselves.
I see.
If so, can setting change of "View/Sort By" option to "Date" be a work around?
(instead of sorting by column header click).

Following is behavor of Seamonkey 1.1.2.
When Date column header click, "View/Sort By" is set to Date, but when Thread column click(toggle ascend/descend/unthread) after Date colmn click, order is changed to "Order received"(View/Sort By" is also changed to "Order Received").
But when set "View/Sort By" to Date turu menu, Thread column click kept Date of "View/Sort By".

Comment 5

[Nelson Bolyard (seldom reads bugmail)]

Wada, let me express my complaint another way.  

When you select View -> Threads -> Threads With Unread, you get a message list
pane in which the messages are grouped by threads, and all the threads are 
collapsed.  Each thread appears as just one line.  I am concerned with the
order of those single-line-threads.  

Most of the time, I want those threads to be ordered by the Date of the oldest
message in the thread.  Another way to say that is, Oldest thread first.  

At other times, I want those threads to be ordered by the Date of the most
recent message in the thread.  I want the threads to be in ascending order, 
but based on the Date of the most recent message in the thread, not based 
on the date of the oldest message in the thread.  
Call this: "Most Recently Updated Thread Last.  

Either way (Oldest Thread First, or Most Recently Updated Thread Last), 
I want the messages in each thread to be ordered in ascending order by their Date (not Received).  As you may recall, I HATE sorting messages in "order 
received".  I always want all messages to be sorted based on the date in the 
Date: header, and never by the date in some Received header.  

Today, the order of the threads themselves is inextricably tied to whether
the messages are ordered by Date, or by "Order Received".  In order to get
the threads in "Oldest Thread First" order, I have to tolerate seeing
the messages in each thread be in "Order Received" order, not in "Date" 
order.  That burns me up.  

The order of the threads needs to be separated from the order of the messages
in the thread (or in the folder, when not threaded). 

Comment 6

[WADA:World Anti-bad-Duping Agency]

I see what(In reply to comment #5)
> The order of the threads needs to be separated from the order of the messages
> in the thread (or in the folder, when not threaded). 

I've understand problem you are talking now.
I could observe problem easily by "View/Sort by"=Date thru menu instead of column header, "Collapse All Threads" and mail folder for bugzilla mail.
And I could confirm phenomenon of next in comment #0.
> when the messages are sorted by Date, the threads appear in 
> the order of the most recently received message in the thread.
(And could experience bug 262316 by "move a mail in a thread then move back")

There are at least three problems which relates threading.
 Problem-1. This bug. Order of main thread.
 Problem-2. bug 262316. Order in a thread.
 problem-3. My comment #4. "View/Sort by"=Date thru column header is reset
            to "Order Received" by click of column header for threading.
I was confused due to problem-3 in my observation.
Sorry for confusion and misunderstanding.

Comment 7

[WADA:World Anti-bad-Duping Agency]

FYI.
Bug 369620 is found for problem of comment #4.

Comment 8

[WADA:World Anti-bad-Duping Agency]

Bug 254159 also reported problem of "threads order by most recently received message in the thread", though report when "Order Received" is chosen.
> Bug 254159 – With Order Received sorting, order threads by date of last/latest message in thread 

Comment 9

[WADA:World Anti-bad-Duping Agency]

I think this bug is DUP of Bug 254159, but uncertain. So set dependency only. 

Comment 10

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #5)
> Either way (Oldest Thread First, or Most Recently Updated Thread Last), 
> I want the messages in each thread to be ordered in ascending order by their
> Date (not Received).

I seems to have been still confused and misunderstanding...
Bug 254159 was for order of thread, instead of order in a thread.

I absolutely agree with you on order of mails in a thread.
Order of thread should be selectable, but Order of mails in a thread is to be absolute(chronological sequence), is to be based on In-reply to and Date: header, in order to reflect sequence of communication by mail.

Because Bug 262316 is already fixed, I think View/Sort By/Date,Ascending(or Descending)&Threaded can be a partial workaround. (Ascending/Descending looks to affect on order of thread only.) 
Is it right?

Please note that problem still remains if parent mail of In-Reply-To: or References: is physically placed after the mail(larger "Order Received" value=Offset in mail folder file).

Comment 11

[WADA:World Anti-bad-Duping Agency]

Bug 369620 

Comment 12

[WADA:World Anti-bad-Duping Agency]

Bug 369620 has been fixed.
With Tb trunk BuildID=2008041503, with mailnews.thread_pane_column_unthreads=true, mails are sorted by Date column when threaded. By this change, mails in a thread(and thread itself too) are sorted by Date.

Note: Default of mailnews.thread_pane_column_unthreads looks to be already changed to false on Trunk.

Comment 13

[WADA:World Anti-bad-Duping Agency]

"Order Received" column value was;
  (A) IMAP : UID == Order received
  (B) NEWS : Article number  == Order received
  (C) Local mail folder(mails are not copied to it) :
             Offset of mail data in mail folder file == Order received
  (D) Local mail folder(mails are copied to it) :
             Offset of mail data in mail folder file != Order received
To support (4), I think new sequence number(represents chronological order, call CLO here) based on timestamp(call Mail Time Stamp, MTS here) is a simple solution.
  MTS : (1-a) date/time of Date: header
              (As you say, this should be used. And if clock of mail sender's)
              (PC is correct, this will display conversation in correct order.)
        (1-b) date/time of latest(top in headers) Received: header.
              If not found, date/time of Date: header
              (Some users may want this behaviour because clock of mail sender's)
              (PC is frequently wrong. They may prefer threading of spams :-)   )
        (2) If no Date: header or malformed Date: header,
            date/time in envelope line of Unix Mbox("From ..." line).
            (Sm/Tb puts download local time without GMT+nn/GMT-nn)
        (3) If no date/time in "From ...", rebuild-index time or Epoch time.
  CLO : Local mail folder : Sequence number of MTS in a mail folder file
                            (similar to UID of IMAP, although varies)
  CLO : IMAP :
        If UID is used, same issue as "order by Received: header" on local mail
        folder arises. Date: header should be used in IMAP case too?   
  CLO : NEWS :
        If article number is used, same issue as "order by Received: header"
        on local mail folder arises. But Date: header is always reliable?
        No problem usually because In-Reply-To: is usually written?

By the way, "Order of threads" in bug summary is slightly misleading. I feel "mail order in a thread" or "order in a thread" or something is better.

Comment 14

[Nelson Bolyard (seldom reads bugmail)]

> "Order of threads" in bug summary is slightly misleading. I feel
> "mail order in a thread" or "order in a thread" or something is better.

Then you're confused about what this bug is about.  
This bug is NOT about the order of individual mails in a single thread.
This bus is about the order of the threads, relative to one another, 
e.g. which thread comes first, which thread comes second, and so on.

Today, the order of the threads, and the order of the individual messages 
in the thread are controlled by the same setting.  This bug says they should
be separate controlled.

Comment 15

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #14)
> This bug is NOT about the order of individual mails in a single thread.
> This bus is about the order of the threads, relative to one another, 

Oh, sorry for my repeated confusion.

> This bug says they should be separate controlled.

I think it's applicable to Bug 479969 too, about the order of individual mails in a single thread, which is cause of my confusion.  Nelson, can you post comment about "what should be" to Bug 479969?

Comment 16

[WADA:World Anti-bad-Duping Agency]

Required settings/options for "the order of the threads" :
 (a) Which mail's CLO is used for threads sorting based on date/time:
     (a1) main mail, (a2) newest mail, (a3) oldest mail, (a4) ?, ...
 (b) Order : (b1) Ascending, (b2) Descending
 (c) Sort column or mail attribute : (c1) Date, (c2) Order Received, (c3) Received
Is there any other mandatory setting/option?
If "order of individual mails in a single thread" is absolutely chronological order, I think sorting of threads by current "Order Received" column value(UID,article number) shouldn't be prohibited. What do you think?

Comment 17

[Nelson Bolyard (seldom reads bugmail)]

Wada,  What's a CLO?

It seems difficult to succinctly describe this issue.  I will try again.

1. Every message has (potentially) two dates of interest, which are:
a) the sender's date in the Date: header, which may be WAY off, or 
b) the receiver's date for the receipt of the message.  

Some people think we should always use one, others think we should 
always use the other.  I think we should offer both choices, but that that
choice should be consistently used in ALL places that do ordering based on 
dates.  

2. Threads are ordered by picking one email message out of the thread, and 
letting the date for that mail (either sender's date or receiver's date,
as discussed above) represent the date of that thread.  Call it the thread
representative's date.  Then the threads are ordered according to their 
respective representative's dates.  

The representative message might be:
a) the message in the thread with the oldest date, or
b) the message in the thread with the newest date, or 
c) the "root" of the thread's tree, which might be neither the oldest or 
newest.  

These settings should be separate.  That is, the user's choice of whether
to use the sender's date or the receiver's date should be independent of 
the user's choice for method of picking the representative message in each
thread.  *THE PROBLEM IS THAT TODAY THEY ARE NOT INDEPENDENT.*

Today, when you choose to use the sender's date, then the newest message in
each thread becomes the representative for its thread.  When you choose to 
use the receiver's date, then the oldest message in the thread becomes the
representative for its thread.  Today, you *CANNOT* choose to sort mail 
according to the SENDER's date, AND ALSO use the oldest mail in the thread
as the representative date for the thread.  THAT's the problem.

This bug asks tht these choices be separated.  Have a setting that says:
"Whenever I choose a "date", I want 
 ( ) the sender's date, 
 ( ) the receiver's date.  

Then separately have a setting that says:
When ordering threads, take each thread's date from 
 ( ) the oldest message in the thread
 ( ) the newest message in the thread
 ( ) the root message in the thread's tree (based on references, not dates).

That last item would be very nice, because it really solves the whole date
issue, ordering messages by their references to one another, rather than by
dates.  But if it's outside the scope of this bug, I understand.  

I want to be able to do something I cannot do today, namely, order threads
by the date of the OLDEST message in each thread, using the SENDER's Date
field.

Comment 18

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #17)
> What's a CLO?

CLO(ChronoLogical Order) which I used in Comment #13. Sorry for undefined term.

Comment 19

[WADA:World Anti-bad-Duping Agency]

> I think we should offer both choices, but that that choice should be consistently
> used in ALL places that do ordering based on dates.

Aha. I've understood your requirement slightly better than before.
But I feel next is natural behavior.
 1. Enter Threaded View mode
 2. Click Date column => Threads are sorted based on Date: header value
 3. Click Order Received column => Threads are sorted based on UID/Article number
 4. Click Received => Threads are sorted based on Received: header value
What do you think?

Comment 20

[Brian Murrell]

I don't know if this has been expressed yet -- TBH, my head hurts after trying to follow the postings in this bug -- but I simply want to be able to click on the "Received" column to change the sort of threads/messages either in ascending or descending sort.

For the purposes of sorting the date of a "thread" should be the newest message in the thread when descending sort is chosen or the oldest message when ascending sort is chosen (if that preference is not configurable seperately).

Currently, it appears that for ascending sort, this all works -- threads are sorted by their oldest message but for descending sort the thread is also being sorted by the oldest message in the thread not the newest one.

Comment 21

[Brian Murrell]

Actually, I take that back.  Even in ascending mode, the sort is not quite working, even for individual messages.  Even though I have clicked on the Received column such that the arrow is pointing down, individual messages are still be sorted by their Date: not their Received: date.

So it's really, all quite broken -- this sorting on Received.

Comment 23

[98a849]

sorry for my ignorance to not reading this entire thread (i got here by bug-report-automatic-web-searcher) ...

i would like to have in my thunderbird-app:
bug: date-(increment or decrement)-email-list-sorting is NOT updated by clicking on comply to "all email folders" for a single-thunderbird-email-folder settings-'tuning' list-entry ...
request: i would like to sort my emails in each folder like in twitter "decrement" (instead of increment email list) .. so the newest email eppears at the top ... but if new windows iso's are released further 'only' at each year or so, thats my herewith demanded thunderbid-feature is maybe not needed (because i can/could sort each of my thunderbid-email-folders by my self). You are very wellcome (i like thunderbird a lot and I think its the BEST email app ever written/developed as far as i know:)