1257826 - Incorrect sort order in unified folders using threaded view when sorted by "Received" (see comment #14)

Status: RESOLVED FIXED

(Thunderbird :: Folder and Message Lists, defect)

Description

[V@no]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0
Build ID: 20160303134406

Steps to reproduce:

Set sort to:
Received
Descending
Threaded

Have at least 2 messages in a thread and have other message received during time between first and last threaded messages.


Actual results:

The thread shows at the bottom in unified view, but when switching to the original folder with exact same sorting, the thread shows on top.


Expected results:

Thread should be shown on top of the list, because the received date of the last message in the thread is newer.

Comment 1

[Jorg K (CEST = GMT+2)]

Attachment: Sorting by date in thread pane

I've never understood how sorting by date in the thread pane is supposed to work, so perhaps this is the bug where I will learn it.

As you can see, the treads are not sorted by the date shown.

What seems to happen is that the sorting is by the date of the last message in the thread, whose date is not visible if the thread is collapsed. That's slightly confusing, but correct. Aceman, can you confirm.

Reporter: Can you attach an annotated screenshot to show what you mean.

This behaviour hasn't changed and no one else has complained so far.

Comment 2

[:aceman]

Yes, I can see that. However I do not use threads so I will not try to decide what the correct/non-confusing behaviour should be.

Comment 3

[Jorg K (CEST = GMT+2)]

<off topic>
You don't use threads?? I thought all the hip people use threads to be more like Gmail ;-) I only use them in folders where I put mailing list content; threads help with getting order into that. I also use threads in my "Bugzilla" folder to group all messages re. a certain bug. Try it one day ;-). Hitting \ collapses the threads and then you have one bug per line, that's not bad.
</off topic>

Let's ask Magnus (the question is in comment #1).

Comment 4

[V@no]

I think the problem is not related to unified folder vs original folder, but rather then sorting by received date is incorrectly sorting threads (NOT messages inside of the thread)

As I understood is that thread (collapsed or not) uses first message's received date for sorting, even when ordered descending, which is wrong, because in descending order it should use last message's received date.

So if you received first message 2 years ago and last message today, the thread will not be on top of the list of all other messages, but at the bottom. When new message arrives you have to scroll and hunt for the thread with it, instead of having the thread on top.
If sorted by date - the thread moves to the top as it should.

Comment 5

[Jorg K (CEST = GMT+2)]

OK, I've changed the summary. Magnus is our "thread man", let's wait for his comment.

Comment 6

[:aceman]

(In reply to Jorg K (GMT+1) from comment #3)
> <off topic>
> You don't use threads?? I thought all the hip people use threads to be more
> like Gmail ;-) I only use them in folders where I put mailing list content;
> threads help with getting order into that. I also use threads in my
> "Bugzilla" folder to group all messages re. a certain bug. Try it one day
> ;-). Hitting \ collapses the threads and then you have one bug per line,
> that's not bad.
> </off topic>

The really hip people do not use Gmail. Also, Gmail (and other Google webapps) interface is totally unreadable crap for me.
Anyway, you can also use quick filter when you want to group bugzilla messages :)

Comment 7

[Jorg K (CEST = GMT+2)]

I don't understand your last paragraph. There should be any great difference between sorting be "Date" and sorting be "Received". The difference should be a couple of minutes or less in most cases, depending on the transmission time of the message.
> If sorted by date - the thread moves to the top as it should.
Sure, it you don't sort be date, you need to go hunting.

Can we please use "most recent" or "newest" instead of "last".

Let's look at examples.

Sort descending, newest on top:

Thread 1   - 03/03/2016
  Thread 1 - today at 15:29
Thread 2   - 17/03/2016
  Thread 2 - today at 15:25.

That seems right, Thread 1 is more up-to-date since it contains a message which is more recent (15:29) than thread 2 (15:25).

Now I sort ascending instead, newest at the bottom.

Thread 2   - 17/03/2016
  Thread 2 - today at 15:25.
Thread 1   - 03/03/2016
  Thread 1 - today at 15:29

This time the thread with the most recent message goes to the bottom.

So as far as I can see, the "date of the thread" is the date of the *most* recent message in the thread.

(And in reply to Aceman: I dislike Gmail, but threads are useful ;-))

Comment 8

[V@no]

Attachment: Sorted by date

Here is screenshot of sorted by received, as you can see it uses newest message date for the order.

Comment 9

[V@no]

Attachment: Sorted by date

Damn it, I need coffee...can't delete my previous comment/attachment.
Lets try this one more time
On this screenshot it sorted by date, the thread uses newest message date for the order. - that is correct.

Comment 10

[V@no]

Attachment: Sorted by received

On this screenshot it sorted by received, the thread uses oldest message for the order. - that is wrong.

Comment 11

[Jorg K (CEST = GMT+2)]

(You can obsolete attachments, and you can tag comments as "obsolete")

Now, let's see:
Attachment 8732529 [details]: Sorted by Date descending, looks correct.
Attachment 8732530 [details]: OK, now I finally see what you're saying.
Why didn't you attach the screen shot when you logged the bug ;-)

So when you sort by "Received", the thread is positioned according to the first message in the thread, the oldest one, so here: Thu 0:03. It should instead use Fri 1:04, the most recent message in the thread like it does for dates.

Right?

I guess that's a bug ;-)

Comment 12

[Jorg K (CEST = GMT+2)]

Attachment: Sorted by received, works for me

OK, here I am sorting by received. It gets sorted by the date of the most recent message in the thread. If there is no thread, it gets sorted by received of the "singleton" message.

Wouldn't you say that this is correct? Am I missing something?

Comment 13

[Jorg K (CEST = GMT+2)]

Take a look at bug 254159, it has a long discussion about this.

Comment 14

[V@no]

Hmmm it seems it's indeed related to unified folder after all as I originally thought.


What I'm getting right now is inconsistency in sorting in unified folder when order is changed in received column (non-unified folder sorts/order correctly as per your screenshot)

So what I'm getting is when order is descending, changing it to ascending and then again to descending the thread sortes by most recent message. Changing order again to ascending and then back to descending sorts thread by oldest message. Repeat changing order will sort thread by most recent message again. And there it goes in the loop like that.

If it makes any difference I'm using IMAP servers.

Comment 15

[Jorg K (CEST = GMT+2)]

Confirmed on a local unified folder. When sorting by "Received" the sort order is wrong every second time.

Screen shot coming.

Comment 16

[Jorg K (CEST = GMT+2)]

Attachment: Sorted by received doesn't work on unified folder.

Once it works, and then it doesn't. Puzzling!

Comment 17

[Jorg K (CEST = GMT+2)]

Aceman, we have defined what is "right": When sorting by date or received, the sort order of a thread is the date or received of the most recent message. This is the current accepted behaviour (ignoring the discussion in bug 254159).

The bug reported here is the following: When you have a *unified* folder and you change the sort order by received from descending to ascending and back, and to ascending again and back, you get different orders in the "descending" state (and maybe in the "ascending" state also, we haven't looked). On time the order is correct and another time the order is wrong, see attachment 8732537 [details].

Would you be interested in looking into it? Surely there is a line missing somewhere that does a reset on something ;-)

Comment 18

[Wayne Mery (:wsmwk)]

(not good idea to have confirmed bug reports in Untriaged component)

Comment 19

[Harry]

I'm surprised that such a serious bug has gone unfixed for two years.

As the others I have sort order set to *received* *descending* and view set to *threaded*

What I'm seeing is that the sort order of my unified inbox is only correct if I toggle threaded *off* then back on again.

If I do not toggle the threading, then:

* new incoming messages unrelated to a thread appear correctly at the top
* new incoming messages related to an existing thread get positioned correctly within that thread, but the thread stays where it was, months or even years down. 

This is quite a serious problem because it means I am unlikely to see replies to threads that are more than a day or two old unless I remember to toggle off the threaded view and toggle it back on. A user who is unaware of this bug is unlikely to do this, and would therefore never see a substantial number of new incoming messages.

This is with thunderbird 52.4.0 32 bit.

Comment 20

[Irv Spalten]

Same problem here!!!

On TB 52.8.0.

Why not fixed? I am losing mail sent to be as I do NOT see the replies.

I have the ORIGINAL message I sent a replt to, and it might be 3 days ago. A reply comes in for my reply and folds in underneath that original message I replied to, but it is OFF the screen due to the length of time between the original I replied to. I can get the SORT to be correct by going to the menu bar and VIEW and SORT and selecting another sort and then clicking Received. Then the message list is SORTED properly. I have to do this EVERY time or I will lose sight on new replies.

The reason I have to use this sort is due to my ISP poor handling of e-mails. I have Spectrum that was Brighthouse. They use RoadRunner for e-mail. However it first goes to the old Time Warner servers that Spectrum also owns and then sent to the servers with my ID on it. The DELAYS can be significant, especially for e-mail with the same CLF.RR.COM domain. Sometimes over 48 hours. So sorting on SENT also keeps the NEW e-mail way down the list and off-screen. At least with RECEIVED sort the original is on top (sort so the newest is on the top)and I get to see it no matter when sent. However this bug does cause me to MISS replies. PLEASE FIX...

Comment 21

[Wayne Mery (:wsmwk)]

Irv, Harry, can you reproduce what is described in comment 17?

Comment 22

[Irv Spalten]

It still does happen.

I know this because I use MAILWASHER PRO that sees the incoming email. It sorts out the junk and they starts TB after is deletes 'spam/unwanted' emails. I can see the replies back and to be honest, I can read them in it, and even reply, but I wait for it to come into TB to reply.

Of course that specific email that causes this problem in TB is put under the original, with a '>' in front of the original, but is DOES NOT move the original up to the top where the reply time should be.

Now IF I had deleted the original (to Trash), the new reply would be in the correct spot in the list. All this is the list is threaded of course.

Since I use MailWasher though I know to look for it. If I didn't have that, I'd probably never see the replies and I'd never think to look for them.

This report is really old, I've learned to 'live' with it.

Comment 23

[Harry]

Irv, Harry, can you reproduce what is described in comment 17?

Yes I can. It's a bag of worms ... but that on its own is not very helpful so I've tried to come up with a logical explanation.

The problem seems to be very much related to threaded view: there is a very obvious (and more easily explained) difference in the way threaded messages are sorted, compared with unthreaded view:

Here's a comparison:

• on the left is my default unthreaded view for my unified inbox, as seen when initially opening thunderbird
• in the centre is the result of toggling to threaded view
• on the right is the result of expanding all of the visible threads in the threaded view

Hmm, I don't seem to have permission to post images or attachments. There is more to say, but it will make no sense without the image.

Comment 24

[Harry]

Attachment: TBsortOrder

Belongs with https://bugzilla.mozilla.org/attachment.cgi?bugid=1257826

Comment 25

[Harry]

Comment 26

[Harry]

Attachment: TBsortOrder2

Compare threaded view on initial opening with result after turning threading off and back on again

Comment 27

[Harry]

Attachment: TBsortOrder2 replacement

Updated version of TBsortOrder2

Comment 28

[Harry]

To continue (see attachment first):

The three columns show the three different views of the topmost messages shown in my unified inbox.

I've highlighted two threads in green an yellow. There is one problem in both of them, which may not be immediately obvious, but I'll come back to that later.

I show three columns in the first image, but the full explanation needs a fourth column. For that, see https://bug1257826.bmoattachments.org/attachment.cgi?id=9467294

In that, the **Threaded **column is a repeat of the initial view when starting thunderbird and the other column is the result of turning threading off and back on again.

Both columns are the very top of my unified inbox so they should both be showing the newest messages. But here there's a major problem:

• although the blue thread still appears, and contains the absolute newest message in the inbox, it's demoted to the bottom of the screenshot between two much older messages
• the yellow thread is nowhere to be seen (actually it's buried another screen down, where I'm never likely to see it)

Looking back, there's a clue in that Initial Threaded view : although it has correctly positioned the thread for the newest message, it's showing the timestamp of the oldest message in that thread.

It seems to me that two essential changes are needed:

• when viewing as threads, the position of the thread in the messages panel should always be determined by the newest message in that thread, and
• when viewing threads, any thread which contains unread messages should always be expanded to show at least the unread messages, in bold.

Comment 29

[Harry]

[Updated the image -- use https://bugzilla.mozilla.org/show_bug.cgi?id=1257826 for the second comparison]

Comment 30

[Harry]

Aceman, we have defined what is "right": When sorting by date or received, the sort order of a thread is the date or received of the most recent message. This is the current accepted behaviour (ignoring the discussion in bug 254159).

In case it's not clear from my responses above (slightly confused by bugzilla's inability to paste images inline into the correct position in bug reports) the sort order in my tests above show that when viewing in descending order at least, threads are currently being sorted according to their oldest message, not the most recent message.

Comment 31

[Irv Spalten]

I, like Harry, understand the situation of where the reply goes, and that the primary initial email never moves based on when it was sent, and ALL NEW replies to it goes under (in the list) the original email and never moves up in the list. This makes it almost guaranteed not to be seen if the time between the initial and next email is more than a few minutes or so (like an email sent to many and replies to it are quick), however, long running emails between me and one other person, due to delays in replying will fall under the initial email well down in the list and not noticed as new.

I can see the way some might like it though and the problem it would create? Why is an OLD email up at the top of the list? Not noticing the little < on the left side indicating it is a thread?

There are many ways to mitigate this. All of course depending on how the uses had the email display choice, but the problem basically is for users with closed threading.

• Bring the thread up into the list of emails only looking at the last email in the thread. Again, this will show dates out of order. However, force an open of the thread and it would be clear to the user why the original email was up in the list.
• Bring the entire thread up to the proper spot depending on the LAST email, but MARK the thread (if closed) with either BOLD or ITALIC indicating a new reply in it.
• Duplicate the email, the reply up top in sort order, and also below under the original email, and this could/should be an optional choice under VIEW THREADING probably.

Still, to me, the real problem is SEEING a reply to an email when the old initial email is fairly old and down the list of view with the new emails.

Comment 32

[Hartmut Welpmann [:welpy-cw]]

Attachment: Bug 1257826 - Fix Sort by Received for search db views. r=#thunderbird-reviewers

When a message header is added to a threaded search db view, its thread is moved to the right place if the view is sorted by date and the message is the newest in that thread. The same happens if the sort order is changed, since the view is rebuilt in this case. Unfortunately this mechanism wasn't enabled when sorting by date received.

Comment 33

[Irv Spalten]

(In reply to Hartmut Welpmann [:welpy-cw] from comment #32)

Created attachment 9467839 [details]
Bug 1257826 - Fix Sort by Received for search db views. r=#thunderbird-reviewers

When a message header is added to a threaded search db view, its thread is moved to the right place if the view is sorted by date and the message is the newest in that thread. The same happens if the sort order is changed, since the view is rebuilt in this case. Unfortunately this mechanism wasn't enabled when sorting by date received.

================

Well, I use Date Received as I've gotten too many where the clock was wrong on the sender's PC or server where it was auto-generated, causing me NOT to see the email.

So the code should be added to Date Received as a 'quick fix', no? Most of the time the difference between the 2 columns is no different, but there are some minutes to over and hour apart, and I've seen some a day or more, and a few random ones YEARS different.

Comment 34

[Harry]

(In reply to Irv Spalten from comment #33)

Well, I use Date Received as I've gotten too many where the clock was wrong on the sender's PC or server where it was auto-generated, causing me NOT to see the email.

So the code should be added to Date Received as a 'quick fix', no? Most of the time the difference between the 2 columns is no different, but there are some minutes to over and hour apart, and I've seen some a day or more, and a few random ones YEARS different.

Even assuming the every sender's clocks are correct, Date Received is the safer sort order -- because email can be delayed by hours or even days in transit, especially if the receiving ISP's mail server is overloaded with a lot of spam.

Comment 35

[Irv Spalten]

(In reply to Harry from comment #34)

Even assuming the every sender's clocks are correct, Date Received is the safer sort order -- because email can be delayed by hours or even days in transit, especially if the receiving ISP's mail server is overloaded with a lot of spam.

Oh, I agree. I've used the CTRL-U to look at the source.

Here is one from Office Depot... TB lists it as Received @ 2/19/2025 11:23PM and Sent @ 2/18/2025 2:23PM. 1 day and 9 hour difference.

In the SOURCE I see this for SENT --> Date: Tue, 18 Feb 2025 19:23:00 +0000
I see this as RECEIVED --> Thu, 20 Feb 2025 04:23:03 +0000

The email itself contains "Date: Tue, 18 Feb 2025 19:23:00 +0000" as the date, but it was sent later it seems.

My timezone offset is -5 so RECEIVED is correct. SENT as well. So the delay was somewhere in between it seems.

It appears my ISP got it directly from Office Depot as well. No intermediately servers listed?

======================
Received: from p-impin014.msg.pkvw.co.charter.net ([47.43.26.155])
by p-mtain002.msg.pkvw.co.charter.net
(InterMail vM.9.01.00.037.1 201-2473-137-122-172) with ESMTP
id <[e-mail]>
for <ispalten@xxx>; Thu, 20 Feb 2025 04:23:04 +0000
Received: from r211.em.officedepot.com ([20.57.66.211])
by cmsmtp with ESMTP
id ky5XtEXluyc4Uky5Xt942a; Thu, 20 Feb 2025 04:23:03 +0000
X-ORIG-RCPT: ispalten@xxx

Comment 36

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/022d28b4be01
Fix Sort by Received for search db views. r=tobyp