263678 - Sort by Date, Grouped by Sort results in wrong partitioning of messages into groups

Status: RESOLVED WORKSFORME

(Thunderbird :: Mail Window Front End, defect)

Description

[Ben]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; rv:1.7.3) Gecko/20041001 Firefox/0.10.1
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; rv:1.7.3) Gecko/20041001 Firefox/0.10.1

Setting Sort by Date, Descending, Grouped by Sort.
Today is Oct 10 (important to understand the following lines) and I see the
following in my INBOX:

Today
01:52 mail....
Last Week
04.10.2004 mail ....  [should be This Week?]
Yesterday
08.10.2004 mail...   [should be Day Before Yesterday or This Week]
Two Weeks Ago
01.10.2004

It seems that there is a group missing: Where does a date between Yesterday and
Last Week go? How is Last Week defined? Is this the current week or the week
before the current week? Shouldn't that be 

Today
Yesterday
This Week = dates between Yesterday and Last Week      !!! missing !!!
Last Week
Two Weeks Ago  [IMO this could be dropped and corresponding mails grouped in Older]
Older 

Reproducible: Always
Steps to Reproduce:
1.
2.
3.




It seems that there is a group missing: Where does a date between Yesterday and
Last Week go? How is Last Week defined? is this the current week or the week
before the current week? Shouldn't that be 

Today
Yesterday
This Week !!! missing !!!
Last Week
Two Weeks Ago  [IMO this could be dropped and corresponding mails grouped in Older]
Older

Comment 1

[David :Bienvenu]

today is today (the current day, not the last 24 hours, if that makes sense)
last week is really the current week, or messages that arrived in the last week,
except for messages that arrived today.
2 weeks are message the remaining messages that arrived in the last two weeks,
that don't fall into the first two categories.

Comment 2

[David :Bienvenu]

is this bug basically about changing the labels? I think the labels I used are
similar to what Outlook uses...

Comment 3

[Ben]

Attachment: Comparison Outlook/TB

Comment 4

[Ben]

In reply:

Problem of terminology I guess, but not very intuitive.
Let's take an example: Today = Wednesday
- Messages dated Tuesday (the previous day) will be grouped in Yesterday
- Messages dated Monday (the day before yesterday) will be grouped in Last
Week??? Shouldn't that be Current Week?
Messages dated Friday of the previous week will be Two Weeks Ago? Although this
is in fact the Previous Week.

Enclosed is a screenshot comparing with Outlook. You will notice that TB puts
04.10.2004 into Two Weeks Ago. I noticed that TB uses the Sent date for
grouping, but displays the Received date. That's confusing. 

Outlook uses (in descending order):
Today  eg. let's say onThursday
Yesterday eg. Wednesday
Tuesday
Monday
...
Last Week

Comment 5

[Ben]

(In reply to comment #4)
Also, the sort order within the groups should correspond to the group sort
order, i.e. descending in the example shown.

Comment 6

[Ben]

(In reply to comment #2)
> is this bug basically about changing the labels? I think the labels I used are
> similar to what Outlook uses...

Not really, more a question of definition. For me, Last Week is the week which
ends witht the past weekend. It seems to me that that between Yesterday and Last
Week, something like Current Week is missing. Outlook groups the current week
into its weekdays: Today (eg. Thursday), Yesterday, Wednesday, Tuesday,..., Last
Week (in descending order). I'd rather prefer:
Today, Yesterday, Current Week, Last Week

Comment 7

[Ben]

(In reply to comment #2)
> is this bug basically about changing the labels? I think the labels I used are
> similar to what Outlook uses...

The attached screenshot shows the confusion: While 07.10 to 10.10 (green)
rightly belong to Last Week, 11.10 (yellow)doesn't fit and should be listed
under a new group labeled This Week.

Comment 8

[Ben]

Attachment: Last Week contains element from the current week

Comment 9

[Ben]

(In reply to comment #8)
> Created an attachment (id=161956)
> Last Week contains element from the current week
> 

Assuming that "Week" is defined as "from Monday to Sunday"

Comment 10

[David :Bienvenu]

Attachment: fix some miscategorization of messages

I switched to 32 bit int math, and adjusted for GMT. I also fixed a leak,
removed some unused code, and fixed a problem where headers were out of date
order in the underlying folder.

Comment 11

[David :Bienvenu]

I've fixed the categorization to do what I intended, but that's not what you
wanted, I don't think...

It's true we're not doing what Outlook does. Outlook does:

today
yesterday
Tuesday
Monday
Sunday
Last Week (which is previous Sunday - Saturday)

I'm not sure what Outook does if today is Monday - does it say:

today
yesterday
Last Week?

or if today is Sunday, does it say:

today
yesterday
Friday
...
Monday
Sunday
Last Week?

I could try to duplicate that - it would be a little bit of work (I'm not even
sure how to figure out what ordinal day of the week it is with our date/time
library...). What do you think, Scott?

Comment 12

[Ben]

(In reply to comment #11)
> I've fixed the categorization to do what I intended, but that's not what you
> wanted, I don't think...
> 
> It's true we're not doing what Outlook does. Outlook does:
> 
> today
> yesterday
> Tuesday
> Monday
> Sunday
> Last Week (which is previous Sunday - Saturday)
> 
> I'm not sure what Outook does if today is Monday - does it say:
> 
> today
> yesterday
> Last Week?
> 

I don't know either, you would have to wait until next Monday when I can try.


> I could try to duplicate that - it would be a little bit of work (I'm not even
> sure how to figure out what ordinal day of the week it is with our date/time
> library...). What do you think, Scott?

IMO, I don't think this is too useful, something like Current Week should do for
the days preceding Yesterday back until Sunday of the current week. It's
probably not going to be much easier, because you would have to take into
consideration that the week starts on Monday in Europe, while in the US it
starts on Sunday. Sorry about that ;-). I hope there is no country where the
week starts on Saturday....

Comment 13

[Ben]

Attachment: Branch 2004-10-17 Wrong order

The attached screenshot still shows the wrong order. In addition, the
individual dates within each group should be sorted in descending order.

Comment 14

[David :Bienvenu]

that's a separate issue, having to do with the order of the groups themselves,
as opposed to which buckets each message goes into. I've seen that problem too,
but don't have a reproducible case.

Comment 15

[Ben]

(In reply to comment #14)
> that's a separate issue, having to do with the order of the groups themselves,
> as opposed to which buckets each message goes into. I've seen that problem too,
> but don't have a reproducible case.

I have also noticed an error in the submenu (see screenshot) "Grouped by Sort G".

Branch 2004-10-17 crashes when successively and randomly selecting messages in
different groups. I don't know if this is related.

Comment 16

[Ben]

> Branch 2004-10-17 crashes when successively and randomly selecting messages in
> different groups. I don't know if this is related.

Seems to be resolved after restart of system.

Comment 17

[Ben]

Attachment: 2004-10-17 correct order

You might be interested in the fact that the same mail account which showed the
wrong sort order on Sunday shows the correct sort order on Monday. See
screenshot.

Comment 18

[David :Bienvenu]

Attachment: fix sorting of date categories

we were sorting the date categories by the date of the newest message in the
thread containing the first message to be grouped instead of the date of the
message itself - that usually worked out to be right, but not always...

Comment 19

[Markus Faßbender]

I have my mails order by date descending. If I select "group by sort" than TB
group the mails and order it ascending in the groups.

Comment 20

[Mike Cowperthwaite]

(In reply to comment #17)
> Created an attachment (id=162453)
> 2004-10-17  correct order
> 
> You might be interested in the fact that the same mail account which showed
> the wrong sort order on Sunday shows the correct sort order on Monday. See
> screenshot.

Um, no; that screenshot continues to show the groups ordered by Date, Descending 
while the messages within the group are sorted by Date, Ascending.


(In reply to comment #14)
> that's a separate issue, having to do with the order of the groups themselves,
> as opposed to which buckets each message goes into. I've seen that problem
> too, but don't have a reproducible case.

Groups by Date, Descending is the only case where this seems to be actually a 
problem.  Bug 267754 has been opened about this issue.

Comment 21

[Mike Cowperthwaite]

*** Bug 271179 has been marked as a duplicate of this bug. ***

Comment 22

[sairuh (rarely reading bugmail)]

using 200411220x-0.9 tbird bits (the three main platforms), message grouping by
descending date (most recent at top of thread pane) is now:

- today (0 days old)
- yesterday (1 day old)
- last week (2-6 days old)
- two weeks ago (7-13 days old)
- old mail (14+ days old)

if this is expected now, I'll verify this bug as fixed.

Comment 23

[Duncan Miller]

Except that doesn't address the problem of the sort order of messages within the
group not honouring the selected order. eg Date, Descending applies to the
grouping but the messages within the groups always appear as date Ascending.

Comment 24

[Markus Faßbender]

(In reply to comment #23)
> Except that doesn't address the problem of the sort order of messages within the
> group not honouring the selected order. eg Date, Descending applies to the
> grouping but the messages within the groups always appear as date Ascending.

But this wasn't the error described in this bug. 
For your error see bug 267754

Comment 25

[Ben]

(In reply to comment #24)
> (In reply to comment #23)
> > Except that doesn't address the problem of the sort order of messages within the
> > group not honouring the selected order. eg Date, Descending applies to the
> > grouping but the messages within the groups always appear as date Ascending.
> 
> But this wasn't the error described in this bug. 
> For your error see bug 267754

It was, though: "Grouped by Sort results in wrong order of groups *and* mails".
Whatever, just make sure one of the two bug reports 263678 or 267754 take care
of it.

Comment 26

[Mike Cowperthwaite]

Attachment: Screenshot showing miscategorization

Some messages are still not being put in the right buckets on group-by-date.
The attached image shows a TB 1.0 screenshot taken about five minutes ago
(that is, about 1:00PM locally, 9-Dec-2004).  Note one 3-Dec message appearing
in the "two weeks ago" and two 3-Dec messages appearing in the "last week"
bucket.  The reason the one message is shown in "two weeks ago" is because the
actual Date header is:
  Thu, 2 Dec 2004 23:05:59 -0800

I'd say the timezones should be taken into account when making these partitions
-- the dates displayed are relative to my timezone, and the partioning should
follow those.  (Altho ISTR there is a hidden preference that allows the date
header to be displayed as received without adjusting; and there is the bug for
displaying mail by Date Received instead of the Date header...)

Comment 27

[Mike Cowperthwaite]

Updating summary to avoid further confusion with bug 267754.

Comment 28

[Mike Cowperthwaite]

*** Bug 273472 has been marked as a duplicate of this bug. ***

Comment 29

[Duncan Miller]

Attachment: view of inbox sorted by

As new emails come overnight a new group is created for the AM ones but is
always named Yesterday instead of Today (see attachment). As soon as I refresh
the list, by a re-sort or just moving to another folder and back again, it is
corrected.

Comment 30

[David :Bienvenu]

if you leave a folder open for hours, we don't migrate messages from today to
yesterday, when the day changes. Would that explain the issues people are seeing?

Comment 31

[Duncan Miller]

Not really. It appears that messages are going to Yesterday as appropriate and
new messages (todays) get a new group but the group is named Yesterday.
Otherwise the behaviour seems correct.

The attachment shows how I found it first thing this morning. the lower group
named Yesterday does contain yesterdays mail. The top group though is actually
todays mail.

So it is really just that I have to two groups named Yesterday.

Comment 32

[Mike Cowperthwaite]

(In reply to comment #30)
> if you leave a folder open for hours, we don't migrate messages from today to
> yesterday, when the day changes. Would that explain the issues people are
> seeing?

No, not in my case -- I don't normally use grouped mode, I was just testing 
something when I saw that symptom and remembered this bug.

Comment 33

[Mike Cowperthwaite]

OK, ran into another situation with this.  Unfortunately, I didn't grab a 
screenshot during the critical period.

I had left the Inbox in Group, By Date overnight (computer on standby) and when 
I dialed in this morning, a few messages arrived, most actually dated yesterday.
I looked at the grouping to see how they'd fared; I didn't see any "Today" group 
(because the Today messages were both spam) but in the Yesterday, one of the 
older (read) messages had a twisty, as if there was threading going on.  I 
opened the twisty and several more messages became visible, altho none shared 
the same sender or subject.

Unfortunately, I did not get a screenshot at this point -- however, my 
recollection is that this pseudo-thread of messages consisted of messages I 
received two days ago -- that is, by this point, they ideally would have been in 
the "last week" bin.  (I'm not complaining that they weren't automatically moved 
to that bin, just that the display got weird.)

However, the current behavior is definitely wrong: all the messages from 
yesterday (5) *and* two days ago (3) appear in the "Yesterday" group -- after 
not only reselecting the folder, not only leaving and re-entering group mode, 
but even after exiting, deleting the .MSF file, restarting TB and re-grouping.

Lately arrived new mail appears in the Today group as expected.

Comment 34

[Vernon Squire]

*** Bug 279159 has been marked as a duplicate of this bug. ***

Comment 35

[Steven Grimm]

I see the behavior described in comment #33 almost daily; I usually leave
Thunderbird open at night and in the morning, it typically seems very confused
about where it wants to put messages. It does not seem 100% consistent. Mail
from today sometimes shows up in "Yesterday" or vice versa, and sometimes
there's a "Today" group with no contents even though there's mail from today in
the "Yesterday" group. When I delete one of the new messages, Thunderbird
sometimes fails to advance to the next message.

For me, though, switching to another folder then back to my inbox fixes the
problem, and I've gotten in the habit of always doing that first thing in the
morning to make the groupings correct.

Since I'm doing that anyway, it'd sure be nice if Thunderbird would save me the
effort and just recompute the groups when it gets the first new message on a
given day, or maybe at the stroke of midnight. I wouldn't mind things shuffling
around onscreen -- I'd rather the display always show the correct grouping than
try to remain consistent at the cost of putting messages in the wrong place. In
fact, it'd be kind of reassuring to see messages migrate from "Yesterday" to
"Last Week" on their own.

I guess the thought experiment is, if you left Thunderbird open while you went
on vacation for two weeks, would you be satisfied seeing week-and-a-half-old
mail piled up in the "Today" group when you got back? That's probably what you'd
see with the current behavior.

Comment 36

[Steven Grimm]

Attachment: Two "Yesterday" groups

A message dated the previous day arrived shortly after midnight, and this is
how Thunderbird displayed it. This screenshot was taken at 12:15AM on
2005/02/04, so in theory the 2005/02/02 message ought to be in a "Last Week"
group rather than in the same "Yesterday" group as a message from the next day.
The last message in the list was, I believe, listed under "Today" until the
moment the previous message (the one selected in the screenshot) arrived
shortly after midnight, at which point the display changed to what you see
here.

Comment 37

[Steven Grimm]

Attachment: Missing header pane line for a message

Here's another one I've seen several times. My header pane had a bunch of
messages in "Old Mail" and one message each in "Two Weeks Ago", "Last Week",
and "Today". I read the "Today" message and deleted it. Thunderbird deleted the
message and the message pane displayed the "Last Week" message, as expected.
However, that message's entry in the header pane vanished, leaving an expanded
but empty "Last Week" group heading, as you can see in the screenshot. When I
collapse that group heading by clicking on its "-" button, it vanishes too, as
does the message in the message pane. Switching to a different folder and back
again makes everything look like it should.

Comment 38

[Ulf]

(In reply to comment #37)
I have a similar failure.
I had my emails sorted by "date" and "descending" and group by sorting.
Today I got a spammail with the date 01.01.1970 (dd.mm.yyyy) it was right sorted
at the end of my main window.
And a other email from yesterday (I download them all today, not yesterday) was
always false sorted. With a group (I think also a second one) "yesterday" at the
end of the main window.

And that is not a descending sorting.
I will attach now a picture explaining it (similar to the other attached one).

Comment 39

[Ulf]

(In reply to comment #38)
> (In reply to comment #37)
> I have a similar failure.
> I had my emails sorted by "date" and "descending" and group by sorting.
> Today I got a spammail with the date 01.01.1970 (dd.mm.yyyy) it was right sorted
> at the end of my main window.
> And a other email from yesterday (I download them all today, not yesterday) was
> always false sorted. With a group (I think also a second one) "yesterday" at the
> end of the main window.
> 
> And that is not a descending sorting.
> I will attach now a picture explaining it (similar to the other attached one).

Sorry ... I forgot to say:
I use Win XP + SP2 (with the newest patches and updates) with Thunderbird 1.0.6
with this extensions:
Enigmail, AttachmentToosl, ContactsSidebar, HeaderScrollExtension, Quote
Collapse, QuoteCOllors, Remove Duplicate Messages, Sender Verification, Show
SMTP Username and Minimize to tray.

Comment 40

[Ulf]

Attachment: Shows the false sorting of one email in the main window

Shows the false sorting by Thunderbird 1.0.6 in the main window (I explained it
in a answer at the bottom of the site). Ulf: [e-mail]

Comment 41

[Markus Faßbender]

(In reply to comment #40)
> Created an attachment (id=192828) [edit]
> Shows the false sorting of one email in the main window
> 
> Shows the false sorting by Thunderbird 1.0.6 in the main window (I explained it
> in a answer at the bottom of the site). Ulf: [e-mail]

I saw this behavior, too. But it's very difficult to reproduce and it only
occurs rarely

Comment 42

[David :Bienvenu]

there are several fixes for problems like this in trunk builds...and an other
fix will go in tomorrow.

Comment 43

[Markus Faßbender]

(In reply to comment #42)
> there are several fixes for problems like this in trunk builds...and an other
> fix will go in tomorrow.

Maybe it work with the trunk. I use the trunk since 2 or 3 weeks and I didn't
saw this problem again. But before it appears not often.

Comment 44

[Simon]

Attachment: Date sort / group bug with mockup of ideal behaviour

Still problems, both with getting the date bucket right and also what the buckets should be. Using 1.0.7 (20050923).

Comment 45

[David :Bienvenu]

has anyone tried a 1.5beta 2 or rc1 build?

Comment 46

[Markus Faßbender]

(In reply to comment #45)
> has anyone tried a 1.5beta 2 or rc1 build?
> 

Yes, I use the trunk respectively beta 1 and beta 2 since august and I couldn't reproduce the problem anymore.

Comment 47

[David :Bienvenu]

ok, thx, marking wfm.

Comment 48

[Seth McFarland]

The date grouping appears to be working correctly for me, too (1.5 Beta 2 under Windows XP) but I still think the date grouping labels are confusing.  Specifically "Last Week" and "Two Weeks Ago".

"Last Week" implies an email that was received in the previous calendar week, rather than in the past 7 days.  Something more like "Within the last week" or "Within the last 7 days" seems more appropriate.  The same goes with "Two Weeks Ago". 

I don't particularly like my ideas alternatives, but I think the current labels should be tweaked a bit to be clearer.  What do you guys think?

Comment 49

[Mike Cowperthwaite]

(In reply to comment #48)
> The date grouping appears to be working correctly for me, too ...
> but I still think the date grouping labels are confusing. 
> Specifically "Last Week" and "Two Weeks Ago".
> 
> "Last Week" implies an email that was received in the previous calendar week,
> rather than in the past 7 days.  Something more like "Within the last week"
> or "Within the last 7 days" seems more appropriate.  The same goes with "Two
> Weeks Ago". 

Bug 315114