Closed Bug 166254 Opened 22 years ago Closed 17 years ago

Received date field

Categories

(MailNews Core :: Backend, enhancement)

Product:
MailNews Core
Component:
Backend
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED
Milestone:
mozilla1.9alpha8

People

(Reporter: james, Assigned: Bienvenu)

References

(Depends on 1 open bug, Blocks 3 open bugs)

Details

(Keywords: fixed1.8.1.5)

Attachments

(5 files, 24 obsolete files)

proposed fix
5.89 KB, patch
mscott
: superreview+
Details | Diff | Splinter Review
only used received: hdr if no Date: header or user sets hidden pref
4.21 KB, patch
mscott
: superreview+
mscott
: approval1.8.1.5+
Details | Diff | Splinter Review
fix toggling of received column
10.46 KB, patch
Details | Diff | Splinter Review
New Received column hiding code
1.47 KB, text/plain
Details
v4, with SearchDialog.js related changes as proposed in comment #241 [checked in]
103.66 KB, patch
mnyromyr
: review+
Bienvenu
: superreview+
Details | Diff | Splinter Review 
'Sent date' is not a good field for ordering mail by. The problem is, if the
sender has the incorrect date set, or is in a different timezone, finding the
unread mail in a folder requires scrolling up and down to find it.

Having a 'Received date' field as well as the 'Sent' date field would alleviate
this problem entirely.
Use: View->Sort by->Order received

duping to bug 72463 for the Received date field

*** This bug has been marked as a duplicate of 72463 ***
Status: UNCONFIRMED → RESOLVED
Closed: 22 years ago
Resolution: --- → DUPLICATE
verified as duplicate
Status: RESOLVED → VERIFIED
reopening - bug 72463 has now been fixed, so "order received" is possible, but
this bug is requesting a field for "date received", which isn't the same. (I
guess this would need a back end change as well, but I'm not sure...)
Status: VERIFIED → UNCONFIRMED
OS: Windows 2000 → All
Hardware: PC → All
Resolution: DUPLICATE → ---
-> NEW for consideration
Status: UNCONFIRMED → NEW
Ever confirmed: true
Dupe of 123786?
possibly - depends what happens to that bug.

based on the original report, bug 123786 is just a dupe of bug 72463. but
further comments imply an actual received date, in which case this a dupe of
that bug.

either that bug should be duped to the original 72463, or that bug should be
clearly changed to be a request for an actual received date field and this bug
should be duped to it.
*** Bug 200955 has been marked as a duplicate of this bug. ***
IMO the "Order received" field should be replaced by a "Date received" field
since the number in the order received field is completely irrelevant for most
end users. If the "Date received" field gets added, the current "Date" field
should be renamed to "Date sent" or at least have its tooltip changed accordingly. 

In order to make this work right even after moving between folders, it should
probably be implemented using what is suggested in bug 190337, therefore making
dependent on that bug.
Depends on: 190337
FYI, the From_ header in mozilla's mbox-format mailbox files appears to already
contain the date received, as mentioned in bug 190337.
(In reply to comment #8)
> IMO the "Order received" field should be replaced by a "Date received" field
> since the number in the order received field is completely irrelevant for most
> end users. ... 

Completely agree to this as moving a message from folder to folder changes order
(at least here using an IMAP mailbox with Cyrus mail server) which is not kept
any more in the real order of incoming so this is not a workaround I could use.
I just had a look at some mail messages' sources. It is a good idea to store he
date mozilla received a message. But it would be better to use the date one's
_mail server_ received the mail message:

Received: (qmail 25558 invoked by uid 65534); 26 Apr 2004 14:13:26 -0000

That would be a _very_ meaningful timestamp.

Maybe we should have three dates stored and displayed:

1.) Sent
2.) Received (by my mail server)
3.) Downloaded
(In reply to comment #11)

Completely agree.

...
> 
> Maybe we should have three dates stored and displayed:
> 
> 1.) Sent
> 2.) Received (by my mail server)
> 3.) Downloaded


IMAP specifies an "internal date" field which is the date received at your 
server.  This seems to be what Outlook Express uses for both sorting and 
display, and it certainly makes the most sense...
I have done a little bit of research into how to fix this bug.

While there is an IMAP to supply the date an email is received there is not one
for POP. However, according to the mail RFCs the 'Received' fields must be top
appended, not interfered with, and the receiving server must append a received
field. This means that the date received can be reliably extracted from the
topmost 'Received:' field in the email. Currently the complete 'Received:'
fields string is stored in the mail database under the token
kReferencesColumnName = "references".

So a proposal of how to fix the bug is to add a case statement to
nsMsgDBView::GetLongField for the date-received sort type where the function
returns the computed date received from the references string (easy if you know
how to work strings and time in mozilla, unfortunately I do not) [? using
functions  nsMsgHdr::ParseReferences or nsMsgHdr::GetStringReference?]. All that
is then required in a few back end tweaks to open up this sort option to
javascript (easy) and the associated front end tweaks (easy).
The IMAP date is not ideal because shares one of the problems with "order
received" - when you move a message from one folder to another, the IMAP date of
the message in the destination will be the date you moved it, not the date of
the message.

I think the date in the top-most Received: header is the one to go for.
(In reply to comment #15)
> The IMAP date is not ideal because shares one of the problems with "order
> received" - when you move a message from one folder to another, the IMAP date 
of
> the message in the destination will be the date you moved it, not the date of
> the message.

RFC 2060 says: "In the case of messages delivered by the IMAP4rev1 COPY 
command, this SHOULD be the internal date and time of the source message."

I believe Cyrus IMAP does this correctly; I'll confirm this on request if 
anyone has any doubt.

From my experiences of the chaos caused by not restoring the internal date 
correctly I'm confident that it is reliable for a "Date Received" field, though 
as mentioned this would only work for IMAP and for POP3 the topmost Received: 
header would have to be parsed.
Blocks: 216033
Product: Browser → Seamonkey
*** Bug 272065 has been marked as a duplicate of this bug. ***
I've noticed that email by some providers is missing a Date: field, in which
case Thunderbird dates the email at 12/31/1969 despite the presence of several
other keyable fields it could use for a more accurate date.  This results in
email being shoved in strange order when it's viewed.  I like the idea to switch
to received date, but also ensure that the date has a fallback to sort to.
Flags: blocking-aviary1.1?
Assignee: sspitzer → mail
Attached patch parse received: headers... (obsolete) — — Splinter Review 
this patch makes us parse the received: headers for a date, preferring the last
header, which will usually be the server closest to the sender. If we fail to
parse a Received hdr, we will use the date: header, as before.
Assignee: mail → bienvenu
Status: NEW → ASSIGNED
Attachment #199772 - Flags: superreview?(mscott)
(In reply to comment #19)
> this patch makes us parse the received: headers for a date, preferring the last
> header, which will usually be the server closest to the sender. If we fail to
> parse a Received hdr, we will use the date: header, as before.

I think that using the first date, which will be the server closest to the
_receiver_, will be more likely to be reliable as an indicator of date received.
 The server closest to the receiver is likely to be a constant factor for all
emails received, whereas the server closest to the sender will depend on the
sender and may even be spoofed.
I can do that. That was the wrong patch, anyway.
Attachment #199772 - Attachment is obsolete: true
Attachment #199772 - Flags: superreview?(mscott)
Attached patch proposed fix — — Splinter Review 
OK, use the first received: header. And if there's no Date: or Received: hdr,
use PR_Now()
Attachment #199777 - Flags: superreview?(mscott)
Attachment #199777 - Flags: superreview?(mscott) → superreview+
(In reply to comment #22)
> Created an attachment (id=199777) [edit]
> proposed fix
> 
> OK, use the first received: header. And if there's no Date: or Received: hdr,
> use PR_Now()

Sorry to bother you, I wish to know if this patch is going to be included in the
1.5 release of Tb, because it is likely to resolve another RFE (number 216033),
which has been issued over two years ago. Thanks in advance.
If I understand correctly, this will change the "date" field shown to be the
received date. From me, that would be great (this issue is pretty much the only
thing stopping us from switching at work), but I suspect there will be people
who will prefer the current behaviour of displaying and sorting by the sender's
date and will consider this fix a backwards step.

For example, a previous work email system I used collected email infrequently
and added a Received header at that point, so the received date would be one of
a couple of times each day, and you wouldn't be able to tell if a message had
been sent in the morning of the current day or the evening of the previous day.

Or I may be misunderstanding...
The original bug report asked for an _additional_ column for "Received date".
Presumably the existing header should be renamed to "Sent date" to avoid
confusion. This is what I'd like, anyway.

IMO, "Sent date" should be taken from the RFC 2822 "Date" header, and if not
present then perhaps the first (bottom-most) "Received" header. This can, of
course, be spoofed. The "Received date" should be taken from the IMAP internal
date if possible, and where not available should come from the last (topmost)
"Received" header.

Outlook Express on OS 9 does this perfectly.
(In reply to comment #24)
> If I understand correctly, this will change the "date" field shown to be the
> received date. From me, that would be great (this issue is pretty much the only
> thing stopping us from switching at work), but I suspect there will be people
> who will prefer the current behaviour of displaying and sorting by the sender's
> date and will consider this fix a backwards step.
> 
> For example, a previous work email system I used collected email infrequently
> and added a Received header at that point, so the received date would be one of
> a couple of times each day, and you wouldn't be able to tell if a message had
> been sent in the morning of the current day or the evening of the previous day.
> 
> Or I may be misunderstanding...

IMHO I think that the current behaviour presents at least one serious drawback:
if the sender has incorrect date / time set (as for a system with bad CMOS
battery), that email message, for Mozilla (and also Thunderbird) is displayed as
sent at the wrong date/time. This is ALWAYS reproducible.
(In reply to comment #26)
> IMHO I think that the current behaviour presents at least one serious drawback:
> if the sender has incorrect date / time set (as for a system with bad CMOS
> battery), that email message, for Mozilla (and also Thunderbird) is displayed as
> sent at the wrong date/time. This is ALWAYS reproducible.

The "Date sent" is always going to actually be "The date the sender claims that
the email was sent" and this is unavoidable. Therefore, if the sender's date is
wrong, then this field is going to be wrong.

If this is a problem, then we should have a "Date received" column, which is
likely to be correct, but has shortcomings as discussed. This is exactly what
this bug is about.

I think both columns should be present. I don't care which the default is;
that's a separate argument. Users can pick and choose which they want to use.
(In reply to comment #26)
> if the sender has incorrect date / time set (as for a system with bad CMOS
> battery), that email message, for Mozilla (and also Thunderbird) is displayed as
> sent at the wrong date/time. This is ALWAYS reproducible.

Yes, and this can be easily the case. I had the problem, that for example
spammers used sent-dates in the future in their mails to be on top in my inbox.

Some people might want to see the date sent also. IMHO the best is to have an
additional field for that. So you either can have both visible in the list.
Ok - my fault for starting it, but please don't add any further comments to this
discussion. The issues have been raised; there's no need for us all to discuss
it in the bug as more comments just makes it harder for the developers to find
the important information in the bug later.
David, I have some questions on your implementation/improvements.
(Q1)Which has higher priority in displaying date as "Mail date" when both exist?
    Date: header? or Received: header?
(Q2)What will be done when malformed Date: header or malformed Received: header?
    Same as No Date: header/No Received: header case?
(Q3)Is your "new mail date" displayed in current Date field in thread pane(mail
list pane)?

this only affects newly downloaded headers, or newly parsed folders (e.g., after
deleting .msf file)
Status: ASSIGNED → RESOLVED
Closed: 22 years ago → 19 years ago
Component: MailNews: Main Mail Window → MailNews: Backend
Product: Mozilla Application Suite → Core
Resolution: --- → FIXED
In what I checked in, the first Received: hdr that has a parseable date has
highest priority; failing that, the date: header, and failing that, we use the
download time as the date to display. We're only displaying and storing one
date. And that's pretty much all I intend to do with this. If anyone else wants
to do more, they're more than welcome to submit a patch.
> the first Received: hdr that has a parseable date has highest priority

Urgh, I'm appalled. I didn't expect such a "solution" which also isn't what the
reporter wanted.
Ok, in best case date/time from Received and Date headers are only seconds appart.
But I guess most users trust that the time in the date column is the one the
user composed the mail. And not few users compose the mail offline to send them
in a batch. In that case Received and date can differ hours and days.
Also now it's no more possible to see the senders time zone without looking at
the source.

I promise not few people will ask about wrong dates in support forums.
Again, and what if the system time is wrong (knowingly or not)?
I think that the solution proposed is equal to the "normal" behaviour of many
email clients (of which Outlook Express is not the last) and I don't think so
many people are complaining 'bout that.
Sure, if both fields were present, it would be better.
I'm happy to back it out, or go back to my original idea of using the last
received: header, which is going to be the one closest to the time the user sent
the e-mail. But that was shot down...
(In reply to comment #35)
> I'm happy to back it out, or go back to my original idea of using the last
> received: header, which is going to be the one closest to the time the user sent
> the e-mail. But that was shot down...

Excuse me, then maybe I haven't understood WHICH was the final patch you
proposed. I thought it was the idea of using the last received: header. Am I wrong?
what's checked in now uses the first Received: header it sees, and ignores the
others. The first Received: header seen happens to be the last header added,
since each server adds its own Received: header at the top. So it's the server
closest to the recipient whose date gets used. Presumably, that's the most
trusted server. The tradeoff is between using the most likely to be accurate
time, vs using the least likely to be spoofed time.
(In reply to comment #37)
> what's checked in now uses the first Received: header it sees, and ignores the
> others. The first Received: header seen happens to be the last header added,
> since each server adds its own Received: header at the top. So it's the server
> closest to the recipient whose date gets used. Presumably, that's the most
> trusted server. The tradeoff is between using the most likely to be accurate
> time, vs using the least likely to be spoofed time.

And so I don't understand who your comment (#35) was referred to...
Can't you include all two headers in the sorting columns list, i.e. the last AND
the first?
I don't want to make the UI and code more complicated than it already is by
adding a second date column.
> I don't want to make the UI and code more complicated than it already is by
> adding a second date column.

Well, I do understand that and despite what I wrote in comment #33 I'm thankful
someone works on this and provided some functionality for the way to fix this bug.
But IMHO letting this code be the whole fix is wrong.

I'd expected at least a pref (yes I know, another one) and a separate field
(where the priorities are first Received header, now and Date header) at best.
And for a Received date field using the first (last added) Received header is ok.
But my position is, that everything else then the Date header's value ist wrong
for a Date field.
If the Received field is used, how should the messages without that field be
dealt with? (You can have those if you place a copy in the mailbox as you send a
message.) Use the Date field instead?
David, thanks for taking this bug forward.

I for one argue for the date from the topmost received header (as explained in
comment 14, the RFC analysis being better than my then understanding of the
mozilla codebase). This date is most likely to allow new mail to arrive in order
when sorting by date.

However, I do see a good argument for a ‘date sent’ (declared by mail client)
and a ‘date received’ (topmost received header). Often large institutions have
mail server issues which lead to email to be delayed before leaving the
institution network. This means it is desirable to 1) use ‘date received’ to
sort email, 2) use ‘date sent’ while interpreting the content of the email.

Unfortunately (from what I, subsequently, inferred from the code) this entails
adding an additional mail database field.
 
Hi all,

Could we wait a couple of days, to see how the second patch looks? It just seems
like the patch is being shot down by everyone.

Thanks a lot for the patch David - it's sad, that it doesn't make the 1.5, but I
do understand why.
(In reply to comment #39)
> I don't want to make the UI and code more complicated than it already is by
> adding a second date column.

I can understand it, and I can explain to users about design/spec after your
latest patch.
But using first Received: header (last added Received: header) changes meaning
of Date column from "date in Date: header" to "arrival date at POP3 server of
mail receiver".
This will produce new confusion on term "date", in message filtering, in junk
purging, etc. (and many unwanted INVALID bugs... ;-)
This should be user choosable, should be intentional change by user, I believe.
David, what do you think? Can you add new pref.js entry and use it?
(In reply to comment #31)
> this only affects newly downloaded headers, or newly parsed folders
> (e.g., after deleting .msf file)

If folder name contains special character(e.g. "/", "#"), file name for the
folder becomes hexa-string(converted name).
In this case, deletion of ".msf" causes loss of original folder name because the
original folder name is saved in ".msf" file.
David, is there any easy way in this case?
User have to keep folder name and file name pair in a text file before deletion
of ".msf"?
User have to rename hexa-string folder name to original folder name after restart? 


I agree that displaying the date from the received headers instead of the Date:
header by default was a bad idea. So I'm going to do the following:

1. Use the Received: date if there's no Date: header (and only use the download
time if neither header exists)
2. Possibly add a hidden pref, default to false, to use the received by header
instead of the Date: header for the Date field.
(In reply to comment #46)
> I agree that displaying the date from the received headers instead of the Date:
> header by default was a bad idea. So I'm going to do the following:
I dunno - a whole heck of a lot better than nothing!  Consistently altering the
date column to mean "best guess of time recieved" ought to be fine, IMHO.  If
people really want, adding back a "best guess of time sent" column.  Frankly, I
don't want my mail sorted by sent time - recieved time is far more useful, as
long as the mail headers are left alone.

> 1. Use the Received: date if there's no Date: header (and only use the download
> time if neither header exists)
If you are going to do this, then I'd use the oldest (last) Received: date,
otherwise you are swapping the start and end of transit time, even though the
later headers are more likely to be untrustworthy and/or broken.

> 2. Possibly add a hidden pref, default to false, to use the received by header
> instead of the Date: header for the Date field.
OK, but a separate (default disabled) displayable/sortable column would be
better, IMHO.
> > 2. Possibly add a hidden pref, default to false, to use the received by header
> > instead of the Date: header for the Date field.
> OK, but a separate (default disabled) displayable/sortable column would be
> better, IMHO.

I agree, they are two separate concepts and they should have two independent
columns.
> I don't want to make the UI and code more complicated than it already is by
> adding a second date column.

I suggest that the "Order received" column is removed from the UI, which would
be completely obsolete now.

I still don't get the notion that people want the date received as close to the
sender as possible - this bug was opened to resolve that issue once and for all.
There are many legit reasons for keeping the date received as close to the
receiver as possible (people working with deadlines, across timezones etc). 

If this is too much of a change then the user interface should clearly indicate
which date is being displayed and sorted on - thus both date fields should be
selectable from the interface and this should not be hidden at all.

Again, the "order received" field can be removed now and thus this would not
"net" add options, only add a clearly understandable one in place for a
non-intuitive field.

If I understood well it is difficult to add one more column to UI. So as the
order received field is not really needed this field could be used to store the
most recent (first) received date. So we could have the original date unchanged
(if used by junk mail controls etc. and the order received contains the received
date which in descending order is quite the same as the order received was - and
I guess nobody interested interested in that counter, but the date is more
information.

Is this possible? - Or are there problems because of probably different variable
types?
(In reply to comment #49)
> I suggest that the "Order received" column is removed from the UI, which would
> be completely obsolete now.

No, it is not.  Order Received has its uses, obscure as they may be -- for 
instance, that shows the order that messages were copied into a (local) folder, 
regardless of the date information on the messages.


But, if we must choose only one of the two columns (I think that THIS is the
main develop problem, am I wrong?), with little or no doubt the majority of
people would prefer to have date received as close to the receiver as possible,
again IMHO.
Yes, indeed. And I really hope that there will be a solution in the near future.
It's been (at least) since 2002 that this issue is talked about. I really would
like to propose Thunderbird and OpenSource to other people but such
circumstances do not make it easy in a lot of cases. In my case it was
completely annoying and finally I talked our network admin into finding a
solution and he patched the mail server in the way that the original date sent
header is overwritten for each incoming mail. But folks, that's really not the
way to do it! If Outlook would behave that way, thousands of people would argue
at Microsoft... - SCNR.

If it is really so difficult to fix this and/or either to add a new column then
there is something going rather wrong. It probably is time to rework some core
things completely.
it's easy to add another column. The problem is that it's easy to add many
columns, until you have a bloated UI and a bloated DB. It's a slippery slope. We
could do like OE does and change the name of the column and displayed header to
Received, but I'm sure there would be a percentage of people unhappy with that too.
There was already the proposal of adding an option for this. So everybody could
be happy. So in my opinion - and I guess I am not alone - there is no more doubt
that this change would be are really needed thing.
(In reply to comment #54)
> it's easy to add another column. 

I really don't want to add to all the bugspam (why doesn't bugzilla have a
seperate area for discussion of a bug?) but I have an idea.  I don't now if this
is workable, but I hope so.  How about a hidden pref, say DateIsDateReceived
(true/false, default false).  This pref, if true, changes the Date column header
to "Date Rec'd" and from then on puts the current time and date into the date
field in the database, disreguarding any dates in the email headers, and doesn't
change any dates already stored in the database.  If the pref is false, things
are basically the same as they are now, except use David's code to help with
munged dates.  How does that sound?  Is it possible?
Code bloat is always going to be a problem with any sophisticated tool

The more things we want a tool to do the more code is required, don't get me
wrong, code bloat is something to be avoided. but most of what I consider code
bloat is not added features but its poor programming practises. 

TB NEEDS a date recieved column and sort function, maybe not for every one but
certainly for many of us, enough to make it an issue. a serious one! those
people who don't see the need can ignore it, but for those of use who do it is a
***glaring*** ommission from TB.

for my vote it should be as close to the reciever as possible. if there is no
way to seperate it from the senders date then TB should add it.

Is it so wrong that a client adds a header? perhaps X-DATE-RECIEVED-TB.
headers are added all the time. I have no problems with TB adding a few.

The sent date is also important, I often use it to determine the date on the
senders computer. I have often solved a users problem (wrong date) using this
field but it is easy to look at the headers and get it it does not really need
to be a ui matter, unless lots of people need it. 

I agree with the sentiment that the order recieved column should go, not the
field just the UI column. retain the field for whatever use it is currently put to.
(In reply to comment #54)
> it's easy to add another column. The problem is that it's easy to add many
> columns, until you have a bloated UI and a bloated DB. It's a slippery slope. We
> could do like OE does and change the name of the column and displayed header to
> Received, but I'm sure there would be a percentage of people unhappy with that
too.

I'm one in the "percentage of people unhappy" :-)

How about changing "Order received" column to "Date received by your server"?

As Comment #49 says, and already opened bug for removing "Order recieved" says,
I think "Order recieved" is not so usefull, especially after "Global Inbox".
Meaning of "Order recieved" column is very near to server arrival date/time
sequence, and I think "Order" is better to be sequence of server arrival
date/time, when Global Inbox, and when moved/copied mail.
(I think this is also applicable to IMAP.)
And changing existing colomn satisfys your comment #39, except making code
complicated a little.
> I don't want to make the UI and code more complicated than it already is
> by adding a second date column.

My prefer.
(1) Change current "Order received" column to "Date received by your server".
(2) Add an option to display date in last added Recieved: Header in date column
 when no/corrupted Date: header.
(3) If option of (2) is on, display date colmn as Outlook Express does, as you
say in your Comment #46.

Almost all logic for (3) is already implemented, or is ready to implement, by you.
An if it's optional by (2), we, who don't want MS's behaviour, will not be unhappy.

And if (1) is implemented, this bug's original request is satisfied, without
making people, who don't want MS's behaviour, unhappy.
 - "Having a 'Received date'" is bug opener's request. See comment #0.
Since last added Received: header is added by POP3 sever which mail receiber is
using, clock time of the server is usually proper, and I think user can afford
incorrect date display due to his server misconfiguration, his server operator's
error etc.
If the mail server is maintained by himself, incorrect date display is caused by
himself, then he can afford to it. 

David, what do you think?
 ------- Additional Comment #51 From Mike Cowperthwaite  2005-10-19 11:43 PDT  
(In addition to comment #58) 
As Mike says in comment #49, "Order Received" has its uses and is currently
used, then removing "Order received" may be impossibble.
But many users want Received: header column, even though developers never want
increasing number of columns...
How about user choosable "Order received" column content?
 - Add an option for data in "Order received" column :
   (a) Download sequence number by my PC(current one)
   (b) Server arrival sequence(Date/time in GMT in last added Received: header)
       If this option, option for GMT display or Local time display. 
 - Save both in ".msf", for both compatibility with older Thunderbird
   and future enhancement.

I think this is not an excellent solution, but I feel practical solution for us
  and practical solution for developers.
David, what do you think? 
Corrrection. Sorry for spam, and my misoperation in copy&paste.
"As Mike says in comment #51"  <== "As Mike says in comment #49"

The term "date" is underspecified - why not just make it mean "date received"? 
Yes, I agree that "date sent" is occasionally useful, even perhaps to sort by,
but what matters most for general mail is when *I* saw it, not when the sender
claimed to send it, not the order that it got filed into my various folders.

Proposal:
this bug: make the "date" column mean "date received". Include a tooltip
defining the term "date" and make sure it gets into the FAQ.  If you want to see
date sent, look at the header(s). This immediately corrects the very serious TB
deficiency without any visible UI impact.
new bug1: add a "sent" column to UI, hidden by default.
new bug2: retitle the "date" column as "received"

In any case, this bug should probably be reopened...
Accordingly to what Michael Thome said, who I agree with, I think that this
"alternate format" could be managed (is not importantwhich is the default) with
an entry in the prefs.js, and a description in the right places (in MozillaZine,
just to make an example). How does it sound?
(In reply to comment #61)
> The term "date" is underspecified - why not just make it mean "date received"? 

 It's a really bad idea to change the meaning of a part of a UI that is already
well established - Thunderbird's Date column means "Date Sent" and that's the
way it should stay. There is a group of users who have a strong preference for
the Date column to show the date sent. I'm one of that group. I'd object to a
change in the Date column display.

 David's understandable reluctance to add complexity to the UI notwithstanding,
I'd rather see the implementation of a separate "Date Received" column.

 Failing that, a pref for which of date sent or date received is shown in the
Date column could also be a satisfactory solution.

 But, please, let's not fight over which date is the "right" or "best" date to
show in the Date column because it shouldn't have to be a choice between one or
the other - we need both.

David, could you re-open this bug, please.

Since your current patch changed meaning of "date column" from "Date sent(Date:
header) to "Date received by server(Received: header)", this bug's request,
"Received date field", was implemeted, then this bug was changed to FIXED.
But, as you say in your comment #46, you are going to change your patch.
The new patach will not be for this bug. The new patch will be for Bug 73565(and
may be for Bug 32216).

Because you changed this bug FIXED, and because you are going to change your
ptach, I asked you some questions about how to display "Received date" in
cuurently existing column with minimum changes/workload.
But, as jwq says in comment #63, both "Sent date" and "Received date" are needed.
I absolutely agree with jwq.
I believe request of this bug, Bug 166254, should be kept.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
David, thanks for quick re-open of this bug.
only used Received hdr for date column if there's no Date: header, or user sets a hidden pref.
Attachment #205084 - Flags: superreview?(mscott)
(In reply to comment #66)
> Created an attachment ...(snip)
> only used received: hdr if no Date: header or user sets hidden pref

The patch aparently is not solution of this bug(request of Received column, see comment #0).
If the patch will be checked-in as fix for this bug, it will produce confusion in bug tracking/fix verification testing. 
I believe the patch should be proposed as the solution of Bug 73565.
pinging Scott for SR...
Blocks: 318323
David, thanks for referencing this bug. Bug #209501 also is about reading dates.

From, Received and Date all may start lines that have date info shown in mail source. But I didn't see anything in my mail source that looked like it could be converted into a date about 3 days in the future. Some msgs I get are typically only one day in the the future, and are from particular other mailing lists.

Bug #209501 noted that there could be a problem in headers with punctuation in dates, the presence or absence of a comma between a dayname (Thur, for example) and the actual date. I noted that the format of dates isn't always the same. Sometimes day of the month is before month, sometimes after. Some header times say -0000, others say GMT. Parsing all that seems tricky.

As noted by IRC, some spams actually are future-dated. But the mailing list msgs with future dates show up with reasonable dates and times when opened.

I don't have any suggestions as to which date info should be used to form msg lists and write to the msf file, except that it seems better to make that info consistent with when the msg was sent. 


Hi, I'm somewhat confused about the resolution of this bug.  I belong to the group for whom "Date Received" is indispensible, and this issue is pretty much the single reason why I've been Evolution rather than Thunderbird.

The last I understood (message #66), the current solution is to use "Date Sent" if the Date: field is defined, and "Date Received" otherwise unless some mysterious secret handshake is performed instead. This is horribly mixed up!

I feel a bit disappointed to read that the developers oppose the identification of the two concepts.  Email is an intrinsically asynchronous process.  It's not reasonable in principle to assume "Date Sent" and "Date Received" are the same thing.  I routinely use "Date Received" for my normal email management, but I may want to use "Date Sent" to check for a missed email which a correspondent says she sent on such-and-such a date. They can easily differ by days.

Why not just have the two columns available separately and let users rearrange their own UI the way they want it?
The other big issue is that any variety of date-sent is inherently untrustworthy.  It would be far better, IMHO, to use NOW (local download time) to fill-in a missing received-date than depend on any possibly bogus time - at least if your localtime is hosed, it is your own fault.

To summarize:
 - time sent is always undependable, both in terms of existance and accuracy (intentional or not).
 - trustable time received should always be gettable from the first received header if you've got something close to a reasonable server.
 - if your server is unreasonable, then you can construct a trustable received time from download time or imap's date (if available)
 - order received is (a) less informative than date received and (b) wrong because imap changes it when you move folders.

The "proposed fix" patch ( https://bugzilla.mozilla.org/attachment.cgi?id=199777 ) comes pretty close, though I worry that if the first received header is malformed, you might end up trusting the next (possibly spoofed) received header.  Also, it would be nice if you saved it back to storage (e.g. an added/modified header) if you needed to use imap.  Also also, it looks like this patch (still) just changes the semantics of the Date column - while I'd prefer this, clearly others do not.  The next patch makes it possible to set a pref to determine if Date is received or a guess at sent, but leaves the default behavior of the Date column even more confused than before.  In any case, no combination of these patches quite addresses the basic request to add received field parsing and column.

Bottom line (for me):
time received is absolutely crucial, time sent is sometimes useful. Reasonable default behavior of the mail reader must not depend on the sender's good will and correctness.
Please note bug #209501 and comment #69 in this bug. Improper punctuation in a header messes up the correct reading of a date and/or time.

Also, any additional headers should not be specific to Thunderbird, Mozilla 1.7x or Seamonkey. They should be generic headers that future versions of Mozilla-family software will be able to read without kludges.


It isn't clear to me what the status is of this bug.  The bug was reopened but nothing has been posted here since March.  In the hope that this bug has been fixed and is in the pipeline, I just downloaded and installed the most recent nightly (well, yesterday's, anyway - Thunderbird build 2006060705).  That version, which is labeled version 3.0a1 does not seem to have this fixed.  There is still just a single "Date" column.  It isn't clear if it is "Date Sent" or "Date Received".  Some of the comments here indicate that the solution might be to change the was date is determined rather than adding this new column.  If that's the accepted solution, please, please, change the column heading so it's clear what date is being used.  Can someone clarify the situation?  Is there going to be a "Date Received" column in Thunderbird?
Comment on attachment 205084 [details] [diff] [review]
only used received: hdr if no Date: header or user sets hidden pref

ping - we're getting complaints about using the received date header instead of the date header, so I'd like to get this, or something similar (perhaps not using the pref) checked in.
(In reply to comment #74)
> (From update of attachment 205084 [details] [diff] [review] [edit])
> ping - we're getting complaints about using the received date header instead of
> the date header, so I'd like to get this, or something similar (perhaps not
> using the pref) checked in.
> 

please note the global effect this might have on RSS feeds' date.  while email sent/received may (in the non spoofed aggregate) be on the order of seconds apart, RSS feeds are fundamentally different.  blogs and forum posts are only useful for the sent date, usually verified by phpbb (for example in case of forums), or other.  and RSS feeds are by nature batched with half hour or more TTL; not sure how Tb would handle.  identical received times would render threaded discussion reading useless. 

imo, two columns are required due to the importance of being clear with the meaning of Date.
Repeating David's ping from two months ago; Scott, sr is being sought here.

I'm not opposed to having the pref, but I want the displayed Date header to match the Date column. and the Date header *must* be from the RFC822 Date header if that's available.
I never understood why there can't be an additional column for that, so everyone has the choice to display one or the other - or even display both...
Would any of the suggested changes affect how dates of newsgroup posts are displayed?
(In reply to comment #76)
> ...
> I'm not opposed to having the pref, but I want the displayed Date header to
> match the Date column. and the Date header *must* be from the RFC822 Date
> header if that's available.
I'm curious - Why is the RFC822 date (if present) useful?  Or is it merely a concern that column heading should match header lines?  Given how large a percentage of the time it is wrong/misformatted/spoofed/missing, I personally would prefer to ignore it completely, as it simply not trustable.
(In reply to comment #79)
> Given how large a
> percentage of the time it is wrong/misformatted/spoofed/missing, I personally
> would prefer to ignore it completely, as it simply not trustable.

I would agree with you - but as there was discussion about it I do not see any reason why not to store all three, see:

(In reply to comment #11)
> [...]
> Maybe we should have three dates stored and displayed:
> 
> 1.) Sent
> 2.) Received (by my mail server)
> 3.) Downloaded

(In reply to comment #79)
> I'm curious - Why is the RFC822 date (if present) useful?  Or is it merely a
> concern that column heading should match header lines?  Given how large a
> percentage of the time it is wrong/misformatted/spoofed/missing, I personally
> would prefer to ignore it completely, as it simply not trustable.

It is generally not spoofed or missing except on spam, which you wouldn't want to read anyway.

If the sender is legitimate and using a normal mail client, it is typically correct.

It is useful for the same reason that is customary to write a date on a paper letter. The recipient knows the point when the email was written which is very helpful to interpret the content of most business emails, especially if many correspondents and/or emails are involved.

I think the RFC2822 date MUST be displayed. If you choose to ignore it, Thunderbird already has an option to turn off columns.

No other MUA I know of excludes the "Sent" (RFC2822) date. Microsoft Outlook has the concept of both a "Sent" and a "Received" (presumably top Received: line or IMAP internal date) date, both of which are available to a user.

Given that no other MUA does this I think you need to provide a rationale for excluding this from Thunderbird, not the other way round.



I've just read through the comments, and I think the following solution will cause no problems for anyone:

Define "Received" as the first of these that is available:
  1. IMAP internal date
  2. POP3 fetch date
  3. Date listed in the top "Received:" RFC2822 header line

Define "Sent" as the RFC2822 Date: field.

For either "Sent" or "Received", if the date is unavailable using the above logic, then we do not know the corresponding date. Ideally this would be represented by an "unknown" value like null, and the GUI would work accordingly (just list "unknown" or leave a space or whatever). I don't think this is essential, as "Received" would normally be available in all cases as there will be at least one "Received" line.

Both Sent and Received columns should be available in the message index view. I suggest that the default is Received, as the user would expect emails to arrive at one end or the other, but not in the middle.

The GUI already allows the user to add and remove columns to suit, and both would be available, so what the default is doesn't affect functionality. I'm happy for whoever submits the patch to choose.

When viewing a message, the Date shown in the message (rather than the index) should be the date Sent, and the text should be changed from "Date:" to "Sent:" to make this clear. This is analogous to how someone would read snail mail.
(In reply to comment #81)
> If the sender is legitimate and using a normal mail client, it is typically
> correct.

Unless the clock on the sender's computer is wrong - at work we have a couple of contacts who send emails with a selection of dates and times in January of 2000.

> Given that no other MUA does this I think you need to provide a rationale for
> excluding this from Thunderbird, not the other way round.

The rationale for swapping one for the other is based on the decision that there can be only one.

> I've just read through the comments, and I think the following solution will
> cause no problems for anyone...

You either missed or ignored the comments which said that having 2 date fields to choose from (which I'd actually be in favour of) adds complexity (because the user when picking fields has more things to choose from), and they don't want to do that.
(In reply to comment #81)
> (In reply to comment #79)
> > I'm curious - Why is the RFC822 date (if present) useful? 
> 
> It is generally not spoofed or missing except on spam, which you wouldn't want
> to read anyway.
You are correct that spam often spoofs or drops it (though not so much any more), but several major help-desk systems drop it, including those used by major ISPs.  Furthermore, even in these days of ntp, it is extremely common for me to get email from people with significantly mis-set system clocks, especially on windows machines pre-NT.  I'm also not sure I understand your argument about disregarding the RCF822 date for spam - IMHO, that is precisely the point: you don't want to read spam, but mis-dated spam/etc will get inappropriately sorted into your inbox (for instance), leaving sorting by sent date less useful.

> If the sender is legitimate and using a normal mail client, it is typically
> correct.
but see above.

> It is useful for the same reason that is customary to write a date on a paper
> letter. The recipient knows the point when the email was written which is very
> helpful to interpret the content of most business emails, especially if many
> correspondents and/or emails are involved.
displaying this date in a message != providing a column and expecting to be able to use it to sort messages.

> I think the RFC2822 date MUST be displayed. If you choose to ignore it,
> Thunderbird already has an option to turn off columns.
> 
> No other MUA I know of excludes the "Sent" (RFC2822) date. Microsoft Outlook
> has the concept of both a "Sent" and a "Received" (presumably top Received:
> line or IMAP internal date) date, both of which are available to a user.
> 
> Given that no other MUA does this I think you need to provide a rationale for
> excluding this from Thunderbird, not the other way round.
I'm not arguing directly for dropping sent date: I am arguing that received date must be a first-class message parameter (with column) because it is both critical and dependable (unlike Sent), and that the mozilla family of MUAs are at a major disadvantage to their competitors because of this omission.

> I've just read through the comments, and I think the following solution will
> cause no problems for anyone:
Except that the GUI folks seem to be balking at adding a Received Date column, so we're still stuck here.  The excuse was to avoid "code bloat" but this seems more like a boogeyman than anything else, given how many bugs have been opened and generated so much discussion for so long: check out the discussion on all of the blocklist.

> Define "Received" as the first of these that is available:
>   1. IMAP internal date
>   2. POP3 fetch date
>   3. Date listed in the top "Received:" RFC2822 header line
I was under the impression that IMAP and POP3 dates changed on folder transfers... personally, I'd recommend reversing the order.


Any chance we could at least get the two active patches applied in time for thunderbird 2?  While they fall short of addressing either the summary or the original request, they do represent significant progress toward fixing this problem and we could then open a new RFE on the other side to get the fields included in the GUI.

As an aside, this open bug is #35 on the Core/MailNews* most votes list, and nearly identical bug 216033 is #12 on the Thunderbird/* list.
Flags: blocking-thunderbird2?
(In reply to comment #83)
> Except that the GUI folks seem to be balking at adding a Received
> Date column, so we're still stuck here.  The excuse was to avoid
> "code bloat" but this seems more like a boogeyman than anything
> else, given how many bugs have been opened and generated so much
> discussion for so long: check out the discussion on all of the
> blocklist.

It seems to me that the argument that adding an additional date column would result in too many options is a bit silly.  There are currently (v1.5.0.5) 16 column options to choose from.  One more isn't going to push this over some limit into unusability.  In any case, even if it did, I would gladly give up Unread, Total, and Account columns in a trade for a Received Date field and possibly a future draft choice.

> > Define "Received" as the first of these that is available:
> >   1. IMAP internal date
> >   2. POP3 fetch date
> >   3. Date listed in the top "Received:" RFC2822 header line
> I was under the impression that IMAP and POP3 dates changed on folder
> transfers... personally, I'd recommend reversing the order.

That's my understanding, as well, and it's the reason the value of the Order Received field is somewhat limited.  While we're talking about traiding fields, a Received Date could easily replace the Order Received field, as far as I'm concerned.
(In reply to comment #85)
> While we're talking about traiding fields,
> a Received Date could easily replace the Order Received field, as far as I'm
> concerned.

That would be logical.  If the UI argument is that adding a new field is cumbersome, then the argument could be made that we are not adding a new field, we are making the existing 'Order Received' field more usable.  To the end user, the 'Order Received' field is a meaningless number (they don't even appear consecutive), and does not always sort by received date either.
> If the UI argument is that adding a new field is
> cumbersome, then the argument could be made that we are not adding a new field,
> we are making the existing 'Order Received' field more usable.  To the end
> user, the 'Order Received' field is a meaningless number (they don't even
> appear consecutive), and does not always sort by received date either.

I will respectfully disagree with your comment that 'Order Received' is a meaningless number, as has already been pointed out in comment #51. I would not like to see it replaced, but I do find the lack of 'Date Received' distressing.

The "Date Received" column should really not be competing with other columns, it should just be added as a new column, separate from 'Date' and 'Order Received', and as a matter of personal opinion I think it should reflect the date in the final (top-most, last in chronological order) received header.
we won't stop ship for this but that doesn't mean we won't end up approving the patch for the branch.
Flags: blocking-thunderbird2? → blocking-thunderbird2-
(In reply to comment #88)
> we won't stop ship for this but that doesn't mean we won't end up approving the
> patch for the branch.

Scott, Any reason not to SR the patch for the trunk?
The patch fixes a regression on the trunk (bug 341548).


(Modification of my previous comment #67)
> If the patch will be checked-in as fix for this bug, it will produce confusion
> in bug tracking/fix verification testing. 
> I believe the patch should be proposed as the solution of Bug 73565.

It seems to have been wrong.
After I reconsidered about David's second patch by Comment #66 and I wrote Bug 341548 Comment #10, I was aware that my understanding about "Date column" was wrong.
 (Wrong)   "Date column" means "Date: header" display.
 (Correct) "Date column" means "mail date" display which held in ".msf".
           The "mail date" is date used for mail age calculation etc.   
           (I don't know correct term for "mail date", so call it "mail date") 
  Current      : "mail date" is always obtained from "Date: header"
  First patch  : "mail date" is always obtained from "Received: header"    
  Second patch : A) When user doesn't set mailnews.use_received_date=true
                     "mail date" is obtained from "Date: header".
                     Only if no "Date: header", from "Received header".
                     =>
                     User can continue to see value in "Date: header"
                     in "Date column" as current. 
                     But user can't taste fruit of "mail date from Received:".
                     Bug 73565 is resolved by this enhancement.
                 B) When user sets mailnews.use_received_date=true
                     "mail date" is always obtained from "Received: header".
                     =>
                     User looses "Date: header" value in "Date column",
                     and user gets "Received: header" value in "Date column". 
                     Column name is unchanged(still "Date").
                     Bug 73565, Bug 32216 and some other bugs which complaint
                     about old date or future date set by spammer, will be
                     automatically resolved by this enhancement.

David, is it right?

As far as user sets mailnews.use_received_date=true intentionally, and as far as user can accept difference from current(value in "Date column" is changed from value in "Date: header" to value in "Received: header"), second patch can be a solution of this bug, although it satisfy this bug's request partially.
And user can taste fruit by second patch ;
  (1) User needn't see future date in "Date column" of mail list pane.
  (2) spam of future Date: header is deleted normally/shortly by usual
      Junk Purge setting such as "Delete after 14 days".
I believe many users can accept difference from current like me.

However, it is true that many users need both two columns - column for "Date: header" value and column for "Received: header" value, as seen in many comments in this bug.
I think next enhancement will be sufficient for this requirement, once your second patch will be released.
 - Implement "Date: header value" column and "Received: header value" column.
   (No change is required for "mail date"="Date column")
Although "Date column" value becomes same as value of one of two new columns, it is not a big issue, I believe.

David, what do you think? (still hate new column? :-))
Bug opener and who want both "Received: header value column" and "Date: header value column", what do you think?
(In reply to comment #90)
> and who want both "Received: header value column" and "Date: header
> value column", what do you think?

For me this solution is sufficient.
(In reply to comment #90)
> However, it is true that many users need both two columns - column for "Date:
> header" value and column for "Received: header" value, as seen in many comments
> in this bug.
> I think next enhancement will be sufficient for this requirement, once your
> second patch will be released.
>  - Implement "Date: header value" column and "Received: header value" column.
>    (No change is required for "mail date"="Date column")

This change would be fine - indeed, it is no more or less than what the original RFE requested 4 years ago.
we could potentially do this with an extension, if we added a hidden pref to collect the Received date header and store it in the msf file. bug 348504 adds the ability for extensions to add custom columns, so an extension could add the received: date.
(In reply to comment #93)
> we could potentially do this with an extension, if we added a hidden pref to
> collect the Received date header and store it in the msf file.
> bug 348504 adds the ability for extensions to add custom columns, so an
> extension could add the received: date.

Oh good news.
I wrongly guessed the reasons why "developers don't want to add new column" were "don't want to add new fields in msf" and "msf compatibility with older versions". 
Is enhancement request bug for hidden prefs/storing in msf already opened?

I don't like extension for enhancement which I like, because of compatibility issues when upgrade of Mozilla family, which always caused/will certainly cause many bugs or many forum threads of "not working well after software update!".
If there are several predefined customizable columns, I think new columns for sorting requirement can be satisfied in many cases although not in all cases.
 - Several customizable columns, and a special column handler for it
 - User can change column name text for a customizable column
 - User can choose mail header from predefined mail header names
 - Predefined mail header names can be limited because mail header (which is
   very convenient if sorting at mail list pane can be done) is not so many.
 - If Recieved: header and Date: header are included in predefined headers,
   this bug's remaining issue will automatically resolved.
   (If unix mail box separator of "From - ..." is included,)
   (user can sort mails by "Download local time".)
One of biggest issues is when/how invoke msf re-creation for all mail folders if option is changed...

David, is there any possibility that such request for enhancement will be accepted by Mozilla mail&news/Thunderbird developers?
Or this kind of enhancement should be done by extension?

By the way, David, when will your second patch proposed in 2005/12 be applied to trunk?
I'm enjoying "mail date" from "Received:" every day and is very satisfied.
But I don't want to read duplicate bugs of bug 341548, don't want to do DUP closing work. 
*** Bug 354813 has been marked as a duplicate of this bug. ***
Here's a data point: I just received two pieces of bugmail that for some reason were delayed.  They have the following timestamps:
  Date:  11:27am         Received:  2:55:55pm
  Date:  11:15am         Received:  2:56:45pm
In this case, at least, the correct order is by the Date field, not the Received field.  (I have no idea why they arrived out of order.)

I wouldn't have seen this if I hadn't been running a trunk build, with this bug's checked-in patch, when they arrived.
*** Bug 363971 has been marked as a duplicate of this bug. ***
Blocks: 360982
Blocks: 355113
Having just investigated bug 355113, I'm more in favor of this patch, as the date stored in the index file (the received date, with this patch) is the date used when folder retention is being used.  Given the possibility of automatic deletion of a desired message because of a wrong date -- especially since auto deletion can't be undone, to my knowledge -- it seems more important to use the received date than if it were simply a matter of not being able to find a message in the expected position in the sort order.  This has been the case with junk mail purging all along.

While I'm sympathetic to the concerns of bug 341548, it seems that the big fix requires adding columns for each (probably storing both dates in the index).
Assuming that's not feasible in the near future, I'd say this patch should get into the 1.8.x branch.
Blocks: 227848
I really disagree with comment 99.  
If I may restate it, it says that: 
(a) because the sender's date may be wrong, and because some folders 
automatically purge messages older than a certain number of dayes, a 
message sent by a system with the wrong date may get purged a day or 
two before it should have been, because of the bad date.

(b) the risk of purging a message a day or two early (due to wrong date)
is SO great and consequences SO terrible that we should continue to force
ALL users to put up with the also-HUGE usability problem of showing the 
wrong dates in the message list pane.  

I totally disagree with that last conclusion.  Yes, I use folders that
expunge messages based on a retention limit.  Obviously, I don't lose any
sleep about those messages not being retained.  It they get purged a day
or two early, so what?  I surely do not want to continue to put up with 
the problem of wrong dates being displayed forever because of this.
(In reply to comment #1)
> Use: View->Sort by->Order received
> 

Problem with using Order Received is if you move mail from one folder to another it changes the Order Received number. Meaning if you are going though your inbox moving messages and don't start from oldest to newest then in the sub folder they are not in true order received. 

One of the bigger problems with the date field that is being used is spammers seem to be taking advantage of this and setting there dates to a date years in the future (like 2083). 

It would be nice to be able to choose either the date and time the e-mail server sent it or received it.



Can someone give some indication of where this is in the process?  This thread is four and a half years old.  There are posts from six month ago about patches.  I'm running version 2 beta 2 and have this line:

user_pref("mailnews.use_received_date", true);

in prefs.js.  Nevertheless, my Date column is still giving me the "Date: header" rather than the last "Received: header".  So, two possibilities:

1) my prefs.js entry is wrong
2) This patch is not in the beta

Which is it?  If my entry is wrong, can someone please give me the correct entry?  If this patch is not in the beta can someone please comment on when we're likely to see any of this?

The patch is not in the beta.  You can get a nightly trunk build, which does have the patch, at:
  ftp.mozilla.org/pub/thunderbird/nightly/latest-trunk

Comment on attachment 205084 [details] [diff] [review]
only used received: hdr if no Date: header or user sets hidden pref

Scott MacGregor,  
Is there anything I can do to get you to review & approve this patch?
Is there any dollar figure that will do the job?

Please look at bug 341548 comment 10 to see why this is important.
Comment on attachment 205084 [details] [diff] [review]
only used received: hdr if no Date: header or user sets hidden pref

looks good David.
Attachment #205084 - Flags: superreview?(mscott) → superreview+
i just installed thunderbird-3.0a1.en-US.win32 created 4/12/07 and
the date column is not using the received date and time. Unless the patch just does not update already received e-mails. Below is source from an e-mail I received.

From - Thu Apr 12 13:12:14 2007
X-Account-Key: account2
X-UIDL: 00017238c2441865
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
X-Mozilla-Keys:                                                                                 
Return-Path: <XXXX@XXX.XX>
Received: from yokoyama1 (000.000.000.00) by smtp.mailserver.com with ESMTP
 (Eudora Internet Mail Server X 3.2.6); Thu, 12 Apr 2007 13:09:45 -0400
Return-Path: <XXXX@XXX.XX>
Received: from 00.000.00.00 (HELO mail.XXX.XX)
     by mailserver.com with esmtp (LF-:7V'6,2E= *N4'8)
     id G.=VK;-T00AU1-JU
     for XXXXX@XXXXXX.XXX; Thu, 12 Apr 2007 17:10:20 -0900
Date:	Thu, 12 Apr 2007 17:10:20 -0900

If you sort by date it is showing this e-mail was sent 4 hours in the future.

Actually this is exactly what is the problem - spammers send e-mails in the future as some sort of "keep on top"-"feature".

On the last e-mail server I was using they had implemented a server side exim script that did overwrite the Date header with the received date. Currently on a new mail server unfortunately I get future-e-mails again.

And what is more annoying: Thunderbird is showing even other e-mails not necessarily in their order of receiving which can be very confusing.

Shortly: the Date stamp is not reliable. The only one reliable date is the last - or spoken from the order of the headers the first "Received" stamp.

So please: Take the date from the first "Received" header instead for that column in Thunderbird.

For the people who would like to say to use the order column instead: Whenever I move the e-mail to another folder I get new order numbers so this is not an option.
This bug is specifically a request that "received date" be added as a first-class concept to thunderbird and other mozilla mail components.  Any other solution would not address the problem and so would be non-responsive to the bug.

The reason that this is an issue is precisely that there is no current way to reliably order email temporally that is not either (a) under control of the sender or (b) destroyed by inbox/folder maintenance.

The traditional "Date" field while "RFC required" is absolutely undependable in many, many instances - yes, spammers often spoof them, but it is also clear that many people are incapable of keeping their system clocks in sync with reality.  Even worse, several major billing, support, and helpdesk systems don't bother to put any Date field at all on their messages, leading to thunderbird hiding such generally critical messages at the beginning of time.  All this conspires to make sorting by Date worse then useless: it is asking for trouble.  Date *might* be genuinely useful as a idiot detector.

The "Received Order" field is interesting, but is reset when messages are moved between folders and boxes (depending on mailserver behavior).  Obviously, while it is possibly useful *within* a single folder, it is only a partial ordering when (for instance) merging views from multiple boxes.  Furthermore, it is only useful for sorting purposes, not for viewing, making it a waste of screen space.  IMHO a much more useful value here would be define "order received" as "filed date": there would be absolutely no difference in sort order and the associated information would be independently useful.

I can understand the reluctance to change the semantics of the Date column (even if I disagree - I seriously don't see any redeeming value in display or order-by Date header...), but I don't understand why adding received date as a column is in any way a less attractive option than a "fix" that allows only expert users to change Date=Date to Date=Received... Better than nothing, I suppose - at least it will finally become possible for expert users to order their email in a sane way.
I could hardly disagree more. While I agree that other means must be found if the Date: field is not present, or perhaps if it's utterly outside sane bounds, anything other than the Date: header (or perhaps the first Received header date) means that messages close in time will commonly be mis-ordered due to the vagaries of the Internet.

If people you care about are sending emails with bogus Date fields, tell them so. I'm sure they will be happy to fix their clocks, and have their reminders go off at the right time and so on. And if spam is a problem, use a spam filter. It can note the bogus date to do its job better.

This proposed patch seems to me to be exactly the right thing, and it should be checked in ASAP.

Gerv
I agree completely with Michael Thome and completely disagree with Gervase Markham .

Like Michael, I consider the bug to be the reason that it was raised for - that a "Date Received" column (regardless of what it is called) is required.

As Michael said, the patch is useful as an obtuse workaround but doesn't not resolve the problem, and therefore would not resolve this bug.

My vote for this bug is for a "Date Received" column to exist. This is what the bug was raised for and this is what I suspect many votes represent.

Obviously there is a division on this matter. Can I propose that this be split into two bugs. Then it will be clear who is voting for what.
I also vote for a new column but it seems to me from the discussions (not only in this issue) that it is too difficult to add a further
column. In that case I can perfectly live also with an option somewhere (hidden
or not) where I can decide to use the received date in the date column.

Telling everybody to fix their clock is something I could do if I am very out
of work (if I know the name where to drop that notice) otherwise I assume that
they will find out that their clock is wrong themselves. However I cannot
assume that others will solve my problems.

I have integrated spam filters on nearly all accounts I am using but as all of
you might experience - those do not always find out every spam.

And actually I cannot understand how it can be more work to add that option
rather than discussing it over years...
Yes, I too agree with Michael Tome. The (obvious for me) reason for the existence of this bug was lack of Date Received column. This can (and probably should) be an additional column - most end users are have no interest in original (as per rfc) message date, they want to know when a message was received into their own, private mailbox!
> they want to know when a message was
> received into their own, private mailbox!

And presumably they love having their discussions out of order if one message gets delayed and a subsequent one does not? This behaviour happens to me regularly under the current regime - far more regularly than receiving non-spam messages with bogus Date: headers.

The only way to guarantee that a conversation is threaded in the order that the people involved wrote the messages is for it to be threaded by Date: (i.e. when it was written) and for the Date headers to be correct. The latter is, in my experience, almost always the case for non-spam mail. Therefore, we should do the former so that the two combine together to achieve the correct result.

But it is correct that there is no point in arguing about "what this bug is for". The fact remains that there's a patch here that solves the problem for some people without making life more difficult for anyone else. So let's get it checked in. If you want to argue for something else, you can then do so.

Gerv
(In reply to comment #109)
> I could hardly disagree more. While I agree that other means must be found if
> the Date: field is not present, or perhaps if it's utterly outside sane bounds,
> anything other than the Date: header (or perhaps the first Received header
> date) means that messages close in time will commonly be mis-ordered due to the
> vagaries of the Internet.
I'm not trying to be argumentative here - I'm genuinely baffled as to why one would care (let alone trust) that the date that someone *claims* to have sent a message is anywhere near as important as when you got it - when one gets physical mail, what matters is when it arrived in your mailbox.  Sure, I *occasionally* check the postmark on an envelope when it arrives in poor condition or later than I expect, but having someone tell me "the check is in the mail" isn't something that I'm likely to ever trust.  The bottom line for me is that the Date header is, in my experience the Date header is often wrong and surprisingly often significantly so, even for legitimate mail.  Received is always trustworthy.  I would vastly prefer a trustworthy measure of absolute ordering than a known untrustworthy, often a bit off, and sometimes totally bogus measure... when you are dealing with a good citizen with decent connectivity, then the measures are going to be nearly identical, but when senders are badly behaved, the "Date solution" rewards the bad behavior, where the "Received solution" is neutral to it.

> If people you care about are sending emails with bogus Date fields, tell them
> so. I'm sure they will be happy to fix their clocks, and have their reminders
> go off at the right time and so on. And if spam is a problem, use a spam
> filter. It can note the bogus date to do its job better.
Ah - Well, so far, Verizon Wireless has ignored my complaints, and my ISP uses a commercial help desk system that doesn't send Date at all. It took a remarkably long time for people to fix their machines w/the recent Daylight Savings Time change.  And ntp is nowhere near ubiquitous, though it is slowly getting better since windows added it in.  What *really* grates on me is how often when I complain, the response is "Well, maybe you ought to use a decent email program".  This *ought* to be truly horrifying to anyone maintaining Thunderbird - it is an extremely serious competitive disadvantage that impacts users far more than most of the new features and bugfixes that have been addressed in the 4.5 years since this bug was opened.

The implicit assumptions behind your response are that (1) I know the people sending me legitimate email (often not true), and that (2) entities with broken clocks/MTAs care that they are annoying me (often not true - see above).  I get legitimate email with bad Date fields that are broken enough to result in visibly misordered email almost every day - thunderbird's baffling lack of the trivial capability to properly order my messages is not something that is possible for me to ignore - the defect is in my face *constantly*.

Threading email discussions cannot reliably be done with the Date header, not only because it is often wrong, but if Date is not approximately the same as Received already, then you cannot know when the author of a message saw the message that they are responding to.  On the other hand, real message threading is perfectly and properly accomplished using the already fully implemented In-Reply-To headers.

FWIW, this bug is the 7th most voted for open MailNews backend bug (out of 570).  The closely related bug #216033 is the 8th most voted for and the 38th oldest open bug in Thunderbird (out of 5051).
(In reply to comment #114)
> I'm not trying to be argumentative here 

I'm sure :-)

> - I'm genuinely baffled as to why one
> would care (let alone trust) that the date that someone *claims* to have sent a
> message is anywhere near as important as when you got it 

Because the people I correspond with don't lie about it, and because there is no other way of accurately putting messages in the order which they were written. Because email is not an instant-delivery medium.

> - when one gets
> physical mail, what matters is when it arrived in your mailbox.  

I don't tend to have physical mail conversations with multiple people participating in the same conversation. And if I were involved in some sort of round-robin multiple-recipient letter scheme, I would doubtless look at the dates written at the top of the letters to ascertain in what order they were written, so I could see who was responding to what.

> Sure, I
> *occasionally* check the postmark on an envelope when it arrives in poor
> condition or later than I expect, but having someone tell me "the check is in
> the mail" isn't something that I'm likely to ever trust.  

Utterly irrelevant example, surely?

> The bottom line for
> me is that the Date header is, in my experience the Date header is often wrong
> and surprisingly often significantly so, even for legitimate mail.  Received is
> always trustworthy. 

It's an absolutely trustworthy measure of when a particular message was received by a particular machine. That doesn't mean it's always *what you want to know*.

For example, I go offline and write some messages, and press Send Later. Meantime, other people are writing messages in the same email conversation. When I go back online, the Received header would erroneously imply I wrote those messages after theirs, when in fact that was not the case. My MUA has carefully noted when I wrote them, and stored the value in the Date: header so that other participants in the thread can see when I wrote it.

Gerv
>For example, I go offline and write some messages, and press Send Later.
>Meantime, other people are writing messages in the same email conversation.
>When I go back online, the Received header would erroneously imply I wrote
>those messages after theirs, when in fact that was not the case. My MUA has
>carefully noted when I wrote them, and stored the value in the Date: header so
>that other participants in the thread can see when I wrote it.

Errr, sorry, but it would not. It is *received* date, not a *sent* date.

BTW - have anyone made any assesment of message threading popularity (ie. how many people use it)? Issues with threading seem to be an important argument in this discussion, but out of my ~10 friends using Thunderbird, no one uses threads (and all complain about lack of Received Date column/field...).
(In reply to comment #115)

> For example, I go offline and write some messages, and press Send Later.
> Meantime, other people are writing messages in the same email conversation.
> When I go back online, the Received header would erroneously imply I wrote
> those messages after theirs, when in fact that was not the case. My MUA has
> carefully noted when I wrote them, and stored the value in the Date: header so
> that other participants in the thread can see when I wrote it.

And the date field would erroneously imply that the others who had
sent their emails when they finished writing them had read
yours. That's what conversation threading is for. Your message should
indicate what it's replying to and fall quite nicely into the threaded
conversation.

(More generally)

I personally care more about Date Received than I do about Date,
though I believe that all three of Date, Date Received, and Order
Received have their place as first class columns, and there are times
I'd want to be able to see all three of them. My vote for this bug was
for a fix to that effect. All three columns, no hacks, no extras. This
is a basic MUA feature.

Seems to me that this discussion has the obvious conclusion that tb/mailnews needs to have both these columns and let us pick and choose which we want to use and when. Sometime's I'll sort some messages by Date Received, sometimes by Date Sent, sometimes by something else.

In that context I don't like the proposed patch. Too often such temporary fixes serve as an excuse for allowing serious issues to remain unresolved (thing of bug 2920 for example).

> BTW - have anyone made any assesment of message threading popularity (ie. how
many people use it)?

This is an interesting question but this bug page is not the place to discuss it.
Seems to me that this discussion has the obvious conclusion that tb/mailnews needs to have both these columns and let us pick and choose which we want to use and when. Sometime's I'll sort some messages by Date Received, sometimes by Date Sent, sometimes by something else.

In that context I don't like the proposed patch. Too often such temporary fixes serve as an excuse for allowing serious issues to remain unresolved (thing of bug 2920 for example).

> BTW - have anyone made any assesment of message threading popularity (ie. how
many people use it)?

This is an interesting question but this bug page is not the place to discuss it.
Re comments #114 and #115, some people use email like a chatroom, sending one-liners back and forth with time differences of seconds or minutes, not hours or days. I just read a pile of msgs from 10K to 80K passed back and forth with one-line additions, and deleted a bunch of them. I would suggest using an irc or local chat service, not email, for such light banter.

I too must say that the received-by date is the only reliable one for the end recipient of an email. If minute-by-minute messages get out of order, that's tough bippies. Use a more suitable medium for chat.




Re comments #114 and #115, some people use email like a chatroom, sending one-liners back and forth with time differences of seconds or minutes, not hours or days. I just read a pile of msgs from 10K to 80K passed back and forth with one-line additions, and deleted a bunch of them. I would suggest using an irc or local chat service, not email, for such light banter.

I too must say that the received-by date is the only reliable one for the end recipient of an email. If minute-by-minute messages get out of order, that's tough bippies. Use a more suitable medium for chat.




This patch gives users a choice.  Today, users have NO choice.  
Clearly, no one solution is preferred by everybody.

The people  who do the work on this code say that adding another column
is too much work.  I believe them.  No one who wants another column is
volunteering to do the work.  That means: it won't happen soon.

In the meantime, we have a patch that gives us all the choice.
It should go in.

For myself, I absolutely don't care about dates in spam, because they all
get filtered/deleted, so they never show up in my folders.

I care MOST that successive emails sent by the same individual show up 
in the order SENT by that individual, even if that is not the order 
received.  Today, without this patch, I simply don't have that option.
Anyone else seeing "midair collision" msgs resulting from Different senders (not the same sender) sending messages at the same time? It seems to me that if the senders are different, it shouldn't be considered a midair collision by bugzilla.
As the person who originally reported this bug, let me just chime in again.. This seems like a pretty cut and dry issue to me. Some people want date sent, some people want date received. Regardless of your opinion of which is more useful, several, if not all of the most popular MUA's out there today have this ability, making this a serious competitive failure. It's clear that vast numbers of people would appreciate this functionality whether everyone agrees with it or not.

So, can we all agree on the intention and move onto implementation details? In my opinion, making this a "hidden" setting is a stop-gap solution which doesn't really address the problem properly. Use cases have been proposed by people who want both a date sent field and a date received field, so the ability to switch it via a prefs file from one to the other is lacking, at best.

Adding a new column may be difficult, but I don't for a second believe that's a good enough reason to implement a half hearted hack which barely solves the problem. Surely there's a developer out there willing and able to do the work to add it properly?

I was gearing up to send this bug a "Happy 5th birthday" card in a few months. Perhaps we can have a goal of getting a resolution before that happens.
I agree that having separate columns for Date Sent (or date ostensibly sent, if you prefer), Date Received, and Order Received is the ideal situation.  I've found myself wanting all three at various times and the fact that I have two of them is a good start.  Having a Date Received column would be a great addition.  This is, by the way, how many other systems report date.  At work, where I use MS Outlook for mail, I can have either or both date fields.  I switch back and forth regularly.  I also appreciate that the date field defaults to newest at the top since that's what I want nine times out of ten.  That's a subject for a different bug thread, I suppose.

It's clear that the folks able to do this are not willing.  I don't hold that against them.  I'm quite grateful for all they have done and continue to do.  Others of us, myself included, are quite willing but not particularly able.  Frankly, I wouldn't know where to start.  If someone wanted to look at the code involved, where would he start?

(In reply to comment #124)
Congrats on (so far) having outlived your bug!  I agree, the bug is very straightforward.  The caption says: "Received date field"

When I switched from Outlook a few years back, this was one of the more noticeable differences; the order of emails is subtly incorrect when I sort by Date.  No, I cannot ask people to fix their computer time; it would be unprofessional to ask this in most cases.  Order Received is (with all due respect) useless since it is reset too easily by moving a file, or even just deleting an attachment.

How about, for now, implement the bug fix for "Date Sent" and RENAME the UI column from "Date" to "Date Sent" so it is more clear what it is.  However it does not fix the problem that the bug is for SO PLEASE DON'T CLOSE THE BUG until we have DATE RECEIVED.

Sorry about the bug spam but this bug is important.
Mattias wrote:
> Order Received is (with all due respect) useless since it is reset 
> too easily by moving a file, or even just deleting an attachment.

Maybe that's a bug and should be fixed.
Order received has it's use but become inacturate if you move an e-mail to another folder. That is not what we are reporting here.

The date collum is not a bug as it works the way it was written. it sorts the date the e-mail client said it was written. (if it's faked it's faked)

I think we are asking that a feature be put in to sort by date received like
most of the major e-mail clients on the market do.

granted 99% of the e-mails I get that have wrong dates are spam, there are a few that i get that are from customers that have to change there computer dates.

then you get the ppl who can't set there time zone correctly so that messes up there sent date. 

as for the person that likes to have thread style e-mail conversations. might i suggest a message board.



> The date collum is not a bug as it works the way it was written. 
> it sorts the date the e-mail client said it was written.

Actually it does NOT do that, not in trunk builds.  
I would be happy if it did, but it does not.  

See comment 90. The trunk now behaves as described there for "first patch".
It's horrible (IMO :).
Why is it most everyone talks about IN THEIR EXPERIENCE and then ignore other peoples experience like it doesnt matter.

We all have different experiences

If you have good spam filters thats nice, but thats you. 

I get hundreds of items of spam a day, and in my role as a webmaster I can't automatically filter all of it. I have to err on the side of caution so a lot of bulk email ends up in my inbox.

The same role prevents me from sending back a "fix your date you idiot" message even if I could be bothered in the first place. We are not our brothers keepers.

This bug as pointed out by micheal in #114 is one of the "most popular" which REALLY means most annoying, it is also one of the longest running

IMHO the rfc that is used as a basis for many of the arguments IS DATED it doesnt account for the changing netscape (lol)

But the RFC allows for MTA's to add headers so adding a received header is not going to break it.

The solution that seems to resolve all wants is an additional column "Received Date". taken from the added received header. Oh yes the MUA is also an MTA so it has the legal right to add the header :)

The user should be able to pick this column out of the column picker NOT via some internal switch, How many users are able to do that?

The bulk of people don't stick with it and submit bug reports and argue the point, they simply up and leave. Why would you want to alienate more people?
 
As a programmer I struggle to see it as being too hard to add a column. but then I haven't investigated the source code so perhaps some bad design decisions have left Thunderbird in this (appalling) state.

Those that want threading perhaps you should start a request for a threading mechanism 'Sent' date was not designed for that process and will never reliably work for that process. But the main point is this bug report is not about threading. Why are you even mentioning it? and no one is saying get rid of the sent header anyway so you can continue to use it.

Hmmm actually ignore my posting. I give up I am going back to outlook

(In reply to comment #127)
> Mattias wrote:
> > Order Received is (with all due respect) useless since it is reset 
> > too easily by moving a file, or even just deleting an attachment.
> 
> Maybe that's a bug and should be fixed.
> 

I was just pointing out why the Order Received column is irrelevant to the bug at hand.  And it would not take the place of Date Received even if it was fixed.

Developers who can fix this are obviously unwilling.  You must be reading these comments too.  Considering all the amazing features of TB; to a user it seems like a small deal to add a Date Received column, and that it would make a lot of people happy to do so.  Perhaps one of you could explain what the actual problem is that prevents this bug from being fixed so that us laypeople can appreciate it?
Nelson, I've landed the patch on the trunk; sorry for the delay.

re adding a Date Received column, I'd be happy to help someone who wanted to do a patch for that. It's not hard, just tedious.
Comment on attachment 205084 [details] [diff] [review]
only used received: hdr if no Date: header or user sets hidden pref

There's a certain amount of clamoring for this
Attachment #205084 - Flags: approval1.8.1.5?
Comment on attachment 205084 [details] [diff] [review]
only used received: hdr if no Date: header or user sets hidden pref

a=mscott for mailnews only change to 1.8.1.5.
Attachment #205084 - Flags: approval1.8.1.5? → approval1.8.1.5+
support for hidden pref landed in 1.8.1.5 - I'm going to add the fixed1.8.1.5 keyword, even though the main bug itself (a whole new received by date column) is not fixed on the trunk or branch.
Keywords: fixed1.8.1.5
Again, as the original requester of this feature, I'd like to re-iterate that I don't consider a hidden preference a great fix. To my mind, there are acceptable use cases for both received and sent date fields, and practically every other MTA available has received as not only an option, but the default.

I appreciate the hidden preference as a stop-gap solution, but I don't believe this bug can ever be truly resolved until the option is more obvious to your every-day user. I can't recommend Thunderbird to my parents if I have to tell them to open a user prefs js file, or go edit config variables, in order to get their mail in the same way they've received it for years with other clients.
The instructions for changing this setting using the config editor are only a little bit more involved than they would be for finding an option in the prefs UI:

Tools | options | advanced (general) - config editor
type received
right click on the line that says "mailnews.use_received_date" and pick toggle
That sets it to false.

Setting it to true by default provoked a lot of negative reaction so we're not going there again.

If someone wants to write a patch to add a received column, again, I'm happy to help.
(In reply to comment #137)
> If someone wants to write a patch to add a received column, again, I'm happy to
> help.

David, first of all thanks for your efforts about Thunderbird code.
I'm not a C++ coder, so I can't help with the patch, but I'd like to ask you if it would be possible to have the received date as separate properity of nsIMsgDBHdr.
In this way it would be very easy to create a custom column to display it.


CCing self.

(In reply to comment #137)
> The instructions for changing this setting using the config editor are only a
> little bit more involved than they would be for finding an option in the prefs
> UI:
> 
> Tools | options | advanced (general) - config editor
> type received
> right click on the line that says "mailnews.use_received_date" and pick toggle
> That sets it to false.

I don't see that option.  *shrug*

> 
> Setting it to true by default provoked a lot of negative reaction so we're not
> going there again.

You're saying that ordering e-mails by the Delivery-date instead of Date, a vastly more sensible method and the same as OE and Outlook do it, provoked negative reaction?  By who?  Idiots?  It's just way better and more sensible.

> 
> If someone wants to write a patch to add a received column, again, I'm happy to
> help.

I think I might try to hack this.  I doubt the columns usable for sorting use much C++, it's more likely XUL and JavaScript.
> I don't see that option.  *shrug*
You're probably not running the latest trunk or 2.0.0.5-pre build
>You're saying that ordering e-mails by the Delivery-date instead of Date...
I'm saying have a column that says "Date" and then putting the data from the Received header instead of the Date header bothered people.

>I think I might try to hack this.  I doubt the columns usable for sorting use
>much C++, it's more likely XUL and JavaScript.

Right - the only C++ required is to set a dateReceived property on the msg hdr when we're parsing the incoming message. We have code to do something very similar now, obviously, but it would need to set a dateReceived property, not the date property. I could help you with that part, if needs be. I wouldn't add it to nsIMsgHdr.idl, but rather, just do something like

msgHdr->SetUint32Property("dateReceived", intDate);

and pull that date out for the Received column with GetUint32Property. Actually, unless you use a custom column, you need a little bit of code in nsMsgDBView.cpp to add a case for the received date column.
David: Why does your code try to parse the Received: header first?  Shouldn't it look at the Delivery-date header, then fall thru to Received?
Jeremy, I think you're right - we should be looking for the Delivery Date header first. Do you want me to make a patch to do that? It would be a little while before I can get to it.
Sure.  I'm currently trying to figure out the nightmarish dark arts of hacking Mozilla, and being reminded of why I try to avoid it whenever possible, so I might come out with something before.  Have to see.
Jeremy: You've come a little late to the party, you're speaking a little too
loudly, and you keep referring to the caviar as "bean dip".  It's awkward.

I completely agree with you that the current implementation is incorrect, but I
think that "idiots" is a strong word for those who feel it's better than the
alternatives. We've done our best to keep this thread civil; please don't spoil
it now.  

If you refer to the past five years of comments, you'll see we've all been
discussing what the best behavior is and what the limitations of the current
architecture are.  You may wish to review these at your leisure to save some time.

ObActualContribution:  Delivery-date: is a non-standard header added, I
think, only by exim and mh. On other mailers, it won't be created or modified
and is therefore easily forgeable, leading right back to the Date: problem.
Delivery-date is described here: http://www.faqs.org/rfcs/rfc2156.html - but if it's easily forgeable, then that's a potential problem (not sure if it's a real world problem, though). Jeremy, what's the advantage of using the Delivery Date over the topmost received header? 

I wonder if one way of dealing with forged dates is to trust the date closest to the current download time, or conversely, trust dates past the current download time less, since they're most likely forged.
(In reply to comment #144)
...
> ObActualContribution:  Delivery-date: is a non-standard header added, I
> think, only by exim and mh. On other mailers, it won't be created or modified
> and is therefore easily forgeable, leading right back to the Date: problem.
> 

... and ...
(In reply to comment #145)
> Delivery-date is described here: http://www.faqs.org/rfcs/rfc2156.html - but if
> it's easily forgeable, then that's a potential problem (not sure if it's a real
> world problem, though). Jeremy, what's the advantage of using the Delivery Date
> over the topmost received header? 

Correct me if I'm wrong, but surely everything in a POP-format e-mail is easily forgeable... by the MTA.  However, we trust the MTA *waaaay* more than anything that passed the e-mail to the MTA.  Otherwise we'd have a pretty useless e-mail service.  So, I'm taking the approach that if we can say the MTA added it, we should assume that's the most trustworthy info we have.

You could just look at the topmost received header, I guess.  I was just thinking that it might possibly not represent the time when the MTA received the e-mail, or there might be a few seconds' difference.  Delivery-date is specifically designed for the MTA to tell you when it received the mail, so I'd give it top priority; obviously fall through to Received for MTAs that don't add that header.  In other words, I'd put the order of priority as:
1. Delivery-date
2. Received
3. Local system time (PR_Now())
4. Date

... where 4 should probably not be used because some idiot could (and many spammers do) set it to 2011 or 1995 and make it appear in a silly place in the list.  You start looking at 1., and fall through to 3.

> 
> I wonder if one way of dealing with forged dates is to trust the date closest
> to the current download time, or conversely, trust dates past the current
> download time less, since they're most likely forged.
> 

That assumes that the user's local date/time are correctly configured, which is very dangerous.  At the end of the day, I'd give 1. and 2. in the above list priority over the local system's date/time.  In other words, if I set my system to June 6th, 1997 and get an email in with a Delivery-date header of July 3rd, 2007, I want it to display the latter.  It's more likely I misconfigured my system than that the MTA lied, and if the MTA lied, I should find a better MTA. :-P

By the way, I've compiled Thunderbird version 3.0a1pre (2007070321), toggled the mailnews.use_received_date value to true, and yet an e-mail with a bogus Date header (but valid Received date) is still having its date listed as the bogus date.  Perhaps you could give me some idea of why this might not be working, David.
@David: Oops, I should have been more precise about "non-standard".  Delivery-Date: isn't in RFC2822.  RFC2156 describes only how X.400-to-MIME (MIXER) gateways should work.  RFC4021, the standards-track registration of MIME header fields, lists it as MIXER-only and "not for general use" - and RFC4021 itself is only a proposed standard, step 1 of 3 in becoming a standard.  So it's not standard.

But more to the point, it's not particularly common, let alone ubiquitous, and so I don't think it can really be trusted the way Received: can.  There's an option to create Delivery-Date: in exim, and there's another option to remove any existing header; I don't know if those options are defaults.  Other MTAs, say sendmail or postfix, would simply pass it through if it already existed.

@Jeremy: You're exactly right - we trust our local MTA.  But we can't assume that the local MTA is the one that added Delivery-Date: unless (a) we are running exim and (b) we've configured exim to add that header (or at least remove such an existing header).  Otherwise, it could have been added anywhere along the chain.

Received:, at least, is nearly-ubiquitously added - and is additive, not, er, replace-a-tive - and so thus more trustworthy when looked at in reverse order.
(In reply to comment #140)
> Right - the only C++ required is to set a dateReceived property on the msg hdr
> when we're parsing the incoming message. We have code to do something very
> similar now, obviously, but it would need to set a dateReceived property, not
> the date property. I could help you with that part, if needs be. I wouldn't add
> it to nsIMsgHdr.idl, but rather, just do something like
> 
> msgHdr->SetUint32Property("dateReceived", intDate);
> 
> and pull that date out for the Received column with GetUint32Property.
> Actually, unless you use a custom column, you need a little bit of code in
> nsMsgDBView.cpp to add a case for the received date column.
> 

David, how is this safe?  It looks like a PRTime type, which your patch is using to store the parsed Received: date, is defined as a 64 bit unsigned int.  Wouldn't setting it as a uint32 property lose data?
at some point maybe you have to convert from milliseconds to seconds - you want to be storing seconds, in any case.
Attached patch Pre-Alpha patch ;-) (obsolete) — — Splinter Review 
Added a pre-alpha patch.  This sort of starts to work - it adds a 'Received' column but all the received dates are off.  I'm guessing this is because it's trying to cram a PRTime (64 bits) into a 32 bit attribute, David.  Please advise on how to fix.
Look at http://lxr.mozilla.org/mozilla/source/mailnews/db/msgdb/src/nsMsgHdr.cpp#527

and when you get ready to ask for a review, please remove all those JEZ comments :-)
Attached patch Pre-Alpha patch 2 (obsolete) — — Splinter Review 
OK, added an updated version.
Attachment #271465 - Attachment is obsolete: true
Attached patch Alpha patch v0.3 (obsolete) — — Splinter Review 
David,

I just realised what you meant by those 64-32-64 bit helper functions.  New patch actually seems to display the Received dates correctly!  It displays a space " " if there's no, or an invalid, Received date.

I've changed a bit of your date generation code.  It used to fall back to the Date header, and then PR_Now(), if it couldn't find a Received header.  I've just made it display a space " " character.  My rationale behind this is that the Received header should be a very trustworthy indicator of when the MTA received the e-mail.  It's nearly always there, and when it isn't, I want to know about it and I don't want the sender to have the opportunity of 'fooling' the user into thinking the Received date is what they specified as Date.  If you want to know the Date header, you can look at the Date column.  The Date column is much more of a 'probably received at this date' than Received, which is why it just falls back to PR_Now() if the Date header isn't there - it's much more prone to spoofing... but that won't matter if we deprecate its use, and display & sort by Received by default, instead of Date.
Attachment #271500 - Attachment is obsolete: true
looks like it's coming along, good job!

+            <treecol id="receivedCol" persist="hidden ordinal width" flex="2"
+                     label="Received" tooltiptext="Received tooltip"/>
+            <splitter class="tree-splitter"/>

the tooltip text wants to be in a dtd file, like the other tooltips (this needs to be fixed in several places)

+        if (!LL_IS_ZERO(m_receivedTime)) {
+          PRTime2Seconds(m_receivedTime, &rcvTimeSecs);
+          m_newMsgHdr->SetUint32Property("dateReceived", rcvTimeSecs);
+        }
+        else {

this file uses this braces style:

if (a)
{
}
else
{
}

and in general, you should follow the prevailing braces style

why are these lines changed? Is your editor messing with the line endings, or were they broken in this file?

+// set to 0 if you don't want to ignore timestamp differences between
+// local mail folders and the value stored in the corresponding .msf file.
+// 0 was the default up to and including 1.5. I've made the default
+// be greater than one hour so daylight savings time changes don't affect us.
+// We will still always regenerate .msf files if the file size changes.
+pref("mail.db_timestamp_leeway", 4000);
+

-pref("mailnews.use_received_date", false);
-

If you get rid of this pref, you basically break users who want to use Received Date for things like sorting within threads (we use the date value for things other than the date column). I'm not sure what we want to do about that, unless it's to switch to using the received date for that kind of sorting...

Is your plan to break this out into a shared routine so that both date columns can share this code?
+  nsDateFormatSelector dateFormat = m_dateFormatDefault;
+//  if (explodedCurrentTime.tm_year == explodedMsgTime.tm_year &&
+//      explodedCurrentTime.tm_month == explodedMsgTime.tm_month &&
+//      explodedCurrentTime.tm_mday == explodedMsgTime.tm_mday)
+//  {
+//    // same day...
+//    dateFormat = m_dateFormatToday;
+//  }
+//  // the following chunk of code causes us to show a day instead of a number if the message was received
+//  // within the last 7 days. i.e. Mon 5:10pm.
+//  // The concrete format used is dependent on a preference setting (see InitDisplayFormats), but the default
+//  // is the format described above.
+//  else if (LL_CMP(currentTime, >, dateOfMsg))
+//  {
+//    // some constants for calculation
+//    static PRInt64 microSecondsPerSecond;
+//    static PRInt64 secondsPerDay;
+//    static PRInt64 microSecondsPerDay;
+//    static PRInt64 microSecondsPer6Days;
+//
+//    static PRBool bGotConstants = PR_FALSE;
+//    if ( !bGotConstants )
+//    {
+//      // seeds
+//      LL_I2L  ( microSecondsPerSecond,  PR_USEC_PER_SEC );
+//      LL_UI2L ( secondsPerDay,          60 * 60 * 24 );
+//
+//      // derivees
+//      LL_MUL( microSecondsPerDay,   secondsPerDay,      microSecondsPerSecond );
+//      LL_MUL( microSecondsPer6Days, microSecondsPerDay, 6 );
+//
+//      bGotConstants = PR_TRUE;
+//    }
+//
+//    // setting the time variables to local time
+//    PRInt64 GMTLocalTimeShift;
+//    LL_ADD( GMTLocalTimeShift, explodedCurrentTime.tm_params.tp_gmt_offset, explodedCurrentTime.tm_params.tp_dst_offset );
+//    LL_MUL( GMTLocalTimeShift, GMTLocalTimeShift, microSecondsPerSecond );
+//    LL_ADD( currentTime, currentTime, GMTLocalTimeShift );
+//    LL_ADD( dateOfMsgLocal, dateOfMsg, GMTLocalTimeShift );
+//
+//    // the most recent midnight, counting from current time
+//    PRInt64 todaysMicroSeconds, mostRecentMidnight;
+//    LL_MOD( todaysMicroSeconds, currentTime, microSecondsPerDay );
+//    LL_SUB( mostRecentMidnight, currentTime, todaysMicroSeconds );
+//
+//    // most recent midnight minus 6 days
+//    PRInt64 mostRecentWeek;
+//    LL_SUB( mostRecentWeek, mostRecentMidnight, microSecondsPer6Days );
+//
+//    // was the message sent during the last week?
+//    if ( LL_CMP( dateOfMsgLocal, >=, mostRecentWeek ) )
+//    { // yes ....
+//      dateFormat = m_dateFormatThisWeek;
+//    }
+//  }

David: Talking to you on IRC about the FetchDate function would be useful.  Unless I'm mistaken, it isn't working properly.
A bit of research for the record: Outlook Express uses the 'Received' date first, then drops to 'Date', then uses the local system time, to mark the received date.  I think we should do the same.  I initially said that you should totally trust the MTA to put a Received header in, but on the rare occasions it doesn't, Date should be used (or the local system time) so that the Received column can be more comprehensive for sorting e-mails.  IOW, you're OK with the Received date being potentially wrong because it only applies to very few e-mails that for some reason don't have a Received header.
Attached patch Beta patch v0.5 (obsolete) — — Splinter Review 
(In reply to comment #155)
> David: Talking to you on IRC about the FetchDate function would be useful. 
> Unless I'm mistaken, it isn't working properly.

Seems I was mistaken; it does work properly.  Ignore that.  :-P

(In reply to comment #154)
> looks like it's coming along, good job!
> 
[...]
> 
> the tooltip text wants to be in a dtd file, like the other tooltips (this needs
> to be fixed in several places)

I've done this with the new patch.

> 
> this file uses this braces style:
> 
> if (a)
> {
> }
> else
> {
> }
> 
> and in general, you should follow the prevailing braces style

Damn, I hate that braces style.  :-)  But it's fixed in the new patch.

> why are these lines changed? Is your editor messing with the line endings, or
> were they broken in this file?

Good question.  I've re-checked out the file from CVS and re-made the changes, and it doesn't seem to be diff'ing these lines now.

> -pref("mailnews.use_received_date", false);
> -
> 
> If you get rid of this pref, you basically break users who want to use Received
> Date for things like sorting within threads (we use the date value for things
> other than the date column). I'm not sure what we want to do about that, unless
> it's to switch to using the received date for that kind of sorting...

That seems like a good idea.  The Received date column is intended to be more reliable than or as reliable as the Date column, in all cases, so this sorting should be switched to use Received by default.

> Is your plan to break this out into a shared routine so that both date columns
> can share this code?

It wasn't, but that seems like a good idea now.  :-)  I've done this in the new patch with the help of an optional bool argument, set to true if you want the Received date.

I've also made the FinalizeHeaders() function always set 'Received' to something, with the order of priority going Received -> Delivery-date -> Date -> PR_Now().  The assumption is that an MTA might somehow not set Received, but might set Delivery-Date, which is slightly more reliable than Date (almost always set, but often spoofed to be wrong).

The problem with PR_Now(), I guess, is that it will calculate the date wrongly if you 'rebuild index' for the folder.  It will set the date to the date when you rebuilt the index, instead of the date it was tagged with when the e-mail was actually received.  I'm not sure if there's an easy way to get around this, and it probably isn't important because so few e-mails will actually need to resort to PR_Now() anyway.

A review of this new patch would be appreciated.  AFAICT, the only thing needed now is to allow sorting by Received column, and (I think) set it to be displayed and used to sort messages by default upon Thunderbird install, instead of Date.
Attachment #271505 - Attachment is obsolete: true
(In reply to comment #157)
> > -pref("mailnews.use_received_date", false);
> > -
> > 
> > If you get rid of this pref, you basically break users who want to use Received
> > Date for things like sorting within threads (we use the date value for things
> > other than the date column). I'm not sure what we want to do about that, unless
> > it's to switch to using the received date for that kind of sorting...
> 
> That seems like a good idea.  The Received date column is intended to be more
> reliable than or as reliable as the Date column, in all cases, so this sorting
> should be switched to use Received by default.

Using Date for threads is really the preferred thing IMO. Elsewhere I wouldn't care so much which is used, but in threads I definitely want Date to read it the order it's written, not the order it happens to be received. 

Date is now used on trunk, but on 2.0 it's still threaded by Order Received which probably corresponds quite well with Received - and that causes problems surprisingly often.
> I've changed a bit of your date generation code.  It used to fall back to the
> Date header, and then PR_Now(), if it couldn't find a Received header.  I've
> just made it display a space " " character.

I think it needs some kind of fallback.

While you might trust the MTA to insert a Receceived header, you cannot trust all mails to be handled by an MTA. Locally generated mail could well be sent directly to the MDA or even appended to an MBOX file (or Maildir) without passing the systems normal MDA.
(In reply to comment #159)
> > I've changed a bit of your date generation code.  It used to fall back to the
> > Date header, and then PR_Now(), if it couldn't find a Received header.  I've
> > just made it display a space " " character.
> 
> I think it needs some kind of fallback.

Yep, I changed my mind on this one.  The latest patch does fall back.
Attached patch Patch v1.0 (obsolete) — — Splinter Review 
Patch v1.0 submitted.  This one does quite a bit more; adds the received column, and the ability to sort by it, as well as group messages by it.  It hides itself for folders other than Inbox, Trash, and other POP folders (except unsent, sent, and drafts).
Attachment #271837 - Attachment is obsolete: true
Attachment #273672 - Flags: review?
Attachment #273672 - Flags: review? → review?(bienvenu)
Attachment #273672 - Flags: review?(mscott)
Iif you change the sort type enums, you break all the .msf files with saved sort orders, because we write out the ordinal value of the current sort in the .msf file. So I don't think you can do that. Why did you want to that?
I thought I saw somwhere in the codebase that 'byCustom' had to be last?
I don't know why that would be - mxr link?
Attached patch Patch v1.1 (obsolete) — — Splinter Review 
OK, I tried adding the const after 'custom' and recompiled - seems to work.  Here's the new patch.
Attachment #273672 - Attachment is obsolete: true
Attachment #273672 - Flags: review?(mscott)
Attachment #273672 - Flags: review?(bienvenu)
Attachment #273854 - Flags: review?(bienvenu)
Attachment #273854 - Flags: review?(mscott)
Comment on attachment 273854 [details] [diff] [review]
Patch v1.1

Looking pretty good. Some things I noticed:

 // sort_type is an int value reflecting nsMsgViewSortType values
 //   as defined in nsIMsgDBView.idl (byDate = 18, byId = 21 etc.)
 pref("mailnews.default_sort_order", 1); // for Mail/RSS/... (nsMsgDatabase)
-pref("mailnews.default_sort_type", 18); //
+pref("mailnews.default_sort_type", 19); //
 pref("mailnews.default_news_sort_order", 1); // for News (nsNewsDatabase)
-pref("mailnews.default_news_sort_type", 21); //
+pref("mailnews.default_news_sort_type", 22); //

are these values right? Since you're not changing the enum anymore, it doesn't seem like you should be changing the news default, since it can't be received...

This code looks like it has a tab in it, and the else if brace should be on a line by itself.

same with this line:

+        if (!LL_IS_ZERO(m_receivedTime)) {  // Upgrade 'Received' to Received: ?
+          PRTime2Seconds(m_receivedTime, &rcvTimeSecs);


+        else if (deliveryDate) {  // Upgrade 'Received' to Delivery-date: ?
+          PRTime resultTime;
+          PRStatus timeStatus = PR_ParseTimeString (deliveryDate->value, PR_FALSE, &resultTime);
+          if (PR_SUCCESS == timeStatus)
+            PRTime2Seconds(resultTime, &rcvTimeSecs);
+		}
+		m_newMsgHdr->SetUint32Property("dateReceived", rcvTimeSecs);
+

and here:

        if (date) {  // Date:

I still haven't had a chance to apply this patch because my tree has some changes to the view code to implement stable secondary sorts, which I'm still finishing up, but I should have that done in the next few days.
Attached patch Patch v1.2 (obsolete) — — Splinter Review 
(In reply to comment #166)
[...]
>  // sort_type is an int value reflecting nsMsgViewSortType values
>  //   as defined in nsIMsgDBView.idl (byDate = 18, byId = 21 etc.)
>  pref("mailnews.default_sort_order", 1); // for Mail/RSS/... (nsMsgDatabase)
> -pref("mailnews.default_sort_type", 18); //
> +pref("mailnews.default_sort_type", 19); //
>  pref("mailnews.default_news_sort_order", 1); // for News (nsNewsDatabase)
> -pref("mailnews.default_news_sort_type", 21); //
> +pref("mailnews.default_news_sort_type", 22); //
> 
> are these values right? Since you're not changing the enum anymore, it doesn't
> seem like you should be changing the news default, since it can't be
> received...

Well spotted.  I've fixed this now.

> This code looks like it has a tab in it, and the else if brace should be on a
> line by itself.

Looks like you missed out the code you meant to reference here?

> same with this line:

I've been through with a hex editor :-) and fixed the two lines I could see that I added that had tabs in, as well as the brackets.  By the way, there's a bunch of existing code with inconsistent bracketing and tabs spattered all over the place.  Maybe you should give up on the tab consistency thing.  :-P

> I still haven't had a chance to apply this patch because my tree has some
> changes to the view code to implement stable secondary sorts, which I'm still
> finishing up, but I should have that done in the next few days.

I also fixed some code that prevented the 'sort by' menu from having the Received option selected or the 'group by sort' option enabled, when sorting by Received.

The one remaining thing I can see that might be wrong is the behaviour when sorting e-mails in threaded mode.  However, I really can't understand how TB is meant to do this and I'll need to look into it more.  It seems to sort the threads by the Received date of the root thread e-mail, rather than the Received date of the most recent thread entry.  Looking at the official TB release I have, it looks like it should do the latter.  Not sure what code is messed up there.
Attachment #273854 - Attachment is obsolete: true
Attachment #274283 - Flags: review?(mscott)
Attachment #274283 - Flags: review?(bienvenu)
Attachment #273854 - Flags: review?(mscott)
Attachment #273854 - Flags: review?(bienvenu)
(In reply to comment #167)
[...]
> The one remaining thing I can see that might be wrong is the behaviour when
> sorting e-mails in threaded mode.  However, I really can't understand how TB is
> meant to do this and I'll need to look into it more.  It seems to sort the
> threads by the Received date of the root thread e-mail, rather than the
> Received date of the most recent thread entry.  Looking at the official TB
> release I have, it looks like it should do the latter.  Not sure what code is
> messed up there.

This error happens when sorting by Received and turning threading on.

In addition, when you sort by Date and then turn on threading, the threads get sorted by the Received date of the thread's most recently Received message, when it should be using the Date of the thread's most recently Dated message.  The behaviour is what I want when sorting by Received and turning threading on, but not when sorting by Date and turning threading on.

So both seem to be behaving incorrectly at this point.  The threading code is complex.
FYI.
There are at least 5 bugs which can be observed while thread display.
(some are fixed on trunk) 
 1. Bug 254159 : Order of main thread.
 2. Bug 262316 : Order in a thread.
 3. Bug 181446 : Thread will be broken when child message is followed by parent
                 message if subjects are not the same
 4. Bug 264941(Thunderbird) / Bug 388205(Seamonkey)
    View->Threads->"Threads with Unread" changes sorting to "Order Received"
 5. Bug 369620(Thunderbird) / Bug 389935(Seamonkey)
    Threading via column header forces Sort by "Order Received",
    when previous sort order is chosen by column header click
    (not via View/Sort By menu)




 Bug 181446.

(In reply to comment #169)
>  1. Bug 254159 : Order of main thread.

I've advised that this should be WONTFIXed, following the fix to this bug being checked in.

>  3. Bug 181446 : Thread will be broken when child message is followed by parent message if subjects are not the same

It looks like this bug is on the verge of being fixed, with a patch just waiting for a superreview.

>  5. Bug 369620(Thunderbird) / Bug 389935(Seamonkey)
>     Threading via column header forces Sort by "Order Received",
>     when previous sort order is chosen by column header click
>     (not via View/Sort By menu)

I have a working patch for that bug, that I'll submit once the fix to this bug is checked in.
Attached patch Patch v1.3 (obsolete) — — Splinter Review 
Fixed 1 or 2 things so small that I can't remember what...
Attachment #274283 - Attachment is obsolete: true
Attachment #274493 - Flags: review?(mscott)
Attachment #274493 - Flags: review?(bienvenu)
Attachment #274283 - Flags: review?(mscott)
Attachment #274283 - Flags: review?(bienvenu)
OK, sorry for the delay. I've been running with the patch, and I think it does what it intends. I have a couple issues with the experience, however. I'm not sure what the solutions are, but here are the issues:

The Date Received column is blank for existing headers. I understand why, but it's not a very nice out of the box experience. I wonder if the view code could display the Date: value in the Received column if the Received value is 0. Just a thought, some people will barf, I'm sure. The alternative of rebuilding all the summary files isn't very attractive.

Out of the box, we show both Date: and Received columns. That's a waste of space, IMO (though mail.app does it, I believe); I think most users are only going to want one. For new profiles, I'd be happy just showing the Received column; for existing profiles, you can't really switch to showing the Received column instead of the Date column, because they very likely are sorted by the Date: column. And I don't think we can just not show the Received column by default - for one thing, Jeremy's head would explode, and for an other, as the patch stands now, the default sort for new folders is by Received, and having a default sort by a column you don't display is not nice.

Thoughts? The patch looks fine, code-wise, but I'd like to deal with these usability issues.
(In reply to comment #172)
> a thought, some people will barf, I'm sure. The alternative of rebuilding all
> the summary files isn't very attractive.

It does seem rather unpleasant, but would it be unacceptably bad as a kind of once-off "Updating your mailboxes for Thunderbird 3" thing?

The only other options I can think of are (a) if the received column is displayed, then initially display nothing if the received value is 0, but immediately calculate and add the received values for any messages that are visible in the pane, so they'd appear soon after, or (b) substitute "date:" for "received" to start with, but start rebuilding the summary files in the background whenever cycles are spare.
(In reply to comment #173)
> It does seem rather unpleasant, but would it be unacceptably bad as a kind of
> once-off "Updating your mailboxes for Thunderbird 3" thing?

But this might well be implemented before TB3...

> The only other options I can think of are (a) if the received column is
> displayed, then initially display nothing if the received value is 0, but
> immediately calculate and add the received values for any messages that are
> visible in the pane

I'm not sure this is possible with the current code, it's generally all-or-nothing when it comes to calculating message index data.

> (b) substitute "date:" for
> "received" to start with, but start rebuilding the summary files in the
> background whenever cycles are spare.

Again, not sure this is possible with the current code.  I think I prefer David's idea of displaying the Date: value instead for any received fields that contain 0.
(In reply to comment #174)
> Again, not sure this is possible with the current code.  I think I prefer
> David's idea of displaying the Date: value instead for any received fields that
> contain 0.

I don't like this solution at all. There's no way to tell that the value you're seeing in the field is date instead of received.

What's wrong with recalculating the index? I have folders with >~10,000 mails in them on an imap server, and I've had a number of times when I've had to re-download the headers and recalculate. It's never taken longer than I would consider reasonable.. 


(In reply to comment #175)
> What's wrong with recalculating the index? I have folders with >~10,000 mails
> in them on an imap server, and I've had a number of times when I've had to
> re-download the headers and recalculate. It's never taken longer than I would
> consider reasonable.. 

By now my profile is 6.7 GB. It does take quite some time to recalculate the indexes and the application isn't very informative during that time. More so on machines with slow disks and not much RAM.  I know what's going on in the background and what has triggered it. Others don't. So at least an assessment of the impact on the average user should be made.
(In reply to comment #176)
> By now my profile is 6.7 GB. It does take quite some time to recalculate the
[snip]
> of the impact on the average user should be made.

I don't think an average user would have even a 1GB profile, so I suspect >6GB is an exception. I agree, some performance analysis would be nice, but at the end of the day, no matter how slow it is, I think it's preferable to having inconsistent values in the field.
Even if it's a bit slow to recalculate because a user has an enormous mailbox, I think it'll be fairly acceptable if there's an alert message that this is a one-off necessary convert / rebuild.
We can't just indiscriminately blow away .msf files. For example, if you have an imap server that doesn't support keywords on the server, blowing away the .msf file will remove all your tags.

In my experience, you mostly care about the difference between received and date values for new incoming mail, e.g., when some spammer sets the mail to be from the future so that it will show up at the top (or bottom) of your mail.
Attached patch Patch v1.4 (obsolete) — — Splinter Review 
Updated patch.  Now received column echoes date column if it has nothing to display.  Received column is inserted in the columns list just before Date.  Received column will appear when TB is upgraded, and Date is hidden by default on new installs.  Threading behaviour is totally correct, AFAICT.

Unfortunately, whilst grouping works for today's messages and earlier, it is broken for messages whoes dates are in the future - they each get grouped in their own bucket.  I can't for the life of my figure out why, so help!

Apart from that, I think the patch is there, though, feature-wise.
Attachment #274493 - Attachment is obsolete: true
Attachment #275585 - Flags: review?(mscott)
Attachment #275585 - Flags: review?(bienvenu)
Attachment #274493 - Flags: review?(mscott)
Attachment #274493 - Flags: review?(bienvenu)
Comment on attachment 275585 [details] [diff] [review]
Patch v1.4

thx, Jeremy. Finally got a chance to run with this. It looks good - I tried a new profile and verified that it just shows the Received column, not the Date column.
Attachment #275585 - Flags: superreview?(mscott)
Attachment #275585 - Flags: review?(mscott)
Attachment #275585 - Flags: review?(bienvenu)
Attachment #275585 - Flags: review+
Am I to understand that the patch changes the default behavior for new
profiles to show only the Received column and not the Date column?  
If so, I object to that change.  

On a technical note, new code should not use the LL macros, which are 
deprecated.  All supported compilers on all supported platforms now 
support 64-bit types (even if the generated code emulates the behavior).
In new code, when using the PRInt64 and PRUint64 types, ordinary 
arithmetic statements and ordinary comparisons should be used rather 
than the LL macros.  The compilers are likely to generate more efficient
code using their own 64-bit implementations than when using NSPR's macros.
(In reply to comment #182)
> Am I to understand that the patch changes the default behavior for new
> profiles to show only the Received column and not the Date column?  
> If so, I object to that change.  

Using Received is superior to using Date because it's generally more reliable, and it's also what most other mail clients do.
Attached patch Patch v1.5 (obsolete) — — Splinter Review 
This patch fixes the problem where, when you tried to sort by Received in a folder whose index hadn't been rebuilt, it fell back on ID for messages that didn't have the Received attribute.  Now it uses Date instead.
Attachment #275585 - Attachment is obsolete: true
Attachment #277020 - Flags: review?
Attachment #275585 - Flags: superreview?(mscott)
Attachment #277020 - Flags: review? → review?(bienvenu)
Attachment #277020 - Flags: superreview?(mscott)
Attached patch Patch v1.5.1 (obsolete) — — Splinter Review 
Apologies, the 1.5 patch still had a debug printf() in that I left.  This doesn't.
Attachment #277020 - Attachment is obsolete: true
Attachment #277023 - Flags: superreview?(mscott)
Attachment #277023 - Flags: review?(bienvenu)
Attachment #277020 - Flags: superreview?(mscott)
Attachment #277020 - Flags: review?(bienvenu)
Received is UTTERLY USELESS in environments where servers randomly delay 
some emails, by up to 24 hours.  When the messages finally arrive, I want
to see them in the order they were sent.  I don't have a problem with 
senders having bad clocks.  I have a BIG problem with screwey mail relays.
I'm not alone.  All the people served by my various mail service providers
have the same problems.  

This is a change that affects how the product works for ALL the users who 
have used it for years, to benefit a few newcomers who expect it to be 
different.  Do we want to alienate our established user base to entice 
some Outlook Express users?
No offence, but why not get your ISP(s) to fix their MTAs?  MTAs should NOT be doing that on a regular basis.  In my experience, it's a far bigger problem to have misconfigured local system clocks or just plain spoofed dates (usually by spammers) than for MTAs to be broken.  I run my own mail server, I'm not even that much of an expert, and it doesn't hold mail up randomly.
Jeremy: No offence, but why not get your correspondent(s) to fix their machine's local system clocks? If you can say "just fix the MTAs", why can't Nelson say "just fix the local clocks"? After all, having your local system clock wrong can lead to all sorts of other problems too.

BTW, spammers spoofing dates in the future is a feature, not a bug. It causes all the spam (which the filter doesn't catch) to collect together in one place, ripe for deletion. It's the nicest thing they do.

Gerv
I've been thinking for a long time since mailnews.use_received_date=false/true was landed by David that;
 - Nelson can go same way as current by mailnews.use_received_date=false and by
   display of Mail_date(=Date:) column, and can display Received: column if needed.
 - Jeremy can change to new way by mailnews.use_received_date=true and by display
   of Mail_date(=Receieved:) column, and can display Date: column if needed,
   although "rebuild index" is required to change to new way. 
 - Mail_date is always used for sorting, mail aging and so on.
Are these already incorrect? (Sorry but I don't understand new design well yet.)
The obvious answer to this dilemma is to have an "OE compatibility mode".
Lots of products have these modes of compatibility with their competitors'
products (word processors, especially!).  Word has a WordPerfect compatibility
mode.  Excell has (or had) a 1-2-3 compatibilty mode.  If we're trying to 
attract OE users, let's make it clear that we're doing that, and make it 
be an option chosen at profile setup time (default: OFF).

BTW, you can construct a filter to junk or delete all email from the future.  
It shouldn't be necessary to reorder the way you display ALL emails, just to
make it more convenient to delete a few spams that can be filtered out.
(In reply to comment #186)
> This is a change that affects how the product works for ALL the users who 
> have used it for years, to benefit a few newcomers who expect it to be 
> different.  Do we want to alienate our established user base to entice 
> some Outlook Express users?

I'm a user who has used TB for years. I logged this very feature request almost 5 years ago, so it's been at least that long. Prior to that, I used Netscape Mail & News, which also used sent date. Personally, I've always found it to be a major annoyance, and I can't believe I'm the only one. So right off the bat, I'd guess your concerns are probably only shared by a small percentage of current users.

I really don't believe changing this to a default will "alienate [the] established user base". Those who rely on sent date will simply switch it back. They're more likely to realise that sent date is even available as an option than a new user who probably hasn't even considered the possibility that a client might use something other than received.

Usage stats for MUAs is hard to come by, but I can't believe TB has more than maybe 20%. That means there's an 80% market share to be gained, most of whom will be currently using clients which use received date. Is the goal to entice new users? I assume so.. in which case, why not make it easy for them? People who are trialling software are going to be far less forgiving of perceived inadequacies than long term users seeing a change (which can be simply reverted).

The idea of an "OE compatibility mode" is, to my mind, completely over the top. We're talking about one simple column and whether it's on by default. Why not make both on by default and let people remove the one they don't want? Doesn't that seem a lot simpler?
(In reply to comment #191)
> (In reply to comment #186)
> > This is a change that affects how the product works for ALL the users who 
> > have used it for years, to benefit a few newcomers who expect it to be 
> > different.  Do we want to alienate our established user base to entice 
> > some Outlook Express users?
> I'm a user who has used TB for years. I logged this very feature request 
> almost 5 years ago, so it's been at least that long.

The same applies for me - and I NEVER used Outlook Express and also desired that since a lot of years. But that were days with less spam...


(In reply to comment #188)
> BTW, spammers spoofing dates in the future is a feature, not a bug. It causes
> all the spam (which the filter doesn't catch) to collect together in one 
> place, ripe for deletion. It's the nicest thing they do.

I did not think about this. "Just fix the local times" is something I experienced in the past that people simply did not care that much. But with the rising amount of spam we have no THE reason for everybody to take care of there local times. And actually since aggressive anti-spam-software do filter often more than it should people now have a real interest in having their clocks set correctly and so I recognize that actually the original problem does occur less and less. So in the meantime I think I can live without that additional column -  so for me this issue is obsolete.
(In reply to comment #191)
[...]
> The idea of an "OE compatibility mode" is, to my mind, completely over the top.

Indeed.  Whatsmore, calling it a 'compatibility mode' implies that it's only being used for compatibility with something, perhaps to be deprecated or phased out later.  That is not the case.  Received is just a more reliable alternative to Date, in most cases.  I have to say that this is not the kind of thing I was expecting a Holy War to break out over, and i'm quite disappointed that it has.

> We're talking about one simple column and whether it's on by default. Why not
> make both on by default and let people remove the one they don't want? Doesn't
> that seem a lot simpler?

It's not the column that's in contention, it's the default mode for sorting threads.  Say you sort by Subject, then go into threaded mode.  When you expand a thread, what do you sort the child threads by?  Not subject, because the subjects are probably all the same.  Previously it was by Date, which made sense.  Now some want it to be by Received (me included), because we think it's more reliable, others want Date still.  I say make Received the default and have a pref to switch it back.  Nelson, what is your problem with that?  It could use the existing mailnews.use_received_date pref, which if set to true (by default) will cause threads to be sorted by received.
(In reply to comment #193)
> it's the default mode for sorting threads.
I've understood what is the current issue now. Thanks Jeremy.
Question: What about mail aging, filtering based on date related data?
          (i.e. what is used as "mail_date" after your enhancement?)
I don't know what mail_date is.  There's no such token in the sourcecode.
Sorry about the bug spam.  Judging from programming comments, it seems like
you're really close to a resolution of this bug.  Let me just say I very much
appreciate this!

(In reply to comment #186)
> This is a change that affects how the product works for ALL the users who 
> have used it for years, to benefit a few newcomers who expect it to be 
> different.  Do we want to alienate our established user base to entice 
> some Outlook Express users?

As a long time user, I feel compelled to speak up and say I will not be
alienated.  For me, sorting by and displaying Received date is the right thing
to do.  Displaying "Date" is too ambiguous.  You might say "this is a change
that affects how the product works for ALL the users who have used it for
years, finally fixing a long time bug"

I don't really care what the default is, or if it's changed, but it sure makes
sense to me to make it something that is used by most other email clients
(afaik).  For instance, when I moved from Outlook to TB, this bug was obvious,
and a major annoyance to me.  I never even considered the "Date" column in
Outlook because it was always correct.  This was one of the items on the
"minus" side when I considered whether to stay with TB, or return to Outlook.

(In reply to comment #188)
> Jeremy: No offence, but why not get your correspondent(s) to fix their
> machine's local system clocks? If you can say "just fix the MTAs", why can't
> Nelson say "just fix the local clocks"? After all, having your local system
> clock wrong can lead to all sorts of other problems too.

It's a matter of the relationship with the sender.  You might receive email
from clients.  It might not be appropriate to ask your client to fix their
system clock.  To paraphrase, please consider that many who use Thunderbird
might not be in position to ask the sender to correct their system clock.  On
the other hand, if your mail server is giving you an incorrect "received date"
then you only have one entity to deal with, and (without any exception I can
think of) you are dealing with someone who views you as a customer.

Personally I would not even ask friends to fix their system clock.  Why? 
Because they would think I'm a moron if I tried to explain that for my email
client to work correctly, I need their system clock to be correct.
(In reply to comment #193)
> (In reply to comment #191)
> > The idea of an "OE compatibility mode" is, to my mind, completely over the top.
> Indeed.  Whatsmore, calling it a 'compatibility mode' implies that it's only
> being used for compatibility with something, perhaps to be deprecated or

Plus "Date received" is also the default date column for Apple's Mail app. So the use of it can't be reduced to some Microsoft "deviation" from an accepted standard. Although, interestingly, Mail.app does appear to display "date sent" in the mail header itself....

(In reply to comment #188)
> BTW, spammers spoofing dates in the future is a feature, not a bug. It causes
> all the spam (which the filter doesn't catch) to collect together in one place,

For the few spam mails that spoof future dates, anyway. And I receive a number of emails from friends that appear to have been sent before they're received. But I presume this wasn't asserted too seriously. Otherwise it'd be like saying "some weeds have purple flowers, so if I segregate all the purple flowers I can fix my weed problem" :)
good people.

just an observation that as of january this bug already had 100 comments and it will soon hit 200.  It's hard to imagine there is *anything* that hasn't already been said. 

Please let the devs decide what will be implemented based on the patch as supplied. Besides, for now, it's only a patch on trunk.
Attached patch Patch v1.6 (obsolete) — — Splinter Review 
This patch adds the mailnews.use_received_date pref that controls whether grouped/threaded messages are sorted by Received or Date.
Attachment #277023 - Attachment is obsolete: true
Attachment #277116 - Flags: superreview?(mscott)
Attachment #277116 - Flags: review?(bienvenu)
Attachment #277023 - Flags: superreview?(mscott)
Attachment #277023 - Flags: review?(bienvenu)
Comment on attachment 277116 [details] [diff] [review]
Patch v1.6

m_useReceiveDate should be cached in a static global, instead of once per thread. If you have 1000 threads, you'll have 4000 bytes of data with the exact same value in it.

You should be able to find lots of places in the code that cache prefs in a global var.

for example, http://mxr.mozilla.org/mailnews/source/mailnews/db/msgdb/src/nsMsgDatabase.cpp#228
Attachment #277116 - Flags: review?(bienvenu) → review-
the example you give of a static global creates inflexibility - see https://bugzilla.mozilla.org/show_bug.cgi?id=377573

while received date may work well for pop/imap, have we considered that batch received rss feeds would all have the same received date - set at time of poll interval - and it really only makes sense to display sent (posted) date?  date spoofing is not an issue here.  a one time all-or-none private var set at load is not good.

the pref should be an attribute of a folder, a g var, or at least should be reread/updated on folder/account change.

true, you shouldn't be using received date for news or rss feeds. But, the way I'd wrote the code, anyway, if there's no received header, we use the date: value anyway.
Attached patch Patch v1.7 (obsolete) — — Splinter Review 
This patch's functionality is identical to 1.6, but it uses static globals to hold the use received data instead of class members.
Attachment #277116 - Attachment is obsolete: true
Attachment #277170 - Flags: superreview?(mscott)
Attachment #277170 - Flags: review?(bienvenu)
Attachment #277116 - Flags: superreview?(mscott)
Attachment #277170 - Flags: superreview?(mscott)
(In reply to comment #195)
> I don't know what mail_date is.  There's no such token in the sourcecode.

Sorry for my incorrect wording.
My question is:
 When mailnews.use_received_date=true,
 and when "sort order" is based on Date: header value,
 which timestamp value is used for filter of "Age in Days is ..."?
 What for filter of "Date is ..."? (predefined Date, not user added Date)

Note: User added item name Date in filter means Date: header(string), but predefined item name of Date is date value obtained from Date: header or Received: header(date type data). To distinguish these two different "Date", I called the latter "mail_date" in previous question.

(In reply to comment #204)
OK, I think I understand now.

>  When mailnews.use_received_date=true,
>  and when "sort order" is based on Date: header value,
>  which timestamp value is used for filter of "Age in Days is ..."?

The Date: header.

>  What for filter of "Date is ..."? (predefined Date, not user added Date)

The Date: header.
To Nelson Bolyard:
What is your expectation for "Age In Days is ..." and "Date is ..."(Not for Date: header user added) of filter in next four cases?
What is "should be" do you think?
 1. mailnews.use_received_date=false, Thread sorting based on Date:     header
 2. mailnews.use_received_date=false, Thread sorting based on Received: header
 3. mailnews.use_received_date=true,  Thread sorting based on Date:     header
 4. mailnews.use_received_date=true,  Thread sorting based on Received: header
Comment on attachment 277170 [details] [diff] [review]
Patch v1.7

if you'll look at the example I pointed you at before, you'll see that there are two globals - one that says whether we've read the global pref, and one for the global value. This saves us the trouble of reading the pref for every thread that's created.

I'm pretty sure we're going to want to default this behavior to use the date header, not the received header, from what Scott says.

What I'd really like to do is get a patch that does the Received column, off by default, and get that checked in, and do all the other stuff in a subsequent patch(like  your recent changes for this global pref).

That way, something gets landed, and can get used and tested, and that way, the subsequent patches are smaller and easier to review.
Attachment #277170 - Flags: review?(bienvenu) → review-
(In reply to comment #207)
> I'm pretty sure we're going to want to default this behavior to use the date
> header, not the received header, from what Scott says.

Well I don't know how to have this discussion with you anywhere else, because you don't seem to respond outside Bugzilla which is unfortunate, so I'll have to do it here.

Please explain the rationale behind this decision.  A few people want the current behaviour to remain and stuff to get sorted by Date by deault, but it seems to me from reading this bug that for every one of them, there are 2 people who think the current behaviour is bad and want it replaced with sorting by Received.  Why not listen to the majority, especially when they're logically in the right?

> What I'd really like to do is get a patch that does the Received column, off by
> default, and get that checked in, and do all the other stuff in a subsequent
> patch(like  your recent changes for this global pref).

As you say, stuff that gets landed can get used and tested.  Whatsmore, it would be landed in the trunk, which gets 'experimental' stuff landed in it all the time, and is largely for testing purposes.  Why, then, would you want me to take out the thread and group sorting functionality for this patch?  Are you seriously telling me it's too 'big' to review?  Why isn't it better to get this functionality in ASAP so it too can be tested?  Allowing to sort by Received column but not sort threads/groups by received seems inconsistent to me.
I'd like to get the non-controversial parts landed. Is that hard to understand?
Attached patch Patch v1.8 (obsolete) — — Splinter Review 
Well, ok.  Patch v1.8 doesn't include the "controversial" stuff.
Attachment #277170 - Attachment is obsolete: true
Attachment #277591 - Flags: review?(bienvenu)
Comment on attachment 277591 [details] [diff] [review]
Patch v1.8

seems like if we're not showing the received column by default in this phase, we wouldn't need any changes to the UpgradeThreadPaneUI method.

I'll try the patch w/o those changes
What is the purpose of the trunk build?  Seriously?  Is it for something other than testing, because I didn't realise.

If you hide the column, nobody will use it.  Why are you even bothering to check it in?  And please, satisfy my curiosity, are there *REALLY* significant numbers of people (please give me a rough number) who are actually going to the effort of contacting you to complain about a patch that hasn't been checked in yet?  Who are these people?  You seem to have done a virtual U-turn in opinion; you wanted to *replace* the date column with received, and the thread ordering too.  I can understand wanting the pref and everything, I can even understand making it off by default (although I disagree), but my understanding has completely ceased at 'put in the functionality but make sure nobody notices it'.
(In reply to comment #206)
> To Nelson Bolyard:
> What is your expectation for "Age In Days is ..." and "Date is ..."(Not for
> Date: header user added) of filter in next four cases?
> What is "should be" do you think?
>  1. mailnews.use_received_date=false, Thread sorting based on Date:     header
>  2. mailnews.use_received_date=false, Thread sorting based on Received: header
>  3. mailnews.use_received_date=true,  Thread sorting based on Date:     header
>  4. mailnews.use_received_date=true,  Thread sorting based on Received: header

Wada's first question seems to be whether to use Date or Received header in 
FILTERS for filtering of incoming mail.  Do I understand that right?

The next 4 questions seem to be about "Thread sorting", which clearly happens
after filtering (messages don't appear in the message list pane until they 
have been filtered).  I think they are actually asking two questions, which 
are:
a) when mailnews.use_received_date=false, should threads be sorted by Date: header or by Received: header?
b) when mailnews.use_received_date=true, should threads be sorted by Date: header or by Received: header?

My answer to those last questions is the subject of bug 383985, which states
that "Order of *threads* in msg list pane should not be controlled by Date 
Sent vs Date Received".  

But maybe Wada's didn't mean to ask about the order of threads, but rather 
the older of messages (in threads?  when display is threaded?  or not?).

Maybe the questions can be restated?
Nelson:

I think you're failing to realise that Received and Order Received are two very different things.

Received isn't currently supported in TB; this patch aims to make it so.
Order Received is something that is pretty much internal to TB; it's TB's internal track of the order messages were received and has *nothing* to do with the Received header - the name just happens to be similar.

What I think WADA was trying to suggest in his/her question was that 'Age in Days' and 'Date', in the filters, are currently based solely on the Date header, and perhaps some people would want them to be based on the Received header instead.

This is true, but the way I would suggest implementing that is:
'Date' always uses the Date header.
'Age in Days' perhaps uses the mailnews.use_received_date pref to determine which header to use.
To filter by Received, add a 'Received date' filter option.

However, that stuff will definitely not be added in the patch to this bug, and will need to be noted as a new RFE bug.
In answer to comment 212, the trunk is the place where the next MAJOR 
revision of the product is supposed to come from.  It's not a playground.
Changes made there are changed intended to appear in the next major release.
In reply to comment 214, Jeremy, thanks for your reply.

I'm OK with mailnews.use_received_date= consistently controlling whether 
date is taken from Date: or Received: headers.  
I'd certainly want WADA's choices 1 & 2, not 3 & 4!  
IOW, false == date from Date:, true== date from Received, except for 
places where the user has an explicit choice between the two.

(In reply to comment #215)
> the trunk is the place where the next MAJOR 
> revision of the product is supposed to come from.  It's not a playground.
> Changes made there are changed intended to appear in the next major release.

Well, I don't know if the TB trunk is anything like the FF trunk (I assume it is), but things are frequently checked into the FF trunk that aren't necessarily going to stay; for example, recently there was a patch checked in to grey out the non-hostname parts of the URL.  That was removed after a bunch of complaints.  They hadn't properly consulted before checking it in, and it was a bit of a 'playground' addition.  Frankly, I don't see the problem.  Trunk is meant to be for testing, and as long as a patch is relatively sound (which 1.8 is), it should be fair game for checking in.
I applied this patch, minus the "upgrade" code to add the received column, but the received column is still getting displayed, even after I removed it using the column picker. So I need to figure out why that is. I know you can't see any possible reason someone would want to remove the Received column, but I need to fix that before I can check this in :-)
So the problem is the function ShowReceivedColumn - whenever it's called with true, the received column is shown. My bad for not noticing how broken that was before. It basically copies the logic for ShowLocationColumn, but that's not the right thing to do. I'm not sure that this is even needed, since we show the Date value in the Received column, when there's no received value. It would be nice to switch in the Date column for those folders where Received doesn't make sense...but I'm not sure if we can do that easily.
Attached patch Patch v1.9 (obsolete) — — Splinter Review 
(In reply to comment #219)
> So the problem is the function ShowReceivedColumn - whenever it's called with
> true, the received column is shown.

Whoops - well spotted.  Patch v1.9 is the same as v1.8, except it uses a persistent 'washidden' attribute to only show the column if it was selected before.
Attachment #277591 - Attachment is obsolete: true
Attachment #277906 - Flags: review?(bienvenu)
Attachment #277591 - Flags: review?(bienvenu)
Attached patch Patch v1.9.1 (obsolete) — — Splinter Review 
... and of course I manage to screw up msgmail3panewindow a bit in v1.9.  This one is fixed.
Attachment #277906 - Attachment is obsolete: true
Attachment #277907 - Flags: review?(bienvenu)
Attachment #277906 - Flags: review?(bienvenu)
the other issue I saw last night was that threadpane.xul by default hides the date column and shows the received column - that's not what we want right now, so I've fixed it locally.
Comment on attachment 277907 [details] [diff] [review]
Patch v1.9.1

the washidden logic isn't quite right. washidden never gets set unless you switch to a folder that requires hiding of the received column. Otherwise, it never gets set. Thus, if I turn off the received column, shut down, and restart, the received column shows up again. I think washidden needs to get initialized with the hidden status of the received column, on startup. I'll try that.
Attachment #277907 - Flags: review?(bienvenu) → review-
I'm still not sure how to fix this - I think we need to know when the user hides the column, and when the code hides the column. I tried re-working it so the variable is called "temphidden", meaning the code hid it, but I'm not sure when to clear that - maybe when you click on a folder that doesn't require hiding, you clear "temphidden".

It's really not a nit - people are going to be annoyed if the columns keep switching back and forth on restart based on what the last folder they shut down with. I.e., as it stands now, imagine you have chosen to show the received column. If you read a newsgroup, and then shut down, when you restart, the received column will not show on the inbox.
this approach seems to work OK for temporarily hiding/showing the received column. I'd need to make the same changes to the SM version of commandglue.js, of course. Do you want to try it out, Jeremy? If it does work, I'd like to get an sr of what I have in my tree from Scott, and land that, and address remaining issues in smaller patches.

Someone brought up the question of the newsblog feeds - maybe we should be hiding the Received column for them as well.

Which brings up a larger point - I wonder if we should be showing the Date column, if it's hidden and the received column is visible, in these situations, instead of just hiding the Received column.  I think a user who has configured the thread pane to just show the received column is going to want to see some date column when looking at a newsgroup or newsblog folder. I guess we could get feedback from folks, but my gut tells me that's the right thing to do.

I don't see why we need all that logic in upgradefolderpane, to position the received column. By default, it shows up before the received column anyway (I know because I removed that part of the patch in my tree). It seems like a complication we don't need...
(In reply to comment #225)
> this approach seems to work OK for temporarily hiding/showing the received
> column. I'd need to make the same changes to the SM version of commandglue.js,
> of course. Do you want to try it out, Jeremy? If it does work, I'd like to get
> an sr of what I have in my tree from Scott, and land that, and address
> remaining issues in smaller patches.

It didn't seem to quite work for me.  I'm submitting this txt file showing what I think UpdateReceivedColumn should look like instead.  I don't think we even need to look at the old folder's type; just the new one, and some receivedCol attributes.

Also, you should remove the "receivedCol.setAttribute("washidden", true);" line from both msgMail3PaneWindow.js files.

> Someone brought up the question of the newsblog feeds - maybe we should be
> hiding the Received column for them as well.

Could do, I guess... but I can't see any flags (http://mxr.mozilla.org/seamonkey/source/mailnews/base/public/nsMsgFolderFlags.h) that are set for mail folders but not RSS folders, so I'd recommend doing that in a later patch, not this one.  It may well require extra code elsewhere, which is a horrible prospect for getting this checked in anytime this millenium.

> Which brings up a larger point - I wonder if we should be showing the Date
> column, if it's hidden and the received column is visible, in these situations,
> instead of just hiding the Received column.

Again, could do, but I'd recommend doing it in a later patch, not this one.  That's beyond core functionality.

> I don't see why we need all that logic in upgradefolderpane, to position the
> received column. By default, it shows up before the received column anyway (I
> know because I removed that part of the patch in my tree). It seems like a
> complication we don't need...

It's very little extra complication, and it just makes sure the new column is placed correctly.  I seem to recall the column being placed somewhere else when I initially created the code, so that's why I added the UI upgrade code.  I think it'd be safer to leave it in.
I think this version works, and is a bit cleaner - does it work for you?

Checking if a folder is in a blog account is not hard - you just get the server for the folder and see if it's of type "rss"
I could do with the full .patch...
Attached patch proposed patch for checkin (obsolete) — — Splinter Review 
this is the complete patch in my tree.

Re the thread pane upgrade code, I think it's a lot safer to leave it out for now, since the code seems to work without it, and we can't really point to something that it fixes. If we can figure out what it fixes, we can check it in later.

Re your irc comment, "var isHidden = ((receivedColumn.getAttribute("hidden") == "true") && (receivedColumn.getAttribute("ignoreincolumnpicker") == "true"));"

Can you say why that change is needed? What's the bug that it fixes? From my reading of the code, "ignorecolumnpicker" and "temphidden" should always have the same value, except that "ignorecolumnpicker" is not persistent, so it's a little less reliable (if you shut down with the received column temporarily hidden, "temphidden" will be true, but "ignorecolumnpicker" won't be. Unless the tree code persists it automatically, in which case they're identical)

Thx for working on this - I know it can be a painful process.
Attachment #277981 - Flags: superreview?(mscott)
(In reply to comment #229)
> this is the complete patch in my tree.

I think you should remove function ShowReceivedColumn...

> Re the thread pane upgrade code, I think it's a lot safer to leave it out for
> now, since the code seems to work without it, and we can't really point to
> something that it fixes. If we can figure out what it fixes, we can check it in
> later.

Create a new profile in something like TB 2.0.0.4, with an Inbox.  Then, close it and run the build of TB with this patch applied.  For me, the Received column appears at the bottom of the list, below 'Order Received'.  That UI upgrade code is needed in this patch, to move the column to the right place.

> Re your irc comment, "var isHidden = ((receivedColumn.getAttribute("hidden") ==
> "true") && (receivedColumn.getAttribute("ignoreincolumnpicker") == "true"));"
> 
> Can you say why that change is needed? What's the bug that it fixes?

If you unpick Received and then move to a folder that shouldn't show received, it's still in the picker.

> Thx for working on this - I know it can be a painful process.

You're telling me.  :-)  Thanks for the .patch addition.
> I think you should remove function ShowReceivedColumn...

oops, yes.

> Create a new profile in something like TB 2.0.0.4, with an Inbox. ...

OK, I'll try that.

> If you unpick Received and then move to a folder that shouldn't show received,
> it's still in the picker.

OK, I'll look at that.

Unfortunately, it won't be until tonight since I'm at the office now, away from the machine with this patch.

> > Re your irc comment, "var isHidden = ((receivedColumn.getAttribute("hidden") ==
> > "true") && (receivedColumn.getAttribute("ignoreincolumnpicker") == "true"));"
> > 
> > Can you say why that change is needed? What's the bug that it fixes?
> 
> If you unpick Received and then move to a folder that shouldn't show received,
> it's still in the picker.

... however, my suggested change there seems to break the code by always causing Received to appear when moving from a non-rcvd-showing to a rcvd-showing folder.

I just advise that you go with (maybe a compacted version of) my code, in attachment #277947 [details], which I've tested and seems to work correctly.
David: Another thing I just realised is that your patch doesn't include any of the nsParseMailbox.cpp or nsParseMailbox.h changes; these are obviously necessary.
(In reply to comment #232)
> ... however, my suggested change there seems to break the code by always
> causing Received to appear when moving from a non-rcvd-showing to a
> rcvd-showing folder.
> 

may i suggest that 'hardcoding' whether any column should be available to the picker or view based on folder type is unnecessary complexity.

plenty of columns don't make sense in rss or news folders, in the same way some feel (Sent) Date doesn't belong in imap/pop folders.  but there is a perfectly fine working solution that allows folder level column customization via the Mnenhy extension.  karsten, are you around?

integrating folder level column views would be a great enhancement to Tb3 and likely make everyone happy in this regard.
Attached patch Patch v2.0 (obsolete) — — Splinter Review 
Thought I'd submit a patch for the tree I compiled this weekend.  It seems to work fine with my new column handling code.
Attachment #277907 - Attachment is obsolete: true
Attachment #278302 - Flags: review?(bienvenu)
Attached patch Patch v2.1 (obsolete) — — Splinter Review 
Oops.  Removed the 'washidden' column setting.
Attachment #278302 - Attachment is obsolete: true
Attachment #278303 - Flags: review?(bienvenu)
Attachment #278302 - Flags: review?(bienvenu)
Comment on attachment 277981 [details] [diff] [review]
proposed patch for checkin

clearing request - still tweaking the patch...
Attachment #277981 - Flags: superreview?(mscott)
Attached patch fix problems jez noted (obsolete) — — Splinter Review 
this fixes the problem handling of the column picker that jez noticed, adds back the thread pane upgrade ui code, and includes the local src diffs.

Scott, this patch basically just adds a received column, defaulted to hidden, and adds sorting and grouping for that column..it also hides the column for newsgroups, and sent/drafts folders, which may turn out not to be to people's liking, but we can try it out...
Attachment #277953 - Attachment is obsolete: true
Attachment #277981 - Attachment is obsolete: true
Attachment #278303 - Attachment is obsolete: true
Attachment #278369 - Flags: superreview?(mscott)
Attachment #278303 - Flags: review?(bienvenu)
Comment on attachment 278369 [details] [diff] [review]
fix problems jez noted

looks good. You'll want Karsten to review the seamonkey UI changes.
Attachment #278369 - Flags: superreview?(mscott)
Attachment #278369 - Flags: superreview+
Attachment #278369 - Flags: review?(mnyromyr)
s/want/need. :-)
Comment on attachment 278369 [details] [diff] [review]
fix problems jez noted

This patch breaks SM's UI, firstly by a conflict in shared usage and secondly by some missing DTD changes.

>Index: mailnews/base/resources/content/commandglue.js
>===================================================================
> /* keep in sync with nsMsgFolderFlags.h */
>-var MSG_FOLDER_FLAG_VIRTUAL = 0x0020;
>+const MSG_FOLDER_FLAG_MAIL = 0x0004;
>+const MSG_FOLDER_FLAG_VIRTUAL = 0x0020;
> const MSG_FOLDER_FLAG_TRASH = 0x0100;

>Index: mail/base/content/commandglue.js
>===================================================================
> /* keep in sync with nsMsgFolderFlags.h */
>+var MSG_FOLDER_FLAG_MAIL = 0x0004;
> var MSG_FOLDER_FLAG_VIRTUAL = 0x0020;
> var MSG_FOLDER_FLAG_TRASH = 0x0100;

TB defines these values as vars to work around strict JavaScript clashes, while SM properly defines them as consts. But adding the const MSG_FOLDER_FLAG_VIRTUAL here will result in the shared SearchDialog.js to break, because it, too, defines this name: <http://lxr.mozilla.org/mozilla/source/mailnews/base/search/resources/content/SearchDialog.js#71>

I think we should replace that line with a comment that this value is defined in commandglue.js which is included by SearchDialog.xul.

>Index: mail/locales/en-US/chrome/messenger/SearchDialog.dtd
>===================================================================
>+<!ENTITY receivedColumn.label "Received">
>+<!ENTITY receivedColumn.tooltip "Click to sort by date received">

These need to go into /suite/locales/en-US/chrome/mailnews/SearchDialog.dtd also.

>Index: mail/locales/en-US/chrome/messenger/messenger.dtd
>===================================================================
>+<!ENTITY sortByReceivedCmd.label "Received">
>+<!ENTITY sortByReceivedCmd.accesskey "v">

These need to go into /suite/locales/en-US/chrome/mailnews/messenger.dtd also.

>+<!ENTITY receivedColumn.label "Received">
>+<!ENTITY receivedColumn.tooltip "Click to sort by date received">

These need to go into /suite/locales/en-US/chrome/mailnews/threadpane.dtd also.


With the above problems fixed, the patch runs smoothly in SM, too.

BTW: I'm not quite sure if the distinction between Drafts and Templates is useful. Showing "Received" for Drafts or Sent doesn't make much sense, but why should it for Templates?
Attachment #278369 - Flags: review?(mnyromyr) → review-
Attached patch Patch v3.0 (obsolete) — — Splinter Review 
Fixes problems Mnyromyr noted.
Attachment #279768 - Flags: review?
Attachment #279768 - Flags: review? → review?(mnyromyr)
Comment on attachment 278369 [details] [diff] [review]
fix problems jez noted

>Index: mailnews/base/resources/content/msgMail3PaneWindow.js
>===================================================================
> function UpgradeThreadPaneUI()
> {

For SeaMonkey trunk, there's nothing to migrate yet, so

  try
  {
    if (pref.getIntPref("mailnews.ui.threadpane.version") < 6)
      pref.setIntPref("mailnews.ui.threadpane.version", 6);
  }
  catch (ex) 
  {
    dump("UpgradeThreadPane: ex = " + ex + "\n");
  }

suffices here.


And as a correction of my last comment:
>>Index: mail/locales/en-US/chrome/messenger/SearchDialog.dtd
>>===================================================================
>>+<!ENTITY receivedColumn.label "Received">
>>+<!ENTITY receivedColumn.tooltip "Click to sort by date received">
> 
> These need to go into /suite/locales/en-US/chrome/mailnews/SearchDialog.dtd
> also.

This is not necessary, the threadpane.dtd already does this. Sorry for that.


BTW:

>Index: mail/base/content/msgMail3PaneWindow.js
>===================================================================
> function UpgradeThreadPaneUI()
> {
[...]
>+      receivedCol.setAttribute("washidden", true);

This attribute doesn't seem to be used anywhere...
Attached patch Patch v3.1 (obsolete) — — Splinter Review 
Fixes problems mentioned by Mnyromyr.
Attachment #279768 - Attachment is obsolete: true
Attachment #279772 - Flags: review?(mnyromyr)
Attachment #279768 - Flags: review?(mnyromyr)
Comment on attachment 279772 [details] [diff] [review]
Patch v3.1

>Index: mailnews/base/resources/content/commandglue.js
>===================================================================
> /* keep in sync with nsMsgFolderFlags.h */
>-var MSG_FOLDER_FLAG_VIRTUAL = 0x0020;
>+const MSG_FOLDER_FLAG_MAIL = 0x0004;
>+// this value is defined in commandglue.js which, is included by SearchDialog.xul:
>+// const MSG_FOLDER_FLAG_VIRTUAL = 0x0020;

This does not fix the search dilaog, but breaks commandglue also...
Attachment #279772 - Flags: review?(mnyromyr) → review-
This repairs v3.1 by implementing the SearchDialog.js related changes of comment 241, by request of jez.
David, are you okay with that?
Attachment #278369 - Attachment is obsolete: true
Attachment #279772 - Attachment is obsolete: true
Attachment #279785 - Flags: superreview?(bienvenu)
Attachment #279785 - Flags: review+
Comment on attachment 279785 [details] [diff] [review]
v4, with SearchDialog.js related changes as proposed in comment #241 [checked in]

Note for checkin, TWIMC: don't submit all the whitespace changes in SearchDialog.js, only the first hunk is relevant.
Comment on attachment 279785 [details] [diff] [review]
v4, with SearchDialog.js related changes as proposed in comment #241 [checked in]

yes, that's fine, Karsten, thx for asking.
Attachment #279785 - Flags: superreview?(bienvenu) → superreview+
Comment on attachment 279785 [details] [diff] [review]
v4, with SearchDialog.js related changes as proposed in comment #241 [checked in]

Landed on trunk. Whew!
Attachment #279785 - Attachment description: v4, with SearchDialog.js related changes as proposed in comment #241 → v4, with SearchDialog.js related changes as proposed in comment #241 [checked in]
Status: REOPENED → RESOLVED
Closed: 19 years ago → 17 years ago
Resolution: --- → FIXED
I'm not sure if this is valid etiquette, but I'd just like to say a big thanks to everyone involved in implementing this feature request. It obviously took a lot of effort most of which will undoubtedly go unrewarded by anything other than the vast appreciation of those of us who have wanted this feature for so long. If any of the developers involved are ever in the Seattle area, I'd be happy to take you to lunch or dinner or something.
Filed bug #396338 for allowing grouping/sorting by Received date.
QA Contact: olgam → backend
Target Milestone: --- → mozilla1.9 M8
Depends on: 402594
Who has set fixed1.8.1.5 in keywords: field?
WADA, In the upper right corner of this (or any) bug page, immediately below 
the reporter's name, you will find a link labelled "(View Bug Activity)".
That link will take you to a page that will show you who changed what and 
when for all changes since this bug was created.  

It tells me the keyword was added by 
bienvenu@nventure.com  	2007-06-30 19:07:13 PST  	
Oh sorry, I missed it(too many rows in View Bug Activity...)

Only following patch looks to be landed on branch.
> only used received: hdr if no Date: header or user sets hidden pref 2005-12-05 14:15 PST
This resolved problem of Bug 73565 (see Comment #64), and this resolved this bug's request(Received column) partially by changing Date column meaning to "Received date:" if hidden pref is set.

Current implementation/enhancement for this bug(Received column) was implemented by patches after above patch, which are landed on trunk only.
Keeping "fixed1.8.1.5" in keywords: field is proper action?

(In reply to comment #254)
> Only following patch looks to be landed on branch.
> > only used received: hdr if no Date: header or user sets hidden pref 2005-12-05 14:15 PST

That's very old and I didn't know about it actually, still...

> This resolved problem of Bug 73565 (see Comment #64), and this resolved this
> bug's request(Received column) partially by changing Date column meaning to
> "Received date:" if hidden pref is set.

I think this bug's resolution is a significantly better way to do things, as it exposes the difference between Received and Date in the UI, allowing the user to select either column (or both).

> 
> Current implementation/enhancement for this bug(Received column) was
> implemented by patches after above patch, which are landed on trunk only.
> Keeping "fixed1.8.1.5" in keywords: field is proper action?
> 

Dunno why bienvenu has set that.  He's a strange guy.

;-)
The first work on this issue was landed on the 1.8.1 tree, iirc. All the later work, of course, was not, hence the confusion, I assume.
Product: Core → MailNews Core
Hi folks, thanks for improving the "date" field, but this still is not working the way I would have expected.

My email has been slow lately, and that highlights the problem.  For example, a colleague sent an email at 09:25 (his computer's time), and the first server received it at 09:26 (serer time).  Then, my provider sat on it for a while (they are having trouble with their email queue), and sent it to me at 10:44 (server time).  However, it didn't actually get to my inbox until 12:15 (my computer's time).

In this example, I would expect to see 09:26 in the Date column, but it actually shows 10:44.  This seems a bit arbitrary - I'm not really interested in the internal functionality of mail servers shuffling email between themselves - I just want to know when the message was sent.

So I would propose changing this feature from using the "last available server date/time", to "first available server date/time".  I don't know how this forum works - should I start a new bug ticket for this question?  Thanks, and sorry for the trouble.


-------------------------------
----- Relevant headers:   -----
-------------------------------
From - Wed Sep 03 12:15:53 2008
Return-Path: <foo@unc.edu>
Delivered-To: foo@foo.se
Received: (qmail 43742 invoked by alias); 3 Sep 2008 14:44:23 -0000
Delivered-To: foo@foo.se
Received: (qmail 48133 invoked from network); 3 Sep 2008 13:26:04 -0000
Received: from unknown (HELO bw2.loopia.se) ([194.9.94.195])
          (envelope-sender <foo@unc.edu>)
          by s60.loopia.se (qmail-ldap-1.03) with SMTP
          for <foo@foo.se>; 3 Sep 2008 13:26:04 -0000
Authentication-Results: bw2.loopia.se
	header.from=foo@unc.edu; domainkeys=neutral (no sig)
Received: from smtp.unc.edu (smtpsrv1.isis.unc.edu [152.2.1.140])
	by bw2.loopia.se (BorderWare Security Platform) with ESMTP id 56CB09143C1DE8BA
	for <foo@foo.se>; Wed,  3 Sep 2008 15:26:03 +0200 (CEST)
Received: from [127.0.0.1] (pha00111.dhcp.unc.edu [152.2.146.120])
	(authenticated bits=0)
	by smtp.unc.edu (8.13.4/8.13.4) with ESMTP id m83DPaJ5025884
	for <foo@foo.se>; Wed, 3 Sep 2008 09:25:37 -0400 (EDT)
Message-ID: <48BE9050.9090703@unc.edu>
Date: Wed, 03 Sep 2008 09:25:36 -0400
From: Foo Foo <foo@unc.edu>
Mattias: I'd say that's correct behaviour.  It's called 'Received' because the idea was that it tells you when your mail server received it.  In your example, I'd say it is exhibiting the expected behaviour.
Jeremy: I really appreciate the work that has gone into this!  The field in TB is called 'Date'.  As a result I expected this column to behave like in other email software (for example Outlook, Outlook Express) with a 'Date' column.  With delayed email, it's showing some arbitrary time that is somewhere inbetween when the email was sent and when I got it.  It's neither 'Sent date' nor 'Received date', it's more like 'On-the-way date'.

I do feel that the fix that was posted vastly improves the situation (for example, when the sender's date/time is incorrect), but in the case when mail is delayed between servers, it still does not work perfectly.  See Comment #25 by Robie, and Comment #186 by Nelson.

One resolution that might put this to rest could be to offer some hidden tweaks where the user can choose whether Date should show 'Date Received' or 'Date Sent', with some pretty specific explanation of each.  Right now, I have no control and it is not doing what I expected.  Thanks!
It's only useless if you expect it to be a sent date.  Any work I did on the patch was specifically designed such that the e-mails would be displayed in the order in which they were received at the server on my end; this was and is intentional.  I think it provides better continuity, because I always know which server's clock that received date is referring to - mine.
Jeremy: can we not have 260 comments worth of debate again? The point is that 

* some people want messages in the order they were sent in order to eliminate the effect of network delays, choosing to trust the clocks of their senders, 

and 

* other people want them in the order they were received, choosing to eliminate the effect of clock skew at the sender but choosing to trust the network to deliver everything in about the same amount of time.

I'm not sure this question has a right answer for everyone. I suspect we just need to have two columns - "Sent", which uses the Date header, and "Received", which uses the time of the last mailhop.

Gerv
...which is exactly what we have at the moment.
I assume this is all still in the 3.0 beta?  I don't see it in 2.0.0.16.
(In reply to comment #263)
> I assume this is all still in the 3.0 beta?  I don't see it in 2.0.0.16.

It should have been fixed for Thunderbird 2.0.0.5 but you are right, I even don't have this field. I can see it with the latest trunk build but not with Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.16) Gecko/20080707 Thunderbird/2.0.0.16 Mnenhy/0.7.5.0 ID:2008070710
I don't think it's been backported into TB2.  It was tough enough getting it checked in, let alone backported.
Then we should remove the fixed1.8.1.5 keyword. David any thoughts?
See comment 135, and please try desperately not to add any more.
If there is a delay between mailservers then I might receive one email "I have a problem - nevermind" before I receive the email saying "I have a problem".  The "date" column really isn't reliable for me.  See comment #257 and comment #259.

Can someone point me to a plugin or something that will create a "sent date" column (that's not the sender's computer's date), or which bug already reports this problem?  Sorry for the additional comment on this bug which I realize is just about getting a "received date" which is not really what I'm looking for.
I just received a message where the "Date" column reports 2:40PM, but when I open the message, the "Date:" field says 12:42PM.  Should they be the same date/time, or different?  What is "date" in this case?  Is that related to this bug, or a new bug, or not a bug?
This bug is closed. Like always, file a new bug for any issue that isn't "this bug didn't get checked in" or "didn't do anything of what it was supposed to".
As far as I can tell, either this bug didn't get checked in, or didn't do anything of what it was supposed to.

My understanding was that the patch in comment 246 was supposed to add a "date received" column that can be displayed and sorted on.  I just downloaded production Thunderbird 3.0, and it's not there.  What am I missing?

I apologize for bringing up this bug yet again, but comments 263 through 274 feature several people wondering about this feature, and no one quite explaining when we can see this new column in production Thunderbird.  I have been following this bug for years, literally waiting for today and 3.0, and I hope someone can explain to me what's going on.
It was checked in, and does do what it was supposed to.

The list of columns you can display and sort by now includes "Received", and it works great.
Aha, but not on IMAP due to bug 402594.  Nevermind!
Oh. You're right. I guess it's time to vote for another bug. =(
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: