266434 - if msg doesn't have a Date: header, we use the time we received the message

Status: REOPENED

(Thunderbird :: Mail Window Front End, defect)

Description

[Henry Lander]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; rv:1.7.3) Gecko/20040913 Firefox/0.10
Build Identifier: Thunderbird 20040913

The date for the emails appears to be the date/time at which the email was
downloaded into Thunderbird.  This is not very useful.  If I open Thunderbird,
and three new emails are downloaded, they all are displayed the current
date/time (the time at which I opened Thunderbird).  What I really want is the
date/time they were recieved on my email server.  

When I look at the email headers I can see that the time of deliver is available
there.  For example right now I have an email which has the following header:
Delivery-date: Tue, 26 Oct 2004 11:59:27 -0500
Yet this email has the following date/time on it in the list: 10/27/2004 3:45
PM.  This is the date/time that the email was downloaded to my PC when I opened
Thunderbird.

Reproducible: Always
Steps to Reproduce:
1. I have thunderbird configured to download the emails when I start Thunderbird.
2. If I have any new emails they show up with the current date/time, not the
date/time at which they were received on the server.
3. I am unhappy because I cant sort the emails in a meaningful chronological order.
4. I file a lengthy and detailed bug report in Bugzilla.

Comment 1

[David :Bienvenu]

the date we display is the date set by the sending e-mail program, at send time,
not the date you received the message on your computer. There is already a RFE
file to use the received by: header for a timestamp.

Comment 2

[Henry Lander]

(In reply to comment #1)
> the date we display is the date set by the sending e-mail program, at send time,
> not the date you received the message on your computer. There is already a RFE
> file to use the received by: header for a timestamp.

I am quite sure that I am seeing the current time.  I just launched Thunderbird.
 I had one new email with a time of 11:44AM (without the date because its the
current date) -- exactly to the minute the current time on my PC, not the time
the email was sent nor received on my email server.  If someone gives me an
email address, I can send you an image of my screen which shows the date
displayed by TBird and the headers of the email message so that you can see it
for yourself.

Comment 3

[Mike Cowperthwaite]

You can attach a screenshot to the bug, you don't have to mail it to anyone; use 
the Create New Attachment link above.  And please keep it reasonably small.

What is the actual   Date:   header in the mails that are supposedly showing the 
wrong time?

Comment 4

[Henry Lander]

Attachment: Shows an email with the wrong time and all headers (41K).

Comment 5

[Henry Lander]

I have added an attachement which shows the problem.  The hilighted email shows
a time of 11:44AM (current date is 10/28/04).  I also show the header of the
email which has various date/time stamps in it.  I am not familiar with email
protocols, and so I do not know how to interpret these headers.  However, I want
to reiterate that 11:44AM is the time at which I opened the TBird client. 

Sorry about not includeing the attachment before -- I am still learning how to
use bugzilla.  Thanks.

Comment 6

[Mike Cowperthwaite]

That message apparently doesn't *have* a Date header!  I wonder where the 
Delivery-date header is being added.

Comment 7

[David :Bienvenu]

so all these messages don't have a Date: header? Yes, in that case, this would
really be a dup of the bug that says we should parse the received by headers.

Comment 8

[Mike Cowperthwaite]

See bug 238532 about the Delivery-Date: header.
See bug 73565 about the missing Date: header.

Comment 9

[Mike Cowperthwaite]

(In reply to comment #7)
> so all these messages don't have a Date: header? Yes, in that case, this would
> really be a dup of the bug that says we should parse the received by headers.

I don't necessarily agree with that -- that's bug 216033, which seems to be 
about displaying *all* mails according to Date Received, whereas I prefer the 
original Date: header to be used when possible.

Comment 10

[Mike Cowperthwaite]

*** Bug 270176 has been marked as a duplicate of this bug. ***

Comment 11

[Gervase Markham [:gerv]]

This is an automated message, with ID "auto-resolve01".

This bug has had no comments for a long time. Statistically, we have found that
bug reports that have not been confirmed by a second user after three months are
highly unlikely to be the source of a fix to the code.

While your input is very important to us, our resources are limited and so we
are asking for your help in focussing our efforts. If you can still reproduce
this problem in the latest version of the product (see below for how to obtain a
copy) or, for feature requests, if it's not present in the latest version and
you still believe we should implement it, please visit the URL of this bug
(given at the top of this mail) and add a comment to that effect, giving more
reproduction information if you have it.

If it is not a problem any longer, you need take no action. If this bug is not
changed in any way in the next two weeks, it will be automatically resolved.
Thank you for your help in this matter.

The latest beta releases can be obtained from:
Firefox:     http://www.mozilla.org/projects/firefox/
Thunderbird: http://www.mozilla.org/products/thunderbird/releases/1.5beta1.html
Seamonkey:   http://www.mozilla.org/projects/seamonkey/

Comment 12

[Gervase Markham [:gerv]]

This bug has been automatically resolved after a period of inactivity (see above
comment). If anyone thinks this is incorrect, they should feel free to reopen it.

Comment 13

[ISHIKAWA, Chiaki]

Just thought to update that there was a bug in later code that did not handle
the parsing of Received: header when Date: header is missing.

It seems that back in 2004-2005, such messages ended up with the received time (i.e., current time of TB when it receives such e-mails.)

However, the later code changes broke this behavior somehow.
Basically, every e-mail ended up with the UNIX epoch time if Date: was missing
MOST of the time. 
There are two issues: one had something to do with incorrect 
index calculation when Date: was missing, which resulted in garbage memory buffer being passed to the parsing routine for date/time string. If the garbage memory area contains a valid date/time string, it would have been picked up. Yes, the chance was low, but it could happen. But more serious problem is that if the garbage memory buffer didn't contain '\0', TB could have gone over the end of the memory buffer to the unallocated territory.
 
The problem(s) were fixed lately this year and the patch is being baked to see if this is not too disruptive and can be landed safely. As you notice, the date order is now
correct (or the same date/time that would have been given by TB in 2004-2005 period.)

Since the incorrect memory access is involved when the bogus memory buffer is passed around, the particular bug entries are being marked as security-related.
Once the patches would land, we will see a better behavior for the e-mails without
Date:  headers, I hope. 

Just hit this bugzilla entry when one of my colleagues suddenly saw his outgoing e-mail timestamped with a date and time of a different timezone, and searched for similar problems.
 
 
TIA

Comment 14

[lumartineru]

Attachment: TB Date bug

Comment 15

[lumartineru]

I just installed TB for the first time and downloaded by Yahoo Email. I have a message from 2010 with this problem. As you can see in the attachment 820534 [details], when sorting by date, it is shown as today (Oct. 22, 2013 12:34 pm), whereas the Date: and Received: headers is Mon, 14 Jun 2010 07:46:39 PDT. I have more messages since 2008, 2009, 2010, and so without this problem.

Comment 16

[ISHIKAWA, Chiaki]

(In reply to lumartineru from comment #15)
> I just installed TB for the first time and downloaded by Yahoo Email. I have
> a message from 2010 with this problem. As you can see in the attachment
> 820534 [details], when sorting by date, it is shown as today (Oct. 22, 2013
> 12:34 pm), whereas the Date: and Received: headers is Mon, 14 Jun 2010
> 07:46:39 PDT. I have more messages since 2008, 2009, 2010, and so without
> this problem.

I think this should be investigated to see, at least, to find out whether 
we need to open a new bug or not.

Reporter can you check that other e-mails from the same sender show up correctly in terms of the
chronological order?

TIA

Comment 17

[lumartineru]

Attachment: Other emails with problem

The sender is myself (From and To my Yahoo account) and I have several more emails with no problema. Yesterday I found more emails. I'll attach an image where you can see from serviciosbancomer.com. The ones at the top have no Date: header and the one below does have Date: header. The problema seems to be again.

Comment 18

[ISHIKAWA, Chiaki]

(In reply to lumartineru from comment #17)
> Created attachment 820978 [details]
> Other emails with problem
> 
> The sender is myself (From and To my Yahoo account) and I have several more
> emails with no problema. Yesterday I found more emails. I'll attach an image
> where you can see from serviciosbancomer.com. The ones at the top have no
> Date: header and the one below does have Date: header. The problema seems to
> be again.

Hmm...

No Date Header ... I thought such an e-mail may have received the timestamp of 
                   POSIX epoch-time 00:00 Jan 1, 1970, (See Bug 73565 - Messages that lack a Date: header are displayed as "sent in 1969/12/31"(or 1970/01/01, Epoc time) but
                   in any case, an e-mail without Date: header is tread as special case.
                   OK, to start with, I think as the discussion in Bug 73565 stated I thought
                   such an e-mail would have used epoch time 1971/01/01 or 1969/12/31.
                   So maybe there has been a change since then. OK, Bug 216033 and Bug 166254 
                   seemed to take care of this. 

With Date Header ... Can you kindly show the exact header line like you showed in the previous upload?
                 I suspect that the Date header line may have a format bug in it (in which case TB
                 decides to assign a timestamp of its choice as in the case of "No Date Header line."
                 

So then the question I would like to ask
 - please show the exact header line of the problematic e-mail like you did in the previous upload.

 - which mailer did you use to send this problematic e-mail to you?
   If it is TB, then I think we should create a new bugzilla entry to start the investigation afresh.
 
 - Did other e-mails you(?) sent also use the same mail client?
   Can you show the exact and whole Date header line of such e-mail for comparison?  

Now I notice the problem, Bug 73565 was from the year 2000, and some improvements have been
made since then, but we may still see some problems even today. Interesting.

Comment 21

[Matthew Jurgens]

Attachment: tb1.JPG

Comment 22

[Matthew Jurgens]

This is still an issue with TB 64 Beta 2 using an IMAP server with maildir.

The only date header that these mails contain is 
Delivery-date: Mon, 19 Mar 2018 09:36:43 +1100

However, the date is shown as the date that the mail was downloaded from the server (presumably the file modification date or something). I just added the IMAP account to a new computer and hence the mails were downloaded and the date shown in the GUI as the download date/time.

See the attachment: https://bugzilla.mozilla.org/attachment.cgi?id=9025855 for the GUI view.

Return-Path: <[e-mail]>
Delivered-To: [e-mail]
Received: from s311.syd1.hostingplatform.net.au
	by s311.syd1.hostingplatform.net.au with LMTP id 6DNPMPvprlqQPAQAXzbHgg
	for <[e-mail]>; Mon, 19 Mar 2018 09:36:43 +1100
Return-path: <[e-mail]>
Envelope-to: [e-mail]
Delivery-date: Mon, 19 Mar 2018 09:36:43 +1100
Received: from mg-aaus-gamma.mailguard.com.au ([13.210.145.47]:45820)
	by s311.syd1.hostingplatform.net.au with esmtps (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256)
	(Exim 4.89_1)
	(envelope-from <[e-mail]>)
	id 1exgus-001Amw-N3
	for [e-mail]; Mon, 19 Mar 2018 09:36:43 +1100
Received: from mg-aaus-gamma.mailguard.com.au (mg-aaus-gamma.mailguard.com.au [127.0.0.1])
	by mg-aaus-gamma.mailguard.com.au (Postfix) with ESMTP id 321891802BC9;
	Mon, 19 Mar 2018 09:35:58 +1100 (AEDT)
Received: from AUS2DOM01.TUPPERWARE.COM.AU (unknown [150.207.160.28])
	by mg-aaus-gamma.mailguard.com.au (Postfix) with ESMTPA id 8D8B81803F5F
	for <[e-mail]>; Mon, 19 Mar 2018 09:35:31 +1100 (AEDT)
From: [e-mail]
To: [e-mail]
Message-ID: <[e-mail]>
Subject: Tupperware Order Confirmation - # 26561268

Comment 24

[GiovanniP]

I can confirm this on 60.6.1: email without a Date: header are shown with date of IMAP sync (the date of the file on the local thunderbird disk). This morning this produced a big family quarrel on an email sent by our alarm/intusion system.
Please fix it, and make the last Received header date as the displayed date, and not the file download date.

Comment 26

[osdm]

In my case this bug seems to be fixed by setting "mailnews.customDBHeaders" to "Received" in config editor. Some other users also confirm that this fixed their problems (see http://blog.dmitryleskov.com/small-hacks/putting-the-received-column-in-thunderbird-to-work/). Is it a possible solution?

Comment 27

[qdjilali.b]

I experienced this bug and can confirm it is still happening as of the current latest Thunderbird 78. I imported a large imap account to thunderbird with thousands of emails, and a handful of them apparently had improperly formatted headers so the dates of those emails were shown as being received during the import process, instead of several years ago. The blog post mentioned by osdm fixed it for me, and now all of those emails are sorted correctly during the date which they were actually received.

Comment 28

[bugzilla]

I have this problems with email sent by my alarm/intusion system.
Please fix it, and make the last Received header date as the displayed date, and not the file download date.

Comment 29

[ISHIKAWA, Chiaki]

(In reply to Matthew Jurgens from comment #22)

This is still an issue with TB 64 Beta 2 using an IMAP server with maildir.

The only date header that these mails contain is
Delivery-date: Mon, 19 Mar 2018 09:36:43 +1100

However, the date is shown as the date that the mail was downloaded from the
server (presumably the file modification date or something). I just added
the IMAP account to a new computer and hence the mails were downloaded and
the date shown in the GUI as the download date/time.

See the attachment: https://bugzilla.mozilla.org/attachment.cgi?id=9025855
for the GUI view.

So the question is either we may want to add "Deliver-date:", which is not quite standard IMHO,
See Bug 612990
or
do something with the latest "Received:" header lines.
Actually, Bug 612990 comment the first Received header we see from the start (the last added Received: )
usually is very close to Delivery-date:.
So we might simply do that.

One thing I noticed lately during the fix for a different bug 1699968
is that
TB may not properly handles the timestamp when mDatabase is null.
It does not call a function to set time stamp or something to that effect.
So this observation may not affect a particular message in question, but that may need be kept in mind.
(This is actually a reminder to myself.)

Comment 30

[ISHIKAWA, Chiaki]

(In reply to bugzilla from comment #28)

I have this problems with email sent by my alarm/intusion system.
Please fix it, and make the last Received header date as the displayed date, and not the file download date.

So there is a sizeable different time between Recevied header date and the file download date?

Comment 31

[ISHIKAWA, Chiaki]

Of course, the problem is with the sender (maybe automated web CGI/JS or whatever) which does not bother to add "Date:" line.
But in communication, the golden rule is try to be strict about the protocol conformance with what we send out,
but a bit lenient about what we receive.

Comment 33

[Alfred Peters [:infofrommozilla]]

All you need is to enable the download of the Received header (Bug 402594).
You can do so through adding 'received' to the 'mailnews.customDBHeaders' pref (Bug 1361643 bug comment #22).

Or use Menu: View -> Messages -> Customize... which will set 'mailnews.customHeaders' pref which will also effect customDBHeaders.
https://searchfox.org/comm-central/rev/3a01c489963cb3bc423bb3b78e9cbecbec6b99fe/mailnews/local/src/nsParseMailbox.cpp#475

Comment 35

[Alfred Peters [:infofrommozilla]]

(In reply to ISHIKAWA, Chiaki from comment #29)

So the question is either we may want to add "Deliver-date:", which is not quite standard IMHO,
See Bug 612990

Regarding standard or non-standard, I have no concerns. The question is rather whether we should edit the content of the e-mail at all. Or would it be more appropriate to introduce an X-Mozilla-DeliveryDate header?

In any case, this would only be a solution for POP3 or local folders. Uploading the modified mail back to an IMAP account is probably out of the question.

Comment 38

[Anje]

I've always understood that:
'Date' is the date the composer created the email. That email may have been created at night, but not actually sent until the morning.
That date may appear to be a different day if significant timezone is involved.

'Received' date is the date the receiving server for mail account actually received the email. In headers that would be the one at the top.

The downloaded date is different again because email may have been on server for days, but computer with Thunderbird installed was not switched on etc. Although in most cases, emails are received by server and downloaded soon thereafter.

The dates displayed can be adjusted depending upon the date/time/timezone set up on computer which is running Thunderbird.
But you would assume all computers do have correct date/time setup.

Example of date:
Bugzilla email says comment changed in bug: at 2022-06-16 18:36:21 PDT

'Date' in received email says: Fri, 17 Jun 2022 01:36:22 +0000
'Received' on server date: Fri, 17 Jun 2022 02:36:25 +0100

But the info at top of 'Source View' says : From - Fri Jun 17 11:12:30 2022 and that was the time I started Thunderbird and downloaded the email. Email various dates - clearly adjusting for timezones as I'm in the UK.

I have preference: mailnews.customDBHeaders Received
It works perfectly on both pop and imap accounts.

In my case, I had inadvertantly set this up whilst doing some filtering some time ago - I had no idea this had fixed the bug, so was perplexed when others had the problem.

This is a request for consideration to perform a quick fix on this bug, so it can be marked as resolved.
Perhaps we could resolve this bug if this preference was set by default: mailnews.customDBHeaders Received
Can this be done for the next update as it is not too complicated?
Thoughts on whether this would be a reasonable action?

Comment 39

[ISHIKAWA, Chiaki]

(In reply to Alfred Peters from comment #35)

(In reply to ISHIKAWA, Chiaki from comment #29)

So the question is either we may want to add "Deliver-date:", which is not quite standard IMHO,
See Bug 612990

Regarding standard or non-standard, I have no concerns. The question is rather whether we should edit the content of the e-mail at all. Or would it be more appropriate to introduce an X-Mozilla-DeliveryDate header?

In any case, this would only be a solution for POP3 or local folders. Uploading the modified mail back to an IMAP account is probably out of the question.

Right. I was thinking of POP3 or local mail folder situation more or less.
I need someone with heavy IMAP usage experience to comment on the matter.

Comment 40

[Alfred Peters [:infofrommozilla]]

(In reply to Anje from comment #38)

I've always understood that:
'Date' is the date the composer created the email. That email may have been created at night, but not actually sent until the morning.
That date may appear to be a different day if significant timezone is involved.

Correct. The Date header. But some mail clients don't set this header.

'Received' date is the date the receiving server for mail account actually received the email. In headers that would be the one at the top.

The "Received Header" is a receipt stamp generated by the respective server through which the message was transported in order to be able to trace the transport route. The date field is only a side information and IIRC even optional.

The downloaded date is different again because email may have been on server for days, but computer with Thunderbird installed was not switched on etc. Although in most cases, emails are received by server and downloaded soon thereafter.

Correct. We store this date in our folder database. You can display it in the folder pane by showing the 'Received' column.
Unfortunately, this information is lost when you perform a "Repair Folder".
Therefore, it would be helpful to additionally store this information in the header of the mail itself:

(In reply to ISHIKAWA, Chiaki from comment #39)

(In reply to Alfred Peters from comment #35)

Regarding standard or non-standard, I have no concerns. The question is rather whether we should edit the content of the e-mail at all. Or would it be more appropriate to introduce an X-Mozilla-DeliveryDate header?
Right. I was thinking of POP3 or local mail folder situation more or less.

I agree with you.

(In reply to Anje from comment #38)

I have preference: mailnews.customDBHeaders Received
It works perfectly on both pop and imap accounts.

IMAP is a different case - handled by Bug 402594.

Comment 41

[Alfred Peters [:infofrommozilla]]

What we are talking about here is the Date column. As source we should then use the earliest available date. If available of course also the Received header. This should be available when receiving via POP3.

Comment 42

[GiovanniP]

I agree with Alfred Peters: in this thread we are talking about on what is shown on the Date column, and how this column is ordered.
An e-mail without a Date: header currently shows the date of the 1st download from the IMAP server in the Date column, so you can never find that email in the correct chronological order.
Thunderbird should guess the value to show on the Date column using the Received: headers, when Date: header is missing.
No modifications to the e-mail are needed.

Comment 43

[Alfred Peters [:infofrommozilla]]

(In reply to Alfred Peters from comment #40)

The "Received Header" is a receipt stamp generated by the respective server through which the message was transported in order to be able to trace the transport route. The date field is only a side information and IIRC even optional.

The downloaded date is different again because email may have been on server for days, but computer with Thunderbird installed was not switched on etc. Although in most cases, emails are received by server and downloaded soon thereafter.

Correct. We store this date in our folder database. You can display it in the folder pane by showing the 'Received' column.

I have just tested a email without Date header and am surprised by the result.
With POP3, the Received column shows me the date from the Received header and the Date column shows me the time of receipt in the TB. This should be the other way around in any case.

It should be:
Date column -> sent date -> earliest date
Received column -> receive date -> latest date

Has it always been like this? Or could this be an regression? Maybe by Bug 1707548?

Comment 44

[ISHIKAWA, Chiaki]

(In reply to Alfred Peters from comment #43)

(In reply to Alfred Peters from comment #40)

The "Received Header" is a receipt stamp generated by the respective server through which the message was transported in order to be able to trace the transport route. The date field is only a side information and IIRC even optional.

The downloaded date is different again because email may have been on server for days, but computer with Thunderbird installed was not switched on etc. Although in most cases, emails are received by server and downloaded soon thereafter.

Correct. We store this date in our folder database. You can display it in the folder pane by showing the 'Received' column.

I have just tested a email without Date header and am surprised by the result.
With POP3, the Received column shows me the date from the Received header and the Date column shows me the time of receipt in the TB. This should be the other way around in any case.

Which version of TB did you use?

I have checked using the latest C-C TB locally compiled.
I send an e-mail without Date header (that is what I thought, but read on)
the Received and Date fields showed the time it was sent.

Well, I thought /usr/lib/sendmail under linux once allowed me to send an e-mail without Date: header line attached from
the sender, but it seems that it might have added Date: header automatically (or did the dovecot server that received the e-mail added it or did TB add that?).
I found that the header lines in the e-mail contained Date: header line.
E.g.:

From - Mon Jun 20 13:56:24 2022
X-Account-Key: account1
X-UIDL: 000157b2533eeb2c
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
X-Mozilla-Keys:                                                                                 
Return-Path: <[e-mail]>
X-Original-To: ishikawa
Delivered-To: [e-mail]
Received: by ip030.example.localdomain (Postfix, from userid 1000)
	id D0CDF271; Mon, 20 Jun 2022 13:38:42 +0900 (JST)
From: [e-mail]
Subject: 13:38
Message-Id: <[e-mail]>
Date: Mon, 20 Jun 2022 13:38:24 +0900 (JST)  <--- This obviously was the time it was sent.

test without Date mail

I read the mail at about 13:57.

OK, I further checkedd. So actually, /usr/lib/sendmail on my Debian GNU/linux is one of the postfix suite programs in disguise.

ishikawa@ip030:/NREF-COMM-CENTRAL/mozilla/comm$ strings /usr/lib/sendmail | grep -i post
libpostfix-global.so
libpostfix-util.so
/usr/lib/postfix
the Postfix sendmail command has set-uid root file permissions
the Postfix sendmail command must be installed without set-uid root file permissions
option -V is deprecated with Postfix 2.3; specify -XV instead
option %s is deprecated with Postfix 2.3; specify -X%s instead
/etc/postfix
postqueue
%s/postdrop -r
postalias
/usr/lib/debug/.dwz/x86_64-linux-gnu/postfix.debug
ishikawa@ip030:/NREF-COMM-CENTRAL/mozilla/comm$ 

And postfix automatically add Date: header line if the mail message is missing it if configuration file says so.
From: https://www.postfix.org/postconf.5.html

always_add_missing_headers (default: no)
    Always add (Resent-) From:, To:, Date: or Message-ID: headers when not present. Postfix 2.6 and later add these headers only when clients match the local_header_rewrite_clients parameter setting. Earlier Postfix versions always add these headers; this may break DKIM signatures that cover non-existent headers. The undisclosed_recipients_header parameter setting determines whether a To: header will be added.

BUT I don't see |always_add_missing_headers| in main.cf for postfix. But Debian is known to tweak configuration files of programs for maintenance purposes and I may have missed it.

Now as for , my pop3 server that I use for testing is part of dovecot suite. When I send the e-mail with

/usr/lib/sendmail ishikawa
From: ishikawa
Subject: test header line

    ...
.

The mail is delivered to local dovecot mail server and that could be when missing Date: header line may have been added.

So many environmental setup factors to get right.

It should be:
Date column -> sent date -> earliest date
Received column -> receive date -> latest date

I agree. At least Received should be the latest. But Received is the received time the last mail server from which one reads
received the e-mail. Not the time you read the e-mail from the mail host.

So in my case, since the delivery to the final mail server is about 20 seconds I sent the mail. So Received: shows very similar time to Date.

Now, I have to figure out how to send an e-mail without Date: reliably.
(Maybe I replace postfix-in-disguise sendmail with a real sendmail.)

Has it always been like this? Or could this be an regression? Maybe by Bug 1707548?

I thought it was OK before. I have to find an environment to test this correctly.

Comment 45

[Magnus Melin [:mkmelin]]

(In reply to Anje from comment #38)

Perhaps we could resolve this bug if this preference was set by default: mailnews.customDBHeaders Received
Can this be done for the next update as it is not too complicated?

It could be considered in general. It would have known and unknown performance - and perhaps other - implications though. So it would need time and analysis.

Comment 46

[Ping Chen [:rnons] (away)]

(In reply to Alfred Peters from comment #43)

I have just tested a email without Date header and am surprised by the result.
With POP3, the Received column shows me the date from the Received header and the Date column shows me the time of receipt in the TB. This should be the other way around in any case.

It should be:
Date column -> sent date -> earliest date
Received column -> receive date -> latest date

Has it always been like this? Or could this be an regression? Maybe by Bug 1707548?

pop3-js didn't do any headers processing, parsing and saving are delegated to nsIPop3Sink.

Comment 47

[Anje]

Perhaps it would be easier to understand if there was more clarity on what 'Received' actually means rather than leave it's contents be interpreted on an adhoc basis.
It has always been 'received on server' - so would it help clarify the meaning of what people see if that column header was renamed to 'Server Received'. So it's functionality and contents do not change and adding the 'Received' to the mailnews.customDBHeaders then works as well.

If you have the 'Received' aka 'Server Received' date working it solves getting the Server received date instead of a duplicate created date in the Date column. This is a quick fix for one part of the problem.

'Date' has always been the date the email was composed and not the date/time sent. Although in most cases it would appear to be about the same.
For example email composed at 16:41 hrs and use 'Send Later'. Wait until eg: 16:45 and then then send unsent messages.
The received incoming email will have 'Date' as 16:41 (obviously this was composed and sent same day) and not 16:45hrs.

But for email sent with no date then it seems Thunderbird applies alternative being 'Download Date'.
I see the options being:

1. Either do nothing with 'Date' as it seems entering the download date is a good alternative for those few emails with no date.
2. Leave 'Date' empty if no date and offer another new 'Download Date' column.

I think there is a good arguement for creating a new 'Download Date' column anyway as sometimes you want to know the difference in time between:
Date email was created.
Date email was received on server.
Date email was downloaded.

Knowing when someone composed email and prepared to send, when server received it and when you downloaded from server would be convenient.
Thoughts ?

Comment 48

[GiovanniP]

Adding extra columns to fix this sorting problem is not an option for me: my users will never be able to understand the problem, come here and read this thread and then add the missing column by themselves. Thunderbird should not confuse the user by default, not "only if configured" with an extra column.

Filling the missing Date: column information with the download date is WRONG for two reasons:

• when configuring a new thunderbird, all e-mail missing Date: headers will be the last and more recent in the list. Even e-mails that are months old will appear as received now.
• it's misleading: a device sends an alert during the night, without Date: header. You open thunderbird at 11:00 AM. And then the alert appears to be sent at 11:00. This already happened to me and caused some problems in my family.

In my point of view, in the case of missing Date: header, thunderbird should choose to show the user the nearest date to the missing Date: header. And, if I'm not wrong, the nearest date to the e-mail composition is in the last Received: header.

Comment 49

[Alfred Peters [:infofrommozilla]]

(In reply to ISHIKAWA, Chiaki from comment #44)

(In reply to Alfred Peters from comment #43)

I have just tested a email without Date header and am surprised by the result.
With POP3, the Received column shows me the date from the Received header and the Date column shows me the time of receipt in the TB. This should be the other way around in any case.

Which version of TB did you use?

Tested with TB91.10 as well as current Daily.

The mail is delivered to local dovecot mail server and that could be when missing Date: header line may have been added.

The Date header is one of only two headers that are actually mandatory in an e-mail according to RFC 5322.

Therefore RFC 5321 also explicitly allows SMPT servers to add this header afterwards:
RFC 5321 - 6.4. Compensating for Irregularities
| The following changes to a message being processed MAY be applied
| when necessary by an originating SMTP server, or one used as the
| target of SMTP as an initial posting (message submission) protocol:
[...]
| o Addition of a date, time, or time zone when none appears

If a POP3/IMAP server makes up for the omission, no one will complain either. Unfortunately, there still seem to be enough clients/servers that don't do this. Otherwise we would not need this bug.

Comment 50

[osdm]

I agree with GiovanniP. All such tech details are not meaningful for us users. What is meaningful is a sensible behavior. When I install Thunderbird on a new device or when I do a repair for a folder, some old e-mails suddenly move to the most recent. Not only that surprises me, it also makes it hard to find an e-mail when you remember approximate date when it was received. For example, I may remember that an e-mail was about "ticket" (but it's not enough to find it, as search may give me hundreds of such e-mails) and that it was received in summer 2019. If such an e-mail is missing Date: header, then I won't find it, because I won't remember that I need to switch to Received (and I've read above that some other magic with customDBHeaders may be needed, I certainly won't remember about that when I reinstall TB).
A sensible behavior would be using something stable for a column that is displayed by default and used for sorted by default. For me it would be acceptable if you would use "Received" date by default, but that would be a breaking behavior for some other users, I don't know. So, the only "safe" solution is to substitute Received date instead of downloaded date when Date: header is missing (and do all other magic with customDBHeaders by default if needed).

Comment 51

[GiovanniP]

I just examined Roundcube Webmail source code: it seems that by default it uses IMAP INTERNALDATE. And the result is good: e-mails are sorted according to the date they arrived on the IMAP server.
IMAP INTERNALDATE could be an alternative to the 1st Received: header.

Comment 52

[Alfred Peters [:infofrommozilla]]

(In reply to Anje from comment #47)

Perhaps it would be easier to understand if there was more clarity on what 'Received' actually means rather than leave it's contents be interpreted on an adhoc basis.

Agreed

It has always been 'received on server'

Okay, then I must have remembered that wrong.

• so would it help clarify the meaning of what people see if that column header was renamed to 'Server Received'. So it's functionality and contents do not change and adding the 'Received' to the mailnews.customDBHeaders then works as well.

Again: That is a solution for IMAP. Which is covered in Bug 402594.

For POP3 we already have the header. Its date is already in the Received column. If we don't have a better date, we should just copy it from there.

Actually, that's probably the plan - it just doesn't work:
https://searchfox.org/comm-central/rev/57645c6b44f6de29a414784105b8e2316e218428/mailnews/local/src/nsParseMailbox.cpp#1358

        // 'Received' should be as reliable an indicator of the receipt
        // date+time as possible, whilst always giving something *from
        // the message*.  It won't use PR_Now() under any circumstance.
        // Therefore, the fall-thru order for 'Received' is:
        // Received: -> Delivery-date: -> date
        // 'Date' uses:
        // date -> 'Received' -> PR_Now()

If you have the 'Received' aka 'Server Received' date working it solves getting the Server received date instead of a duplicate created date in the Date column. This is a quick fix for one part of the problem.

Agreed

'Date' has always been the date the email was composed and not the date/time sent. Although in most cases it would appear to be about the same.

JFTR: RFC 5322 - 3.6.1. The Origination Date Field
| The origination date specifies the date and time at which the creator
| of the message indicated that the message was complete and ready to
| enter the mail delivery system. For instance, this might be the time
| that a user pushes the "send" or "submit" button in an application
| program. In any case, it is specifically not intended to convey the
| time that the message is actually transported, but rather the time at
| which the human or other creator of the message has put the message
| into its final form, ready for transport. (For example, a portable
| computer user who is not connected to a network might queue a message
| for delivery. The origination date is intended to contain the date
| and time that the user queued the message, not the time when the user
| connected to the network to send the message.)

But for email sent with no date then it seems Thunderbird applies alternative being 'Download Date'.

Actually, only if no other date is available, this should be the last choice.

If I understood him correctly, Chiaki ISHIKAWA's suggestion is now to cache this date in a separate header when downloading for the first time so that it is not lost by a "repair folder".

Comment 53

[mail]

It seems to me making the behaviour achieved by using Bug 266434 Comment 26 the default fixes this issue. Potentially the internal date could also be used to fix this.