612990 - Date cannot be retrieved from delivery-date: header

Status: RESOLVED DUPLICATE of bug 32216

(Thunderbird :: Untriaged, defect)

Description

[SR]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 (.NET CLR 3.5.30729)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6

I've received an email which Date: header is not set (header below).
There are two header fields that are set with different times :
"From -" (not From:)
"Delivery-date:"

The date from "From -" is used in the UI and I see no way to use another field or to display the "Delivery-date" field in the UI in order to sort according to its value.

The original (emails modified) is :

From - Wed Nov 17 20:58:53 2010
X-Account-Key: account1
X-UIDL: 000096e4491dc341
X-Mozilla-Status: 0011
X-Mozilla-Status2: 00000000
X-Mozilla-Keys:                                                                                 
Return-path: <[e-mail]>
Envelope-to: [e-mail]
Delivery-date: Wed, 17 Nov 2010 13:27:28 +0100
Received: from smtp08.smtpout.orange.fr ([80.12.242.130] helo=smtp.smtpout.orange.fr)
	by mx1.lost-oasis.net with esmtp (Exim 4.69)
	(envelope-from <[e-mail]>)
	id 1PIh6m-0003Ju-KR
	for [e-mail]; Wed, 17 Nov 2010 13:27:28 +0100
Received: from wwinf1g18 ([10.232.37.62])
	by mwinf5d15 with ME
	id YCTU1f00C1LTB0S03CTUnN; Wed, 17 Nov 2010 13:27:28 +0100
From: sender <[e-mail]>
Reply-To: sender <[e-mail]>
To: recipient <[e-mail]>
Message-ID: <12455942.77767.1289996848220.JavaMail.www@wwinf1g18>
In-Reply-To: <[e-mail]>
References: <[e-mail]>
Subject: =?UTF-8?Q?re:_Rep=C3=A9rage_=C3=A9lectrique?=
MIME-Version: 1.0
Content-Type: multipart/alternative; 
	boundary="----=_Part_77766_4065525.1289996848195"
X-Originating-IP: [90.25.234.3]
X-Wum-Nature: EMAIL-NATURE
X-WUM-FROM: |~|
X-WUM-TO: |~|
X-WUM-REPLYTO: |~|
X-Antivirus: avast! (VPS 101117-0, 17/11/2010), Inbound message
X-Antivirus-Status: Clean


Reproducible: Always



Expected Results:  
The email has been downloaded at 20:58 in thunderbird but I already read it around 13:30 through my webmail.
Correct time should be 13:30.

If the source field cannot be determined automatically, at least a manual way should allow me to set the correct date or to display it in the message list.

Comment 1

[WADA:World Anti-bad-Duping Agency]

"From -" line is mail separator of unix mbox file, and is added by Tb. Timestamp of the "From -" line is "mail download time of your PC", in local time, without timezone offset. It's not in GMT, local time of your PC.    

Tb doesn't use Delivery-date: header for timestamp of a mail. Tb uses Date: header as mail's timestamp for threading, and Tb shows "Date" column and "Received" column which corresponds to Date: header and top(last added) Received: header respectively.
Timestamp of Date: header is required for correct threading when clock of mail sender's PC is correct. If no Date: header, or if malformed/corrupted Date: header, Tb uses Epoc time(1970/01/01 GMT+0000) or current date(local time of download in "From -" line, if POP3).
For user's convenience in such cases, Tb introduced "Received" column which Outlook Express already had have.

As almost all servers add Received: header, and as top most(last added) Received: header is usually added by server who received the mail, and as nearest server's clock is usually correctly set, and as it's usually almost same as timestamp of Delivery-Date: header if Delivery-Date: is added by server, timestamp of top most(last added) Received: header can be considered as "accurate mail arrival time at destination server" almost always, even when Delivery-Date: header is not added by server.
Use "Received" column instead of "Date" column for sorting by timestamp, if you receive many mails of no Date: header(==RFC violation of mail sender==bug of mailer), or of malformed/corrupted Date: header, or of wrong timestamp in Date: header, please.
If IMAP, see bug 402594, and do workaround, please.

By the way, for me, Epoc Time or too old timestamp or future timestamp at "Date" column is one of most convenient indicators of Junk mail :-)

Comment 2

[SR]

Thank you very much for your clear and complete answer.
It's a pleasure to see that you haven't just closed the bug just telling me that's the expected behaviour with no explanation.

It was not obvious to me that "received" column (translated "reçu" in my thunderbird version) was a date.
It is now clear and I have added the column in the UI.

I understand that the message I've received does not conform to RFC.
I also understand that, when you don't have conformant message, you use the "From -" header, added by Tb.

So I'm not filing a bug anymore, but a suggestion for changing the behaviour of the "Date" information.
Actually, why not using, when it is set :
  1. Date: header
  2. Received: header
  3. Tb synthetised From - header

Thus, if I understand correctly, the more accurate information available will be used in the "Date" column.

Comment 3

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #2)
> So I'm not filing a bug anymore, but a suggestion for changing the behaviour
> of the "Date" information.
> Actually, why not using, when it is set :
>   1. Date: header
>   2. Received: header
>   3. Tb synthetised From - header

I know(needless to say, developers too) such user's want really exists. It's single Date column for his convenience, for his use case of a column for timestamp which is related to a mail, in his environment such as many malformed mails.

Currently, Tb's Date column and Received coulmn is timestamp in Date: header and timestamp of Received: header. These two headers are for absolutely different purposes; Date: is timestamp of mail composition, and top(last added) Received: is timestamp of mail arrival(almost same as timestamp of Delivery-Date: and usually same.) Meaning of both colums of Tb is same since ititial of implementation of the column.
Meaning of Date column is never changed from Netscape mail. So, Date column==timestamp of Date: header is widely accepted for long time. It's one of reasons why new Received column is added instead of "merge timestamp of other headers or other data like current data to Date column value".

I think "Consolidate Mail Date for user" column or "Unified Mail Date for user" column is better implemented for requirement like yours, if important for many many users. It's similar to "Unified Who column", recipient if sent mail and sender if received mail, which Eudora has, and which may be implemented by Penelope(next Eudora based on Tb).
Such enhancements for user's convenience is not mandatoy feature of a mailer. So, it's a good candidate of add-on.
 
> Thus, if I understand correctly, the more accurate information available will
be used in the "Date" column.

What is your definition of *accurate* information about mail composition time of mail of no Date: header or malformed Date: header?
Guessing it from Received: header for spammers is so important? Threading by guessed timestamp from Received: for spams is so important?
If you need to sort mails in "accurate mail arrival time" always, Received column(same as Delivered-Date: in almost all mails) is better than "Unified Mail Date for user" column, isn't it?

Comment 4

[SR]

(In reply to comment #3)
> (In reply to comment #2)
> > So I'm not filing a bug anymore, but a suggestion for changing the behaviour
> > of the "Date" information.
> > Actually, why not using, when it is set :
> >   1. Date: header
> >   2. Received: header
> >   3. Tb synthetised From - header
> 
> Currently, Tb's Date column and Received coulmn is timestamp in Date: header
> and timestamp of Received: header. These two headers are for absolutely
> different purposes; Date: is timestamp of mail composition, and top(last added)
> Received: is timestamp of mail arrival(almost same as timestamp of
> Delivery-Date: and usually same.) Meaning of both colums of Tb is same since
> ititial of implementation of the column.
> Meaning of Date column is never changed from Netscape mail. So, Date
> column==timestamp of Date: header is widely accepted for long time. It's one of
> reasons why new Received column is added instead of "merge timestamp of other
> headers or other data like current data to Date column value".
> 

Well, I'm not sure to follow completely what you mean. You told me that the "Date" column is the value of :
  1. Date: header
  2. Tb synthetised From - header

And that's what I see with the example I've posted above in comment #1.

So "Date column==timestamp of Date: header is widely accepted for long time." does not look correct according to this behaviour.

The fact that there already is a complexity behind the "Date" column is the reason why I proposed a little change. I thought, and maybe I was wrong, that as the "Date" column already wasn't a "pure" "Date:" header transcription, it would not be a fundamental evolution.

> I think "Consolidate Mail Date for user" column or "Unified Mail Date for user"
> column is better implemented for requirement like yours, if important for many
> many users.

Whatever its name is, as long as I can use it :)
But I think that, if the "Date" column has to be set with something that is not the "Date:" header, then then the "received:" header value is probably a better source of information than the "From -" header value.

We know that we are talking about what should be displayed when a non conformant mail is received.
We're not switching regular mail "Received:" and "Date:" header meanings.


> Guessing it from Received: header for spammers is so important? Threading by
> guessed timestamp from Received: for spams is so important?

I don't get that point. I don't thread spam, I check if it is not a false positive (which it is not in most cases) and I delete it.
And if you say that the "Date" column does not reflect the real value of the "Date:" header in some cases, well yes, it's the current behaviour, isn't it ?

> If you need to sort mails in "accurate mail arrival time" always, Received
> column(same as Delivered-Date: in almost all mails) is better than "Unified
> Mail Date for user" column, isn't it?

Perhaps, but this is not my need.
My point is to say that if we don't know the date of composition, but we know when the message was sent (through received: header) then the date of the received: header is probably closer to the date of composition than the date of the POP3 message download.
As date of message download is already used in the "Date" column (through "From -" synthetised header) then why not using a closer date *in this particular circumstance* ?

This is probably not the most elegant solution.
Maybe the best change is not the suggestion made in comment #3, but to create a Date column with a complex, user-defined, behaviour, with a default behaviour to discuss. And to have "Received date", "Composition date" columns that are given raw values extracted from headers, with no other behaviour.

Thus, you have both flexibilty, user friendliness, and information correctness.
But I guess, that such a change may look like a revolution (and it's perhaps not relevant, as I don't pretend to be an expert).

Comment 5

[WADA:World Anti-bad-Duping Agency]

Oh, you are right. Current Date column != timestamp of Date: header if no Date: header or malformed Date: header, because timestamp of Date: header never exists. And, Tb's behaviour on Date column value is not consistent when no Date: header or malformed Date: header.
  timestamp of download in some cases, but Epoc time in other cases.
  timestamp of download in some cases, but in such cases, Epoc time or
  rebuild-index time if rebuild-index(repair folder) is executed.
IIRC, Epoc time was always used initially(perhaps since Netscape mail). Current time was probably used in some codes by complaints around sorted order, and the "current time" is download time or import time or rebuild-index time. 
Report for phenomenon of "changed Date column value after rebuild-index" exists.

Behaviour you are proposing is almost same as behaviour of Outlook Express 6. I'm who checked Outlook Express's behaviour in bug report related to no/malformed Date: header. Outlook Express used next Received: header's value, if timestamp of top most Received: header is corrupted, and used next of next if next is corrupted, ....

How about bug open like next?
  Date column value is changed by rebuild-index.
  Date column value should be consistet even if no Date: or malformed Date:.
  One of best ways to keep consistency is Outlook Express like way,
  and Outlook Express like way is resonable and convenient for users.
It's never complaint on Tb's Date column value for spam. It's never enhancement. It's report of real bug and proposal of a solution.
Note: If I open such bug, it'll be like next :-)
 Date column value is inconsistent. Use Epoc Time always if no/malformed Date:.

Comment 6

[SR]

I'm sorry, I did not understand what you were asking in your previous comment.

I proposed two ways of evolution.
One that look "light", in comment #2.
One that looks more complex, at the end of comment #4

I have no idea of how Tb works, so these ideas are maybe just garbage.
I'm convinced that something has to be done, hence the suggestions.

In my opinion, using 1/1/1970 in Date column doesn't help a lot.

Comment 7

[WADA:World Anti-bad-Duping Agency]

I wanted to say next in my previous comment.
- As you say, there is no reason of "current Epoc time or download time is
  right action" but "timestamp of Received:/Delivery-Date: is wrong action"
  when no Date: header or malformed Date: header.
- Your 'One that look "light", in comment #2' is almost same as what Outlook
  Express does.
- There is inconsistency in Date column value setting by Tb, if no Date: or
  malformed Date:
- "Reasonable Delivery-Date: is not used or inconvenience in daily use"
  is too weak as reason why improvement is needed.
- This kind of request tends to be considered as merely complaint or merely
  "nice to have". I think "inconsistency should be corrected" is needed as
  reason why improvement is required.  
- If I open new bug, I perhaps request 1/1/1970 which doesn't help you
  as a way to get back consistency of Date column value, 
  because 1/1/1970 is sufficient for me, and because I know that change back
  to "1/1/1970 always" is far easier than your suggestion.

If improvement is requested with proper and acceptable reason in new bug, and suggestion of "Delivery-Date: use" is done in addition to your suggestion of comment #2, this bug can be replaced by the new bug.

Comment 8

[Wayne Mery (:wsmwk)]

duplicate of bug 32216?

Comment 10

[SR]

I don't see this bug as a duplicate because bug 32216 is about malformed (because of localization or anything else) Date: header.
Here, there is no Date header, so this bug is about :
 -where do we get the best information when no Date: header exists
 -how do we display it

There is no problem with the decoding of a non conformant Date: header.

Comment 11

[WADA:World Anti-bad-Duping Agency]

(In reply to SR from comment #10)
I agree with you.
There are three kinds of malformation around Date:
(a) No Date: header. Start point of this bug. Your suggestions or requests are available.
(b) Broken Date: header. Extracting yyyy/mm/dd(and hh:mm:ss) is impossible.
(c) Maformed but bad Week part only, produced by over localization at mail sender side.
    So yyyy/mm/dd hh:mm:ss can be extracted if incorrectness of Week part is ignored.

For (b), same solution as one for (a) should be applied.
For (c), same solution as one for (a) can be applied, but "skipping validity check of Week part upon extracting timestamp from Date: header" is also a pretty reasonable solution of this case. "Extracted timestamp from existent Date: header" is far important than "accurate Weekday string for the timestamp", because this is interpreting message header and this is not mail composition by Tb.