367268 - delivery receipts (DSN) overview

Status: RESOLVED FIXED

(MailNews Core :: Internationalization, enhancement)

Description

[Matthias Wimmer]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1
Build Identifier: Version 1.5.0.9 (20061207)

Currently incoming delivery receipts (DSN) are just displaied as normal e-mail messages by Thunderbird. Many normal end-users do not understand these delivery receipts. I want to propose a UI improvement for handling these receipts.
The DSN messages should be parsed by Thunderbird and they should be used to update a new status field in the sent folder and the DSN message should be hidden. The new status field for messages in the sent folder should probably know the following states: not yet delivered / delivery failed / delivered / unknown due to passing of a MTA that does not support DSN - and it may have an additional state "read" which could be set if a MDN is received.
As "delivery failed" reports may be of special interest to the sender, receivment of such a DSN should trigger a special indication, that is shown to the user. But such a DSN should not be displayed as the mail, but as a Thunderbird message (therefore localized to the user's language), that explains what happened.

Reproducible: Always

Comment 1

[zug_treno]

Bug 339307 request something similar, but only for return receipts, duplicate?

Comment 2

[Matthias Wimmer]

Yes, I think that Bug 339307 is a duplicate of this bug.

Comment 3

[Magnus Melin [:mkmelin]]

Let's keep this about DSN and the other about MDN.

Comment 4

[Tom Nolf]

AFAIK there was no such thing as DSN in TB earlier then version 3. Please correct me if I am wrong. I wonder because this proposition for enhancement is dated at early 2007. Did real DSN already exist in TB back then?

Anyway, what's the status of this enhancement? Do you know if there were already extensions implemented that I just might not have found? 

I just installed TB3 beta3 and found out that there is a new option in "Options > Delivery Status Notification" that initiates a DSN if the receiving MTA supports this feature. 
This means that the backend is already working and it shouldn't be impossible to realize this enhancement which, btw, I find really useful too.

Comment 5

[Magnus Melin [:mkmelin]]

Yes it existed on trunk since bug 93085.

Comment 6

[Tom Nolf]

I want to take this proposed enhancement and work on it. Can someone tell me how to be assigned to this "Bug"?

Comment 7

[Magnus Melin [:mkmelin]]

Here you go!

Comment 8

[Tom Nolf]

I think TB must be extended in order to allow an extension to associate the returned DSN message with the original sent message.

Reason:
RFC3463 (http://www.rfc-editor.org/rfc/rfc3464.txt) defines the message format for Delivery Status Notifications (DSN). As you can see in "2.2.1 The Original-Envelope-Id field", which I quoted below, the field Original-Envelope-Id of the DSN message mirrors the EnvelopId given in the "MAIL FROM:" command of the ESMTP dialogue to give the MUA means to associate the DSN with the sent message.

The problem is that TB seems to not memorize this EnvelopeId and thus it can't be used to do the association stuff.
In nsSmtpProtocol.cpp (method SendHeloResponse) the method msg_generate_message_id(..) is used to create the EnvelopeId but it is not stored anywhere. 
See: http://mxr.mozilla.org/comm-central/source/mailnews/compose/src/nsSmtpProtocol.cpp#643

I think that this should be extended to handle the EnvelopeId like the MessageId and make it available in TB at some place.

Does anyone know how to proceed with this? Should this be posted as a bug? I will also ask for advice on the #maildev IRC  channel (thanks to Magnus for that hint). Is there some other place where this can be discussed?

Thanks, Tom

From: http://www.rfc-editor.org/rfc/rfc3464.txt

2.2.1 The Original-Envelope-Id field

The optional Original-Envelope-Id field contains an "envelope
identifier" that uniquely identifies the transaction during which the
message was submitted, and was either (a) specified by the sender and
supplied to the sender's MTA, or (b) generated by the sender's MTA
and made available to the sender when the message was submitted. Its
purpose is to allow the sender (or her user agent) to associate the
returned DSN with the specific transaction in which the message was
sent.

If such an envelope identifier was present in the envelope that
accompanied the message when it arrived at the Reporting MTA, it
SHOULD be supplied in the Original-Envelope-Id field of any DSNs
issued as a result of an attempt to deliver the message. Except when
a DSN is issued by the sender's MTA, an MTA MUST NOT supply this
field unless there is an envelope-identifier field in the envelope
that accompanied this message on its arrival at the Reporting MTA.

The Original-Envelope-Id field is defined as follows:

original-envelope-id-field =
"Original-Envelope-Id" ":" envelope-id

envelope-id = *text

There may be at most one Original-Envelope-Id field per DSN.

The envelope-id is CASE-SENSITIVE. The DSN MUST preserve the
original case and spelling of the envelope-id.


NOTE: The Original-Envelope-Id is NOT the same as the
Message-Id from the message header. The Message-Id identifies
the content of the message, while the Original-Envelope-Id
identifies the transaction in which the message is sent.

Comment 9

[Jernej Horvat]

is there a way to save/filter DSN replies to predefined folder ?
i could not find a proper msg filter header to use it on DSN replies...

Comment 10

[Tom Nolf]

There is no message header for DSN messages. You can try to use a filter that matches the string "Original-Envelope-Id:" in the body of the message. That's an indicator of a DSN message but keep in mind that this string could also appear in replies or forwards so it's not perfect.

Comment 11

[Jernej Horvat]

what about using 

Content-Type: multipart/report;
...
	report-type=delivery-status


problem is that sometimes it's wrapped and sometimes  not.

Comment 12

[Tom Nolf]

This would be possible too and is even better if you can create a filter for Content-Types of the body section.

Comment 13

[Tom Nolf]

Btw: The add-on I am working on for this "bug" will set a special header for incoming DSN messages so that people can create their own filters for it.

Comment 14

[Jernej Horvat]

(In reply to comment #13)
> Btw: The add-on I am working on for this "bug" will set a special header for
> incoming DSN messages so that people can create their own filters for it.

it would be nice if DSN handling would be integrated in TBird.

Comment 15

[Tom Nolf]

Attachment: A patch to TB trunk that uses the messageId as the DSN envid

This patch provides means to hand over an envid to the nsSmtpProtocol which is then used in the ESMTP command MAIL FROM. Until now the envid was generated in nsSmtpProtocol and abolished after its single usage. Thereby it was not possible for add-ons to relate an incoming DSN message to a sent message.

This patch also extends nsMsgSend to pass over the messageId of a message as the envid. The reason for using the messageId is because it complies with the requirements of the envid according to the standards and the messageId is already persistent and a key in the message database which allows fast queries.

Comment 16

[David :Bienvenu]

Comment on attachment 410538 [details] [diff] [review]
A patch to TB trunk that uses the messageId as the DSN envid

thx for working on this. You should use the newer string interfaces, so

+  attribute string dsnEnvid;

becomes attribute ACString dstEnvid

and then use a CString here:

+      char* dsnEnvid = nsnull;
+
+      // get the envid from the smtpUrl
+      rv = m_runningURL->GetDsnEnvid(&dsnEnvid);
+        
+      if (!dsnEnvid) {

so it becomes

nsCString dsnEnvid;
rv=m_runningURL->GetDsnEnvId(dsnEnvid);

if (dsnEnvId.IsEmpty()) 
  dsnEnvId.Adopt(msg_generate_message_id(senderIndentity));

and then

buffer += dsnEnvId.get();

and no need for the PR_Free(dsnEnvid);

the get/set methods on nsSmtpUrl become simpler as well.

nsSmtpProtocol doesn't seem to be using m_messageId.

Comment 17

[David :Bienvenu]

also, instead of changing the smtp service idl interface, you could just make the caller set the dsnEnvid on the returned url, and change the caller to pass in an out parameter to get the url back. That would make the patch smaller...

Comment 18

[Tom Nolf]

Attachment: Rev. 1: Incorporated improvements of first patch review

This patch includes all proposals of the first review and thus is much smaller then the first version.

Comment 19

[David :Bienvenu]

Eric, are you ok with using the message-id as the dsnEnvId?

Comment 20

[David :Bienvenu]

Comment on attachment 410944 [details]
Rev. 1: Incorporated improvements of first patch review

r=me, with some nits:

you don't need the braces here:

+      if (dsnEnvid.IsEmpty()) {
+          dsnEnvid.Adopt(msg_generate_message_id(senderIdentity));
+      }

and

+    nsCOMPtr<nsIURI> returningUrl;

returningUrl isn't a great name for the var - runningUrl would be better.

thx for the patch. If you could attach a new patch addressing my comments, and request sr from [e-mail] or standard8, that would be great. I've cc'ed the author of the original code to make sure he's OK with this change.

Comment 21

[Tom Nolf]

Attachment: Revision 2: Incorporated suggestions of second review

Thanks for your effort, David.

Comment 22

[[e-mail]]

Comment on attachment 412382 [details] [diff] [review]
Revision 2: Incorporated suggestions of second review

>+      
Nit: spaces on "blank" line (there are apparently 3 lines like this)

>+    // set envid on the returned URL. Thus the idl interface stays unchanged
Not quite sure what the second part of the comment is supposed to mean...

>+    nsCOMPtr<nsISmtpUrl> smtpUrl(do_QueryInterface(runningUrl, &rv));
>+    NS_ENSURE_SUCCESS(rv, rv);
>+    smtpUrl->SetDsnEnvid(nsCString(mCompFields->GetMessageId()));
GetMessageId() never returns null, so please use nsDependentCString.

>+      buffer += dsnEnvid.get();
Remove the .get(), it actually causes more work.

sr=me with these fixed.

Comment 23

[Tom Nolf]

Attachment: [checked in] Revision 3: Incorporated suggestions of first super review

Thanks for your suggestions, Neil. I included them in revision 3.

Comment 24

[Tom Nolf]

Thanks for super reviewing, Neil.

Are there further steps necessary or can this patch be submitted to the trunk? Who would do me the favour?

Comment 25

[Mark Banner (:standard8)]

(In reply to comment #24)
> Are there further steps necessary or can this patch be submitted to the trunk?
> Who would do me the favour?

Just in case you didn't already realise, adding checkin-needed to the keywords (like someone has already done) is the way to get this submitted.

I have noticed a minor error in your patch, in that you didn't change the uuid of nsISmtpUrl, however I'll fix that when I check it in in a couple of minutes.

Comment 26

[Mark Banner (:standard8)]

Checked in, with uuid fixed: http://hg.mozilla.org/comm-central/rev/656fe281bf22

Tom: it is unclear to me if this patch is all that is needed to fix this bug - if it is now fixed, can you mark it as such please?

Comment 27

[Tom Nolf]

Thank you very much for checking in the patch.

The other part of this bug is being developed as an add-on. I would propose to close this bug once the first version of the add-on is ready.

Comment 28

[Tom Nolf]

I run into a problem with my TB add-on made to solve this bug and an IMAP account today. 

It seems that sent messages are retrieved from IMAP (or at least added to the internal message header database) only when I open the sent-view (click on the 'Sent' folder) or when I am already in this folder. 

The result is that a sent message related to the returning DSN can only be found by the add-on if the user opened the sent folder.

I tried to set the property "Check this folder for new messages" and enabled the configuration mail.check_all_imap_folders_for_new=true but this doesn't seem to have any effect: sent emails on the IMAP account appear in TB only when I open the ''sent'' folder.

Does anyone know about this issue and may can give me a hint?

With this issue solved the add-on would already be very sophisticated and could be published as beta.

Comment 29

[[:Aureliano Buendía]]

Where I can set it as wanted-thunderbird 3.1?

Comment 30

[[:Aureliano Buendía]]

set flag for wanted-thunderbid3.1 request

set helpwanted keywords because comment #28

Comment 31

[Nikolay Shopik]

(In reply to comment #28)
> I run into a problem with my TB add-on made to solve this bug and an IMAP
> account today. 
> 
> It seems that sent messages are retrieved from IMAP (or at least added to the
> internal message header database) only when I open the sent-view (click on the
> 'Sent' folder) or when I am already in this folder. 
> 
> The result is that a sent message related to the returning DSN can only be
> found by the add-on if the user opened the sent folder.
> 
> I tried to set the property "Check this folder for new messages" and enabled
> the configuration mail.check_all_imap_folders_for_new=true but this doesn't
> seem to have any effect: sent emails on the IMAP account appear in TB only when
> I open the ''sent'' folder.
> 
> Does anyone know about this issue and may can give me a hint?
> 
> With this issue solved the add-on would already be very sophisticated and could
> be published as beta.

When synchronization enabled sent emails appears after synchronization downloads them, AFAIK. If you migrated from TB2 you may have not enabled that.

Comment 32

[Wayne Mery (:wsmwk)]

Tom's email address is dead. 
Does anyone know if the add-on ever got made?

The "Thunderbird piece" got fixed circa 3.0, so closing. If there's any other bits to be done a new bug should be opened.

Comment 33

[chrizilla]

(In reply to Wayne Mery (:wsmwk) from comment #32)

Does anyone know if the add-on ever got made?

I don't know if it's the result of this bug, but the knowledgebase mentions Notification Viewer, a (defunct) add-on that seemed to satisfy the request here long ago (TB 2.0 - 3.1).

The "Thunderbird piece" got fixed circa 3.0, so closing.

Was there any user-accessible capability added to Thunderbird?
If not, should this bug really be closed? The bug was created in 2007 as a Thunderbird bug (and remained so during work on this bug until 2016) and to my knowledge none of the reporter's (excellent) ideas and requests for Thunderbird in comment #0 have been implemented. So should the reporter's request be closed as fixed when it apparently isn't ?

There is also an inconsistency in moving this Thunderbird bug (DSN overview) to MNC while keeping its brother bug 339307 (MDN overview) a Thunderbird bug.