Open Bug 1545930 Opened 5 years ago Updated 9 months ago

API to send messages

Categories

(Thunderbird :: Add-Ons: Extensions API, enhancement)

Product:
Thunderbird
Component:
Add-Ons: Extensions API
Type:
enhancement

Tracking

(Not tracked)

Status:
ASSIGNED

People

(Reporter: Fallen, Assigned: TbSync)

References

Details

(Whiteboard: [Prio2023])

Attachments

(2 files)

Microsoft Outlook Permission
20.88 KB, image/png
Details
Bug 1545930 - Add method to send messages programatically. r=darktrojan
48 bytes, text/x-phabricator-request
Details | Review

It would be great to have an API to send messages in the background. This API should be developed with care obviously, since it also has great power.

At the minimum, this should be gated behind a permission. We could also consider providing the user with an option to allow/disallow add-ons from sending email regardless of if the permission was granted or not.

My use case is to create an add-on for weekly status updates, where I can gather them in a browserAction popup UI, and have a possibility to click send to send it to predefined recipients.

To convert my add-on Mail Merge to a WebExtension, an API to create and save or send messages is indeed needed. The API should cover both sending a message now or later and saving a message as a draft or template.

At the moment Mail Merge creates a new nsIMsgCompose object and interacts with nsIMsgCompose, nsIMsgComposeParams and nsIMsgCompFields. See: function mailmergeutils.compose() in /content/utils.js in Mail Merge 6.1.0.

When you develop such an API, please keep Mail Merge and its use-case in mind, thanks!

As sending emails in the background is indeed a sensitive operation with a (high) security impact, having a special permission for this in the manifest.json sounds reasonable.

I would also be fine, if the user had to confirm the actual delivery once per session. Obviously - in the case of Mail Merge - having the user to confirm the delivery of each individual message would kill the add-on from a usability perspective. (I have users, who send 100+, 1000+ and even 10000+ messages per session.)

I'm also trying to convert one of my extensions to a MailExtension and for me sending a mail is also needed:
The extension after a click on a button in the compose window sends the mail and saves it as .eml to the filesystem (archiving the mail into a Document Management System). Therefore the proposed send API command should also return an identifier of the just sent mail so that it would be possible to save this mail to the filesystem (by using messages.getRaw() and saving the content to a file).

@darktrogen @jorgk Are you able to advise if the "send API" is expected to land with Thunderbird 78 ESR? Thank you

I have no intention of working on this before 78.

Type: defect → enhancement

Would this be available in Thunderbird 78.x, i.e. 78.1 or 78.2? Or does this mean that the "send API" would be expected to land in the next ESR after 78? Thank you

I just don't know. My crystal ball can't see that far.

It might not happen at all. Such a thing has security implications we haven't even begun to tink through yet. (Do you want your Thunderbird used by a malicious extension as a vector for sending spam? Me neither.)

In the meantime for those of you converting existing code: as always, put it in an experiment API and use that. If you think it's generic enough to be used as a built-in API, show us what you've got and we'll consider it.

It might not happen at all. Such a thing has security implications we haven't even begun to think through yet. (Do you want your Thunderbird used by a malicious extension as a vector for sending spam? Me neither.)

  1. The Mail Merge add-on is not a malicious extension.
  2. You can currently Mail Merge with Thunderbird 68 ESR. Shouldn’t that mean that this is issue is not an enhancement but a bug?
  3. You can Mail Merge with Microsoft Outlook. There is a free Mail Merge program that works with Microsoft Outlook and is compatible with 2000, 2003, 2007, 2010, 2013, 2016 and 2019. Please let us know if you want a copy.
  4. Isn’t Thunderbird looking for more traction in the Enterprise space? Hence, is not implementing this issue a backwards step?
  5. Should this issue be brought up at Council Meetings and Status Meetings?

Thank you

I am not saying that Mail Merge or any other extension is malicious. I'm not saying anything about Mail Merge at all.

What I am saying is that having an API function to send email from an extension makes it very easy to create a malicious extension and makes Thunderbird a more attractive target for malicious people. We have to be careful about that.

And yes, I know that an extension could already send email in current and previous versions of Thunderbird. We are creating this for future versions of Thunderbird and we have the opportunity to decide how we want to do that. Doing it right first time is important, because change is hard.

You asked if this would be available in any particular version. I don't know. Currently I have a lot of work which is a higher priority than this bug. It will happen when it happens.

I don't want to escalate anything here. Obviously no pressure to implement this soon, but I want to second that this seems like a necessary API, and therefore a bug, not an enhancement.

One of the most attractive things about extendable software is the ability to automate things, and without that ability extensions are basically just complicated themes. With API functions like onBeforeSend(), a malicious extension can already intercept and modify email messages after you've stopped editing them. That seems equally dangerous, but of course the ability to filter messages is an important part of many extensions, and lacking that functionality would be unacceptable. I would argue the same is true for this.

@darktrogen I understand that you're not ixnaying the concept of a sending API, but I respectfully encourage you to change this back to 'bug' status, because WebExtensions would be fundamentally crippled if they don't support this feature if/when experiments are eventually no longer enabled.

Whether it's is a bug or an enhancement has no bearing on … anything, really. This is a request to add something that doesn't exist, so it's an enhancement.

Sorry for being a little bit off topic in this comment:

It is remarks like these in comment #6 and comment #8 that are so immensly frustrating and demotivating!!

I fully understand that classic / legacy restart extensions have no future. But I don't understand why you have to kill restartless extensions as well - I can see no valid technical reason for this decision. Converting Mail Merge to a restartless extension would have been possible without too much work.

Nevermind... The way into the future is obviousy WebExtensions: But Thunderbird is apparently lacking a lot of APIs with very valid use-cases - like this specific API to send messages.

We add-on developers are told that we can use a technique named WebExtensions Experiments for the time being. But again Thunderbird is lacking a clear commitment that WebExtensions Experiments will be there for years to come!

For my own add-on Mail Merge this means that I can spend the next few weeks or months trying to solve or workaround all the problems I probably face by converting my add-on to a WebExtensions Experiment. And - assuming that I somehow manage this successfully - maybe one year later Thunderbird decides to kill WebExtensions Experiments without all the necessary WebExtensions APIs in place, then all my hard work was wasted.

Again, sorry for being a little bit off topic in this comment, the next two comments will be on topic - I promise!

I also fully understand that an API to send messages will probably not make it into Thunderbird 78.

My basic understanding is, that at the moment the actual sending of the messages is tangled into the compose window. Therefore Mail Merge creates nsIMsgCompFields, nsIMsgComposeParams and nsIMsgCompose objects to setup all the necessary bits and peaces. See: function compose() in /content/utils.js in Mail Merge 6.1.0.

For the actual body of the message to work with embedded images it was necessary to make a little dance with the editor in the compose window. See function send() in /content/progress.js in Mail Merge 6.1.0.

I have heard that sending the messages will be rewritten in pure JavaScript in the future. So I think this rewrite would be a good time to also implement a WebExtension API to send messages.

Of course this also means that there must be a real commitment from Thunderbird to keep WebExtension Exeperiments alive until this rewrite including the WebExtension API to send messages have landed in Thunderbird.

Of course it is correct that an API to send messages has some imminent dangers, i.e. (1) information can be exfiltrated via emails and (2) the API can be used for spamming.

Both dangers are obviously not new to WebExtensions! Both dangers could have been achieved without much work with classic / legacy extensions as well. For more than 10 years the source code of my own add-on Mail Merge is released as free and open source software; Mail Merge clearly demonstrates how to send messages programmatically with "classic" techniques.

Exfiltrating information can obviously be done by other means as well: Think of an evil extension, that has access to the saved messages or the message being drafted in the compose window and has access to the internet. You can trivially read the contents of the messages and send them to a remote server. Yes, there are already all the necessary WebExtension APIs for this evil use-case in place!

Nevertheless a well designed API to send messages could mitigate both risks by (1) asking for a special permission at install time and (2) asking for another special permission at run time. See my own comment #1 as well.

(In reply to Alexander Bergmann from comment #13)

Both dangers are obviously not new to WebExtensions! Both dangers could have been achieved without much work with classic / legacy extensions as well. For more than 10 years the source code of my own add-on Mail Merge is released as free and open source software; Mail Merge clearly demonstrates how to send messages programmatically with "classic" techniques.

Classic/legacy extensions basically had no security/privacy model. I think moving to WebExtensions and creating a new dedicated API is an ideal time to reconsider those models, and Geoff is quite right to raise that.

WebExtensions have an improved permissions model, but there's still things to consider around APIs. For a possible example (without having thought about the implications, and let's not discuss at this stage because it might not even be on the table) maybe we should limit the send APIs to only be allowed to be called as a result of a user action rather than just randomly at any time.

(In reply to Alexander Bergmann from comment #12)

Of course this also means that there must be a real commitment from Thunderbird to keep WebExtension Exeperiments alive until this rewrite including the WebExtension API to send messages have landed in Thunderbird.

Whilst there's been no longer term commitment to keep Experiments alive, there also has been no discussion of a date to remove it except for that fact it might not be available for the long term. They certainly aren't going away in 78 - and so you can easily use similar functionality that you do today. I also think it would be rather silly if Thunderbird developers decided to remove experiments without considering the state of the current APIs/requests and getting feedback of add-on developers. So whilst I agree this is potentially an important API, I really don't see there is any discussion worth having here at the moment with respect to the api and the future of experiments.

I would also like to remind people that Thunderbird has very limited developers available, and lots of add-on developers are currently asking for various new APIs, fixes to APIs, or other things. Having been watching most of the bugs here, I have have to say I'm not seeing add-on developers try to help out by providing fixes to APIs or putting up patches for possible new APIs based on their own development of experimental APIs. That's been a little disappointing. (Disclaimer: Whilst I used to work on Thunderbird, I now work on Firefox, and my volunteer add-on development for Thunderbird is completely separate to my paid Firefox work). Note: if you want to discuss this further, I'd suggest doing so off-line not in this bug.

I think having an API to send would be useful, but agreed it needs a bit more thought, plus, the backend to support it needs a lot of work first.

Would a permission look something like the attached image? Thank you

Let's get this moving. How should this work?

  1. What would be the desired input for such a messages.send() function?
  • a raw message string, similar to what you get when calling messages.getRaw()?
  • a draft (and of course adding ways to create messages/drafts without using the composer)
  1. Input for creating messages/drafts
  • raw message string?
  • headers array and array of mime parts / attachments?

From my perspective it would be great if the message.send function accepts a raw message string.
And separately, it would be ideal if there were a way to create a draft, also using a raw message string, but that isn't strictly related to this bug :)

A third option for input parameters would be ComposeDetails and an Array of ComposeAttachment, to send a copy of a message which is currently being composed.

@John Bieling's proposal would work for my use cases (and I believe for @Alexander Bergmann's as well). One other consideration is how progress would be indicated. It would be helpful to be able to register a callback to get status updates to be able to recreate a "progress bar" in the extension while sending.

As for security, here's an idea: If an extension with "messages.send" permissions is active, a toolbar icon could be added to the main Thunderbird interface with a counter of how many messages have been sent by an extension. Clicking this icon would show a breakdown by extension of how many messages were sent and allow for post-hoc disabling of the "messages.send" permissions. This icon should of course be hide-able.

A downside is that this might be a little in-your-face for the user if they are using the extension to send 10000's of messages.

I don't think I see the use case for sending a copy of a message which is currently being composed, but any of the other proposals would work for me as well.

The reason I agreed more strongly with the "raw message string" idea is that it seems less convoluted for most use-cases than needing to first create a draft message.

Jonathan P-H, by "raw message string" do you mean an object like ComposeDetails? Or do you mean a raw string, headers and all? I don't think it would be very friendly to extension developers if they had to create email headers and encode attachments, etc. all themselves.

It makes sense to use some sort of encapsulation, but the flexibility of manipulating messages as raw strings has always felt invaluable to me. ComposeDetails only exposes a small subset of possible headers, which is too restrictive for my use case.

If ComposeDetails could be extended to operate as a wrapper for a full message, with some ability to inspect and manipulate the underlying raw message string, that would be workable for me.

If your concerns are just the headers, we probably can find other solutions. I feel the API should guarantee to produce valid emails and if the developer needs to prepare the actual message string, I see a lot of things that can go wrong. Encoding being the most prominent one.

I think that should cover it.

Just to make it concrete, my use case is:

* Read message from `Drafts/`
* Look at custom headers
If conditions are met, then send it:
    * Sanitize headers (remove custom headers, reset `Date`, etc)
    * Send message

    If the message needs to recur in the future,
    then modify the original Draft:
        * update a bunch of headers,
        * create new Message-ID,
        * save as new message in `Drafts`

    * remove original Draft

The key feature I need for this use-case is the ability to manipulate any/all of the headers, not just select ones that are hard-coded into the ComposeDetails class. So long as it's possible to do the above I'm happy with whatever implementation.

Again, the part about saving a new message to Drafts is important to me, but not strictly part of this feature request.

Please also allow

* Read messages from the `Templates`
* Read message from any `LocalFolders` add-on folder

Thank you

LocalFolders by Christopher Leidigh

As described in Comment 2 I hope that the function will make it possible to send the mail that is actually been composed in the compose window. In this way it would make possible to create addons that enhance the "send functionality" in a way to interact with the mail the user wants to send at the moment.
As said it would be perfect to get (after the sending was successful) the messageHeader of the sent mail. This way the developer of an addon could also interact with the already sent email (save it or move it to a folder for example).

(In reply to cm-schl from comment #29)

As described in Comment 2 I hope that the function will make it possible to send the mail that is actually been composed in the compose window. In this way it would make possible to create addons that enhance the "send functionality" in a way to interact with the mail the user wants to send at the moment.

Comment 2 sounds like you actually want to use compose.sendMessage for that button and have the "Save" functionality added (xref bug 1747456). Whilst this API could allow extracting a message from the compose window and sending it, I don't see that as its primary function in most cases.

(In reply to Mark Banner (:standard8) from comment #30)

(In reply to cm-schl from comment #29)

As described in Comment 2 I hope that the function will make it possible to send the mail that is actually been composed in the compose window. In this way it would make possible to create addons that enhance the "send functionality" in a way to interact with the mail the user wants to send at the moment.

Comment 2 sounds like you actually want to use compose.sendMessage for that button and have the "Save" functionality added (xref bug 1747456). Whilst this API could allow extracting a message from the compose window and sending it, I don't see that as its primary function in most cases.

Oh, you're completely right, sorry that was my fault!
Thanks

No need to be sorry. This is still at a very early stage and the discussion is important to understand the needs and how the different options we have could work together.

(In reply to cm-schl from comment #29)

As described in Comment 2 I hope that the function will make it possible to send the mail that is actually been composed in the compose window. In this way it would make possible to create addons that enhance the "send functionality" in a way to interact with the mail the user wants to send at the moment.
As said it would be perfect to get (after the sending was successful) the messageHeader of the sent mail. This way the developer of an addon could also interact with the already sent email (save it or move it to a folder for example).

The use case that cm-schl describes in comment #29 is exactly the use case I also need for an addon. Also the possiblity to to get (after the sending was successful) the messageHeader of the sent mail is very important to me.

(In reply to John Bieling (:TbSync) from comment #19)

Let's get this moving.

Thank you for picking this up. I hope I am not too late to join the party! :-)

How should this work?

  1. What would be the desired input for such a messages.send() function?
  • a raw message string, similar to what you get when calling messages.getRaw()?
  • a draft (and of course adding ways to create messages/drafts without using the composer)
  1. Input for creating messages/drafts
  • raw message string?
  • headers array and array of mime parts / attachments?

I think the API should be built around the new object ComposeDetails.

In theory ComposeDetails could reflect the complete state of a message, e.g. the message being composed. You could use ComposeDetails to interact with the current compose window using getComposeDetails() and setComposeDetails(). You could use ComposeDetails to open a new compose window using beginNew(), beginReply() and beginForward(). And you could use ComposeDetails to save or send a new message directly using the new send function.

I have also filed Bug 1749196, because ComposeDetails is currently lacking some essential properties. (Again, the idea in my mind is for ComposeDetails to reflect the complete state of the message.)

Providing a raw message to the API might be handy for some use-cases. But for my own add-on Mail Merge it would directly lead to the problem of how to create the raw message. I can get the ComposeDetails of the current message in the compose window by using getComposeDetails(). But then another function would be needed to convert the ComposeDetails into a raw message string. (Or my add-on would have to reinvent the wheel completely.)

At the moment it is already possible to use ComposeDetails to first create a new compose window using beginNew() with the details like from, to , cc, bcc, subject, body and attachments provided programmatically. And then to call sendMessage to send the message. For example:

let newDetails = {
    from: "from@example.com",
    to: ["to@example.com"],
    cc: ["cc@example.com"],
    bcc: ["bcc@example.com"],
    subject: "Test",
    body: "Hello World",
    attachments: []
};

let newMessage = await messenger.compose.beginNew(newDetails);

await messenger.compose.sendMessage(newMessage.id, { "mode": "sendLater" });

Unfortunately Thunderbird is pretty much unusable while the individual messages are being processed, i.e. a new compose window is opened and the message is being sent. This might be acceptable for a few minutes - when there are only a few individual messages being sent - but not for a lot of minutes or even hours - when there are hundreds or thousands of individual messages being sent.

A new function send, which accepts ComposeDetails and "mode" ("sendLater", "sendNow", "saveAsDraft", "saveAsTemplate") could perform the delivery directly in the background and without the detour, i.e. opening a new compose window. So Thunderbird could stay responsive while the individual messages are being sent. For example:

let details = {
    from: "from@example.com",
    to: ["to@example.com"],
    cc: ["cc@example.com"],
    bcc: ["bcc@example.com"],
    subject: "Test",
    body: "Hello World",
    attachments: []
};

await messenger.send.message(newDetails, { "mode": "sendLater" });

It might also be handy or even necessary to get some kind of feedback, e.g. error handling, progress report, etc.

Severity: normal → S3

I was not able to get this ready for 115. I had to work on other things and I just did not have enough time. I uploaded the current patch, which works but does not yet support all properties.

The idea is to be able to re-use a composeDetails object retrieved from a composer and send the exact same message. There are still open questions about how to handle and interpret all the possible combinations of isPlainText, body, plaintextBody and deliveryMode

Help from the community is appreciated, for example figuring out, why some of the properties do not work.

Attachment #9339724 - Attachment description: WIP: Bug 1545930 - Add method the send messages programatically → WIP: Bug 1545930 - Add method to send messages programatically
Assignee: nobody → john
Attachment #9339724 - Attachment description: WIP: Bug 1545930 - Add method to send messages programatically → Bug 1545930 - Add method to send messages programatically. r=darktrojan
Status: NEW → ASSIGNED

The only remaining unsupported properties are type and relatedMessageId. If possible, I would like to use these (in a follow-up bug) to associate the sent message with an existing one and mark the existing one as a "has been answered" or "has been forwarded".

Whiteboard: [Prio2023]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: