Open Bug 136871 Opened 20 years ago Updated 5 months ago
mail opens new smtp connection for each item sent
. Should reuse server connection, especially for slow or overloaded network .
From Bugzilla Helper: User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.9+) Gecko/20020402 BuildID: 2002040211 If there are many mails queued for transmission (i.e. in disconnected mode) and then the queued mails are sent, Mozilla opens a fresh connection to the mail server on each message. This is a real problem if the mail server is using authentication and is slow to authenticate - our mail server requires about 4 seconds to authenticate a user, so this is 4 seconds per message wasted. Reproducible: Always Steps to Reproduce: 1. Go offline 2. Compose N messages, where N > 1 3. Go online & force a send of unsent messages Actual Results: Mozilla will open a new mail server connection on each message Expected Results: Mozilla would open the socket to the MTA, and then send all pending messages.
Reporter: Can you reproduce this bug with a recent build of Mozilla (for example, 1.4rc2)? If so, then please comment again with details. If not, then please resolve this bug as WORKSFORME. Thanks.
Verified with build 2003050714 - Queued several messages offline, started Ethereal capturing SMTP traffic, and put Mozilla into on-line mode. For each queued message, Mozilla: 1) Opened the connection. 2) Authenticated to the server. 3) Send the mail 4) Closed the socket - full TCP shutdown.
yes, I see the same
Status: UNCONFIRMED → NEW
Ever confirmed: true
*** Bug 228638 has been marked as a duplicate of this bug. ***
I'd fix this by adding the ability to cache a single connection to the smtp server object and add a boolean to the smtp url that says whether or not to cache the connection when done. Default would be false. That flag would be checked when the smtp url was done, and if true, we would leave the connection alive and store it in the smtp server. The send later code would set it to true for each msg sent until the last one, where it would be set to false. The code that gets an smtp protocol object would need to learn to try to get it from the server.
I think this should be Hardware/OS All/All. See the dupe.
I've put a $100 bounty on this one, details at: http://www.markshuttleworth.com/bounty.html
David But the connection can only be reused when sending mails through the same SMTP server as the mail(s) before. So we've three options: 1. Hold the connection until reaching a message that has to be send using another server. That might lead to re-establishing a former connection again when reaching another message. 2. Holding multiple connections at a time, each for a once used SMTP server in case we need it later. They're only closed after all messages have been sent. 3. Sorting the messages to blocks that going to be send using the same SMTP server. And the SMTP protocol code has to know if it has already sent a message to jump right to the MAIL FROM point instead waiting for the servers initial 220. But that should be quite trivial.
>But the connection can only be reused when sending mails through the same SMTP >server as the mail(s) before. So we've three options: that's why I said to put the connection in the smtp server object and to retrieve it from the stmp server. There's one of those per smtp server, so you should be fine there. We'll only ever cache smtp connections when doing a send unsent messages, imo. At no other time do we really know we're going to send multiple messages to the same server. So we'll only cache connections in that case.
Oh, I see what you mean, Christian. A single unsent messages folder can have messages destined for multiple smtp servers...well, a simple way of doing this would just be to iterate over the smtp servers and have each of them close their cached connection, if any, when the send later operation is done. In the normal case (a single stmp server), this doesn't do any extra work, and has the advantage of simplicity. There can only be one unsent messages folder, so I don't think this would cause any problems. Only sending unsent messages should cause a connection to get cached...
Ok, that's option 2, yes? Having messages destined for 3 different smtp servers will produce three cached connections until they're closed at the end. I wouldn't feel that good with it, but it's a simple and functional solution.
yes, that's option 2. It's not great, but Option 1 has the drawback you mention. I'm not sure Option 3 is doable easily w/o reading all the messages into memory - I don't believe which smtp server to use is stored in the db/msg hdr.
> don't believe which smtp server to use is stored in the db/msg hdr. No, have to read the id key from the X-Identity-Key line, translate into identity and from there to the smtpserver. Surely that most difficult and costly solution.
Is this really specific to PC/Linux?
OS: Linux → All
Hardware: PC → All
I've been working on this for a while, and I got the protocol details worked out and tested already, but i'm not sure how to get through the myriad of classes between where each email is picked up out of the outbox and where it's actualy put on the wire. Currently, I have NS_MsgLoadSmtpUrl() passing an array of URL's to nsSmtpProtocol, and nsSmtpProtocol stores the list and sends each one in order. But if I implement it this way, there are still 5 more functions that need to be modified to pass all their arguments in arrays. This sees like an ugly hack to me. The code that currently sends mail is simply not condusive to an easy fix for this bug. I guess it might be possible to reuse the same nsSmtpProtocol object, but I do not know how to go about that. If someone could help me figure out what the best solution is, i'd appreciate it.
I must admit that I never got out of nsSmtpProtocol. But couldn't NS_MsgLoadSmtpUrl build a list of aUrl's and determine wheter it already created a SmtpProtocol object with this URL? If yes, reuse it and its connection. Of course SmtpProtocol must not close the socket when one message has been sent and the object must not get destroyed until the last message has been sent. This way no function has to know all URL at once. Care has to be taken for servers closing a cached connection (timout) while we're sending a message through another one. Reusing such a "connection" and object would be tricky.
I just had an idea. My code currently uses a list of url's, and processes them sequentialy. I could just write a function in nsSmtpProtocol to add URL's to this list one by one, and when all the URL's are added, a different function will kick off sending all the URL's at once. The server won't time out, because we won't start sending untill we have all the URL's, and if the server doesn't support multiple messages at the same time, we can just keep creating more connections. My only problem is how do I save the same instance of nsSmtpProtocol. I'm not very familiar with Mozilla code, but might XPCOM do something of this nature? I'm reading the docs for it now, an XPCOM service I guess is what this would be?
(In reply to comment #18) > I just had an idea. My code currently uses a list of url's, and processes them > sequentialy. I could just write a function in nsSmtpProtocol to add URL's to > this list one by one, and when all the URL's are added, a different function > will kick off sending all the URL's at once. I can't see why this could help against a timeout. See my comment #8, except you sort the URL's by server, you can't be sure you send all mails for server x continuously. If you don't, connection to server x could time out while you're sending a message to server y before returning back to send another message to server x.
> I can't see why this could help against a timeout. See my comment #8, except you > sort the URL's by server, you can't be sure you send all mails for server x > continuously. Heh, i'm not that far yet. I misread your previous post. I think that having multiple messages going to multiple servers is niche case, and that if we had to revert to the current behaviour when multiple servers are involved, it wouldn't be too terrible a loss. I'll worry about that after it works for a single server.
Well, if you think that will do it, it's ok. I only wanted to prevend us/your from changing the whole strategy if multiple servers come into play.
I would vote for solution 3 from comment 8. There you don't need to worry about any timeouts or caching connections.
I agree with comment 22. Most people will only be using one SMTP server.
This seems like a good win.
Assignee: bienvenu → bugmil.ebirol
Flags: blocking-thunderbird3? → blocking-thunderbird3+
The asynchronous smtp sending (bug 440794) will use a queue in the outbox, and should fix this. Moving off of blocking to wanted as per driver discussion, but I feel optimistic that it'll happen soonish.
Depends on: 440794
Leaving wanted‑thunderbird3+, but low priority.
Priority: -- → P5
QA Contact: sheelar → networking.smtp
Target Milestone: --- → mozilla1.9.1
Summary: mail opens new connection for each item sent → mail opens new smtp connection for each item sent (should reuse connection)
bienvenu, is it reasonable that bug 511079 would deliver this? or does this bug come first? (return to reality)
Severity: enhancement → normal
Priority: P5 → --
Target Milestone: mozilla1.9.1 → ---
(In reply to Wayne Mery (:wsmwk) from comment #28) > bienvenu, is it reasonable that bug 511079 would deliver this? > or does this bug come first? They are essentially independent.
(In reply to David Ascher (:davida) from comment #25) > The asynchronous smtp sending (bug 440794) will use a queue in the outbox, > and should fix this. Moving off of blocking to wanted as per driver > discussion, but I feel optimistic that it'll happen soonish. You still feeling it? :-)
This is a fairly serious issue for those of us working primarily offline with infrequent slow connections (e.g., low-speed wireless, satellite) to send queued messages, and it's been open for 10 years (!), so please elevate the Priority. Thank you.
In Bug 1301865 I reported the following issue: In our support forums a user reported a problem by using filters to forward messages. By this filter Thunderbird has to forward 20-30 messages per day. It seems the filter is sending these 20-30 messages in parallel, which leads to 20-30 parallel SMTP connections. The provider has a limit of 3 parallel SMTP conntections. This leads always to SMTP errors and failing the filter (the messages are marked as forwarded, but in fact they are not). It would be necessary to have an option to limit Thunderbirds parallel SMTP connections and/or parallel related filter actions.
SMTP log related to my last comment
You need to log in before you can comment on or make changes to this bug.