mail opens new smtp connection for each item sent. Should reuse server connection, especially for slow or overloaded network.

NEW
Unassigned

Status

defect
18 years ago
3 years ago

People

(Reporter: david.hagood, Unassigned)

Tracking

({perf})

Bug Flags:
blocking-thunderbird3 -
wanted-thunderbird3 +

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

50.33 KB, text/x-log
Details
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.9+) Gecko/20020402
BuildID:    2002040211

If there are many mails queued for transmission (i.e. in disconnected mode) and
then the queued mails are sent, Mozilla opens a fresh connection to the mail
server on each message.

This is a real problem if the mail server is using authentication and is slow to
authenticate - our mail server requires about 4 seconds to authenticate a user,
so this is 4 seconds per message wasted.


Reproducible: Always
Steps to Reproduce:
1. Go offline
2. Compose N messages, where N > 1
3. Go online & force a send of unsent messages

Actual Results:  Mozilla will open a new mail server connection on each message

Expected Results:  Mozilla would open the socket to the MTA, and then send all
pending messages.
Reporter: Can you reproduce this bug with a recent build of Mozilla (for
example, 1.4rc2)? 

If so, then please comment again with details. 
If not, then please resolve this bug as WORKSFORME. Thanks.
Verified with build 2003050714 - Queued several messages offline, started
Ethereal capturing SMTP traffic, and put Mozilla into on-line mode.

For each queued message, Mozilla:
1) Opened the connection.
2) Authenticated to the server.
3) Send the mail
4) Closed the socket - full TCP shutdown.

yes, I see the same
Status: UNCONFIRMED → NEW
Ever confirmed: true
*** Bug 228638 has been marked as a duplicate of this bug. ***
I'd fix this by adding the ability to cache a single connection to the smtp
server object and add a boolean to the smtp url that says whether or not to
cache the connection when done. Default would be false. That flag would be
checked when the smtp url was done, and if true, we would leave the connection
alive and store it in the smtp server. The send later code would set it to true
for each msg sent until the last one, where it would be set to false. The code
that gets an smtp protocol object would need to learn to try to get it from the
server.
I think this should be Hardware/OS All/All. See the dupe.
I've put a $100 bounty on this one, details at:

  http://www.markshuttleworth.com/bounty.html
David

But the connection can only be reused when sending mails through the same SMTP
server as the mail(s) before. So we've three options:
1. Hold the connection until reaching a message that has to be send using
another server. That might lead to re-establishing a former connection again
when reaching another message.
2. Holding multiple connections at a time, each for a once used SMTP server in
case we need it later. They're only closed after all messages have been sent.
3. Sorting the messages to blocks that going to be send using the same SMTP server.

And the SMTP protocol code has to know if it has already sent a message to jump
right to the MAIL FROM point instead waiting for the servers initial 220. But
that should be quite trivial.
>But the connection can only be reused when sending mails through the same SMTP
>server as the mail(s) before. So we've three options:

that's why I said to put the connection in the smtp server object and to
retrieve it from the stmp server. There's one of those per smtp server, so you
should be fine there.

We'll only ever cache smtp connections when doing a send unsent messages, imo.
At no other time do we really know we're going to send multiple messages to the
same server. So we'll only cache connections in that case.
Oh, I see what you mean, Christian. A single unsent messages folder can have
messages destined for multiple smtp servers...well, a simple way of doing this
would just be to iterate over the smtp servers and have each of them close their
cached connection, if any, when the send later operation is done. In the normal
case (a single stmp server), this doesn't do any extra work, and has the
advantage of simplicity. There can only be one unsent messages folder, so I
don't think this would cause any problems. Only sending unsent messages should
cause a connection to get cached...
Ok, that's option 2, yes? Having messages destined for 3 different smtp servers
will produce three cached connections until they're closed at the end.

I wouldn't feel that good with it, but it's a simple and functional solution.
yes, that's option 2. It's not great, but Option 1 has the drawback you mention.
I'm not sure Option 3 is doable easily w/o reading all the messages into memory
- I don't believe which smtp server to use is stored in the db/msg hdr.
> don't believe which smtp server to use is stored in the db/msg hdr.

No, have to read the id key from the X-Identity-Key line, translate into
identity and from there to the smtpserver. Surely that most difficult and costly
solution.
Is this really specific to PC/Linux?
all/all
OS: Linux → All
Hardware: PC → All
I've been working on this for a while, and I got the protocol details worked out
and tested already, but i'm not sure how to get through the myriad of classes
between where each email is picked up out of the outbox and where it's actualy
put on the wire. Currently, I have NS_MsgLoadSmtpUrl() passing an array of URL's
to nsSmtpProtocol, and nsSmtpProtocol stores the list and sends each one in
order. But if I implement it this way, there are still 5 more functions that
need to be modified to pass all their arguments in arrays. This sees like an
ugly hack to me. 

The code that currently sends mail is simply not condusive to an easy fix for
this bug. I guess it might be possible to reuse the same nsSmtpProtocol object,
but I do not know how to go about that. If someone could help me figure out what
the best solution is, i'd appreciate it.
I must admit that I never got out of nsSmtpProtocol.
But couldn't NS_MsgLoadSmtpUrl build a list of aUrl's and determine wheter it
already created a SmtpProtocol object with this URL? If yes, reuse it and its
connection.
Of course SmtpProtocol must not close the socket when one message has been sent
and the object must not get destroyed until the last message has been sent.

This way no function has to know all URL at once.
Care has to be taken for servers closing a cached connection (timout) while
we're sending a message through another one. Reusing such a "connection" and
object would be tricky.
I just had an idea. My code currently uses a list of url's, and processes them
sequentialy. I could just write a function in nsSmtpProtocol to add URL's to
this list one by one, and when all the URL's are added, a different function
will kick off sending all the URL's at once. The server won't time out, because
we won't start sending untill we have all the URL's, and if the server doesn't
support multiple messages at the same time, we can just keep creating more
connections.

My only problem is how do I save the same instance of nsSmtpProtocol. I'm not
very familiar with Mozilla code, but might XPCOM do something of this nature?
I'm reading the docs for it now, an XPCOM service I guess is what this would be?
(In reply to comment #18)
> I just had an idea. My code currently uses a list of url's, and processes them
> sequentialy. I could just write a function in nsSmtpProtocol to add URL's to
> this list one by one, and when all the URL's are added, a different function
> will kick off sending all the URL's at once.

I can't see why this could help against a timeout. See my comment #8, except you
sort the URL's by server, you can't be sure you send all mails for server x
continuously.
If you don't, connection to server x could time out while you're sending a
message to server y before returning back to send another message to server x.
> I can't see why this could help against a timeout. See my comment #8, except you
> sort the URL's by server, you can't be sure you send all mails for server x
> continuously.

Heh, i'm not that far yet. I misread your previous post. I think that having
multiple messages going to multiple servers is niche case, and that if we had to
revert to the current behaviour when multiple servers are involved, it wouldn't
be too terrible a loss. I'll worry about that after it works for a single server.
Well, if you think that will do it, it's ok. I only wanted to prevend us/your
from changing the whole strategy if multiple servers come into play.
Product: MailNews → Core
I would vote for solution 3 from comment 8. There you don't need to worry about 
any timeouts or caching connections.
Assignee: mscott → bienvenu
I agree with comment 22. Most people will only be using one SMTP server. 
Flags: blocking-thunderbird3?
This seems like a good win.
Assignee: bienvenu → bugmil.ebirol
Flags: blocking-thunderbird3? → blocking-thunderbird3+
Keywords: perf
The asynchronous smtp sending (bug 440794) will use a queue in the outbox, and should fix this. Moving off of blocking to wanted as per driver discussion, but I feel optimistic that it'll happen soonish.
Depends on: 440794
Flags: wanted-thunderbird3+
Flags: blocking-thunderbird3-
Flags: blocking-thunderbird3+
Leaving wanted‑thunderbird3+, but low priority.
Priority: -- → P5
QA Contact: sheelar → networking.smtp
Target Milestone: --- → mozilla1.9.1
Summary: mail opens new connection for each item sent → mail opens new smtp connection for each item sent (should reuse connection)
Duplicate of this bug: 470722
Product: Core → MailNews Core
No longer depends on: 440794
Assignee: bugmil.ebirol → nobody
bienvenu, is it reasonable that bug 511079 would deliver this?
or does this bug come first?
(return to reality)
Severity: enhancement → normal
Priority: P5 → --
Target Milestone: mozilla1.9.1 → ---
(In reply to Wayne Mery (:wsmwk) from comment #28)
> bienvenu, is it reasonable that bug 511079 would deliver this?
> or does this bug come first?

They are essentially independent.
(In reply to David Ascher (:davida) from comment #25)
> The asynchronous smtp sending (bug 440794) will use a queue in the outbox,
> and should fix this. Moving off of blocking to wanted as per driver
> discussion, but I feel optimistic that it'll happen soonish.

You still feeling it? :-)
This is a fairly serious issue for those of us working primarily offline with infrequent slow connections (e.g., low-speed wireless, satellite) to send queued messages, and it's been open for 10 years (!), so please elevate the Priority. Thank you.
Duplicate of this bug: 1301865
In Bug 1301865 I reported the following issue:

In our support forums a user reported a problem by using filters to forward messages. By this filter Thunderbird has to forward 20-30 messages per day. It seems the filter is sending these 20-30 messages in parallel, which leads to 20-30 parallel SMTP connections. The provider has a limit of 3 parallel SMTP conntections. This leads always to SMTP errors and failing the filter (the messages are marked as forwarded, but in fact they are not).

It would be necessary to have an option to limit Thunderbirds parallel SMTP connections and/or parallel related filter actions.
Posted file smtp.log
SMTP log related to my last comment
No longer depends on: sendinbackground
See Also: → sendinbackground
Summary: mail opens new smtp connection for each item sent (should reuse connection) → mail opens new smtp connection for each item sent. Should reuse server connection, especially for slow or overloaded network.
You need to log in before you can comment on or make changes to this bug.