mail opens new smtp connection for each item sent. Should reuse server connection, especially for slow or overloaded network.

NEW
Unassigned

Status

MailNews Core
Networking: SMTP
16 years ago
a year ago

People

(Reporter: David Hagood, Unassigned)

Tracking

({perf})

Bug Flags:
blocking-thunderbird3 -
wanted-thunderbird3 +

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

50.33 KB, text/x-log
Details
(Reporter)

Description

16 years ago
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.9+) Gecko/20020402
BuildID:    2002040211

If there are many mails queued for transmission (i.e. in disconnected mode) and
then the queued mails are sent, Mozilla opens a fresh connection to the mail
server on each message.

This is a real problem if the mail server is using authentication and is slow to
authenticate - our mail server requires about 4 seconds to authenticate a user,
so this is 4 seconds per message wasted.


Reproducible: Always
Steps to Reproduce:
1. Go offline
2. Compose N messages, where N > 1
3. Go online & force a send of unsent messages

Actual Results:  Mozilla will open a new mail server connection on each message

Expected Results:  Mozilla would open the socket to the MTA, and then send all
pending messages.

Comment 1

15 years ago
Reporter: Can you reproduce this bug with a recent build of Mozilla (for
example, 1.4rc2)? 

If so, then please comment again with details. 
If not, then please resolve this bug as WORKSFORME. Thanks.

Comment 2

15 years ago
Verified with build 2003050714 - Queued several messages offline, started
Ethereal capturing SMTP traffic, and put Mozilla into on-line mode.

For each queued message, Mozilla:
1) Opened the connection.
2) Authenticated to the server.
3) Send the mail
4) Closed the socket - full TCP shutdown.

Comment 3

15 years ago
yes, I see the same
Status: UNCONFIRMED → NEW
Ever confirmed: true

Comment 4

15 years ago
*** Bug 228638 has been marked as a duplicate of this bug. ***

Comment 5

15 years ago
I'd fix this by adding the ability to cache a single connection to the smtp
server object and add a boolean to the smtp url that says whether or not to
cache the connection when done. Default would be false. That flag would be
checked when the smtp url was done, and if true, we would leave the connection
alive and store it in the smtp server. The send later code would set it to true
for each msg sent until the last one, where it would be set to false. The code
that gets an smtp protocol object would need to learn to try to get it from the
server.

Comment 6

15 years ago
I think this should be Hardware/OS All/All. See the dupe.

Comment 7

15 years ago
I've put a $100 bounty on this one, details at:

  http://www.markshuttleworth.com/bounty.html

Comment 8

15 years ago
David

But the connection can only be reused when sending mails through the same SMTP
server as the mail(s) before. So we've three options:
1. Hold the connection until reaching a message that has to be send using
another server. That might lead to re-establishing a former connection again
when reaching another message.
2. Holding multiple connections at a time, each for a once used SMTP server in
case we need it later. They're only closed after all messages have been sent.
3. Sorting the messages to blocks that going to be send using the same SMTP server.

And the SMTP protocol code has to know if it has already sent a message to jump
right to the MAIL FROM point instead waiting for the servers initial 220. But
that should be quite trivial.

Comment 9

15 years ago
>But the connection can only be reused when sending mails through the same SMTP
>server as the mail(s) before. So we've three options:

that's why I said to put the connection in the smtp server object and to
retrieve it from the stmp server. There's one of those per smtp server, so you
should be fine there.

We'll only ever cache smtp connections when doing a send unsent messages, imo.
At no other time do we really know we're going to send multiple messages to the
same server. So we'll only cache connections in that case.

Comment 10

15 years ago
Oh, I see what you mean, Christian. A single unsent messages folder can have
messages destined for multiple smtp servers...well, a simple way of doing this
would just be to iterate over the smtp servers and have each of them close their
cached connection, if any, when the send later operation is done. In the normal
case (a single stmp server), this doesn't do any extra work, and has the
advantage of simplicity. There can only be one unsent messages folder, so I
don't think this would cause any problems. Only sending unsent messages should
cause a connection to get cached...

Comment 11

15 years ago
Ok, that's option 2, yes? Having messages destined for 3 different smtp servers
will produce three cached connections until they're closed at the end.

I wouldn't feel that good with it, but it's a simple and functional solution.

Comment 12

15 years ago
yes, that's option 2. It's not great, but Option 1 has the drawback you mention.
I'm not sure Option 3 is doable easily w/o reading all the messages into memory
- I don't believe which smtp server to use is stored in the db/msg hdr.

Comment 13

15 years ago
> don't believe which smtp server to use is stored in the db/msg hdr.

No, have to read the id key from the X-Identity-Key line, translate into
identity and from there to the smtpserver. Surely that most difficult and costly
solution.

Comment 14

15 years ago
Is this really specific to PC/Linux?

Comment 15

15 years ago
all/all
OS: Linux → All
Hardware: PC → All

Comment 16

14 years ago
I've been working on this for a while, and I got the protocol details worked out
and tested already, but i'm not sure how to get through the myriad of classes
between where each email is picked up out of the outbox and where it's actualy
put on the wire. Currently, I have NS_MsgLoadSmtpUrl() passing an array of URL's
to nsSmtpProtocol, and nsSmtpProtocol stores the list and sends each one in
order. But if I implement it this way, there are still 5 more functions that
need to be modified to pass all their arguments in arrays. This sees like an
ugly hack to me. 

The code that currently sends mail is simply not condusive to an easy fix for
this bug. I guess it might be possible to reuse the same nsSmtpProtocol object,
but I do not know how to go about that. If someone could help me figure out what
the best solution is, i'd appreciate it.

Comment 17

14 years ago
I must admit that I never got out of nsSmtpProtocol.
But couldn't NS_MsgLoadSmtpUrl build a list of aUrl's and determine wheter it
already created a SmtpProtocol object with this URL? If yes, reuse it and its
connection.
Of course SmtpProtocol must not close the socket when one message has been sent
and the object must not get destroyed until the last message has been sent.

This way no function has to know all URL at once.
Care has to be taken for servers closing a cached connection (timout) while
we're sending a message through another one. Reusing such a "connection" and
object would be tricky.

Comment 18

14 years ago
I just had an idea. My code currently uses a list of url's, and processes them
sequentialy. I could just write a function in nsSmtpProtocol to add URL's to
this list one by one, and when all the URL's are added, a different function
will kick off sending all the URL's at once. The server won't time out, because
we won't start sending untill we have all the URL's, and if the server doesn't
support multiple messages at the same time, we can just keep creating more
connections.

My only problem is how do I save the same instance of nsSmtpProtocol. I'm not
very familiar with Mozilla code, but might XPCOM do something of this nature?
I'm reading the docs for it now, an XPCOM service I guess is what this would be?

Comment 19

14 years ago
(In reply to comment #18)
> I just had an idea. My code currently uses a list of url's, and processes them
> sequentialy. I could just write a function in nsSmtpProtocol to add URL's to
> this list one by one, and when all the URL's are added, a different function
> will kick off sending all the URL's at once.

I can't see why this could help against a timeout. See my comment #8, except you
sort the URL's by server, you can't be sure you send all mails for server x
continuously.
If you don't, connection to server x could time out while you're sending a
message to server y before returning back to send another message to server x.

Comment 20

14 years ago
> I can't see why this could help against a timeout. See my comment #8, except you
> sort the URL's by server, you can't be sure you send all mails for server x
> continuously.

Heh, i'm not that far yet. I misread your previous post. I think that having
multiple messages going to multiple servers is niche case, and that if we had to
revert to the current behaviour when multiple servers are involved, it wouldn't
be too terrible a loss. I'll worry about that after it works for a single server.

Comment 21

14 years ago
Well, if you think that will do it, it's ok. I only wanted to prevend us/your
from changing the whole strategy if multiple servers come into play.
Product: MailNews → Core

Comment 22

14 years ago
I would vote for solution 3 from comment 8. There you don't need to worry about 
any timeouts or caching connections.

Updated

13 years ago
Assignee: mscott → bienvenu

Comment 23

10 years ago
I agree with comment 22. Most people will only be using one SMTP server. 
Flags: blocking-thunderbird3?
This seems like a good win.
Assignee: bienvenu → bugmil.ebirol
Flags: blocking-thunderbird3? → blocking-thunderbird3+

Updated

10 years ago
Keywords: perf
The asynchronous smtp sending (bug 440794) will use a queue in the outbox, and should fix this. Moving off of blocking to wanted as per driver discussion, but I feel optimistic that it'll happen soonish.
Depends on: 440794
Flags: wanted-thunderbird3+
Flags: blocking-thunderbird3-
Flags: blocking-thunderbird3+

Comment 26

10 years ago
Leaving wanted‑thunderbird3+, but low priority.
Priority: -- → P5
QA Contact: sheelar → networking.smtp
Target Milestone: --- → mozilla1.9.1

Updated

10 years ago
Summary: mail opens new connection for each item sent → mail opens new smtp connection for each item sent (should reuse connection)

Updated

10 years ago
Duplicate of this bug: 470722
(Assignee)

Updated

10 years ago
Product: Core → MailNews Core
Depends on: 511079
No longer depends on: 440794
Assignee: bugmil.ebirol → nobody

Comment 28

7 years ago
bienvenu, is it reasonable that bug 511079 would deliver this?
or does this bug come first?
(return to reality)
Severity: enhancement → normal
Priority: P5 → --
Target Milestone: mozilla1.9.1 → ---

Comment 29

7 years ago
(In reply to Wayne Mery (:wsmwk) from comment #28)
> bienvenu, is it reasonable that bug 511079 would deliver this?
> or does this bug come first?

They are essentially independent.

Comment 30

7 years ago
(In reply to David Ascher (:davida) from comment #25)
> The asynchronous smtp sending (bug 440794) will use a queue in the outbox,
> and should fix this. Moving off of blocking to wanted as per driver
> discussion, but I feel optimistic that it'll happen soonish.

You still feeling it? :-)

Comment 31

6 years ago
This is a fairly serious issue for those of us working primarily offline with infrequent slow connections (e.g., low-speed wireless, satellite) to send queued messages, and it's been open for 10 years (!), so please elevate the Priority. Thank you.

Updated

2 years ago
Duplicate of this bug: 1301865

Comment 33

2 years ago
In Bug 1301865 I reported the following issue:

In our support forums a user reported a problem by using filters to forward messages. By this filter Thunderbird has to forward 20-30 messages per day. It seems the filter is sending these 20-30 messages in parallel, which leads to 20-30 parallel SMTP connections. The provider has a limit of 3 parallel SMTP conntections. This leads always to SMTP errors and failing the filter (the messages are marked as forwarded, but in fact they are not).

It would be necessary to have an option to limit Thunderbirds parallel SMTP connections and/or parallel related filter actions.

Comment 34

2 years ago
Created attachment 8793636 [details]
smtp.log

SMTP log related to my last comment

Updated

a year ago
No longer depends on: 511079
See Also: → bug 511079
Summary: mail opens new smtp connection for each item sent (should reuse connection) → mail opens new smtp connection for each item sent. Should reuse server connection, especially for slow or overloaded network.
You need to log in before you can comment on or make changes to this bug.