Closed Bug 81387 Opened 23 years ago Closed 6 years ago

Check POP/IMAP before SMTP send (SMTP-after-POP)

Categories

(MailNews Core :: Networking: SMTP, defect)

Product:
MailNews Core
Component:
Networking: SMTP
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: matthew, Unassigned)

References

Details

(Whiteboard: [parity-becky]) 

The ability to have Mozilla automatically poll POP/IMAP prior to any SMTP
traffic would be great for those of us using this mechanism for anti-relaying on
our servers.  Simply connect to POP, issue USER and PASS and then QUIT once it
has validated prior to sending any SMTP traffic.
For reference, "Becky" has this functionality and it works wonderfully.
Marking NEW.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Summary: Check POP/IMAP before SMTP send → [RFE] Check POP/IMAP before SMTP send
*** Bug 96312 has been marked as a duplicate of this bug. ***
This is a 4xp bug. NS4.7x has this feature. Therefore, this is not RFE, is it?
Severity: enhancement → major
Keywords: 4xp
Wait.
This has been supposed to work by setting "Use user name" in the SMTP settings,
right?
Now, I do not see this working any more. Mozilla does not ask for password for
the pop account when sending.
So, this is actually a regression of bug 48799.
Changing key work
Keywords: 4xp → regression
Summary: [RFE] Check POP/IMAP before SMTP send → Check POP/IMAP before SMTP send
Depends on: 90507
Bug 90507 might be the blocker of this bug.
Adding dependency.
FYI, 060708 trunk build can send mails with "use username" feature.  However,
090508 trunk build for Mac is failing.
SMTP can also take a username/password pair with the AUTH command so there's
really two different methods that need to be addressed:

authenticating to the SMTP server by sending AUTH [PLAIN|LOGIN|DIGEST-MD5]
authenticating by connecting to IMAP4/POP3 server before send

I'm not sure that both cases are being addressed so just pointing this out.
Sorry if I've missed it.
*** Bug 128235 has been marked as a duplicate of this bug. ***
Has there been any progress on this? I would like to nominate this functionality
for "Mozilla 1.0". SPAM is an always increasing problem and therefore Mozilkla
should support the commin methods against ist, such as "smtp after pop/imap".
Note that this should be impmeneted for either imap or pop3. Our mail relay
post.gaia.de checks both pop and imap logins.
Why has this been marked regression? It never worked in mozilla so far. Removing
"regression" keyword. BTW, how about some triaging of this bug (it has no
assigned milestone)
Keywords: regression
*** Bug 142968 has been marked as a duplicate of this bug. ***
I would like to stress again that this is a feature of growing importance -
especially when fighting spam. Providers need closed-relays and there needs to
be some kind of authentification. Therefore a POP3/IMAP login before sending
SMTP stuff out makes sense. And it would save a lot of support cases with people
using synchronization for both mail and news. None of the "big" Browser supports
this feature on Win right now ...

I would be happy if this could be  implemented in post 1.2 builds ....
Just another note:
I do not mean the mechanism of username authentification inside SMTP. I mean
that when a browser requests POP3/IMAP mail, the current IP address of this user
is listed in a special table on the server. SMTP calls are opened for this
adress for some limited time. See http://www.davideous.com/smtp-poplock/ for
details - here it's described for qmail mail servers.

The only thing that needs to be changed inside Mozilla is the ORDER of mail
processing: first login as POP or IMAP user, then sending SMTP stuff. NOT vice
versa! I can't believe that this is a hard thing to do ;-)
I would like to nominate this as an "essential" for post-1.2b builds.
Anti-relaying through "smtp after pop/imap" is a simple, but vital element of
fighting spammers and therefore very common, at least here in Germany. Mozilla
should support efforts to support this. Can we hope that will be assigned soon ;-)?
Why don't you just add the code from Linux? AFAIK, Everything under Linux seems
to be GNU GPL FREE !
So just get the code out of Kmail or other applications which give POP before
SMTP functionality. I am not a programmer, so don't have much idea about it, but
I think using other people's code seems like sense to me since it saves time and
makes software better.
Over and Out.
atul.

*** Bug 182484 has been marked as a duplicate of this bug. ***
I have 2 accounts in gmx.net:

if I check mail before send, I can send mail OK. I don't mind if it's no made
automatic or no ( but I do prefer Mozilla made this for me).

My problem is I can't find a way to send from my 2 accounts. I can only send
from one account, the other said:

error bla bla bla . the mail server respond: {mp013-rz3} need to authenticate
via  SMTP-AUTH-Login

Am I doing somethig wrong or there is no way to configure the accounts?

Is this a new bug?


Hello.
I hope you are not too annoyed of all the stupid questions you might read here
but sorry, I simply have no clue where I can get help but here. This bug
describes exactly my problem. I use Mozilla 1.4 and still have the same problem
as reported in May 2001. In October 2002 everybody seemed to agree that this is
an "essential" for Mozilla 1.2 builds. I don't want to be rude for I can imagine
that it is a lot of work and takes time to do it but I would like to know if
this problem is about to be solved or not?
yours fabian
fabi_gigi@hamburg.de
*** Bug 213427 has been marked as a duplicate of this bug. ***
I also vote for this to be fixed - as over 1 Million potential domains holder on Strato - one of the leading German cheapo web providers - are affected.
Currently I use the "send later" work around, but the failure to perform a POP3 before any SMTP is the single reason why I cannot install thunderbird on some of my non-techy friend computers.
Characteristics of this bug
===========================
- First noticed on 17.05.2001
- Keyword mail3 mail4 mail6 could be added
- 27.000 ("resolved") dups of this bug have been filed.
- Really Resolved: 0.
- Workarounds: Millions.
- Affected Users:
  - My Ma...
  - Dial-Up users not able to poll mail regularly (which would do the job)
  - All clients of providers running DRAC/popbsmtp-alike anti-spam solutions
  - In germany: at least all strato customers. (>1.000.000!) 
  - Widely used in japan: Google for "pop before smtp eudora"
- Look at http://www.activatormail.com/smtpauth.htm
  Currently, only The Bat! and Pegasus seem to have useable implementations.
- Comment:
  Spam (and its abuse) is a every-day topic. The techniques used (by strato
  for example) may not be perfect, but may at least somewhat help reducing 
  spam.  Users are unable to force providers to disable this feature.

"Reasons" for that "bug"
===========================
Note that it's not about RFCs 2554 or 2195. (Comment #6)!
Pop after Smtp is used for spam abuse.
At least two implementations seem to be used by "various" (?) providers:
- POP before SMTP : 
           http://spam.abuse.net/adminhelp/smPbS.shtml
  Home (?) http://popbsmtp.sourceforge.net/
- Dynamic Relay Authorization Control :
  http://mail.cc.umanitoba.ca/drac/
A different approach to the problem: 
http://whoson.sourceforge.net/

popbsmtp is a perl script, scans logs and incorperates with postfix.
drac consists of sendmail patches & uses rpc with pop/imap servers.

Both products store a hash table of "active" (30 mins in case of drac),
pop/imap sessions. They are used (instead of/in addition to regular smtp 
authentication [RFC above]) by sendmail/postfix to verify the 
sender's authencity. DRAC seems to be more widely used.

Workarounds
===========================
Every half a year, I have to tell my ma, that whenever she faces a
cryptic message like "Please use POP before SMTP", that she needs
to fetch mail before sending. The message she faces is the SMTP error
reply created by the mailserver - which isn't that helpful to my ma.
But why are such workarounds needed in a cutting-edge mail suite
(that supports spam filters on the client side...)?

... life could be nicer...
Proposed Solution
===========================
For every outgoing SMTP session, we need to make sure to
check/login to the mailbox of the corresponding POP/IMAP account.

Starting point would be to add a additional UserPref, specifying
a Pop/Imap-Account to be used to check before sending. Like

	userpref.smtpserver-1.popbsmtp="popaccount-2";
	 ...or for imap...
	userpref.smtpserver-1.imapbsmtp="imapaccount-1";
	(Volunteers may implement it into the xul interface.)

(Some pseudo code scrambled with real function names):
Affected files:

/mozilla/mailnews/local/src/nsPop3Service.cpp --> CheckForNewMail()
 Using this call might be easiest way to go, although the request for
 new mails is not needed. As mentioned in a related (duped) bug report,
 a simple Pop3Server::login() + auth() would be sufficent...

/mozilla/mailnews/compose/nsSmtpService.cpp
	fix funciton SendMailMessage (around line 145), add call to new
	smtpServer->GetTryPopBeforeSmtp(&identity); // fetch userprefs
	smtpServer->GetTryImapBeforeSmtp(&identity);

	if (pop/imap server defined for this smtp server) exec...:

	func GetTryPopBeforeSmtp(&identity)
	{
	 if (pref.this.mailserver.popbsmtp)
	  return pop3Service::CheckForNewMail( &identity )
	 else
	  return false;
	} 

	func GetTryPopBeforeSmtp(&identity)
	{
	 if (pref.this.mailserver.imapbsmtp)
	 {
		//printf("Imap-Before-Smtp: Fix Milestone #2\n");
		//ImapService?::CheckForNewMail( userpref.popbsmtp.value )
	 }
	 return false;
	}

Questions: Can these messages be sent at any time to pop3service instance?
Had only little time to investigate, but could be the right places to
hook in. Will this cause a big slowdown or are these methods available anyway?

Any hint of a core developer wheter that's the approach to go would
encourage me to investigate further.


Alternatives
===========================
- Ask Strato et al to be more friendly to spammers.
- Tell Ma to use a diffrent mailclient. (NO!! Nightmare!!!)
- Let people pay billy g. some money to keep our mailboxes clean.
- Tell us all that we're stupid and close this bug as "wontfix" ?
- Tell us that it's already done for thunderbird?


cya, Jan
Matthew Peddlesden, the original reporter, pointed out to me by mail:

- "Becky" he refered to, was not a patch by a Person called "Becky" to mozilla,
  but just another mail reader ( http://www.rimarts.co.jp/becky.htm )
  that supports exactly what we need.

- He also made a proposal, which would be the deluxe-way to solve this issue:
----quote----
Essentially all it needs it the ability to make sure it issues a POP3/IMAP
login request prior to sending any email out on a matched SMTP server (i.e.
for SMTP server A check POP3 server A first, for SMTP B use POP3 B first and
so forth).  If the SMTP message fails to send due to spam, perhaps also then
try to check all of the POP3/IMAP registered servers one at a time until the
message sends or you run out of servers, recording whatever was successful,
as this will then auto-configure it in the case of something slightly less
intuitive for the end user.

It might even do that without the user requesting it, try sending the smtp
message, if it bounces immediately with a "you cannot relay" then try
checking POP3/IMAP and resending, if that works, flag it as such in the
config and do that in future, that way users don't have to find an option
and tick a box - which many probably wouldn't be able to understand :-)
----/quote----

I really like his proposal!
Anyway, I'll try the first step / easier approach - as soon I get some feedback.

*** Bug 237127 has been marked as a duplicate of this bug. ***
> But why are such workarounds needed in a cutting-edge mail suite

POP before SMTP as "technique" itself is a workaround.
I couldn't even find a RFC for it.

If a server wants a SMTP client to authenticate, it should use SMTP-Auth, that's
what it is for.
This shouldn't be too hard to do - basically, we'd add a new smtp server pref
that's the account to check new mail before connecting to the smtp server. The
tricky part is that checking for new mail via pop3 or imap is asynchronous, and
we'd need to wait for the check to finish before doing the smtp send. I wonder
if we could take advantage of the nsIMsgLogonRedirectionRequester feature to do
this. We'd add a custom redirector that did a check for new mail on the account,
and when it's finished, tell the smtp protocol to go ahead with the original
user name, password, etc.
Assignee: mscott → bienvenu
David, I am not shure if we really need a serialized sequence where smtp only
starts after all pop/imap connections have ended. The common implementations of
"smtp after pop/imap" simply put the IP adress of the requesting pop/imap client
into a table (simply a berkeley dbfile), immediately after the POP/IMAP
authentication. No matter how long the mail download takes, the user's IP is
know to the smtp server from this point and thes smtp channel is kept open for
at least 15-20 mins.

It should be sufficient if smtp sending is just delayed for a few seconds after
the pop/imap auth has started. 20-30 seconds should be enough.
Hmm, the timer approach would be a pain - you'd have to wait 20 or 30 seconds
for each message send whereas it probably only takes a second or two to
authenticate.
(In reply to comment #25)
> David, I am not shure if we really need a serialized sequence where smtp only
> starts after all pop/imap connections have ended.
[snip]
> authentication. No matter how long the mail download takes, the user's IP is
> know to the smtp server from this point and thes smtp channel is kept open for
> at least 15-20 mins.

Yep, that works. When I get the "no relay" error, I hit retrieve and then
immediately resend the email. So, they work asynchronously as long as POP has
logged in. Of course, it is possible that the POP server could be slower to
respond than the SMTP server (if they're different machines) which would result
in a "no relay" situation again. I don't know if you can get an event when POP
has successfully logged in; that would be best.
If we'd issue POP login just for authentication purposes we could add a
two-liner in AuthFallback() right after
  m_nsIPop3Sink->SetUserAuthenticated(PR_TRUE);

Something like
  if(justAuthenticate)
    m_pop3ConData->next_state = POP3_SEND_QUIT;


But as I wrote earlier, the whole SMPT after POP is a piece of crap.
David; I am not an expert in pop3/imap4 protocols but as far as I understand,
there is only one authentication at the beginning of a mail transmission (e.g.
POP3 RFC1081: USER/PASS command from client answered by OK+ from the server
meaning "maildrop locked and ready" and similar in RFC2060 for imap). So, the
delay until the startup of SMTP send would be all the same, no matter how many
messages are waiting for being fetched by client.

Shure, the smtp after pop is "is a piece of crap" but popup blockers also
occured only after people/companies have misused the "popping feature" of
windows. Mozilla should give a hand in fighting that kind of misuse, I think ;-)
SMTP after POP is better than nothing.
But people here talk if it were the only remedy for spam and one isn't a good
user if he don't use it.

SaP is more complex, needs more resources and is less secure than SMTP-Auth. I
wonder why (normally) economically thinking companies use it.
Thomas, I was referring to this part of your comment:

>It should be sufficient if smtp sending is just delayed for a few seconds after
>the pop/imap auth has started. 20-30 seconds should be enough.

I think it would be better to do the smtp send right after the pop/imap auth,
which requires knowing when the pop/imap auth has completed. I don't know if
it's better to do a full get new mail, or just do a user + pass - it seems like
rather a waste to logon to the pop3 server and not do anything...I'm starting to
think it might be better to only do this if the smtp send failed - is there a
well-known error code (not string, but error code) returned in this case?
The common message is: "550 Relaying denied, authenticate with POP3 or IMAP4
first!" RFC1893 states in chapter "3.8 Security or Policy Status"

----------------------
[...]
       X.7.1   Delivery not authorized, message refused

          The sender is not authorized to send to the destination.
          This can be the result of per-host or per-recipient
          filtering.  This memo does not discuss the merits of any
          such filtering, but provides a mechanism to report such.
          This is useful only as a permanent error.
----------------------

So, the "550" is a general message for any kind of unauthorized access to smtp.
It is not really specific for the "smtp after pop/imap" case. And there is no
way how one could check the "smtp after pop/imap" capability beforehand (like
the CAPABILITY command in IMAP). I will check with some other mail clients how
they implemented this in detail ...
OK, the results of my little research were not very successful: none of the
common free clients I checked (kmail, mutt, sylpheed) implements the "smtp after
pop" now. The only client I could find was EUDORA:
http://www.active-venture.com/support/cp/email-client-eudora.htm#pop describes
it from the user side. But this is "closed-source", no chance to grab the code
... Maybe its useful to contact the sylpheed guys: they mention "smtp after pop"
as a ToDo feature (see http://sylpheed.good-day.net/#todo). They should have
made some suggestions on this ...
Chris, in Comment #28 you write...
> But as I wrote earlier, the whole SMPT after POP is a piece of ****.
Full ack on this, but what should users do who are facing that ****
and have to live with it?

... and in Comment #30
> SaP is more complex, needs more resources and is less secure than SMTP-Auth. I
> wonder why (normally) economically thinking companies use it.
As far as I could see, large scale providers decided to use it to share
auth information across multiple smtp relays. Oppinion was, that it takes
less time to fetch auth information from a DB instead of having to
deal with smtp auth for each request. (?)

In comment #33 Thommie writes, that only Eudora supports SaP. 
The Bat! and Pegasus also support it, as visible on
http://www.activatormail.com/smtpauth.htm

Some providers offer SMTP-Auth and/or SaP. Some (large ones) stay running SaP
only,as by doing so, they also support (bad) SMTP-Auth-unaware mail readers...
In a few years, this feature may have become useless --- but...
the original bug was filed 2001, so...
Thanks for re-engaging on this topic.
OK, I talked to few mail-server admins and user-support people: The best
solution would be a simple user interface where people can:

a) decide for a smtp server that "pop/imap before smtp" is needed (simple yes/no
decision with "no" as default)
b) a menu where one can select the pop/imap server which needs to be contacted
for authentication (in most cases, but not in all, this will be the same DNS
name for smtp and imap/pop)
c) a two digit field where people can select the "delay time" between the
successful pop/imap auth and the start point of the smtp send command. A default
value of some seconds (5-10) should be sufficient in most cases.

http://www.faqs.org/rfcs/rfc1939.html shows the typical POP3 case. It is NOT
necessary to grab all waiting messages first! The POP3 login can be reduced to
an absolute minimum, just for authentication:

1) INIT 
S:  +OK POP3 server ready

2) AUTHORIZATION state.
C: USER mrose
S: +OK mrose is a real hoopy frood
C: PASS secret
S: +OK mrose's maildrop has 2 messages (320 octets)

-> At this point, "SMTP send" could be triggered, with an optional/default
"delay time" as stated  in c)

3) QUIT Command

C: QUIT
S: +OK dewey POP3 server signing off

A similar approach could be done for IMAP.

The idea of using the error code from the smtp server (550) and triggering "smtp
after pop" automagically, is not really good: The error message is not
standardized enough and, moreover, this could be a security threat: people could
automatically grab POP/IMAP login passwords with a "fake pop/imap server".
Therefore, users should always have full control whether they want to send their
password for "smtp after pop" authentication or not.
I am in this same boat with Mozilla 1.6.  My web hosting service proclaims it 
to be a known issue with work arounds for Outlook/Outlook Express.

Here's my hosting service page:
   http://kb.discountasp.net/article.aspx?id=10006

Here's the Microsoft page:
   http://support.microsoft.com/default.aspx?scid=kb;en-us;289945

Any and all help would be greatly appreciated.
Still no change in 1.7 and in Thunderbird 0.7. It's a real trouble! It's no
problem for me to check mail before sending, but I can't make ordinary users do
it every time...
Still no change in 1.7 and in Thunderbird 0.7. It's a real trouble! It's no
problem for me to check mail before sending, but I can't make ordinary users do
it every time...
Product: MailNews → Core
*** Bug 271708 has been marked as a duplicate of this bug. ***
Summary: Check POP/IMAP before SMTP send → Check POP/IMAP before SMTP send (SMTP-after-POP)
If the server remembers the IP number that checked POP3/IMAP for 15-20 minutes
as we see in comment #25 than all the backend code we need is simply making sure
that email is checked every 10 minutes. If the user switches on th "Check POP
before SMTP" UI, an ugly  pop-up window (remember it is an ugly authentication
method, after all) saying "The browser changes your preferences to automatically
check your mail every 10 minutes" would be helpful addition.
Aloha David, This bug has become a major issue since the release of TB1.0 When
will it be solved? 
Instead of doing proper SMTP authentication (no idea why ISP's don't implement
this in a more widespread fashion, all they have to do is enter a few more lines
in their "how to set up your mail client" sections), it seems this kludge is
being used. Basically, once you do an IMAP login, somehow the IP address is kept
in some sort of stack so that you can send SMTP through their SMTP servers. But
if you have not logged in to IMAP (no idea if you need an active connection, or
if it's just login, that times out after a specified period), then the SMTP
server will reject your IP as un-authenticated.
I agree with #28 From Christian Eyrich of course the workaround "is ****" but
not having it; results in: http://forums.mozillazine.org/viewtopic.php?p=1109867   
About 2000 views within 3 weeks. 
*** Bug 280361 has been marked as a duplicate of this bug. ***
It seems to be an ignorant world around user friendly SMTP solutions.

This bug exists for more than 3 years now. And there is another major Bug around
SMTP in Mozilla / Thunderbird: The SMTP GUI (thanks to ch.ey for his work on
this). There is much time spent in functions like "spell as you type", so users
have mails with correct spelling, but they are not able to send the mails...

In Germany we would say "Ohne Worte".
The idea to check Mails automatically every xx minutes (see comment #40) would
be a not good idea for users, which are not online all the time.

A Login into the related POP account just before sending should be the smartest
solution. Maybe we are able to check if there is / was a successfull login the
last minutes, to avoid to much (unnecessary) logins?
We run our own Exim server. I was just looking at some packets with Etherpeek, and got the following:

Line  1 :  -ERR Unknown AUTHORIZATION state command <CR> <LF> .. 100

this is followed by:

TCP Checksum: 0x4877 Checksum invalid. Should be:  0x0724


Any idea if this is the right bug? Mail seems slow, but does come through. Thanks.
FYI.
Many ISP's started "Outbound Port 25 Blocking" recently, then lazy administrators of SMTP server should enable "Mail Submission Port", thus they have to enable SMTP AUTH in order to support "Mail Submission Port". This will certainly reduce number of lazy SMTP servers who force "POP before SMTP" without enabbling SMTP AUTH support.  
To David Bienvenu(person who is set in Assigned To: currently):

Why still "not WONTFIX"? (See Bug 228198 Comment #18)
Is there any reason to keep this bug open? Is there any plan to implement "POP before SMTP" for lazy administrators of SMTP server?
Product: Core → MailNews Core
(In reply to comment #47)
> To David Bienvenu(person who is set in Assigned To: currently):
> 
> Why still "not WONTFIX"? (See Bug 228198 Comment #18)
> Is there any reason to keep this bug open? Is there any plan to implement "POP
> before SMTP" for lazy administrators of SMTP server?

bienvenu ping.
and resetting QA
QA Contact: esther → networking.smtp
I'm conflicted about this - it's a silly way for authentication to work, but it exists in the real world, and it' snot hard to implement on the client, and other clients to it, as I understand it.
"SMTP after POP" was dying even when this bug was filed 8 years ago. Almost all (serious) ISPs don't do it anymore (AFAICT, etc.), not even as a fallback.
So while this may be nice-to-have if someone comes up with a patch, we shouldn't waste precious development time on it, IMO. ;-)
Assignee: bienvenu → nobody
I agree with Karsten, It is no longer an issue in the times of "normal" smtp authentication methods. This should be closed with status "wontfix"
taking a cue from comment 51 => wontfix
Severity: major → normal
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → WONTFIX
Whiteboard: [parity-becky]
You need to log in before you can comment on or make changes to this bug.