Last Comment Bug 279525 - Thunderbird Sends IP Address for EHLO
: Thunderbird Sends IP Address for EHLO
Status: RESOLVED FIXED
: dataloss, fixed1.8.1
Product: MailNews Core
Classification: Components
Component: Networking: SMTP (show other bugs)
: Trunk
: All All
: -- major with 3 votes (vote)
: ---
Assigned To: Philip Prindeville
:
:
Mentors:
: 420743 (view as bug list)
Depends on: 322956
Blocks:
  Show dependency treegraph
 
Reported: 2005-01-23 17:51 PST by Jerry Baker
Modified: 2009-01-24 00:21 PST (History)
17 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
Pseudo-code (1.26 KB, text/plain)
2005-01-24 11:27 PST, Jerry Baker
no flags Details
Proposed patch (2.18 KB, patch)
2005-12-15 14:26 PST, Philip Prindeville
no flags Details | Diff | Splinter Review
Alternate fix (1.92 KB, patch)
2005-12-15 16:03 PST, Philip Prindeville
no flags Details | Diff | Splinter Review
Alternate fix (1.92 KB, patch)
2005-12-15 19:24 PST, Philip Prindeville
no flags Details | Diff | Splinter Review
Alternate fix (1.94 KB, patch)
2005-12-15 19:31 PST, Philip Prindeville
no flags Details | Diff | Splinter Review
Proposed fix as -u diffs. (2.68 KB, patch)
2006-01-02 23:11 PST, Philip Prindeville
no flags Details | Diff | Splinter Review
Fix using prsystem.h interface. (2.61 KB, patch)
2006-01-03 11:51 PST, Philip Prindeville
no flags Details | Diff | Splinter Review
Fix that checks for qualification (FQDN) of gethostname()'s return value. (3.13 KB, patch)
2006-01-10 15:39 PST, Philip Prindeville
no flags Details | Diff | Splinter Review
Proposed fix, not changing existing API (3.14 KB, patch)
2006-01-10 19:01 PST, Philip Prindeville
mozilla: review+
mscott: superreview+
Details | Diff | Splinter Review
is this the right patch for the trunk? (3.41 KB, patch)
2006-07-13 13:05 PDT, David :Bienvenu
no flags Details | Diff | Splinter Review

Description Jerry Baker 2005-01-23 17:51:49 PST
20050123 trunk

Thunderbird sends the local IP address for EHLO instead of the domain. This
causes any mail server with a default Spam Assassin configuration to give
Thunderbird email +4 points on the spam scale.

Here is the text of an SMTP conversation:

-----
220-[server] ESMTP Exim 4.43 #1 Sun, 23 Jan 2005 19:40:22 -0600 
220-We do not authorize the use of this system to transport unsolicited, 
220 and/or bulk e-mail.

EHLO [1.1.1.1]

250-[server] Hello [1.1.1.1] [1.1.1.1]
250-SIZE 52428800
250-PIPELINING
250-AUTH PLAIN LOGIN
250-STARTTLS
250 HELP

AUTH PLAIN AGJha2Vyd2ViAEc4eEdqd3Rk

235 Authentication succeeded

MAIL FROM:<xxx@xxxx> SIZE=397

250 OK

RCPT TO:<xxxx@xxxx>

250 Accepted

DATA

354 Enter message, ending with "." on a line by itself

Message-ID: <41F4520F.80507@bakerweb.biz>
Date: Sun, 23 Jan 2005 17:40:31 -0800
From: Jerry Baker <xxxx@xxxx>
User-Agent: Mozilla Thunderbird 0.6+ (Windows/20050123)
X-Accept-Language: en-us, en
MIME-Version: 1.0
To: <xxx@xxxx>
Subject: TEST
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

TEST
.

250 OK id=1CstDL-0007vu-71

QUIT

221 [server] closing connection
Comment 1 Jerry Baker 2005-01-23 18:11:18 PST
The rule for SpamAssassin is called HELO_DYNAMIC_IPADDR and it assigns 4.4
points if this test is met.

Adding dataloss keyword since Thunderbird's actions against the RFC's are
resulting in the loss of emails.
Comment 2 Christian :Biesinger (don't email me, ping me on IRC) 2005-01-23 18:20:08 PST
(In reply to comment #1)
> actions against the RFCs are resulting in the loss of emails.

(fixed typo in the quote)

note that RFC 2821 (SMTP) says:
"In situations in which the
   SMTP client system does not have a meaningful domain name (e.g., when
   its address is dynamically allocated and no reverse mapping record is

   available), the client SHOULD send an address literal"

so this does not seem to be a violation of the RFC.
Comment 3 Jerry Baker 2005-01-23 18:24:59 PST
Does Thunderbird test for reverse DNS? It doesn't seem to because my IP has a
reverse DNS entry, but THunderbird still uses my IP.
Comment 4 Jerry Baker 2005-01-23 18:39:24 PST
I just confirmed that Thunderbird uses the IP address even for fixed IP
addresses that do have reverse DNS entries. This bug IS in violation of the RFC.
Comment 5 Jerry Baker 2005-01-23 23:54:13 PST
See bug 68877 for the cause of the incorrect behavior. 
Comment 6 Jo Hermans 2005-01-24 03:52:16 PST
Note that we can't do the name-lookup in most cases, because :

- there's often no reverse lookup available, or it might be wrong (not
bi-directional). Spamassassin often complains about that.
- the ip-address is often behind a NAT, so it doesn't even make any sense at all
- when the ip-address is behind a firewall, the client might find a FQDN, but
that might be a private name, so it might not be verifiable on the server side.
Again some bad points from Spamassassin.

And it's not against the RFC 2821. It's true however that the RFC assumes that a
FQDN is passed, and an ipaddress is only given if a host doesn't have a
meaningful name at all. You should read that last part as if the client isn't
capable (or willing) to do the lookup.
Comment 7 Jerry Baker 2005-01-24 07:20:27 PST
I'm sorry if I'm being obtuse, but I can't figure out how the RFC intends "the
argument field contains the fully-qualified domain name of the SMTP client if
one is available" to mean, "the argument field contains the FQDN of the client
unless the client doesn't feel like supplying one." Is that specified somehwere
else in the RFC? The way I read it, the sentence that starts with, "in
situations in which the SMTP client system does not have a meaningful domain
name" means whatever follows should only happen when an FQDN does not exist for
the client.

In addition, if a message is sent with a HELO/EHLO that does not match rDNS,
then SpamAssassin awards that message 2.645 points, but using the IP address for
HELO/EHLO causes it to award 4.4 points. That nullifies any argument that using
the IP address in HELO/EHLO was done because spam filters complained. We went
from a situation where only certain special circumstances would cause mail from
Mozilla/Thunderbird to be awarded 2.645 points to a situation where it will
*always* be awarded 4.4 points. Seems to me that increasing the percentage of
messages flagged as spam, and increasing the severity of that rating, is a
regression and not an improvement.
Comment 8 Jerry Baker 2005-01-24 07:26:17 PST
Also, see section 3.6 of the same RFC:

    The domain name given in the EHLO command MUST BE either a primary host name 
    (a domain name that resolves to an A RR) or, IF THE HOST HAS NO NAME, an 
    address literal as described in section 4.1.1.1.
Comment 9 Jerry Baker 2005-01-24 07:32:42 PST
Section 4.1.4 goes further:

    The SMTP client MUST, if possible, ensure that the domain parameter to the 
    EHLO command is a valid principal host name
Comment 10 Jo Hermans 2005-01-24 07:59:59 PST
(In reply to comment #7)
> That nullifies any argument that using
> the IP address in HELO/EHLO was done because spam filters complained.

I didn't say that we changed it because of the spam filters. We don't seem to
generate a DNS name because it was often impossible (no reverse lookups),
inaccurate (in case of NAT or firewall) and even difficult (in case of
multi-home hosts).

see bug 68877 comment 7 and comment 22

Comment 11 Christian Eyrich 2005-01-24 08:08:05 PST
> (In reply to comment #7)

> see bug 68877 comment 7 and comment 22

Right, while working on bug 68877 we tried to get our host name (e.g.
https://bugzilla.mozilla.org/attachment.cgi?id=145727) but failed AFAIK. CC'ing
Darin, maybe he can make a statement on the necko site of this. SMTP code can't
do more than using the provided services.(In reply to comment #10)
Comment 12 Rob Janssen 2005-01-24 09:58:44 PST
Before bug 68877 was fixed, Mozilla sent the domain-name part of the sending
mailaddress as the HELO name.  This was even worse.

I do not understand why the hostname cannot be retrieved without using reverse
lookup.  Maybe because there are too many different methods to retrieve the
hostname (as many as operating systems Mozilla is ported to?).

Anyway, the current solution is not ideal.
Comment 13 Jerry Baker 2005-01-24 10:06:27 PST
You can eliminate the NAT/firewall problem immediately. There are only three IP
blocks that are reserved by IANA for use in private networks: 192.168/16,
172.16/12 and 10/8. If the machine sending the mail has one of these addresses,
then we don't use rDNS. We have to use rDNS because at least Windows will lie
about its domain name.

Unless I'm missing something, I don't see why it shouldn't work something like this:

PSEUDO-CODE
-----
PRNetAddr iaddr;
PRHostEnt hostentry;
char *hostbuf = (char*)PR_Malloc(PR_NETDB_BUF_SIZE);
nsCOMPtr<nsISocketTransport> socketTransport = do_QueryInterface(m_transport);
rv = socketTransport->GetSelfAddr(&iaddr);

if (NS_SUCCEEDED(rv))
{
  if ((iaddr->inet.ip >= 3232235520 && iaddr->inet.ip <= 3232301055) ||
      (iaddr->inet.ip >= 167772160 && iaddr->inet.ip <= 184549375) ||
      (iaddr->inet.ip >= 2886729728 && iaddr->inet.ip <= 2887778303) ||
      (PR_GetHostByAddr(&addr, hostbuf, PR_NETDB_BUF_SIZE, &hostentry) ==
PR_FAILURE))
      {
        char ipAddressString[64];
        if (PR_NetAddrToString(&iaddr, ipAddressString, sizeof(ipAddressString))
== PR_SUCCESS)
        {
          char ipAddressString[64];
          if (PR_NetAddrToString(&iaddr, ipAddressString,
sizeof(ipAddressString)) == PR_SUCCESS) 
          {
            NS_ASSERTION(PR_IsNetAddrType(&iaddr, PR_IpAddrV4Mapped) == PR_FALSE,
              "unexpected IPv4-mapped IPv6 address");

            if (iaddr.raw.family == PR_AF_INET6)   // IPv6 style address?
              aResult.AssignLiteral("[IPv6:");
            else
              aResult.AssignLiteral("[");

            aResult.Append(nsDependentCString(ipAddressString) +
NS_LITERAL_CSTRING("]"));

          }
        }
      }

      elsif (PR_GetHostByAddr(&addr, hostbuf, PR_NETDB_BUF_SIZE, &hostentry) !=
PR_FAILURE)
        aResult.AssignLiteral(hostbuf);
   }
Comment 14 Jerry Baker 2005-01-24 11:27:47 PST
Created attachment 172277 [details]
Pseudo-code

Eliminated the paste goof and it's more readable as an attachment.
Comment 15 Rob Janssen 2005-01-30 03:32:31 PST
(In reply to comment #13)
> We have to use rDNS because at least Windows will lie
> about its domain name.

Why is that a problem?
I think one should just get the machine name and the local domain, glue
them together, and send that in the EHLO.
When some OS has lied about it (for what reason?) then too bad.

It is not something that is going to be verified and access blocked based on
it, at least not by clueful mail system operators.  Even programs like sendmail
send the local system name, not an rDNS-derived name, in the EHLO.  Why should
Mozilla do anything better in the hypothetical multi-homed system situation?
Comment 16 Jerry Baker 2005-01-30 03:39:46 PST
Windows doesn't lie about the domain, it just assumes that the host name is
whatever your computer name is. That is not always the case, in fact it almost
never is correct. I suppose as long as the domain is correct, there is no reason
not to use gethostname() to grab the domain.
Comment 17 Rob Janssen 2005-01-31 13:14:57 PST
(In reply to comment #16)
> it just assumes that the host name is
> whatever your computer name is. That is not always the case, in fact it almost
> never is correct.

I don't understand what you want to say.  I have never seen a Windows system
whose domain name was different from the host name.  But maybe I have never
seen badly-managed Windows systems?

Or do you mean the "dialup user" scenario where the PC has a fixed name, dials
in to Internet, gets a dynamic address and then the PC name is not related to
the dynamic address?

You are not supposed to fix that.  There is no need to resolve the dynamic
address to a name, the next-hop SMTP server will do that.  The PC just has
to tell the server (in the EHLO command) what its own picture of the system
name is.  It is up to the SMTP server to compare that with the rDNS name and
put those two in a Received: line.  Usually something like:

Received: from EHLO_name (rDNS_name [ip.ad.re.ss]) ....

So just using the hostname is perfectly fine.  rDNS should only be used as
a fallback in case no hostname is available from the system.
Comment 18 Jerry Baker 2005-01-31 13:21:25 PST
For instance, my rDNS looks like this:

wbar12.lax1-4-14-049-041.dsl-verizon.net

When you ask Windows for my hostname, it says:

jerbaker.dsl-verizon.net
Comment 19 Rob Janssen 2005-02-01 00:55:44 PST
(In reply to comment #18)
> For instance, my rDNS looks like this:
> 
> wbar12.lax1-4-14-049-041.dsl-verizon.net
> 
> When you ask Windows for my hostname, it says:
> 
> jerbaker.dsl-verizon.net

Sending EHLO jerbaker.dsl-verizon.net is perfectly OK in this case.
It does not exist as a registered name, but that should not be a problem, it
will result in a received line like:

Received: from jerbaker.dsl-verizon.net
(wbar12.lax1-4-14-049-041.dsl-verizon.net [4.14.49.41]) by smtp.example.com ....

This is very common in mail messages and should not result in them being flagged
as spam.

Remember: sendmail does it the same way.  If there is any de-facto standard, it
is sendmail.
Comment 20 Philip Prindeville 2005-12-10 12:17:17 PST
This has been open for a while, and there seems to be consensus on what to do...  so, I'm not sure why this bug hasn't had more traction.

Using the IP address is incorrect behavior, if only for the reason that passing through a transport layer gateway (IPv4-to-IPv6), a NATing router, etc. will *not* properly rewrite your stream to contain the correct IP address...  and hence you have a high risk of being flagged as spam.

In the case of dial-up users, the ISP should arguably be using dynamic DNS... but even if they don't, it's not a big deal.  It should be enough for the MTA that the domain portion of the name matches between the HELO argument and the rDNS on the connection.  If the host portion doesn't exactly match, it's not critical.

BTW:  How many different notions of "name" does Windows have?  There's the NetBIOS name for the computer, the value of gethostname() from the kernel, the contents of %COMPUTERNAME% (which might come from one of the above)...  anything else?

Comment 21 Rob Janssen 2005-12-11 04:26:20 PST
My position remains that the code should not attempt to perform any tricks using IP address, rDNS or similar to retrieve the hostname.
It should get the hostname from the system and put that into the EHLO command.

When it is only a single-component name, you would want to add a domain part.
Lacking any source for that (depending on the OS it has been compiled for?) you may want to add .local

There is no need to be clever about sending a more correct name than the system owner has configured.
Comment 22 Philip Prindeville 2005-12-11 13:00:44 PST
I largely agree... except perhaps about the .local bit.

If there is no domain, don't invent one.  The machine is most likely misconfigured.  Adding a workaround here will manifest some other odd behavior that will be difficult for the user to understand.

If the domain isn't set and the user doesn't want one, then their MTA should be configured to accept just a plain hostname.  Otherwise, they do need a domain and should fix their configuration.

What OSes don't have gethostname() or uname() or some equivalent?

And what do we do with the return value of getdomainname()?  I.e. gethostname() returns a single component, but getdomainname() is non-null...

But yes, I agree with your conclusion:  don't try to out-clever the user.  In the case where the user has done something explicitly for a reason, there is nothing more annoying than the software trying to "compensate for" or "correct" it and not being able to disable this behavior.
Comment 23 Philip Prindeville 2005-12-15 14:26:15 PST
Created attachment 206014 [details] [diff] [review]
Proposed patch

I'm currently using this and it works fine.  I've also made RPM's available for others to download and test.
Comment 24 Philip Prindeville 2005-12-15 14:30:02 PST
Oh, one other comment.  I didn't look too closely at the string objects that TB uses, but we might be able to use:

buffer += hostName;

instead of:

buffer += nsDependentCString(hostName);

and I'll test that as well.
Comment 25 Jerry Baker 2005-12-15 14:30:41 PST
How does Thunderbird decide whether to issue HELO or EHLO, and does this patch address that?
Comment 26 Philip Prindeville 2005-12-15 15:03:16 PST
That is a separate issue, and not addressed by this patch (this patch does send the hostname for both HELO and EHLO, however).

It looks like ::ExtensionLoginResponse() initially tries to send an EHLO.  The response to that is handled in ::SendEhloResponse() where the response code to sending EHLO failed (i.e. didn't return 250).  So it retries with a HELO instead a few lines down in the function.
Comment 27 Philip Prindeville 2005-12-15 16:03:43 PST
Created attachment 206029 [details] [diff] [review]
Alternate fix

Might be a little more legible, and avoids a sometimes unnecessary constructor.
Comment 28 Philip Prindeville 2005-12-15 19:24:50 PST
Created attachment 206051 [details] [diff] [review]
Alternate fix

Oops.  The string operators aren't compatible with RW, so I had to rewrite the patch.  My bad.  This one built and I'm testing it...
Comment 29 Philip Prindeville 2005-12-15 19:31:21 PST
Created attachment 206052 [details] [diff] [review]
Alternate fix
Comment 30 Johannes Walther 2005-12-19 07:32:01 PST
I'm afraid just using the hostname will be not better in most but worse in some cases. Unfortunately I can't compile TB myself for some time. But what does hostname give for
1. a system without any hostname configured, connected via dial in, assigned a different ip number each time
2. a system with a manually configured hostname in an inhouse behind a NAT dial in router
3. a system in an inhouse network getting assigned all information via a DHCP server being behind a NAT dial in router
4. a system in an inhouse network getting assigned all information via a DHCP server running in an official IP subnet
Comment 31 Jerry Baker 2005-12-19 07:42:13 PST
Worse than having all messages sent by Thunderbird flagged as spam by the default configuration of SpamAssassin?
Comment 32 Johannes Walther 2005-12-19 08:51:19 PST
> Worse than having all messages sent by Thunderbird flagged as spam by the
> default configuration of SpamAssassin?

Not if SpamAssassin is your authority but yes if the RFC 2821 is. And judging from it I still can't see anything wrong providing an IP number.
Comment 33 Jerry Baker 2005-12-19 09:01:58 PST
(In reply to comment #32)
> I still can't see anything wrong providing an IP number.

Except for that the RFC says HELO/EHLO ***MUST*** send a domain name unless one is not available.
Comment 34 Johannes Walther 2005-12-19 09:32:28 PST
> Except for that the RFC says HELO/EHLO ***MUST*** send a domain name unless one
> is not available.

A valid, meaningful, through A RR resolvable domain name. I don't object we'd try finding out that. But Philip's code doesn't look like it will produce always a valid one. And if we can't be sure, a pure IP is not wrong.
Comment 35 Philip Prindeville 2005-12-19 12:14:53 PST
Sending the IP address is no magic bullet.  If you are going through a NATting gateway, then the address the client will be sending is meaningless (as well as wrong).

The RFC's, if I remember, don't object to sending a HELO command with no argument.  So in the case of not having an IP address, we could do that.

But how often will dialing up (and dial is on the decline anyway) not get you a hostname?

Worst case, you force your host to have a hostname by configuring one manually.  It will be retained statically.  Unlike a random IP address out of a dial pool, which changes each time you connect.

Comment 36 Rob Janssen 2005-12-20 15:42:43 PST
(In reply to comment #30)

Again, there is absolutely no point in trying to derive a name from the actually assigned (dynamic?) IP address and send that as a hostname.
The receiving MTA will do the reverse-resolving and usually will put the result in the Received header, no need to do that yourself.

Even highly acclaimed MTAs such as sendmail do not attempt to derive a name from the IP address.  They get the hostname from the system and put it in the HELO/EHLO.
Comment 37 Philip Prindeville 2006-01-02 23:11:43 PST
Created attachment 207401 [details] [diff] [review]
Proposed fix as -u diffs.
Comment 38 Philip Prindeville 2006-01-03 11:51:17 PST
Created attachment 207431 [details] [diff] [review]
Fix using prsystem.h interface.
Comment 39 David :Bienvenu 2006-01-03 11:57:40 PST
ah, thx, I was hunting around for the right function to use, instead of gethostname(). Will this send the IP address if it can't determine the host name?
Comment 40 Jerry Baker 2006-01-03 12:04:35 PST
It looks like Windows is configured to use gethostname when PR_SI_HOSTNAME is called. This never returns an IP address according to Microsoft (http://msdn.microsoft.com/library/default.asp?url=/library/en-us/winsock/winsock/gethostname_2.asp).
Comment 41 Philip Prindeville 2006-01-03 12:17:24 PST
I used the cross-reference tool and followed the function, at least for BeOS and a couple of other unix-like systems, and it seems that it returns the hostname.

Didn't follow it in the case of Windows.
Comment 42 Jerry Baker 2006-01-03 12:19:10 PST
It does, but if a host name is not configured on the system Microsoft isn't clear what happens. Just that it's some sort of name.
Comment 43 Philip Prindeville 2006-01-03 12:23:24 PST
Is there a simple, atomic way to get what the machine thinks is its own name?

Does Win32 have a POSIX compliant uname() function?  If so, we can use the nodename.

Comment 44 Jerry Baker 2006-01-03 13:08:46 PST
gethostname will give you what Windows thinks is the machine's name. I was just pointing out that MS says it will never return an IP address. It's not clear what it will return if there is no host name configured.
Comment 45 Philip Prindeville 2006-01-03 13:15:41 PST
Supposedly the empty string.

If that's the case, then that's fine.

It will either error out and draw the user's attention to the fact that it needs to be configured, or else it will just work (some SMTP implementations accept "HELO" with no parameters...  this is correct behavior since the "HELO" command was originally designed to stop MTA's from looping back and connecting to themselves, causing mail loops).
Comment 46 David :Bienvenu 2006-01-04 11:41:53 PST
Doesn't the RFC say we SHOULD send an IP address if we have an empty host name? I realize you don't want us to send the IP address at all, but the RFC seems to say we SHOULD.
Comment 47 Jerry Baker 2006-01-04 12:15:46 PST
Re-reading the RFC leads me to believe that this patch introduces prohibited behavior. The RFC is pretty specific that the name supplied to the SMTP server should match rDNS. gethostname() will not usually match rDNS in the case of DHCP-assigned IP addresses. If we MUST send a name if one is available, and there is an rDNS record for our IP, then we MUST send that. If there is no rDNS record, then we send the IP, but only if rDNS fails.
Comment 48 Wolfgang Rosenauer [:wolfiR] 2006-01-04 12:33:56 PST
(In reply to comment #47)
> The RFC is pretty specific that the name supplied to the SMTP server
> should match rDNS. gethostname() will not usually match rDNS in the case of
> DHCP-assigned IP addresses. If we MUST send a name if one is available, and
> there is an rDNS record for our IP, then we MUST send that. If there is no rDNS
> record, then we send the IP, but only if rDNS fails.

IMHO this is too exact and not feasible interpretation of the RFC.
You are right and there are mailservers out there (only a few) which don't accept mails from another MTA if HELO/EHLO name and rDNS don't match.
But even MTAs are using the hostname by default so the administrator of the MTA has to make sure that the hostname and the rDNS entry match.
I really don't think that MUAs out there have to follow this exactly.
I would have to check what other MUAs are doing but using the hostname if it exists and using the local IP if not would be the best match *I think*.
Comment 49 Jerry Baker 2006-01-04 12:44:15 PST
It would definitely be better than what's happening currently. Rightly or wrongly though, the RFC does require that the name supplied resolve to an A record. Whether that is done in practice or not, and whether that is wise or reasonable is a subjective opinion that I really haven't considered yet.
Comment 50 Philip Prindeville 2006-01-04 12:46:00 PST
That is indeed what section 3.6 of RFC-2821 says "SHOULD".  However, it doesn't seem to take NAT into account.

It also says "and no reverse mapping record is available" (presumably a DNS INADDR PTR record).  What if you're using NIS+ or /etc/hosts files on a local network with unroutable addresses?

In that case, a name that's locally significant can be used, even if it exists outside of the DNS namespace.

It's a non-trivial task for a machine to detect if it's behind a NATting gateway, and if so what it's real address is.  Given that, it's pretty difficult to detect with certainty one's own externally visible address.

And that's without the issue of transport level tunnels or gateways (i.e. picking out an SMTP stream on IPv4 and reencapsulating it on IPv6).

I've contacted the IETF secretary, and will hopefully have a chance to discuss this issue with Klensin for the definitive word...
Comment 51 David :Bienvenu 2006-01-06 08:56:21 PST
great, thanks. This may have already been discussed, but what do other e-mail clients do? For example, if a lot of other e-mail clients do what you're proposing here, it's less likely to cause a problem with servers.
Comment 52 Jerry Baker 2006-01-06 09:37:00 PST
Outlook sends the host name. Not the FQDN mind you, but just the computer name as configured in Windows. So, for my computer it sends EHLO JBAKER.
Comment 53 Rob Janssen 2006-01-06 10:35:47 PST
(In reply to comment #51)

sendmail sends the configured local system name, not an rDNS-derived name.
as explained before, there should be no problem doing this.
receiving MTAs will tag on extra text in the Received: header when the HELO/EHLO name does not match the rDNS name and/or when the A record for this name does not match the sending IP address, but to SpamAssassin these are minor points, they happen all the time because of mailclusters, multi-homed hosts, etc.
Comment 54 Jerry Baker 2006-01-06 10:43:36 PST
Can we do something like this:

--------------------------------------------------
/* Attempt rDNS */
if (rDNS)
  string = "EHLO rDNS";

/* Failing rDNS, use configured host name + domain */
else if (gethostname())
  string = "EHLO gethostname()";

/* Last resort, use IP address */
else
  string = "EHLO [xx.xx.xx.xx]";
--------------------------------------------------

This way if rDNS works we can be RFC-compliant, and if not we can at least try to do something semi-sane. The overhead for rDNS is a non-issue since we already do lookups every time we make a network request anyway (just not directly).
Comment 55 Philip Prindeville 2006-01-06 13:18:05 PST
The rDNS name is never used (or should never be used).

The RFC (2821) states the candidate values in order of preference are:

* the host's official name

* the host's IP address (sent as an address-literal, i.e. bracketed dotted quad)

In the case of a multi-homed machine, each address might reverse-map
to a different name... which is one reason amongst many that rDNS shouldn't be used.
Comment 56 Jerry Baker 2006-01-06 13:59:07 PST
How do you get the official FQDN of a machine without rDNS? I would definitely be interested in seeing a list of major reasons not to use rDNS, especially since section 4.1.1.1 of the RFC specifically mentions rDNS as a way that should be employed before sending the IP.

"In situations in which the SMTP client system does not have a meaningful domain name (e.g., when its address is dynamically allocated AND NO REVERSE MAPPING RECORD IS AVAILABLE), the client SHOULD send an address literal" (emphasis added)

If you have any desire to even make a half-hearted attempt to follow the RFC, you're going to have to test the host name you get from the system anyway. That's going to require a DNS query, unless you know of a better way to see if a host name resolves to the local IP). Might as well make it an rDNS query and save the trouble.
Comment 57 Philip Prindeville 2006-01-06 14:29:39 PST
I'm reading the current copy of draft-klensin-rfc2821bis-00.txt and seeing in S4.1.1.1:

   In situations in which the SMTP client system does not have a meaningful
   domain name (e.g., when its address is dynamically allocated and no
   reverse mapping record is available), the client SHOULD send an address
   literal (see Section 4.1.3), optionally followed by information that will
   help to identify the client system.

That is not the same as saying that one should use rDNS to derive the name.

I'm assuming that in the UNIX and Win32 world, the name gotten via gethostname(2) should be the same as what the rDNS maps back to.  Even if it isn't (say your laptop is no longer at its usual static IP address on your DSL or cable connection, but is instead on a hotel or airport wireless network), the name that I would still trust is that which the machine believes itself to be, not that returned by rDNS.

Of course, having worked for ISPs and Telcos for a total of 9+ years, I'm disinclined to trust whatever information the carrier gives me and prefer instead what I know to be true (having configured it myself).

If you're using a dynamically managed address that your carrier is providing and they aren't giving you a matching hostname with valid rDNS, then that's a separate issue... and needs to be resolved in parallel.  Breaking SMTP because carriers can't seem to find their posterior with both hands is not the right thing to do.

With respect to your closing paragraph, it's not within the scope of SMTP to enforce the mapping of the hostname to the rDNS of the IP address (and in the case of a multi-homed machine, which interface?).  Think about it: other protocols like FTP, NNTP, etc. have a similar structure.  Do you really want each of those coming up with duplicate (but possibly disagreeing) strategies to enforce configuration sanity?

And in the increasingly prevalent cases of NAT or VPN, you might not know (a) what your real address is, or (b) which interface's address to rDNS to begin with, respectively.

This is a can of worms, and it's too easy to get it wrong.
Comment 58 Jerry Baker 2006-01-06 14:38:27 PST
You are overcomplicating the issue. The RFC states that when a reverse record is not available we should send an IP. When I read that, it makes me think that perhaps they meant someone should see if rDNS works. Call me strange, or too trusting, but I think the best way to see if rDNS works is to actually send an rDNS query. What do you think?

You stated that you are worried about cases where the machine is properly configured, but rDNS is not. I wonder what the consequences to you are if rDNS returns a host name that is different than the name you have configured yourself.
Comment 59 Philip Prindeville 2006-01-06 15:04:43 PST
Sorry, who is overcomplicating the issue?

Me?  I don't think so.  I've been working on IETF protocols since before RFC-1048 came out.

No one who sat down and tried to anticipate a dozen network failure modes when architecting a protocol has ever been disappointed.  Nor has anyone who ever questioned their ISP's ability to botch things up.

Ok... let's suppose you do what you say.  You do a DNS query on the value of gethostname().  You check it against the addresses of your interfaces?  Or do you pick an interface (and in the case of multiple interfaces, how do you pick one?) and rDNS that address and compare that to your hostname?

And let's suppose it fails, or returns a disagreeing value.  What then?  Do you proceed anyway?  Send the IP address instead?  Signal a warning, but send the hostname configured into the system anyway?

To me, the consequences of rDNS returning a disagreeing name is moot, because I don't trust rDNS.  I would proceed normally anyway.

What are you suggesting should be the upshoot of having a disagreeing return value?  And what is your reasoning?

Paul Mockapetris one told me that policy should never be embedded in protocol.  The implementation of a protocol should include the mechanisms to express policy, but policy should never be part of protocol.

If you embed assumptions of what to do when rDNS disagrees into the protocol, then you're mixing policy and protocol.  (The policy here being about preferences of whether external configuration state is to override internal configured state, as well as the assumption about whether DNS failures tend to be transient in nature or not.)

Comment 60 Jerry Baker 2006-01-06 15:12:19 PST
(In reply to comment #59)
> Ok... let's suppose you do what you say.  You do a DNS query on the value of
> gethostname().  You check it against the addresses of your interfaces?  Or do
> you pick an interface (and in the case of multiple interfaces, how do you pick
> one?)

How about you use the same one you are going to use to send the mail???

> and rDNS that address and compare that to your hostname?

No. Your host name doesn't matter.

> And let's suppose it fails, or returns a disagreeing value.  What then?  Do you
> proceed anyway?  Send the IP address instead?  Signal a warning, but send the
> hostname configured into the system anyway?

If rDNS fails, and DNS on your hostname fails also, then send IP address. Simple. Not complicated.

> To me, the consequences of rDNS returning a disagreeing name is moot, because I
> don't trust rDNS.  I would proceed normally anyway.

That's fine, and you are entitled to your opinion. However, I do believe Thunderbird is trying to comply with published specifications and not the personal preferences of individuals.

> What are you suggesting should be the upshoot of having a disagreeing return
> value?  And what is your reasoning?

Disagreeing with what? Are you suggesting that ISPs are going to return a result from rDNS that does not resolve to the IP address you just looked up? Is that even possible?

In the case that whatever you have typed into your machine not agreeing with rDNS, so what. Who cares. It doesn't matter. If you are going to suggest that we should deviate from the specification at least provide some reason for doing so other than some vague notion that you will be bothered by the idea of EHLO sending a host name that you didn't enter yourself somewhere.
Comment 61 Philip Prindeville 2006-01-06 15:49:35 PST
> > Ok... let's suppose you do what you say.  You do a DNS query on the value of
> > gethostname().  You check it against the addresses of your interfaces?  Or do
> > you pick an interface (and in the case of multiple interfaces, how do you pick
> > one?)

> How about you use the same one you are going to use to send the mail???

Well, first off, that's a layering violation; secondly, is highly OS-specific and unportable; thirdly, it breaks in the presence of multi-pathing, policy-based routing, or tunnelling (which many VPN-based services are).

What happens if you're using port-forwarding via SSH to talk to a remote server?  Then the local socket address (as well as the destination one) is going to be 127.0.0.1.  Have you tried an rDNS on that address lately?

> No. Your host name doesn't matter.

Yes, it really does.  Information that you can change and/or verify is always better than information that someone else has sole control over.

What is your reasoning for asserting it doesn't matter???

Sendmail, perhaps the most successful MTA of all time, has been using the value of gethostname() or uname() since its inception with no inherent issues.  Why assume that this is no longer a viable--and even preferable--approach?  What do you know that we don't?

> However, I do believe Thunderbird is trying to comply with published specifications and not the personal preferences of individuals.

The use of rDNS to establish the hostname is an inference that you've made: nowhere is that connection explicitly made.  The spec says:

   In situations in which the SMTP client system does not have a
   meaningful domain name (e.g., when its address is dynamically
   allocated and no reverse mapping record is available), the client
   SHOULD send an address literal (see Section 4.1.3), optionally
   followed by information that will help to identify the client system.

You're making the leap that (a) rDNS is the only test of what's "meaningful", and (b) you're overlooking the case of a statically configured hostname and address.  

For that matter, how do you propose that Thunderbird detect the difference between statically allocated address and a dynamically allocated one?  And even if it could, why would it???

We've got my personal preferences (read: paranoia and distrust) versus your unfounded assumptions and inferences.  ;-)  Well, as long as everyone can bring something to the table.

The only test for meaningfulness that I'll unconditionally agree to is:

    hostname[0] != '\0' && index(hostname, '.') != NULL

Anything more complicated than that would be difficult to prove "correct" in all circumstances.

> Disagreeing with what?

By "disagreeing values" I mean that the rDNS name looked up on your interface (and again, *which interface*) disagrees with your configured hostname, or the DNS address lookup on your hostname doesn't match the address of any interface marked up.

> Are you suggesting that ISPs are going to return a result from rDNS that does not resolve to the IP address you just looked up?

Eh?  rDNS on an address returns a name, not another address.

In the case of dynamic DNS, by the way, the order of events is:

(1) you configure a name;
(2) your machine acquires an IP address;
(3) your machine notifies a DNS server that the mapping of name => address and vice versus is "x";
(4) some time in the future, that dynamic mapping becomes uniformly visible throughout the known universe.

If at any time there's a failure in the rDNS, how do you propose that TB deduce what the nature of that failure is, how it occurred, why it occurred, and how it will most likely be recovered from?

You do you know, for instance, that an rDNS query on an address didn't return the expected value simply because the DNS hasn't converged on a new value (i.e. that updates haven't yet propagated to all secondaries)?  You could requery in 5 minutes and get the correct answer...

Or it could be because you queried a secondary that was temporarily partitioned from the primary and missed a refresh...

DNS is expected to be correct "in general" and over long periods of time.  A temporary error or inconsistency is not the basis for invalidating configuration state coming from a separate source  You're assuming that it is, e.g. that at ANY given time, DNS information is correct and complete whereas a statically configured hostname on the UA host is not.

That's just not how DNS works, neither in theory nor operation.
Comment 62 Jerry Baker 2006-01-06 16:47:54 PST
Look, I don't want to be personal, but I'm starting to wonder what is wrong here. You say the specification doesn't talk about rDNS when that's exactly what it talks about. What part of "when a reverse record in unavailable" doesn't mean rDNS to you?

All of your examples are edge cases. You are suggesting that we purposefully break the specification in order to accomodate potential edge cases. What about all the non-edge cases where rDNS is fine? I'm having a hard time imagining all these Thunderbird users on multihomed machines that are routing through SSH tunnels.

Yes rDNS returns a name. That's the whole point (duh). That's what we're going to send to EHLO.

This conversation is obviously a lost cause. There's no sense in continuing the discussion. When I read the RFC, it's very clear that we should send a host name that resolves to an A record. If we cannot get a host name (including by using rDNS), then we should send the IP. If you read it differently there's nothing I can do to change your mind. I just don't know how they could have wrote it more clearly except to have written the same sentence three times.
Comment 63 Philip Prindeville 2006-01-07 00:17:19 PST
(In reply to comment #62)
> Look, I don't want to be personal, but I'm starting to wonder what is wrong
> here. You say the specification doesn't talk about rDNS when that's exactly
> what it talks about. What part of "when a reverse record in unavailable"
> doesn't mean rDNS to you?

You're leaving half the context out.  It said "when its address is dynamically allocated and no reverse mapping record is available".

First off, not all addresses are dynamically allocated.  A lot of people use static addresses.

Second, there's no reasonable way for TB to deduce if the address was statically configured or dynamically allocated.

> All of your examples are edge cases. You are suggesting that we purposefully
> break the specification in order to accomodate potential edge cases. What about
> all the non-edge cases where rDNS is fine? I'm having a hard time imagining all
> these Thunderbird users on multihomed machines that are routing through SSH
> tunnels.

These aren't edge cases:  these are cases I ran into on a regular basis looking at normal Enterprise networking scenarios during the 5+ years I was at Cisco in the Remote Access Business Unit.

Statically allocated addresses are *hardly* an edge case.

Nor is using Active Directory on Windows PC's and having that populate the DNS dynamically.

Just because you've never seen something before doesn't make it an "edge case".

And designing a protocol that only works under ideal conditions?


> Yes rDNS returns a name. That's the whole point (duh). That's what we're going
> to send to EHLO.

Were you suggesting taking an address, rDNSing that, then taking the resulting value and doing a DNS lookup on that, and comparing that address to the original address?  It wasn't clear what you were saying.

Were you recommending performing a DNS sanity check?

> This conversation is obviously a lost cause. There's no sense in continuing the
> discussion. When I read the RFC, it's very clear that we should send a host
> name that resolves to an A record. If we cannot get a host name (including by

You're reading into the spec.  (1) it's an example, so it's not exhaustive.  And (2), it says quite plainly *when it's a dynamically allocated address*.  No mention is made of static addresses.

You're assuming this works equally well for statically configured addresses.


> using rDNS), then we should send the IP. If you read it differently there's
> nothing I can do to change your mind. I just don't know how they could have
> wrote it more clearly except to have written the same sentence three times.

You can point out to me where it says what to do in the case of static addresses, and show me what exact fool-proof test to do in order to know which one you've got.  (Including having a "fixed" IP address that is discovered by DHCP.)

Comment 64 Jo Hermans 2006-01-07 04:03:25 PST
The problem is that 99% of all mail clients are *lying* in their EHLO header, because they don't know (or even can't know) their real FQDN. The use of multiple interfaces, NAT, private DNS, unreliable rDNS, etc ... just made it worse, and impossible to solve. Toy operating systems that ask the user to configure the name of the computer are making it plain hopeless. I know that I will be flamed, but this isn't just about Windows, but also includes Linux, unless you have a responsible admin (just look at the state of rDNS, it says enough) !

IMHO, Thunderbird does the right thing, and uses the ipaddress because it can't reliably determine the FQDN. That is, it tries to be honest and publish the real ipaddress ; ignoring the multiple interface issue, but at least it tries to be a good net-citizen. Unfortunately, SpamAssassin assigns a bad value to SMTP-clients that uses ipaddresses.

For the people that want to send a hostname, however incorrect it may be - post a path, check it in. Congratulations, you've just "fixed" an email client that actually implements the protocol correctly (or at least tried to do it). Reliability of email has not improved (on the contrary), but you might have a higher chance to get thru a Spam-Assassin filter.

For people that want to determine the real FQDN (using rDNS and whatever) - nice, but it's unfortunately hopeless. There are other ways to prove your true identity, signing the email for instance. We'll have to accept the fact that SMTP is not secure, and we should better focus on improving the protocols (I'm working in a related field).

For people that want a configurable hostname in the GUI (another bug, but related) : congratulations, you're trying to make it easier for spammers. Not the real kind, just the kiddies. There are unfortunately many SMTP implementations out there, that actually believe what you're sending.
Comment 65 Jerry Baker 2006-01-07 07:57:35 PST
(In reply to comment #64)
> IMHO, Thunderbird does the right thing, and uses the ipaddress because it can't
> reliably determine the FQDN.

We don't need the FQDN, we just need a host name that resolves to an A record (and that A record matches our IP address). If I have a machine were foo.com resolves to my IP address, and an rDNS lookup returns xyz.foo-isp.com, it doesn't matter. We still can send xyz.foo-isp.com. I do not believe the authors of the SMTP specification intended that the client should send the host name that the user wishes to be known by, just one that ACTUALLY identifies their host.

Second, Thunderbird cannot know that the FQDN is impossible to determine until AFTER it tries to determine it unsuccessfully.

> For the people that want to send a hostname, however incorrect it may be - post
> a path, check it in. Congratulations, you've just "fixed" an email client that
> actually implements the protocol correctly (or at least tried to do it).

When the protocol requires that we at least try to get a host name that resolves to an A record, and we do not, that doesn't seem like implementing the protocol correctly to me. We don't even try to see if we can get one.

> Reliability of email has not improved (on the contrary), but you might have a
> higher chance to get thru a Spam-Assassin filter.

This sentence read to me, "Reliability of email has not improved, but the reliability of email has been improved." When you get mail through one of the most widely used spam filters, it seems to me that it is improving mail reliability.

I am curious in what manner you believe sending a host name that resolves to an A record would make mail less reliable.

> For people that want to determine the real FQDN (using rDNS and whatever) -
> nice, but it's unfortunately hopeless. There are other ways to prove your true
> identity, signing the email for instance. We'll have to accept the fact that
> SMTP is not secure, and we should better focus on improving the protocols (I'm
> working in a related field).

We're not trying to be secure, we're trying to follow the specification that says we shouldn't send the IP address until after we try to determine the host name and fail.

All we have to do is get a host name that resolves to our IP address. That's it. It doesn't matter if the user wishes it were a different name, or whether the user doesn't like rDNS, or whatever. As long as we can come up with a host name that resolves to an IP address on the client's machine, we're set.

My proposal is that there are far more clients who will be on dynamically allocated addresses, and on single IP machines, so we should start there with the highest probbility of success. I know about fixed IP's, but they are exceedingly rare for the end user. I don't know of a single DSL, cable, satellite, or other broadband provider that issues static IP addresses without specifically requesting it and paying extra.

Right now, every single EHLO command issues by Thunderbird is in direct conflict with the SMTP protocol specifications. If we implement something that at least TRIES to get a host name, then we will reduce the frequency with which Thunderbird sends the wrong thing.

Because most users have dynamic IP addresses, and they do not have any control whatsoever over their host name's DNS records, we should start there. That's why we should first do an rDNS. If that returns a host name, then great we're done. NO ONE CARES IF IT'S NOT THE HOST NAME YOU LIKE.

If rDNS does not return a host name, then we should see if the machine has one configured by using gethostname(). We should see if that host name resolves to an A record by looking it up and seeing if the IP address returned matches an IP on the current machine. If it does, then we're done.

If both rDNS and gethostname() fail, then AND ONLY THEN, should we send the IP address.

That's all I have to say about it. The specification REQUIRES that we send a host name that resolves to an IP address unless we cannot determine one. It doesn't say we should, but only if we want to. It doesn't say that we should as long as we feel like it. It says MUST. You cannot say for certain that a host name could not be determined until you try. That is not opinion, just fact.
Comment 66 Jerry Baker 2006-01-07 08:24:39 PST
getnameinfo() does exactly what we want. It returns the host name associated with an address, and can be flagged to return only host names that can be resolved by DNS. See http://msdn.microsoft.com/library/default.asp?url=/library/en-us/winsock/winsock/getnameinfo_2.asp
http://developer.apple.com/documentation/Darwin/Reference/ManPages/man3/getnameinfo.3.html
http://uw713doc.sco.com/en/man/html.3N/getnameinfo.3N.html
Comment 67 Philip Prindeville 2006-01-07 11:59:35 PST
(In reply to comment #65)
> Thunderbird cannot know that the FQDN is impossible to determine until
> AFTER it tries to determine it unsuccessfully.

And sometimes, not even then...

What about stale data in the case of dynamic DNS?


> All we have to do is get a host name that resolves to our IP address. That's
> it. It doesn't matter if the user wishes it were a different name, or whether
> the user doesn't like rDNS, or whatever. As long as we can come up with a host
> name that resolves to an IP address on the client's machine, we're set.

Says who?

I have a static address with Qwest.  Another user might set up their MTA to block generic Qwest dialup and DSL addresses (because they are pooled), but not my particular machine, since it is reserved.  They might have an ALLOW for my machine specifically that is supposed to get applied before a REJECT for *.qwest.net ...

> My proposal is that there are far more clients who will be on dynamically
> allocated addresses, and on single IP machines, so we should start there with
> the highest probbility of success. I know about fixed IP's, but they are
> exceedingly rare for the end user. I don't know of a single DSL, cable,
> satellite, or other broadband provider that issues static IP addresses without
> specifically requesting it and paying extra.

A solution that works in only a specific subset of cases isn't sufficient.  And static addresses are more common than you think.

So what if you have to specifically request a static address and pay extra for it?  What's $5 a month to have your own website?  It's a lot less than going to  a server farm...

What are you saying?  "Static addresses cost more... therefore they shouldn't be supported."

> Because most users have dynamic IP addresses, and they do not have any control
> whatsoever over their host name's DNS records, we should start there. That's
> why we should first do an rDNS. If that returns a host name, then great we're
> done. NO ONE CARES IF IT'S NOT THE HOST NAME YOU LIKE.

However, in the case of static and dynamic addresses, users have control of their hostname.  Which says to me that that's a much better place to start.

> If rDNS does not return a host name, then we should see if the machine has one
> configured by using gethostname(). We should see if that host name resolves to
> an A record by looking it up and seeing if the IP address returned matches an
> IP on the current machine. If it does, then we're done.

That wouldn't work with VPN or SSH tunnelling in a corporate environment.  Your IP address will be that of the default interface (your dialup or Wifi or whatever).  But the address that you use to send email will be that of the tunnel interface because the VPN software will selectively pick out certain streams based on application or destination address.

> That's all I have to say about it. The specification REQUIRES that we send a
> host name that resolves to an IP address unless we cannot determine one. It
> doesn't say we should, but only if we want to. It doesn't say that we should as
> long as we feel like it. It says MUST. You cannot say for certain that a host
> name could not be determined until you try. That is not opinion, just fact.

Only in the case of dynamic addresses, and the rDNS verification is only a example.  There are other solutions.  It also doesn't require it.  It says "SHOULD", not "MUST".

You can do something else and still be compliant with the specification.
Comment 68 Jerry Baker 2006-01-07 16:46:00 PST
I am completely astonished that such a simple specification is so hard for people to understand. WE HAVE TO SEND A HOST NAME THAT RESOLVES TO OUR IP ADDRESS IF AT ALL POSSIBLE. That's the specification.

Your friend with the filter shouldn't rely on your EHLO to filter. That would be an assinine filter and they deserve to miss their messages if that's how they do it.

All of this is obviously beyond the ability of some to comprehend. I give up. You win. Yay for you.
Comment 69 Jo Hermans 2006-01-07 18:22:16 PST
(In reply to comment #68)
> I am completely astonished that such a simple specification is so hard for
> people to understand. WE HAVE TO SEND A HOST NAME THAT RESOLVES TO OUR IP
> ADDRESS IF AT ALL POSSIBLE. That's the specification.

Jerry,

you don't seem to understand. I give you 2 examples to explain why you can't rely :

- suppose that I have address 138.203.209.163. That's my real ipaddress at the moment, and I can do a perfect rDNS from it. So I can place beantn0021484.ngn.bel.alcatel.be in it, right ? Well, guess again. In the zone where my PC is located, it's a valid rDNS address. But at the receiver side (SMPT_server, or another SMTP-client), it can't be verified, because that data is not accessible from outside alcatel.be.

- suppose I have address 10.0.0.1. That's in the private address space (used in NAT among other things), and you won't be able to resolve that in rDNS. So what do you do then ? How do you even recognize thast it's a NAT address after all. Not every NAT-server uses 192.168/16, 172.16/12 and 10/8. In fact, I helped to install an entire country behind such a system. That address will resolve to a different hostname on this side of the firewall compared to the other one. Both are right.

The bottom line is that nobody cares anymore what's begind the HELO/EHLO. Face it: it's a losing battle. Ok, you don't want an ipaddress ? Then we'll put the (possibly fake) hostname inside. I don't see what it improves, except that it might score less on the SpamAssassin scale. The content itself might not be resolvable by the receiver (or might resolve to a different ipaddress for various reasons). Trying to use rDNS might produce a FQDN that has a better chance to resolve to the same ipaddress, but won't work in every case either. You'll never receive the holy grail of having a correct identifier that is verifyiable by the outside world. You might start lying just as well (as many implementations currently do). Thunderbird currently inserts the ipaddress, which is still valid according to the RFC. And just as valid or invalid as you would like to consider it.

Note that the identifier behind the HELO was never intended to be the /correct/ identifier. It was originally used to preventing loops.
Comment 70 Philip Prindeville 2006-01-10 12:29:34 PST
Can someone please flag this bug as being dependent on 322956?
Comment 71 Philip Prindeville 2006-01-10 15:39:48 PST
Created attachment 208149 [details] [diff] [review]
Fix that checks for qualification (FQDN) of gethostname()'s return value.

Ok, we not only get the value of gethostname() via the blessed interface (PR_GetSystemInfo()), but we check to see that the string is (1) non-empty and (2) contains a domain portion, or at least a dot.
Comment 72 Jerry Baker 2006-01-10 17:12:28 PST
Is the presence of a dot the way you propose to see if the host name resolves to an A record as the SMTP protocol requires?
Comment 73 Philip Prindeville 2006-01-10 17:24:35 PST
(In reply to comment #72)
> Is the presence of a dot the way you propose to see if the host name resolves
> to an A record as the SMTP protocol requires?


I'm pretty sure I addressed this already.  Let's grab that scrollbar and push it upwards... and... voila.  I did indeed:

> The only test for meaningfulness that I'll unconditionally agree to is:
> 
>     hostname[0] != '\0' && index(hostname, '.') != NULL
> 
> Anything more complicated than that would be difficult to prove "correct" in
all circumstances.

What part of that statement should I further clarify?

Comment 74 Philip Prindeville 2006-01-10 19:01:53 PST
Created attachment 208165 [details] [diff] [review]
Proposed fix, not changing existing API
Comment 75 David :Bienvenu 2006-01-19 14:24:11 PST
Comment on attachment 208165 [details] [diff] [review]
Proposed fix, not changing existing API

I guess we can give this a try and see how loud the complaints in the other direction are (e.g., if anything breaks)
Comment 76 Jerry Baker 2006-01-19 17:33:19 PST
(In reply to comment #73)
> > The only test for meaningfulness that I'll unconditionally agree to is:
> > 
> >     hostname[0] != '\0' && index(hostname, '.') != NULL
> > 
> > Anything more complicated than that would be difficult to prove "correct" in
> all circumstances.
> 
> What part of that statement should I further clarify?

None. I just wanted it on the record that you intentionally are writing non-compliant code in case it becomes an issue later on.
Comment 77 Philip Prindeville 2006-01-20 15:04:29 PST
(In reply to comment #76)
> > [snip]
> > 
> > What part of that statement should I further clarify?
> 
> None. I just wanted it on the record that you intentionally are writing
> non-compliant code in case it becomes an issue later on.

Tell you what.  How about you go and administrate networks with static IP addresses, extensive use of dynDNS, and Active Directories for 6 months each, and then come back and tell us if you still believe your solution is the fix-all that you seem to be convinced beyond reason that it is, or if you've learned anything new during that time.

I'll wait.
Comment 78 Philip Prindeville 2006-02-09 22:10:54 PST
David:

So, what needs to happen to get this committed?
Comment 79 David :Bienvenu 2006-02-10 08:01:35 PST
Comment on attachment 208165 [details] [diff] [review]
Proposed fix, not changing existing API

If Scott's OK with trying this on the trunk, I can check it in.
Comment 80 Philip Prindeville 2006-02-15 21:03:02 PST
BTW:  I went and grovelled through dhcp-3.0.1 sources, and from what I could tell the FQDN is always used to set hostname in the client scripting:

client/scripts/netbsd:      hostname $new_host_name
client/scripts/linux:      hostname $new_host_name
client/scripts/openbsd:      hostname $new_host_name
client/scripts/solaris:      hostname $new_host_name
client/scripts/freebsd:      hostname $new_host_name
client/scripts/bsdos:      hostname $new_host_name
client/scripts/nextstep:      hostname $new_host_name

so contrary to what some people have suggested, the return value of gethostname() should always return the FQDN (if known).


Comment 81 Philip Prindeville 2006-02-15 21:04:00 PST
(In reply to comment #79)
> (From update of attachment 208165 [details] [diff] [review] [edit])
> If Scott's OK with trying this on the trunk, I can check it in.

Has Scott been pinged?

Comment 82 Jerry Baker 2006-02-15 21:24:37 PST
(In reply to comment #80)
> so contrary to what some people have suggested, the return value of
> gethostname() should always return the FQDN (if known).

You mean on *nix. On Windows, gethostname() returns whatever name the user has given their computer in their "Control Panel". So, if I go into Control Panel -> System -> Computer Name and change my computer's name to be "ribbit," gethostname() will return "ribbit". At least that's what happens on my XP.
Comment 83 Philip Prindeville 2006-02-15 21:40:11 PST
(In reply to comment #82)
> You mean on *nix. On Windows, gethostname() returns whatever name the user has
> given their computer in their "Control Panel". So, if I go into Control Panel
> -> System -> Computer Name and change my computer's name to be "ribbit,"
> gethostname() will return "ribbit". At least that's what happens on my XP.

What's your point?  That XP isn't imune to misconfiguration?

Ok, neither is Linux.  I can get into /etc/sysconfig/network and put some damaged value into HOSTNAME...

A better analogy would be, "What does the Microsoft DHCP client do when given DHCP Option 12 with a FQDN, or (since it's Microsoft, after all) handle Option 81?"

The litmus test is does it take the value from a DHCPACK containing option 12 or option 81 and put that into the sethostname() buffer?  This would give the most accurate reflection of how Microsoft intended the FQDN to be stored.
Comment 84 Jerry Baker 2006-02-15 21:44:35 PST
I think you misunderstand how Windows networking is configured. The computer's name as configured in the Control Panel is *always* user set and is *never* changed by DHCP. So, in *all* cases Windows users on DHCP will be sending whatever name they have given their computer instead of anything related to DNS.
Comment 85 Philip Prindeville 2006-02-15 21:54:48 PST
(In reply to comment #84)
> I think you misunderstand how Windows networking is configured. The computer's
> name as configured in the Control Panel is *always* user set and is *never*
> changed by DHCP. So, in *all* cases Windows users on DHCP will be sending
> whatever name they have given their computer instead of anything related to
> DNS.

Ok, the question still stands:  Where does the Windows DHCP client service squirrel away the value of DHCP Option 12 or Option 81?


Comment 86 Jerry Baker 2006-02-15 22:02:26 PST
I have spent a few hours digging for that information without success. I did find a document that suggested that on Windows CE at least that information is discarded.
Comment 87 Philip Prindeville 2006-02-15 22:11:37 PST
(In reply to comment #1)
> The rule for SpamAssassin is called HELO_DYNAMIC_IPADDR and it assigns 4.4
> points if this test is met.
> 
> Adding dataloss keyword since Thunderbird's actions against the RFC's are
> resulting in the loss of emails.

Going back and reading this comment, that's not a bug in Thunderbird.

That's a problem with SA being overly pessimistic about dynamic addresses.

I think that you're also not interpreting how HELO_DYNAMIC_IPADDR works correctly.  Network 1/0 is an unallocated bogon network.  So addresses shouldn't be pooled from it.  If you try the test with a real pooled DHCP address, you might find that this rule doesn't fire.

Repeat the test in the description with the actual routable address assigned to your client host.
Comment 88 Philip Prindeville 2006-02-15 22:13:04 PST
(In reply to comment #86)
> I have spent a few hours digging for that information without success. I did
> find a document that suggested that on Windows CE at least that information is
> discarded.

Anyone have a DevNet contract that can ask?


Comment 89 Jerry Baker 2006-02-15 22:14:02 PST
That's how I came across this in the first place. I wasn't able to get my own emails because SpamAssassin was dumping my emails. That's when I discovered 4.4 points for having an IP address sent in the HELO.
Comment 90 Philip Prindeville 2006-02-15 22:15:39 PST
(In reply to comment #89)
> That's how I came across this in the first place. I wasn't able to get my own
> emails because SpamAssassin was dumping my emails. That's when I discovered 4.4
> points for having an IP address sent in the HELO.

Was your address really 1.1.1.1, and if so, WHY?

If it wasn't 1.1.1.1, what happens when you repeat your test with the actual IP address?

Comment 91 Jerry Baker 2006-02-15 22:28:10 PST
No, I just edited to be 1.1.1.1 to avoid disclosing my IP address publicly.
Comment 92 Jerry Baker 2006-02-15 23:00:50 PST
This is all I could find: http://support.microsoft.com/default.aspx?scid=kb;en-us;Q259825

Who knows if it works that way in other versions of Windows or not.
Comment 93 Philip Prindeville 2006-02-21 14:39:18 PST
(In reply to comment #91)
> No, I just edited to be 1.1.1.1 to avoid disclosing my IP address publicly.


Were you using a routable or unroutable address?  What CIDR block was the address in?

Please be careful when attaching logs and then editing them, as this can be misleading and have people chasing their tails, especially if you forget to make all of the edits...

Network 1 as it turns out has special significance in SpamAssassin.


Comment 94 David :Bienvenu 2006-02-21 14:57:08 PST
Scott, I think this is worth trying on the trunk and see if it creates any problems.
Comment 95 Jerry Baker 2006-02-21 14:58:27 PST
The address was a routable address in the 4.0.0.0/8 network.
Comment 96 Philip Prindeville 2006-03-02 10:52:07 PST
(In reply to comment #94)
> Scott, I think this is worth trying on the trunk and see if it creates any
> problems.


Has there been any out-of-band communication on this happening (or not happening) that we could update the comments with a summary of?

Comment 97 Scott MacGregor 2006-03-14 17:35:43 PST
This seems to conflict with Bug 68877 to me. I'd really rather not open up that can of worms again.
Comment 98 Philip Prindeville 2006-03-14 23:42:39 PST
(In reply to comment #97)
> This seems to conflict with Bug 68877 to me. I'd really rather not open up that
> can of worms again.

I've been reading through Bug 68877... ALL OF IT... and a lot of it echos what has already been said here.  Which isn't very surprising.  The wide consensus is that (1) you can't always rely on rDNS working, and that (2) sending the address-literal for the current connection is a reasonable fallback strategy.

Arriving at a fix is often times an iterative process.  I don't see this as opening a can of worms again, as much as elaborating on the solution in the time since 68877 was committed, and what we've learned from the operational experience in the interval.

It could end up being that we've been trying to work-around problems that can't easily be fixed.

For instance, I've been seeing a couple of repeating themes:

* going through a NATting gateway doesn't cause the EHLO parameter to get rewritten if it's an address-literal.  That may be a bug in the way NAT works.

* in a VPN split-tunneling scenario, the host might simultaneously have two addresses, two hostnames, two FQDNs, and two default routes (yes, based on policy routing).  How do you pick one?

* a similar dilemma applies in the case of multi-homed hosts

* in the case of using DHCP with a network operator (say cable or DSL)... should the ISP be responsible for ensuring that the IP address comes with a FQDN, telling you what that FQDN is, and having valid rDNS in place?

We can tweak special case fixes that make the overall solution less automatic (or somewhat more burdensome) in all scenarios.  Or we can come up with a general solution that falls through the cracks in being unable to deal with specific corner cases.

Revisiting this issue is a Good Thing if it's warranted.





Comment 99 Philip Prindeville 2006-03-14 23:44:08 PST
(In reply to comment #97)
> This seems to conflict with Bug 68877 to me. I'd really rather not open up that
> can of worms again.

Forgot to add:  what specifically do you see as being in conflict?  I couldn't find a conflict.


Comment 100 Jerry Baker 2006-03-15 07:26:07 PST
(In reply to comment #98)
> (1) you can't always rely on rDNS working, and that (2) sending the
> address-literal for the current connection is a reasonable fallback strategy.

rDNS does work a lot of the time, though, and it's not reasonable to say that it is so unreliable that it should never even be tried. A "fallback strategy" implies that you have tried to do something, it has failed, and that now you will attempt a second-best option. This patch's strategy is to assume failure of rDNS and use the "fallback" strategy 100% of the time, which, by definition, makes it the primary strategy and not a "fallback" strategy.
Comment 101 Philip Prindeville 2006-03-15 10:42:25 PST
(In reply to comment #100)

> rDNS does work a lot of the time, though, and it's not reasonable to say that
> it is so unreliable that it should never even be tried. A "fallback strategy"
> implies that you have tried to do something, it has failed, and that now you
> will attempt a second-best option. This patch's strategy is to assume failure
> of rDNS and use the "fallback" strategy 100% of the time, which, by definition,
> makes it the primary strategy and not a "fallback" strategy.

For what it's worth Jerry, repeating yourself doesn't add weight to your argument, and I'm getting tired of having the same conversation.

rDNS is no magic bullet.  Why is this so hard to understand?

Even when it works flawlessly, it might still not be appropriate.

Look at actual data from the Internet:

# nslookup 65.220.17.5
Server:         205.171.3.65
Address:        205.171.3.65#53

Non-authoritative answer:
5.17.220.65.in-addr.arpa        name = www.besco.com.
5.17.220.65.in-addr.arpa        name = www.accmgmt.net.
5.17.220.65.in-addr.arpa        name = www.cfadminz.com.
5.17.220.65.in-addr.arpa        name = www.blount911.com.
5.17.220.65.in-addr.arpa        name = www.ceoagenda.com.
5.17.220.65.in-addr.arpa        name = www.microcerv.net.
5.17.220.65.in-addr.arpa        name = www.easttntitle.com.
5.17.220.65.in-addr.arpa        name = www.broadwayelec.com.
5.17.220.65.in-addr.arpa        name = www.cooperhotels.com.
5.17.220.65.in-addr.arpa        name = www.insiteknoxville.com.
5.17.220.65.in-addr.arpa        name = toner.microcerv.com.
5.17.220.65.in-addr.arpa        name = denark.com.

Authoritative answers can be found from:
17.220.65.in-addr.arpa  nameserver = ns2.microcerv.com.
17.220.65.in-addr.arpa  nameserver = ns1.microcerv.com.
ns1.microcerv.com       internet address = 65.220.15.2
ns2.microcerv.com       internet address = 65.220.15.4

#

What would the right thing to do be in this case?

Your rDNS worked!  It came back with an answer.  Several, in fact.

You're overblowing the value of the solution that you propose because it happens to work in your instance.

For people that want to develop a solution that works for everyone as best as it can, that's just not enough.

How about you go back and read comment #77 again.  I don't think it's yet been 18 months since I wrote that.

Comment 102 Jerry Baker 2006-03-15 10:55:28 PST
(In reply to comment #101)
> For what it's worth Jerry, repeating yourself doesn't add weight to your
> argument, and I'm getting tired of having the same conversation.

And yet you continue.

> rDNS is no magic bullet.  Why is this so hard to understand?

Your position is not hard to understand, but that doesn't mean it makes any sense.

> Even when it works flawlessly, it might still not be appropriate.

Your patch doesn't even try. You dismiss it outright without even trying. Any conversation beyond this is moot.

> Look at actual data from the Internet:
> What would the right thing to do be in this case?
> 
> Your rDNS worked!  It came back with an answer.  Several, in fact.

Yes it did, and all of them would be appropriate because they identify the same host. That's what the specification asks for. Not all solutions have only one answer. Your patch is guaranteed to send the wrong thing for every single Windows user using DHCP. EVERY SINGLE ONE.
 
> You're overblowing the value of the solution that you propose because it
> happens to work in your instance.

It works in yours too, you just don't like the answer.

> For people that want to develop a solution that works for everyone as best as
> it can, that's just not enough.

As opposed to your solution which is 100% guaranteed to break it for the majority of users.
Comment 103 Philip Prindeville 2006-03-15 11:06:45 PST
(In reply to comment #102)
> And yet you continue.

Actually, I was addressing Scott.

> > rDNS is no magic bullet.  Why is this so hard to understand?
> 
> Your position is not hard to understand, but that doesn't mean it makes any
> sense.

As opposed to one that only makes sense when you eliminate 95% of the real-world corner cases that one has to deal with.

> > Even when it works flawlessly, it might still not be appropriate.
> 
> Your patch doesn't even try. You dismiss it outright without even trying. Any
> conversation beyond this is moot.

Actually, I've tried it on all of the machines I have access to, both with and without the host name coming from DHCP, and it works fine.

> > Look at actual data from the Internet:
> > What would the right thing to do be in this case?
> > 
> > Your rDNS worked!  It came back with an answer.  Several, in fact.
> 
> Yes it did, and all of them would be appropriate because they identify the same
> host.

Really?  I don't think so.  Go back and read the original description of the bug.  You said that SpamAssassin was rejecting email.

It would very likely reject email if you HELO'd with one address, but tried to send with a different one as the sender's domain.

> That's what the specification asks for. Not all solutions have only one
> answer. Your patch is guaranteed to send the wrong thing for every single
> Windows user using DHCP. EVERY SINGLE ONE.

Poppycock.

You don't know that.

In fact, no one has said conclusively what the Microsoft DHCP client will do.  And if it does the wrong thing... then it needs to be fixed.  Not Thunderbird.  Linux, Solaris, and FreeBSD DHCP clients all do the right thing.

Kludging Tbird to workaround a bug in the DHCP client is not the right thing to do.

> > You're overblowing the value of the solution that you propose because it
> > happens to work in your instance.
> 
> It works in yours too, you just don't like the answer.

It does?  Let's see:

# nslookup 192.168.1.5
Server:         205.171.3.65
Address:        205.171.3.65#53

** server can't find 5.1.168.192.in-addr.arpa: NXDOMAIN

#

Guess again.

> > For people that want to develop a solution that works for everyone as best as
> > it can, that's just not enough.
> 
> As opposed to your solution which is 100% guaranteed to break it for the
> majority of users.

If your DHCP client is doing the wrong thing, you're already broken... you just don't know it.

Comment 104 Philip Prindeville 2006-03-15 11:12:30 PST
(In reply to comment #102)

Instead of a rant, how about coming up with a well documented methodology to test exactly what the DHCP client does?

For instance, you could statically configure a garbage hostname, set a static IP address, and then go into the control panel, and turn on DHCP instead.  Then examine the hostname field again and see if it has been overridden.  If  you know how, you could even attach a monitor to the DHCP service and trap its calls (including seeing what sendto() and sethostname() calls it makes, and what the content of the query and response packets are).

That is, you could bring something concrete and substantive to the table.

Instead of opinions and speculation.
Comment 105 Jerry Baker 2006-03-15 11:18:20 PST
I have a better idea: Why not go to some Windows machines that are actually using a real ISP? See what happens. Stop playing with your in-house DHCP, DNS, and Windows machines. Connect a Windows PC to Verizon, AT&T, AOL, and other ISP's and watch what happens with your solution.
Comment 106 Jerry Baker 2006-03-15 11:21:40 PST
(In reply to comment #103)
> Really?  I don't think so.  Go back and read the original description of the
> bug.  You said that SpamAssassin was rejecting email.

Are you basing your solution on the success of sneaking mail past SpamAssassin or meeting the requirements of SMTP specs?
 
> It would very likely reject email if you HELO'd with one address, but tried to
> send with a different one as the sender's domain.

All of those domains had the same address.
> # nslookup 192.168.1.5
> Server:         205.171.3.65
> Address:        205.171.3.65#53
> 
> ** server can't find 5.1.168.192.in-addr.arpa: NXDOMAIN
> 
> #
> 
> Guess again.

This is the crux of my argument. Your patch will never know that this has happened because it doesn't even try. When this happens, and only when this happens, we should send the IP address.
Comment 107 Philip Prindeville 2006-03-15 11:38:41 PST
(In reply to comment #105)
> I have a better idea: Why not go to some Windows machines that are actually
> using a real ISP? See what happens. Stop playing with your in-house DHCP, DNS,
> and Windows machines. Connect a Windows PC to Verizon, AT&T, AOL, and other
> ISP's and watch what happens with your solution.

Good idea, Jerry.

In the 5+ years that I was at Cisco, 5+ years at France Telecom, 3 years at Ricochet Wireless, 1 year Bellcore and Erol's Internet, it never would have occured to me to pay attention to what happens in actual ISPs.

Nor, for that matter, when I wrote RFC-1048 or worked on the DHCP working group.

If the ISP is sending option 12 or option 81, and the FQDN is being put into the sethostname() buffer as it should be, then what part of my proposal is going to fail, and why?

And if this isn't being done, then explain to me why getting it fixed isn't the correct solution?


Comment 108 Jerry Baker 2006-03-15 11:46:03 PST
(In reply to comment #107)
> If the ISP is sending option 12 or option 81, and the FQDN is being put into
> the sethostname() buffer as it should be, then what part of my proposal is
> going to fail, and why?
> 
> And if this isn't being done, then explain to me why getting it fixed isn't the
> correct solution?

I take it that your post was a long-winded way of saying that you are not going to try hooking up to an actual ISP and see what happens with your patch.
Comment 109 Philip Prindeville 2006-03-15 11:52:01 PST
(In reply to comment #106)
> (In reply to comment #103)
> > Really?  I don't think so.  Go back and read the original description of the
> > bug.  You said that SpamAssassin was rejecting email.
> 
> Are you basing your solution on the success of sneaking mail past SpamAssassin
> or meeting the requirements of SMTP specs?

Both.

But the SMTP specification doesn't say that the HELO parameter, in the case of an FQDN, has to have any validity.  That's something that MTA implementors have added over the years.

Historically, the HELO parameter was just used to detect when a host was talking back to itself (which was easy to do in the days of a BB&N IMP using the 1822 protocol).  Keep in mind that SMTP predates TCP, and you could speak SMTP over an NCP connection...  and many hosts did during the cut-over.

> > It would very likely reject email if you HELO'd with one address, but tried to
> > send with a different one as the sender's domain.
> 
> All of those domains had the same address.
> > # nslookup 192.168.1.5
> > Server:         205.171.3.65
> > Address:        205.171.3.65#53
> > 
> > ** server can't find 5.1.168.192.in-addr.arpa: NXDOMAIN
> > 
> > #
> > 
> > Guess again.
> 
> This is the crux of my argument. Your patch will never know that this has
> happened because it doesn't even try. When this happens, and only when this
> happens, we should send the IP address.

The single most widely used MTA *ever* (Sendmail) has always used, since day 1, the contents of the gethostname() buffer for this argument without problems.

Why are you convinced that this will suddenly break?  Or that it has somehow stopped being the correct thing to do?

In the case of behavior as above, there is no absolute foolproof way of knowing if the returned data is valid or not, and hence having a perfect mechanism for deciding when to use the fallback method.

Can you think of a set of rules that can be used with absolute reliability to decide when to use the rDNS return data and when not to?

What about the case of a split-tunnel, i.e. you're on an ISP but you're also using a VPN tunneled connection to connect to the corporate internet.  The kernel applies a policy map based on (amongst other things) the protocol and port numbers (both source and destination) to decide which interface (you have two:  one for the dsl, and the other for the IPSEC tunnel) to home the packets from on an unbound socket (which is what most client applications use).

Your machine has two interfaces, and therefore two FQDN's.  Both are valid.  Which one do you go with?  And why?

Give me a fullproof mechanical way of detecting the correct answer, and I'd concede that your solution is workable.

Comment 110 Philip Prindeville 2006-03-15 11:55:40 PST
(In reply to comment #108)
> I take it that your post was a long-winded way of saying that you are not going
> to try hooking up to an actual ISP and see what happens with your patch.

What part of "Actually, I've tried it on all of the machines I have access to, both with and without the host name coming from DHCP, and it works fine." was unclear?  If I haven't tried with Windows, it might just be because I don't have a Windows machine here.

If it's being long-winded, it's because I'm limited to using single syllables when explaining things to you.


Comment 111 Philip Prindeville 2006-03-15 11:57:31 PST
(In reply to comment #105)
> I have a better idea: Why not go to some Windows machines that are actually
> using a real ISP? See what happens. Stop playing with your in-house DHCP, DNS,
> and Windows machines. Connect a Windows PC to Verizon, AT&T, AOL, and other
> ISP's and watch what happens with your solution.

You're the person saying that this won't work.

Prove it either way.

I've provided a patch, and operational experience with that patch on a several different clients, and documented my results.

What have you contributed?

Comment 112 Jerry Baker 2006-03-15 12:05:57 PST
> Your machine has two interfaces, and therefore two FQDN's.  Both are valid.
> Which one do you go with?  And why?

This problem exists equally for your solution. Which hostname do you use?

> Can you think of a set of rules that can be used with absolute reliability to
> decide when to use the rDNS return data and when not to?

You do realize that the implication of this statement is that the method I propose is not absolutely reliable, and therefor yours is. Yours is not whatsoever. It will be guaranteed to fail in a situation which millions of user will experience on a daily basis. But, as you say, I don't understand long words so perhaps you'd care to explain why it is that your solution will give us a more reliable solution.

> What have you contributed?

Umm, the bug report. The research on the issue prior to and during the discussion on this bug. I think you have mistaken me for a beta tester of your patches. That assumption is incorrect.
Comment 113 Jerry Baker 2006-03-15 12:12:35 PST
(In reply to comment #112)
> perhaps you'd care to explain why it is that your solution will give
> us a more reliable solution.

It occurs to me that this is likely to be misinterpreted as a "more correct" solution. What I mean is how will your solution do the right things in a larger set of real world circumstances than trying rDNS before sending the result of gethostname or the IP address?
Comment 114 Philip Prindeville 2006-03-15 12:22:36 PST
(In reply to comment #112)
> > Your machine has two interfaces, and therefore two FQDN's.  Both are valid.
> > Which one do you go with?  And why?
> 
> This problem exists equally for your solution. Which hostname do you use?

Is it any wonder I'm long-winded?

Ok, you're not getting this.

The gethostname() buffer is singular and unitary.  There's just one.

There's no ambiguity of which one to use, because there's only one.  Unlike having to interfaces, each with its own rDNS FQDN.

A good VPN client will allow you to configure overriding the hostname buffer (or not).

Similarly, the VPN hub upstream will be configured to send a hostname downstream via DHCP or not...

Because it's a VPN connection, you can assume that the client already has a hostname set from the ISP... and thereby optionally not send an option 12 FQDN.

Tne problem does NOT exist equally.  You aren't grokking this.

There is only ever going to be a single hostname buffer.

> > Can you think of a set of rules that can be used with absolute reliability to
> > decide when to use the rDNS return data and when not to?
> 
> You do realize that the implication of this statement is that the method I
> propose is not absolutely reliable, and therefor yours is. Yours is not
> whatsoever. It will be guaranteed to fail in a situation which millions of user
> will experience on a daily basis. But, as you say, I don't understand long
> words so perhaps you'd care to explain why it is that your solution will give
> us a more reliable solution.

Guaranteed to fail?

Where are you getting this?

You've said yourself in comment #92 that you don't know how it works.  And now you're telling me you're sure that it won't work in millions of cases?

It would be a lot easier to take you seriously if you stopped contradicting yourself and started providing facts.

> > What have you contributed?
> 
> Umm, the bug report. The research on the issue prior to and during the
> discussion on this bug. I think you have mistaken me for a beta tester of your
> patches. That assumption is incorrect.

I've not mistaken you for having tested my patches.  No chance of that.  That was assume diligence and investigative follow through.

No, I'm pretty sure that you're the guy that provided a worm's eye view of what you see, and only what you see, and then very little of a concrete nature beyond that.

I've made the reasonable request that you do a little digging to provide conclusive information about how the DHCP client behaves, and you've not done so.
Comment 115 Philip Prindeville 2006-03-15 12:26:22 PST
(In reply to comment #113)
> (In reply to comment #112)
> > perhaps you'd care to explain why it is that your solution will give
> > us a more reliable solution.
> 
> It occurs to me that this is likely to be misinterpreted as a "more correct"
> solution. What I mean is how will your solution do the right things in a larger
> set of real world circumstances than trying rDNS before sending the result of
> gethostname or the IP address?


It's a good thing one can't turn blue from typing.

I wrote:

"In the case of behavior as above, there is no absolute foolproof way of knowing
if the returned data is valid or not, and hence having a perfect mechanism for
deciding when to use the fallback method."

"Can you think of a set of rules that can be used with absolute reliability to
decide when to use the rDNS return data and when not to?"

What part of that isn't clear?

In the absence of an absolutely foolproof test of knowing if data is tainted or not, you must assume that the data is always TAINTED.

Period.

How can I be more clear?

Getting a result doesn't in and of itself mean the result is valid.  The data pasted in comment #101 makes that painfully obvious.



Comment 116 Don Seiler 2006-03-30 14:49:46 PST
(In reply to comment #51)
> great, thanks. This may have already been discussed, but what do other e-mail
> clients do? For example, if a lot of other e-mail clients do what you're
> proposing here, it's less likely to cause a problem with servers.
> 

Outlook Express behaves like Outlook in comment #52 and just sends the hostname, not FQDN.  On the linux side, I've tested mutt sending with both ssmtp and msmtp. ssmtp would send the FQDN of the box within my LAN, and msmtp just always sends "localhost".

Don't know if you gents are aware the Hotmail seems to be also blocking the bad HELO, even thought it's at the bottom of the chain.  At home I send through SBC's SMTP server (which requires auth), and eventually it routes on to Hotmail, where it just gets sent to their equivalent of /dev/null.  It doesn't even show up in the user's Junk Mail folder.  What's more annoying is that I don't receive any kind of notification that the mail failed to deliver.  From my point of view, SBC's SMTP server accepted it, and that's all I know.  If a friend hadn't alerted me to not receiving expected emails, I might still be blissfully unaware.

Now whether or not we want to cater to the likes of Hotmail, you'll have to admit it's a fairly large pool of recipients and a lot of us are likely to have to send email to them.

I also noticed that OE and msmtp say "with login" on that HELO line, but tbird says "with plain".  Probably a separate issue.
Comment 117 Don Seiler 2006-03-31 08:01:24 PST
(In reply to comment #116)
> Outlook Express behaves like Outlook in comment #52 and just sends the
> hostname, not FQDN.  On the linux side, I've tested mutt sending with both
> ssmtp and msmtp. ssmtp would send the FQDN of the box within my LAN, and msmtp
> just always sends "localhost".

Forgot to note that both ssmpt and msmtp mails *are* accepted by Hotmail, as is the Outlook express.  It seems to just be exceptionally **** IP-based HELO.
Comment 118 Scott MacGregor 2006-03-31 15:14:21 PST
Comment on attachment 208165 [details] [diff] [review]
Proposed fix, not changing existing API

based on comment 98, I'm going to sr this and let's try it out on the trunk only.

Thanks for your patience Philip.
Comment 119 Philip Prindeville 2006-04-05 19:34:53 PDT
(In reply to comment #118)
> (From update of attachment 208165 [details] [diff] [review] [edit])
> based on comment 98, I'm going to sr this and let's try it out on the trunk
> only.
> 
> Thanks for your patience Philip.

No problem.  It's a thorny issue.  There's no denying that.

I'd just rather we all understand the problem rather than rushing a bad fix.


Comment 120 Don Seiler 2006-04-24 10:54:57 PDT
(In reply to comment #118)
> (From update of attachment 208165 [details] [diff] [review] [edit])
> based on comment 98, I'm going to sr this and let's try it out on the trunk
> only.

When might we see this patch in a release?  The hotmail thing is really killing me, as sad as that sounds.  Are there nightly trunk builds for win32 available?
Comment 121 Don Seiler 2006-04-24 10:56:54 PDT
(In reply to comment #120)
> When might we see this patch in a release?  The hotmail thing is really killing
> me, as sad as that sounds.  Are there nightly trunk builds for win32 available?

http://releases.mozilla.org/pub/mozilla.org/thunderbird/nightly/2006-04-24-06-trunk/

Guess it helps if I look first and ask questions later.  I guess my next question would be the typical "is it even remotely stable"?
Comment 122 David :Bienvenu 2006-07-13 11:56:15 PDT
that patch didn't compile on my macbook pro - PR_SI_HOSTNAME_FQDN was not defined.
Comment 123 David :Bienvenu 2006-07-13 13:05:36 PDT
Created attachment 229148 [details] [diff] [review]
is this the right patch for the trunk?

I replaced the thing that didn't compile with PR_SI_HOSTNAME_UNTRUNCATED, the thing you added to NSPR - is that correct? If not, what should I do to get this to compile on the trunk? Also, did PR_SI_HOSTNAME_UNTRUNCATED get added to the 1.8 branch, do you remember?
Comment 124 David :Bienvenu 2006-07-13 13:15:05 PDT
OK, I read the dependent bug, so I see how the enum name changed. I've landed this change on the trunk. Getting it into 2.0 depends on it working well on the change, and the patch for bug 322956 getting approved for the branch as well.
Comment 125 Philip Prindeville 2006-07-13 13:17:24 PDT
(In reply to comment #123)
> Created an attachment (id=229148) [edit]
> is this the right patch for the trunk?
> 
> I replaced the thing that didn't compile with PR_SI_HOSTNAME_UNTRUNCATED, the
> thing you added to NSPR - is that correct? If not, what should I do to get this
> to compile on the trunk? Also, did PR_SI_HOSTNAME_UNTRUNCATED get added to the
> 1.8 branch, do you remember?

If that was the only change (PR_SI_HOSTNAME_FQDN => PR_SI_HOSTNAME_UNTRUNCATED), then yes, that would be the appropriate fix.

Comment 126 Thomas Henlich 2006-11-01 12:20:40 PST
Bug 289731 is related to this bug.
Comment 127 Jerry Baker 2006-11-01 12:31:48 PST
And Thunderbird 20061024 trunk still sends the IP address for EHLO.

----------
220 virt24r.secure-wi.com ESMTP Sendmail Secure/Rabid; Wed, 1 Nov 2006 12:29:25 -0800
EHLO [71.108.210.217]
250-virt24r.secure-wi.com Hello pool-71-108-210-217.lsanca.dsl-w.verizon.net [71.108.210.217], pleased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-8BITMIME
250-SIZE
250-DSN
250-ETRN
250-AUTH LOGIN PLAIN
250-DELIVERBY
250 HELP
Comment 128 Philip Prindeville 2006-11-01 16:16:53 PST
(In reply to comment #127)
> And Thunderbird 20061024 trunk still sends the IP address for EHLO.
> 
> ----------
> 220 virt24r.secure-wi.com ESMTP Sendmail Secure/Rabid; Wed, 1 Nov 2006 12:29:25
> -0800
> EHLO [71.108.210.217]
> 250-virt24r.secure-wi.com Hello pool-71-108-210-217.lsanca.dsl-w.verizon.net
> [71.108.210.217], pleased to meet you
> 250-ENHANCEDSTATUSCODES
> 250-PIPELINING
> 250-8BITMIME
> 250-SIZE
> 250-DSN
> 250-ETRN
> 250-AUTH LOGIN PLAIN
> 250-DELIVERBY
> 250 HELP


On which platform?  Using DHCP or static IP addressing?  What's the return value of gethostname() when this happens?

Please give a little more information than just "it doesn't work", etc.  If you want other people to be able to reproduce what you're seeing, you'll need to specify under which conditions you're seeing it.
Comment 129 Jerry Baker 2006-11-01 16:33:27 PST
(In reply to comment #128)
> On which platform?  Using DHCP or static IP addressing?  What's the return
> value of gethostname() when this happens?

Windows, DHCP, and gethostbyname() returns jerbaker.dsl-verizon.net. This information has not changed since I reported the bug.

> Please give a little more information than just "it doesn't work", etc.  If you
> want other people to be able to reproduce what you're seeing, you'll need to
> specify under which conditions you're seeing it.

My bad. I had thought it was clear since the whole entire debate on this topic was the fact that the patch would fail to do the right thing on Windows DHCP clients.
Comment 130 David :Bienvenu 2006-11-21 08:44:15 PST
fix landed for 2.0
Comment 131 Samuel Sidler (old account; do not CC) 2007-03-30 16:20:14 PDT
I'm not entirely sure how to verify this bug. Jerry, as the original reporter, can you confirm that this bug has been fixed in a 2.0 branch build?
Comment 132 Jerry Baker 2007-11-14 21:20:13 PST
Here is what headers from my email look like as of 2.0.0.9:

Received: from ?192.168.232.99? ( [71.108.205.174])
        by mx.google.com with ESMTPS id k19sm3043869rvb.2007.11.14.21.17.05
        (version=SSLv3 cipher=RC4-MD5);
        Wed, 14 Nov 2007 21:17:06 -0800 (PST)

Note that the IP address is still being sent for the HELO. I don't know what's up with the question marks.
Comment 133 Jo Hermans 2007-11-15 04:30:28 PST
works for me in version 2.0.0.9 (20071031) :

Received: from [172.31.152.44] ([172.31.152.44]) by FRVELSBHS07.ad2.ad.alcatel.com over TLS secured channel with Microsoft SMTPSVC(6.0.3790.2499);
	 Thu, 15 Nov 2007 13:27:02 +0100
Message-ID: <473C3B15.1080500@alcatel-lucent.be>
Date: Thu, 15 Nov 2007 13:27:01 +0100

Comment 134 Jerry Baker 2007-11-15 07:39:49 PST
Without understand your network configuration, I have to say that looks like it's NOT working for you either. It should read something like:

Received: from yourhost.alcatel.com ([172.31.152.44]) by
FRVELSBHS07.ad2.ad.alcatel.com over TLS secured channel with Microsoft
SMTPSVC(6.0.3790.2499);
         Thu, 15 Nov 2007 13:27:02 +0100
Message-ID: <473C3B15.1080500@alcatel-lucent.be>
Date: Thu, 15 Nov 2007 13:27:01 +0100
Comment 135 Jo Hermans 2007-11-15 09:16:44 PST
No, the mail-proxy doesn't have access to my dns-name. But it was a bad example (my setup changes recently), I just wanted to show that I'm not seeing the ? character. I'll try again when I'm home.
Comment 136 Jerry Baker 2007-11-15 09:33:41 PST
My bad. A little miscommunication. I don't know if the question marks are introduced by Thunderbird or some MTA along the way. I'm inclined to think one of the MTAs messed it up.
Comment 137 Phil Ringnalda (:philor) 2008-03-03 10:47:03 PST
*** Bug 420743 has been marked as a duplicate of this bug. ***
Comment 138 Jerry Baker 2008-08-15 01:39:51 PDT
Should I open a new bug for the fact that Thunderbird is still sending IP addresses for EHLO when a reverse DNS would give the correct hostname, or should this bug be reopened. As I originally reported, this is only an issue because it's getting my mail binned by some SpamAssassin configurations.

The ? characters are caused by some MTA replacing the [ and ] characters with a ?.
Comment 139 Magnus Melin 2008-08-15 04:23:58 PDT
New bugs for new/remaining issues please.
Comment 140 Philip Prindeville 2009-01-22 15:26:51 PST
(In reply to comment #138)
> Should I open a new bug for the fact that Thunderbird is still sending IP
> addresses for EHLO when a reverse DNS would give the correct hostname, or
> should this bug be reopened. As I originally reported, this is only an issue
> because it's getting my mail binned by some SpamAssassin configurations.

Jerry: give it a rest.

This is the best fix possible in a complicated world.  There is no fix that will work absolutely correctly in *all* scenarios.  Get over it.

This fix is the least harmful in scenarios where (a) you're passing through a NATting gateway (outbound) that changes the connection's IP but doesn't rewrite the stream contents themselves, or (b) you're passing through an ALG (application level gateway, again outbound) that both rewrites the connection's IP *and* the stream contents.

The fix that works best for you in your narrow scenario might be more harmful to others in somewhat different circumstances.

Consider that a NATting firewall can't rewrite the stream contents on an SMTP/S connection anyway, just change the addresses.

Move on.

Everyone else has.

This is as good as it's going to get.

Get yourself either a static IP address and stop using NAT, or get a firewall with ALG capabilities that can rewrite the stream.
Comment 141 Jerry Baker 2009-01-22 22:58:43 PST
(In reply to comment #140)
> This is the best fix possible in a complicated world.  There is no fix that
> will work absolutely correctly in *all* scenarios.  Get over it.

I don't believe there is a solution that will work correctly in all scenarios. 

A. The SMTP specification states that an address literal should not be used unless "no reverse mapping record is available."

B. The current solution sends address literals when a reverse mapping is available.

C. Therefore, the current solution isn't compliant.

The utility to myself or others, or anybody's willingness to "move on," is not a valid argument for or against complying with the SMTP specification.
Comment 142 Philip Prindeville 2009-01-23 00:02:36 PST
(In reply to comment #141)
> I don't believe there is a solution that will work correctly in all scenarios. 
> 
> A. The SMTP specification states that an address literal should not be used
> unless "no reverse mapping record is available."
> 
> B. The current solution sends address literals when a reverse mapping is
> available.

There is a reverse mapping for the NATted connection's visible *external* address, which is set by the NATting firewall.  It is *not* known to the internal host, which knows only its own unroutable address.

Further:  imagine a scenario where I have a multihomed host with:

(a) an internal subnet address (on a physical network) which may or may not be routable,

(b) no default route,

(c) and two IPSec tunnels putting me on remote subnets.

Which interface do I choose to do the remote DNS lookup on?  And why?  How do I know this is the right choice in all cases?

Explain your reasoning in 5000 words or less, and remember: it has to work in all cases.

The world is more complicated than your one example.

Indeed, it's more complicated than my example.  By a whole lot.  That's why I don't claim to know what will work in all instances.

I know of one company that uses routable addresses on their internal network which is a class B (because they can), but obfuscates their host addresses using NAT going out their firewall (via a class C network) because they believe it makes their network more secure.  And it's perfectly legal to do so.

So even applying a simple test like, "if my address is routable, then rDNS is... and if it's not, don't" wouldn't even work in that case.

And not to make too fine a point of it, but in your example in comment #132, you actually couldn't be more mistaken.

Any address on the unroutable network 172.16.0.0/12 *isn't* be definition routable or unique, and by extension not valid in a reverse DNS mapping (since it isn't globally unique: RFC 1918, section 3, paragraphs 3, 7, and 8).

The only address that has rDNS is the one that your NATting firewall is using to masquerade addresses.  That's the FIREWALL's address, not the sending MTA's address.  NATting, when done properly, is something that the endpoints are unaware of.

Yet your example requires the endpoints to detect the presence of NATting and act differently.

Further, there are Cable and Cell carriers than perform NATting, thus if you have a firewall on their network, you're actually being NATted twice... so even if you could ask your firewall for its *public* address, that address would actually be an unroutable private address on the carrier's customer address pool.

Further, suppose you took your IP address and performed an rDNS on it, and even got something that looked half-way reasonable.  So what?  How do you know there's not some split-horizon DNS server that is giving you answers that are only locally significant (i.e. not valid outside of your administrative domain)?

Did I mention that the real world was more complicated than your example?

> C. Therefore, the current solution isn't compliant.

Compliant with what, Jerry?  There are multiple standards at play here (RFC-2821 is only one of them), and some of them interact with each other in subtle and unanticipated ways.

I'm done trying to explain this to you, Jerry.

You're not getting it.

> The utility to myself or others, or anybody's willingness to "move on," is not
> a valid argument for or against complying with the SMTP specification.
Comment 143 Chad Jones 2009-01-23 10:24:50 PST
I wanted to include the text I originally posted in bug #420743 which was marked as a duplicate of this bug.  The text is below.  I just wanted to make it clear this is an issue that does effect users negatively.  I don't know if changing it will make things worse or not but I do want to make it clear that the current configuration is a problem for some users.  
Perhaps the best solution might be a switch somewhere in the application where if the switch is set it uses the "new" way and if not it uses the "old" way?  Just a suggestion I'm not an expert.  I'm just a Thunderbird user. 

----
Thunderbird message headers are causing problems when sending to hotmail users
because of how the headers are formulated.

The following website details the issue.  Relevant text is included below.
http://seilerwerks.wordpress.com/2006/03/31/hotmail-vs-thunderbird-the-quickening/

----
Hotmail vs Thunderbird: The Quickening

----
I've been sending a lot of email to a lot of vague acquaintances lately, mostly
due to trying to get a lot of people to join us and play some Ultimate.
Obviously a lot of people happen to use Hotmail for their email provider. One
day, one such acquaintance told me that he hadn't received an expected email
from me. Intrigued and suspicious, I decided to do some controlled testing with
a dummy hotmail account I've kept around for just such an occasion.

It turns out that when I send email from Mozilla Thunderbird to a Hotmail
address in which my address is not already in their contact list or in the
"Safe List", it simply gets dropped by Hotmail. It doesn't even get delivered
to that user's Junk Folder or anything for them to deal with. No notification
is sent to me that the mail wasn't delivered, since my mail was accepted by my
ISP's SMTP server just fine, which then sends it to Hotmail. And the recipient
obviously has know way of knowing that an email wasn't delivered, except for my
personally notifying them through non-email channels.

What was even more curious though, is that when I sent mail from ssmtp or msmtp
(via mutt) on my linux box at home, also going through my ISP's SMTP server, it
was received.

Well it turns out that both Thunderbird and Hotmail are at fault (in my
opinion) on this one. Thunderbird sends an IP address in the SMTP HELO command.
This is basically asking to get flagged as spam by any decent spam catcher,
like SpamAssassin. I feel comfortable saying that this is why Hotmail drops the
mail. Thunderbird doesn't even try to get the client machine's hostname or
FQDN. On linux, ssmtp would send the internal (non-resolvable) FQDN of that
machine, and msmtp would just literally send "localhost". Both were acceptable
to Hotmail. Outlook and Outlook Express send just the hostname (not FQDN).
Thunderbird always sends the IP address, no matter what.

Well fine, but then why does Hotmail feel the need to simply drop the message.
As I said earlier, if my email address is in the Hotmail user's Safe List or
contact list, it will be received. But if the user never sees my message in the
first place, they would never know to add me to their list. Plus if I'm trying
to email a bunch of kids (e.g. replying-all to a multi-recipient list), I
shouldn't be expected to first contact them another way and ask them to add me
as a contact so Hotmail won't drop the message. It's just asinine. What is the
point of the Junk folder, I ask? Rest assured that I've gone over my own
Hotmail Junk Filter settings and made them as forgiving as possible. The mail
doesn't get delivered. PERIOD.

Furthermore, as you'd read in the Thunderbird bug, the SMTP RFC does indeed say
to send the IP address if the FQDN is not obtainable. Even though Thunderbird
really isn't making an effort to obtain the FQDN, simply blackholing the email
message is going to result in a lot of data loss.

What's more, it isn't like Thunderbird is sending the "bad" HELO to Hotmail. It
is sending it to SBC's (my ISP) SMTP server, which requires authentication, and
is accepted there just fine. Here is a header snippet to show you what I mean:

Received: from smtp104.sbc.mail.re2.yahoo.com ([68.142.229.101]) by
bay0-mc5-f16.bay0.hotmail.com with Microsoft SMTPSVC(6.0.3790.1830); Thu, 30
Mar 2006 10:58:00 -0800
Received: (qmail 93766 invoked from network); 30 Mar 2006 18:57:59 -0000
Received: from unknown (HELO ?172.16.2.3?) (foobar@ameritech.net@1.1.1.1 with
plain) by smtp104.sbc.mail.re2.yahoo.com with SMTP; 30 Mar 2006 18:57:59 -0000
User-Agent: Thunderbird 1.5 (Windows/20051201)

I've changed my internal and external IPs and ameritech.net login to protect
the innocent. As you can see, the "bad" HELO is buried down in the message.
Hotmail doesn't seem to trust SBC's authenticated SMTP server enough to just
accept it.

And so, if you have Hotmail, there is a chance someone out there is trying to
email you and you simply aren't seeing it. I'd suggest emailing
bill@microsoft.com and let him know. Just don't do it from Thunderbird.

Reproducible: Sometimes

Steps to Reproduce:
1. Send email to hotmail
2. Likely the email was never received and never even reached the "Junk Folder"
of the recipient
Actual Results:  
Email isn't received

Expected Results:  
The user would get the email.

Please fix this problem!
Comment 144 Philip Prindeville 2009-01-24 00:21:20 PST
(In reply to comment #143)
> [snip]

Thanks for going through those steps to collect that info, Chad.  That was helpful.

First, is your platform for Thunderbird Windows?

If so, what does "ipconfig /all" tell you is your hostname?  And is your DHCP client requesting option 12 or option 81?

Second, I was surprised to read that this was actually going through an intermediate relay (SBC).

Indeed, Hotmail should not be second-guessing SBC's own ability to validate email (ham).  Unless there was a Received: line that looked 100% like the known fingerprint of a faked header, it should accept the message as coming from a known/valid source.

I just looked at what Linux FC9 does, and indeed if your hostname is set as FQDN, TB will indeed send that instead of the IP address in the HELO.

Note You need to log in before you can comment on or make changes to this bug.