1414607 - wrong hostname is passed to the proxy resolver. remove realhostname?

Status: RESOLVED DUPLICATE of bug 1483485

(MailNews Core :: Networking, defect)

Description

[Ilya Basin]

User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0
Build ID: 20171024165158

Steps to reproduce:

After migration from gmail to internal Exchange server I changed the IMAP hostname in the account settings.

Gmail is accessible via our socks proxy and Exchange is accessible directly.

The Exchange hostname is in the exceptions list.

Looks like in the dialog I can change "realhostname", but the original "hostname" is passed to the proxy resolver.

mail.server.server5.hostname
mail.server.server5.realhostname



Actual results:

Thunderbird was trying to connect to the Exchange server using the proxy


Expected results:

Thunderbird should connect directly, because the host in the exceptions list.

Comment 1

[Jorg K (CEST = GMT+2)]

Aceman, do you know what "realhostname" and "hostname" are about? I already couldn't work out what they do when looking at bug 1400359.

Comment 2

[Magnus Melin [:mkmelin]]

We have both realhostname and hostname so that you could change the hostname and keep the same mail/news you had in the original place, without redownload and stuff. For opening connections you should always use realhostname.

Comment 3

[Jorg K (CEST = GMT+2)]

Quoting bug 1400359 comment #16:
Summarising: nsMsgProtocol::OpenNetworkSocketWithInfo() is called with the real name [1], but in nsPop3Protocol::LoadUrlInternal() when logging and for calling net_pop3_load_state(), it uses server->GetHostName(hostName) which is the "plain" name [2].

[1] https://dxr.mozilla.org/comm-central/rev/952762964f408e33d060dd1110cac0626ae5fa51/mailnews/local/src/nsPop3Protocol.cpp#579
[2] https://dxr.mozilla.org/comm-central/rev/952762964f408e33d060dd1110cac0626ae5fa51/mailnews/local/src/nsPop3Protocol.cpp#1135

Comment 4

[Jesper Hansen]

I'm reading IMAP not POP3?

I was having my eyes on nsProtocolProxyService::CanUseProxy(nsIURI *aURI, int32_t defaultPort) [1]

[1] https://dxr.mozilla.org/comm-esr52/source/mozilla/netwerk/base/nsProtocolProxyService.cpp#692

Comment 5

[Jorg K (CEST = GMT+2)]

Oops, I'm on the wrong boat, sorry. IMAP, indeed. Looking through nsImapProtocol.cpp, there are various calls to getting a host name. We'd have to check them all to see where this goes wrong.

Comment 6

[Jorg K (CEST = GMT+2)]

Gene, this could do with some clean-up. Can I get you interested?

Comment 7

[gene smith]

Been a while since I worked with proxy settings to access internet. I assume the reporter went to the advanced connection settings and selected "No Proxy" to access local exchange imap server?. I assume he means that after he did that and changed the server url that tb kept using the proxy? (tb restart may be needed.)
With "No Proxy" the exception list (No proxy for:) is gray. Is this the "exception list" he refers to? Did he manually change host name prefs (server url) he mentions with config editor or did he change it on the account setup page? The "steps to reproduce" are a bit unclear to me.

Comment 8

[Jorg K (CEST = GMT+2)]

Hmm, yes, I can't work out either what we need to do to reproduce it. Would the reporter use FoxyProxy? Jesper, in comment #4 you pointed to some Mozilla core code. If that's wrong, please file a bug in Core::Netwerk (no typo).

Comment 9

[gene smith]

Reading it again, I see he did change the server name with the account mgr. Also, I guess he wants to keep using the proxy but just not for the exchange server and he has it listed in the "No proxy for" list. Is he entering an ip address there or a url? Or maybe a known bug in the mozilla "netwerk" stuff?

Comment 10

[Ilya Basin]

I was entering the exact hostname in the "No proxy for" list.

Comment 11

[Ilya Basin]

"Manual proxy configuration" is selected and the "SOCKS Host" and "Port" are filled. SOCKS v5 is selected.

Comment 12

[gene smith]

(In reply to Ilya Basin from comment #10)
> I was entering the exact hostname in the "No proxy for" list.

So the "exact hostname" resolves to the ip address of the exchange server you are trying to access? If it's a static address you might try putting in the ip address instead, just as a test.

Comment 13

[gene smith]

Also, just to be sure, the list is comma separated like this:  localhost, 127.0.0.1, imap.my-exchange.lan
?

And I assume you still need to use the socks proxy for other accounts?

Comment 14

[Ilya Basin]

- Just tried to put the ip address to the "No proxy for" list. This haven't changed a thing
- Nevertheless, Thunderbird should not try to resolve host names before checking for proxy exceptions
- With SOCKS v5 Thunderbird should not resolve host names at all unless network.proxy.socks_remote_dns is set to false (mine is set to true)

Gene you're obviously digging in the wrong direction.

It is verty easy to reproduce the problem even without a real proxy server or 
> FoxyProxy
(don't know what that is)

Just set some invalid socks server, like 127.0.0.1:1234, change the IMAP host name to some other name and add this new name to both your /etc/hosts and proxy exception list. Thunderbird will ignore the exception and will try to use the proxy and will fail to connect.

Comment 15

[gene smith]

(In reply to Ilya Basin from comment #14)
> 
> Gene you're obviously digging in the wrong direction.
> 
> It is verty easy to reproduce the problem even without a real proxy server
> or 
> > FoxyProxy
> (don't know what that is)
> 
> Just set some invalid socks server, like 127.0.0.1:1234, change the IMAP
> host name to some other name and add this new name to both your /etc/hosts
> and proxy exception list. Thunderbird will ignore the exception and will try
> to use the proxy and will fail to connect.

I am going to try something like that but not at a place I can do that right now. Thanks for the suggestion. 
Also, I found a similar report in bug 1263295 but it was never confirmed (but not specifically using socks).

Comment 16

[gene smith]

One more shot in the dark: I don't know what your server url is, but I see references to using the exclusion sting <local> if it doesn't contain dots (a locally accessible hostname). I.e., this might work (??):
localhost, 127.0.0.1, <local>

Comment 17

[gene smith]

Didn't see any benefit to using the <local> string...

I went ahead and set up my own socks5 proxy server and I think I see almost the opposite of what you report. If I have anything at all in the exclusion list (e.g., localhost, mozilla.com, 127.0.0.1 etc) it disables the proxy usage and the connection to gmail server bypasses the proxy and goes direct.

I have the proxy server set up on another computer on the lan listening on port 1080. Data only goes to the proxy host and on to gmail port 993 when I have the "No Proxy for:" box empty with the Socks Host set to "hplt" (my listening socks server host) and port set to 1080 on the other host running tb. I verified this using wireshark on both hosts. I also have the checkbox "Proxy DNS when using SOCKS v5" checked.

The line number referenced in comment 4 above has changed quite a bit. So maybe the mozilla code that handles proxy has changed since you reported the problem. The new version, I think v60, may change what you see. I am running a dev version that I built that is close to the v6X trunk version, which may explain why you can't seem to bypass the proxy while I see it bypassed unless I clear the exclusion list.

Can you explain again your network setup? Is your proxy on your lan in a stand-alone host? What exact items do you have in the "No proxy for" list, etc? Also, how are you verifying that the connection to your exchange server is going through the proxy and not a direct connection?

Comment 18

[Ilya Basin]

> Is your proxy on your lan in a stand-alone host?
Yes

> What exact items do you have in the "No proxy for"
localhost, 127.0.0.1, secureimap.acme.com

> how are you verifying that the connection to your exchange server is going through the proxy and not a direct connection?
The IMAP server is only accessible through the VPN connection established from my workstation. Our proxy has no access to it.

Comment 19

[Ilya Basin]

> If I have anything at all in the exclusion list ... it disables the proxy usage
Did you *change* the server in the existing account or did you create a new account? This is important.

Comment 20

[gene smith]

(In reply to Ilya Basin from comment #19)
> > If I have anything at all in the exclusion list ... it disables the proxy usage
> Did you *change* the server in the existing account or did you create a new
> account? This is important.

Yes, it has changed. When I first set up the gmail account years ago the server name was imap.gmail.com. Then recently, for testing purposes, I changed over to using a local "stunnel" server and I changed the server name to localhost (stunnel handled the tls and connected to imap.gmail.com, similar to socks). Now while working with you on this bug I changed it back to imap.gmail.com with no stunnel and using the new socks server. All done via normal account mgr, not config editor.

Anyhow, yesterday when I checked the hostname and realhostname with config editor, they differed. One was imap.gmail.com and the other was localhost. Don't remember which was which. I set them so both were imap.gmail.com and saw no difference in tb behavior I described in comment 17.

Have you tried setting them both to secureimap.acme.com with config editor and restart tb?

Also, I asked before I think, do you have other accounts in tb that are using the socks proxy OK with your current configuration?

Comment 21

[Ilya Basin]

> Have you tried setting them both to secureimap.acme.com with config editor and restart tb?
Yes and that helped as I wrote in the ticket description.

> do you have other accounts in tb that are using the socks proxy OK with your current configuration?
Yes, I have a gmail account with imap.gmail.com that works via proxy and the other one with secureimap.acme.com that should be direct.

> I set them so both were imap.gmail.com
And now set it back to localhost via account mgr.

Comment 22

[gene smith]

(In reply to Ilya Basin from comment #21)
> > Have you tried setting them both to secureimap.acme.com with config editor and restart tb?
> Yes and that helped as I wrote in the ticket description.

You may have implied that in comment 0 but I don't see where you explicitly said you worked around the issue by setting hostname with config editor to, I assume, secureimap.acme.com. So let me ask one more time just be sure; when you changed
mail.server.server5.hostname from the old original value, something like gmail.com, to the correct secureimap.acme.com that it correctly stopped the socks proxy usage for the exchange account like you want?

Anyhow, sorry for my confusion and I was wrong about there being a problem with the proxy code. It does work OK when the hostname == realhostname and the name is placed in the exclusion list. (I had added some temporary debug code that broke it.)

So the *real* problem is just why does the proxy code use the hostname and not the realhostname. Seem like passing it realhostname should fix it.

(In reply to Magnus Melin from comment #2)
> We have both realhostname and hostname so that you could change the hostname
> and keep the same mail/news you had in the original place, without
> redownload and stuff. For opening connections you should always use
> realhostname.

Also, not sure I understand this rationale for why there is a hostname and realhostname, but I will look closer.

Comment 23

[gene smith]

Attachment: GetHostName.diff

OK, I see that when a new account is created the server url entered is stored in mail.server.serverX.hostname and there is no mail.server.serverX.realhostname yet, i.e., it is empty. The original hostname value is also used as the directory name where the mail files are stored, and the directory name is stored in another variable that does not change if the server url is changed on the server settings page. When the server url is changed, the new url is save in mail.server.serverX.realhostname, now not empty. As the reporter found, you can change the stored original hostname to be the same as the new realhostname and it solves the proxy problem and does not affect anything else adversely (e.g., stored emails in the directory having the original hostname are still OK).

As I mention in comment 22, the url that is compared against the proxy exclusion strings is the value stored in mail.server.serverX.hostname. It is not possible to just pass in instead the correct "realhostname" since the string is extracted from a URI parameter local to the mozilla framework and I don't see a direct way to change that in the mailnews code.

However, in the mailnews code, there are two function GetHostName() and GetRealHostName() that are the only access methods to hostname and realhostname pref variables. The URI in the mozilla code is ultimately derived from GetHostName() that, assuming the user doesn't change it with config editor, provides the originally entered server URL. The attached diff proposes that GetHostName() first check if there exists a realhostname and return it if it exists; otherwise it just returns hostname. This automates the action of going into config editor and changing hostname to equal realhostname. 

With this change, I see the the realhostname is now extracted from the URI in the mozilla proxy code and it solves the problem. I haven't seen any breakage but I could be missing something. I haven't tested this with pop3 or news but I assume the same concepts apply?

Comment 24

[Jorg K (CEST = GMT+2)]

Comment on attachment 9005110 [details] [diff] [review]
GetHostName.diff

Review of attachment 9005110 [details] [diff] [review]:
-----------------------------------------------------------------

I've always been puzzled by the use of two hostnames in the system, so Magnus is possibly the better person to ask, see comment #2.

The code could be written more elegantly without the duplication, so:
const char *hostNameToUse;
if (aResult.IsEmpty())
  hostNameToUse = "hostname";
else
  hostNameToUse = "realhostname";

Otherwise this approach looks reasonable, use realhostname if set, otherwise hostname. Are there cases where realhostname isn't set?

And how about bug 1429040, similar issue?

::: mailnews/base/util/nsMsgIncomingServer.cpp
@@ +1341,5 @@
> +  NS_ENSURE_SUCCESS(rv, rv);
> +
> +  if (aResult.IsEmpty())
> +  {
> +  {

Something funny here.

Comment 25

[Ilya Basin]

I wonder what does GetRealHostName() return when
> there is no mail.server.serverX.realhostname yet

Comment 26

[gene smith]

(In reply to Ilya Basin from comment #25)
> I wonder what does GetRealHostName() return when
> there is no mail.server.serverX.realhostname yet
 
If realhostname is not yet set, GetRealHostName() returns hostname. 

Actually, all I did was duplicate the functionality of GetRealHostName() in GetHostName(). 
(In reply to Jorg K (GMT+2) from comment #24)
 
> I've always been puzzled by the use of two hostnames in the system, so
> Magnus is possibly the better person to ask, see comment #2.

I also requested Magnus for feedback.

> 
> The code could be written more elegantly without the duplication, so:
> const char *hostNameToUse;
> if (aResult.IsEmpty())
>   hostNameToUse = "hostname";
> else
>   hostNameToUse = "realhostname";
> 
> Otherwise this approach looks reasonable, use realhostname if set, otherwise
> hostname. Are there cases where realhostname isn't set?

realhostname only gets defined and set when you change the "server name" on the server config page, I think. Therefore, for most typical accounts it is probably blank/undefined.

> 
> And how about bug 1429040, similar issue?

Sounds like the same thing.

> 
> ::: mailnews/base/util/nsMsgIncomingServer.cpp
> @@ +1341,5 @@
> > +  NS_ENSURE_SUCCESS(rv, rv);
> > +
> > +  if (aResult.IsEmpty())
> > +  {
> > +  {
> 
> Something funny here.

Not sure what happened, looks like it shouldn't compile! I will check when I am on my dev system.

Comment 27

[gene smith]

I did notice one issue caused by my proposed change, but it probably wouldn't affect a typical user. On my test profile for development I have multiple accounts running, but each of them I have changed to connect to localhost at a unique port that my stunnel daemon is listening on. When I look at the imap log, instead of seeing the original server name, e.g., imap.gmail.com, I now see localhost for everything. So this makes it difficult to tell which server/account the various log entries apply to. However, I could work around this by changing each "localhost" entry on the server config page to something unique, like "gmail-localhost" and mapping gmail-localhost to 127.0.0.1 in the /etc/hosts file. (In this example, reallocalhost would then change from "localhost" to "gmail-localhost".) So in the log I should see gmail-localhost for the gmail specific lines.

Comment 28

[gene smith]

(In reply to Jorg K (GMT+2) from comment #24)
> 
 > > +  if (aResult.IsEmpty())
> > +  {
> > +  {
> 
> Something funny here.
 
The extra { isn't in the changed file. The attached diff was cut and pasted from a larger "hg diff" so must be a user error. Running "hg diff" again I don't see the extra { so just ignore it. Sorry.

Comment 29

[Jesper Hansen]

(In reply to Jorg K (GMT+2) from comment #8)
> Jesper, in comment #4 you pointed to some Mozilla core code. If that's wrong, please file a bug in Core::Netwerk (no typo).

I don't recall what this was about

Comment 30

[Magnus Melin [:mkmelin]]

Comment on attachment 9005110 [details] [diff] [review]
GetHostName.diff

Review of attachment 9005110 [details] [diff] [review]:
-----------------------------------------------------------------

Wish we didn't have the realHostname at all :/
It's always messing things up. Very confusing.

I imagine this won't work though. If it does, what's the point of having the realHostname?

Comment 31

[Jorg K (CEST = GMT+2)]

(In reply to Magnus Melin from comment #30)
> I imagine this won't work though.
Please share your imagination, under which circumstances wouldn't that work? If I understand it correctly, the approach is to use realhostname if set, otherwise hostname. What do you suggest to do instead?

Comment 32

[Magnus Melin [:mkmelin]]

Somewhere in the code where we change the hostname, or after such a change? The point was if that if this works, why wouldn't we be able to get rid of realhostname altogether?

Comment 33

[gene smith]

(In reply to Magnus Melin from comment #32)
> Somewhere in the code where we change the hostname, or after such a change?
> The point was if that if this works, why wouldn't we be able to get rid of
> realhostname altogether?

I kind of thought this too but haven't yet checked completely that both "host names" are not needed.

Comment 34

[Jorg K (CEST = GMT+2)]

Those two hostnames are causing more trouble, we should move some solution forward. See for example bug 1483485.

Comment 35

[Magnus Melin [:mkmelin]]

Yes it would be great to remove the "realhostname" since yes it's always causing confusion. But then we need to figure out what the replacement is.

Comment 36

[Steven Singer]

I tripped over a similar issue several years ago in bug 381708, comment 3 has the clearest explanation. Although the way in which things went wrong was different, the underlying cause was the same: the system does a proxy lookup on hostname before doing a network connection on realhostname. In my case, the proxy lookup on the wrong hostname caused a delay, but the proxy configuration coincidentally returned a value which worked for the network connection to realhostname. If my skimming of this bug is correct, the problem here is that the proxy lookup on hostname returns information which is not correct for a network connection to realhostname.

I agree with previous comments that the hostname/realhostname distinction is confusing. I can easily imagine how the code evolved into this state and the separation does serve a real purpose. However, the net result is that there's this juicy API containing the words "HostName" which shouts out to any users of the code as "this is the name of the host for the purpose you're currently thinking of".

In fact there are two purposes: a "stable" name that will be constant forever (and is therefore safe to use as a reference, for example, in an IMAP URL to to specify a target account or folder); and the "network" name to be used on the network layer which may change whenever a corporate IT department needs to commission a new server and/or otherwise change its name (including changing the domain). The use of the hostname to specify the directory name for caching has already been separated out into another variable.

This is not just a corporate network issue. When one of the domain name providers I use outsourced their email to Office365 this left my preferences set up as (companyname is the name of the domain name provider):

mail.server.server4.directory                   C:\Users\…\ImapMail\imap.companyname.co.uk
mail.server.server4.directory-rel               [ProfD]ImapMail/imap.companyname.co.uk
mail.server.server4.hostname                    imap.companyname.co.uk
mail.server.server4.realhostname                outlook.office365.com
mail.server.server4.spamActionTargetAccount     imap://[e-mail]
mail.server.server4.spamActionTargetFolder      imap://[e-mail]/Junk Email
mail.identity.id5.archive_folder                imap://[e-mail]/Archives
mail.identity.id5.draft_folder                  imap://[e-mail]/Drafts
mail.identity.id5.fcc_folder                    imap://[e-mail]/Sent Items
mail.identity.id5.stationery_folder             imap://[e-mail]/Templates

Using realhostname for the first purpose would cause serious problems as it will break IMAP URLs stored elsewhere in preferences (remember, folder references for one ID can refer to IMAP folders for different accounts).

Actually, this begs an interesting question. IMAP URLs always refer to the original name. So, if we (as I believe would be correct) switch to passing realhostname to the proxy configuration, the hostname passed to the proxy configuration will not be the hostname from the URL. I don't know if this is likely to lead to confusion. Is this a unique case? Are there other URL schemes where the hostname use for network connections is not the hostname used in the URL?

I don't know enough about the internals of the Mozilla to make concrete suggestions. Whenever I've come across this sort of API design problem in the past, I've updated the API so every caller has to specify explicitly which variant they want and I've retired the bare name as it's an accident waiting to happen.

So, in this case, if this were a codebase I had complete control over, I'd make hostname and realhostname private to the class maintaining them and expose public APIs that returned stable and network names (where GetStableHostName always returns hostname whereas GetNetworkHostName returns realhostname if set, otherwise it returns hostname). This would then allow the internals to be refactored later.

A review of the codebase might let us know whether hostname is used for anything other than URLs. If not then we could mark URLs as the special case (and make the two APIs GetUrlHostName and GetNetworkHostName; I'd still retire the bare name).

I appreciate this may be a change that's too disruptive for a mature codebase.

There might be a solution that makes the IMAP URLs the place where the aliasing occurs but I can't see a neat solution that doesn't also risk codebase disruption.

Comment 37

[Ray Satiro]

Attachment: Capture.PNG

I had this problem today. I added a gmail account several days ago in Thunderbird 68.4.2 (32-bit) which for some reason showed the hostname of .gmail.com and I had to change it to imap.gmail.com. I ended up with:

mail.server.serverN.hostname = .gmail.com
mail.server.serverN.realhostname = imap.gmail.com

I needed to access gmail without the proxy so I put imap.gmail.com on the proxy list. I tried 'Get Messages' and saw could not connect to .gmail.com which made no sense to me (at the time I didn't know about the settings). By searching for gmail in the config editor I was able to track down the problem.