129056 - Unable to connect to secure news server through application proxy firewall

Status: VERIFIED DUPLICATE of bug 56291

(MailNews Core :: Networking: NNTP, defect)

Description

[Ian Neal]

Using build ID 2002022603 on win32

Trying to connect to secnews.netscape.com as a secure server with an application
proxy firewall inbetween. If you try to subscribe to newsgroup it gives a
connection refused error message. Proxies are setup the same was as NS4.x which
works fine through application proxy firewall (Technologic Interscan). Socks is
blank for both but security points to firewall.

Comment 1

[[e-mail] (gone - use [e-mail] instead)]

Can you try this with a later build?  There were some socks fixes that landed
recently, thanks.

Comment 2

[Ian Neal]

Using Build ID 2002030604 on Win2K SP2

Problem still occurs, I don't have the socks field within proxy settings filled
in as we use an application proxy server plus under NS4.7x this wasn't filled in
and it worked fine connecting through to a secure newsserver.

Comment 3

[[e-mail] (gone - use [e-mail] instead)]

Is it one of these products?  If so, which one?
http://www.antivirus.com/products/internet_gateway.htm

Comment 4

[Ian Neal]

Argh, I'm always getting things confused. We use a Technologic Interceptor
firewall (http://www.esoft.com) in conjunction with a Trend Interscan Viruswall.
The only proxy setting that doesn't point to the viruswall is the SSL one which
goes directly to the firewall (unlike the other settings http, ftp and gopher
that go to the firewall via the viruswall).

Comment 5

[Ian Neal]

Just tried this again with BuildID 2002071208 on secnews.netscape.com port 563
and I still get a connection refused error message.

Comment 6

[Per Weisteen]

Hi, I've got the same problem. I'v run a tcpdump and there is no attempt to use
the application proxy. The connects are done directly to secnews.netscape.com on
port 563 regardless of any proxy setting in Mozilla.

Comment 7

[Henry E. Thorpe]

This should work, assuming that the code in mailnews/news/src/nsNNTPProtocol.cpp
is correct:

    nsCOMPtr<nsIProxyInfo> proxyInfo;

    rv = NS_ExamineForProxy("nntp", hostName.get(), port,
getter_AddRefs(proxyInfo));
    if (NS_FAILED(rv)) proxyInfo = nsnull;

    if (isSecure)
      rv = OpenNetworkSocketWithInfo(hostName.get(), port, "ssl", proxyInfo, ir);
    else
      rv = OpenNetworkSocketWithInfo(hostName.get(), port, nsnull, proxyInfo, ir);

Looks like if network.proxy.nttp and nntp_port are set in user.js, then the
proxy will be used for NNTP (not quite sure what this means...), and if
network.proxy.ssl and ssl_port are set, then for SNEWS connections, the HTTP
CONNECT tunnelling will be used.

However, I'm not sure if the backend code for this was ever written.  Did this
ever work in Mozilla?

Comment 8

[Jesse Glick]

I would like to perhaps broaden the scope of this bug to mention that I cannot
connect to an *insecure* NNTP server through an HTTP proxy, either.

From what I can tell, Mozilla supports proxying NNTP over SOCKS5. (This is just
what I have managed to pick up from reading various things.) Unfortunately, this
is useless for my company's firewall: we have both SOCKS5 *and* HTTP proxies,
but for whatever reason the SOCKS5 proxies are not configured to permit NNTP
traffic to pass. The HTTP proxies *do* though, using CONNECT tunneling. (I
gather this is unusual? since almost no one else seems to ask for it to be
supported?) NS Proxy Server, from what I can tell.

How I determined this to be true: using any NNTP client with no particular proxy
configuration - Mozilla by default, Pan, etc. - try running it with runsocks and
connecting to an external NNTP server. Does not work. (runsocks with the same
configuration does work with other apps such as CVS.) Now get transconnect
(http://transconnect.sf.net/, nice little utility) and configure it to use the
regular HTTP proxy on the usual port, and run the NNTP client with tconn.so in
the $LD_PRELOAD - works fine. (You have to put the NNTP server name in
/etc/resolv.conf so no DNS lookup is attempted, or have a version of
transconnect that proxies DNS - mine doesn't.)

Justification: I currently access external NNTP servers successfully using
transconnect and Mozilla, but it is cumbersome to set up: e.g. official version
of transconnect does not support port restrictions on its proxying, which is bad
because SMTP and IMAP tunneling may not be wanted at all, and HTTP and FTP
proxying already works fine in Mozilla, so a patched version is needed that only
tunnels NNTP; you have to make a custom Mozilla launch script to set up
$LD_PRELOAD; you have to use a separate config file rather than Mozilla
preferences; all NNTP servers must be listed in /etc/hosts to avoid DNS lookups;
etc. It is annoying, especially if you have never set it up before, and I would
very much like to see proper support directly in Mozilla.

Since transconnect - a really trivial program, a few hundred lines of C code -
manages to use CONNECT tunneling without problems, I assumed Mozilla had code to
do this too. After a lot of searching, I found nsNntpService.cpp and tried to
patch it (will attach attempt), using current 1.2 branch sources as a starting
point. nsIProtocolHandler says re. ALLOWS_PROXY_HTTP:

  This protocol handler can be proxied using a http proxy (e.g., http,
  ftp, etc.).  nsIIOService::newChannelFromURI will feed URIs from this
  protocol handler to the HTTP protocol handler instead.

which I understood to mean that if a proxy were set for this protocol, CONNECT
tunneling would be used for it.

Unfortunately it did not work. I have a proxy autoconfig file (which works fine
for HTTP) and tried using a patched Mozilla with this config. Attempting to
contact any NNTP server (e.g. to retrieve groups) did not succeed; no error
would be reported, but the connection would just hang and wait. (Using
transconnect, the same server's groups can be retrieved within a second; and I
still have the /etc/hosts entry so DNS is not relevant here.) I then tried
setting the proxy to be localhost and used netcat. Interesting results: when you
make an HTTP connection, Mozilla just sends a regular HTTP/1.1 request, but with
Proxy-Connection: Keep-Alive. Fine. When you try to make an NNTP connection,
port 8080 is connected to, but Mozilla doesn't send anything at all! It just
opens the socket and waits. It should be printing at least

"CONNECT nntp.external.com:119 HTTP/1.1\r\n\r\n"

Since it didn't do anything, my suspicion is that CONNECT tunneling is just not
written at all. Just a guess. Strange, since it is so trivial to implement.

Any clarification on whether this is supposed to work?

If you could ask Mozilla to tunnel NNTP through an HTTP proxy, implications
would be:

1. For users with no proxy config, none.

2. For users with no proxy config, a new option (I guess?). Proxies seem to be
stored per-protocol. You could add a new Preferences box for NNTP tunneling over
HTTP proxies (distinct from SOCKS4/5 tunneling). Or have some way to indicate
that for miscellaneous protocols (Chat falls into this category, I think - I use
transconnect for this too), if there is no standard proxy system built into the
protocol, permit the user to select: no proxy; HTTP proxy; or SOCKS4/5 proxy.

3. For users with a proxy autoconfig: might be a problem. If FindProxyForURL
were suddenly to start being called for NNTP, then users who do not want or need
to proxy NNTP would suddenly need to change their PAC's to specifically exclude
news: URLs. This might be considered a compatibility problem. (Though
technically I think it is right: if you are given "news://[e-mail]"
as a URL and you return "PROXY proxyserv:8080" then Mozilla IMHO should honor
this request. If you only want HTTP proxying, check the URL protocol in
Javascript.) To avoid the need for anyone to edit existing PACs, you could call
a different function if it exists, e.g. FindProxyForSpecialURL, used only for
miscellaneous protocols like NNTP.

Summary: I would like to see this issue broadened to "Cannot connect via
miscellaneous protocols (NNTP, ...) using HTTP proxy CONNECT tunneling". If I
misunderstood the previous comments in this issue, and this should be filed
elsewhere, please let me know where - I tried searching through Bugzilla,
mozilla.org development docs, ufaq.org, etc. and did not find any more
information about this topic.

http://wp.netscape.com/eng/mozilla/2.0/relnotes/demo/proxy-live.html

seems to imply that at least snews protocol is supposed to obey the PAC.

Comment 9

[Jesse Glick]

Attachment: Attempted patch

Did not work - tried to tunnel NNTP over any configured HTTP proxy. See
explanation in issue body.

Apply against current 1.2-branch sources.

Comment 10

[benc]

In short, SNEWS should (but might not) follow the preferences for "SSL proxy",
which would have been migrated from a 4xp profile as the "Security" setting.

If it doesn't work, that is a bug, and should carry the 4xp keyword.

As for NTTP over HTTP proxy, what you are really saying is "NNTP using the
CONNECT command over an "HTTP"/URL proxy". I don't know if that ever worked,
although there always seem to be a couple things that nobody ever bothered to
tell me about. "network.proxy.nntp" sounds like one of those. If there is
interest in this, it really should be a separate bug.

Comment 11

[Jesse Glick]

In response to [e-mail]:

> As for NTTP over HTTP proxy, what you are really saying is "NNTP using the
> CONNECT command over an "HTTP"/URL proxy".

Yes, exactly.

> I don't know if that ever worked

Neither do I (if by that you mean "in Communicator 4.x").

I am happy to file a separate issue for this if you think it is really an
independent matter (I don't understand the proxy internals enough to know). Let
me know and I will do so, or file it yourself and CC me.

My assumption as a user was that (1) Mozilla lets you specify HTTP proxies, (2)
the HTTP spec permits it to tunnel other protocols using CONNECT, (3) my proxy
server does support this, (4) my PAC requests use of the proxy server
unconditionally, therefore (5) NNTP should go over it. But if this is considered
weird usage (unlike proxying snews) and should be treated as a separate RFE,
that is fine too.

Comment 12

[Bradley Baetz (:bbaetz)]

nttp/ssl over CONNECT isn't supported.

Most proxies in fact are set up to do the other way arround - squid defaults to
only allowing a few ports, for example.

This is a dupe of bug 56291, and is Hard(TM) to fix.

The stuff mentioned in comment #7 is for socket level stuff only - we need to be
at a higher level for connect

*** This bug has been marked as a duplicate of 56291 ***

Comment 13

[[e-mail] (gone - use [e-mail] instead)]

verified dup (thanks, bbaetz!)