180490 - Proxy: add (insecure) NNTP using CONNECT (over HTTP proxy)

Status: RESOLVED DUPLICATE of bug 264981

(MailNews Core :: Networking: NNTP, enhancement)

Description

[Jesse Glick]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020827
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020827

Bug 129056 gives all the details. That was marked duplicate of Bug 56291. Both
of those bugs seem to be dealing with SNEWS. I am requesting specifically
support for proxying of insecure plain news, using an HTTP proxy.

Bug 44995 and Bug 89500 seem to deal with proxying using SOCKS. I presume this
works for NNTP (I have tried, but our SOCKS server does not appear to pass NNTP
traffic).

From what I can tell, there is not intended to be support for this currently in
the Mozilla codebase, so I am marking this an RFE; naturally please mark it a
bug if in fact it is supposed to work already. Please indicate whether a patch
for this would be accepted or whether the request seems so unusual that support
for it is not wanted in the master Mozilla sources.

I will try to be clear, since this request seems to keep on getting
misinterpreted and confused with either secure NTTP or SOCKS support.

I want to connect to some (insecure) NNTP servers. When I am on an open network,
I simply connect to them directly. news.netbeans.org primarily.

When I am on my company LAN, I cannot connect to such servers directly. NNTP
(port 119) traffic is blocked, even outgoing.

For some protocols, e.g. CVS (port 2401), I can connect to an external server
using a SOCKS5 proxy. However, for whatever reason, this does not work for NNTP
traffic.

There is an HTTP proxy (Netscape Proxy Server, I think). While direct HTTP
connections are blocked, they can go through the proxy just fine.

It turns out that NNTP connections will also pass through this (HTTP) proxy,
using the generic CONNECT command. In fact, as far as I can tell, this is the
only way to get an NNTP connection to an external server from inside my LAN at
all. Without it, I cannot read news.

Fortunately a tiny utility

http://transconnect.sourceforge.net/

can help. It is a clever tool that works much like the generic SOCKS5 "runsocks"
command: uses $LD_PRELOAD to interpose the C function connect(). After checking
the host etc., it connects to an HTTP proxy, sends the CONNECT command to it,
waits for a 200 OK header, then returns the open stream back to the application.

In fact using transconnect I can read news with Mozilla (I use 1.1 currently)
from within the LAN. Just as runsocks transparently makes apps work with a SOCKS
proxy, transconnect transparently makes them work with an HTTP proxy. In fact
transconnect will correctly pass HTTP, CVS, etc. if used with command-line tools.

However this is pretty cumbersome to set up - you have to set up environment
variables etc., with no GUI in Mozilla, nor support from Proxy Auto Configs. The
default version of the tool proxies all ports, which is undesirable since
Mozilla already proxies HTTP/HTTPS fine as is, and can report errors better
since Mozilla is in control of the whole process. Furthermore the DNS handling
is not great: you can either hardcode IP addresses of your news servers in
/etc/hosts, which is obviously a pain; or use a version of transconnect that
proxies DNS requests to some public DNS server. This is silly, just an artifact
of the fact that connect() takes an IP address and host resolution is done with
a different function; if the app (Mozilla) knows it is using a proxy, it can
simply CONNECT to the unresolved host name and rely on the proxy server to do a
proper cached name lookup (which it will).

I can't figure out if there is any support for CONNECT proxying in the Mozilla
code base already. Anyway, the implementation of the proxy protocol is dirt
simple; replace the following sequence

1. Resolve host name

2. Open socket to IP address and port

3. Perform protocol-specific communications

with the following

1. Resolve proxy host name

2. Open socket to proxy IP address and port

3. Send "CONNECT ${hostname}:${port} HTTP/1.1\r\n"

4. Send any proxy authorization information

5. Wait for a 200 response (deal with other error codes appropriately)

6. Perform protocol-specific communications

Note that steps 1, 2, 4, and 5 are essentially the same as for regular HTTP
proxying. So in principle it should not be a lot of code to do this.
transconnect does a reasonable job in less than a thousand lines of C source,
most of which deals with reading the config file.

Reproducible: Always

Steps to Reproduce:
1. Set up a proxy autoconfig file which returns e.g. "PROXY proxyhost:8080" for
every unrecognized IP.

2. Try to use news with an external insecure server.
Actual Results:  
Mozilla reports that it cannot connect to the server.

Expected Results:  
Connect to the remote server using CONNECT proxying.

(Note: see my comments in Bug 129056 re. possible backward-compatibility
problems with existing Proxy Auto Config files if "PROXY" is interpreted even
for NNTP protocol. Currently it is not interpreted for NNTP protocol as far as I
can tell. I do not know what happens if you return "SOCKS" for a host given by
NNTP, but I presume it works.)

Comment 1

[Jesse Glick]

Note: there might be other protocols which could benefit from the same generic
proxying capability, i.e. treat HTTP/CONNECT proxying as analogous to SOCKS5.
For example: chat, imap, smtp. I am reporting this for NNTP because that is the
only protocol for which I actually need it, not because that is the only
protocol with which it could be used.

Comment 2

[Bradley Baetz (:bbaetz)]

Yeah, the issue is that some people want to create a channel and tehn access it
as an nsinntpchannel, where what we'd have would be an nsihttpchannel for CONNECT

the ftp-over-http code only works because ftp requests over http really have
nothing to do with teh ftp protocol, so that code isn't run - we just send a GET
requiest for a different scheme. Thats not true with mailnews protocols

I'm sure this is a dupe, becuase I know darin and I discussed it when I did the
proxy rewrite last year.

Comment 3

[benc]

Let's keep this bug just about NNTP, -> NEW

In general that is true, although few people have asked for it. Ari Luotonen
often objected to the name "SSL Tunneling" or "SSL Proxy", because there was no
SSL involved, it is maintains a virtual data circuit. The method of proxy was
created because SSL needed it, not because only SSL can use it.

Comment 4

[(not reading, please use [e-mail] instead)]

sorry for the spam.  making bugzilla reflect reality as I'm not working on these bugs.  filter on FOOBARCHEESE to remove these in bulk.

Comment 5

[Serge Gautherie (:sgautherie)]

Filter on "Nobody_NScomTLD_20080620"

Comment 6

[benc]

clarified summary.