73293 - ftp via proxy broken in newcache

Status: VERIFIED FIXED

(Core :: Networking: HTTP, defect)

Description

[Bradley Baetz]

To reproduce:
1. Set up an ftp proxy (I'm using squid)
2. Go to ftp://ftp.mcgill.ca
3. click on "pub"
4. click on "README"
5. Click back

Expected results:
ftp://ftp.mcgill.ca/pub is shown

Actual results:
The readme file is still shown, even though the urlbar is updated. It is
impossible to visit any other proxied ftp site without deleting the NewCache
directory.

It doesn't matter which directory/file is chosen - once you stop going forward,
it doesn't work any more. Shift-reload doesn't help either, and neither does
typing an ftp url in the url bar.

I first noticed this with gopher - I suspect that the dummy URI in
nsFtpProtocolHandler::NewChannel is confusing something.

Comment 1

[Darin Fisher]

not sure if there is a duplicate bug report opened elsewhere, but this is
definitely a known problem related to the VALIDATE_ONCE_PER_SESSION load
attribute.

Comment 2

[Tom Mraz]

Hmmm, I see a slightly different behavior (Linux build 2001032521). I can
navigate through ftp directories, but when I click back in the mozilla there is
loaded some completely messed up document which contains parts of other
previously loaded documents.

I use squid proxy too.

Comment 3

[Tom Mraz]

Attachment: Screenshot of messed up display after clicking back button

Comment 4

[neeti]

-->gordon

Comment 5

[Bradley Baetz]

[e-mail]: Thats another bug, which is apparently known, although I
couldn't find a bug on it.

OK, despite what I said to darin last night on IRC, this is not a
revalidate_once bug - I can get this behaviour when set to revalidate_always,
its just a tiny bit harder to trigger:

Clear your mozilla cache (by removing the NewCache dir)
Using ns4, load ftp://ftp.mozilla.org/ and ftp://ftp.mozilla.org/bin/ (so that
the proxy has it stored)
Start mozilla
Go to ftp://ftp.mozilla.org/
Click on bin/
Observe that the pages are the same.

Note that about:cache has no ftp entry, but http://test.com does. I'll attach an
nspr.log file for http and ftp - all the cache entries refer to
http://test.com/. This log file shows mozilla sending an If-Modified-Since
header for ftp://ftp.mozilla.org/bin, even though it has never been fetched
before. The proxy still has this url cached, so it sends back a 304 response,
and so mozilla shows the old page.

The htp://test.com thing is bug 58872. After doing this, visiting
http://test.com/ gives you the ftp dir listing, unless you shift-reload.

The cache lookup should probably be keying off the overrideRequestSpec instead.

The attached log files are formed by doing ./mozilla ftp://ftp.mozilla.org/ then
clicking on the bin link. (Clicking on the bin link again tries to load
ftp://ftp.mozilla.org/bin/bin which fails)

Comment 6

[Bradley Baetz]

Attachment: nspr.log

Comment 7

[gordon]

I'm a bit confused since ftp isn't using the new cache yet.  Is the proxy using 
HTTP?

Comment 8

[Doug Turner (:dougt)]

ftp proxies simply forward to http.  Note, that there are a bunch of ftp proxy 
bug scheduled for 0.9.1.

Comment 9

[Bradley Baetz]

> Is the proxy using HTTP?

Yep - its probably an HTTP bug, using the wrong uri. This should be dealt with
before 0.9.1 though - it makes using ftp through a proxy almost impossible
(unless you shift-reload each page, and you need the patch for bug 73447 for
that to work)

Comment 10

[Darin Fisher]

I suspect that this is a dupe of two problems:

1) FTP does not transfer its load attributes to the HTTP channel (bug 73447)

2) HTTP does not handle VALIDATE_ONCE_PER_SESSION properly (bug 73099)

Comment 11

[Bradley Baetz]

darin: No - I retested both with VALIDATE_ALWAYS (or whatever its called), and
with my patch for 73447.

Comment 12

[Darin Fisher]

ok... if using validate always didn't help, then this is not a dupe of 73099.

Comment 13

[Tom Mraz]

Please, please fix this bug in Mozilla 0.9. It really makes the ftp through
proxy completely unusable.

Comment 14

[Darin Fisher]

it sounds to me as though the HTTP proxy channel is not seeing the same load
flags as the FTP channel.  i believe this is because the load group's default
load flags are never anything other than LOAD_NORMAL.

Comment 15

[Bradley Baetz]

Darin: when clicking back/forwards, the load attributes should be load_normal,
shouldn't they? The problem (from the logs) is that the "uri" http://test.com is
always the same, and so the cache thinks that it already has the result for that
uri. The ftp channel could force revalidate_always when talking to the proxy,
but thats a workarround, not a solution.

See bug 58872.

Comment 16

[Darin Fisher]

bbaetz: when clicking back/forward the load flags are to never validate, which
makes sense since it is just browsing through history.  but, i see your point
that this is likely due to HTTP caching everything to the same URL.  why should
HTTP be caching at all?  this seems wrong.  i'm working on a patch which will
add an INHIBIT_CACHING bit to the load flags... FTP (and other protocols that
proxy via HTTP) should really set this bit IMO.

Comment 17

[Bradley Baetz]

darin: Why shouldn't http be caching the response? It is connecting to an HTTP
proxy which is sending back a cachable response, and sends a 304 response (with
an Age header) back to an If-Modified-Since request. Theres very little point in
an organisation setting up a caching proxy if mozilla is going to force it to
always revalidate.

The proxy server can have huristics set up for how long to cache for - squid
uses a minimum 24 hours for both ftp and gopher, by default. (roughly)

My uni's servers (where I get my mail) are all down for upgrades until this
evening - find me on IRC if you want to discuss this :)

Comment 18

[Darin Fisher]

My point was not that the proxy server would not do caching, but rather that we
would not need to do caching on behalf of FTP or other protocols b/c the proxy
would most likely be doing it.

At any rate, I landed changes on Friday along with my patch for bug 75679 that
makes HTTP use the ProxyRequestURI for the cache key instead of the regular URI.
This may very well solve this problem, though I've not tested it. 

Comment 19

[Bradley Baetz]

I think I misunderstood what you were saying earlier.

I presume you meant:

"... but rather that we would not need to do caching on behalf of FTP or other
protocols b/c the proxy *channel* would most likely be doing it."

Anyway, I tested, and its fixed now, but I'll leave it open in case you want to
make those changes.

Comment 20

[Darin Fisher]

bbaetz: i really meant that the proxy server would be caching on behalf of
FTP and other protocols.  my initial feelings on the matter were that FTP
would take care of its own caching.  but, with the simple change of making
HTTP use the OverrideRequestURI for the cache key, we can have HTTP cache
the FTP requests.  i'm okay with this... this should probably be the default
behavior.  though, when dougt starts adding caching of FTP directory lists,
he may want to step back and consider whether or not that's necessary in the
proxy case... it may still be.

marking FIXED per bbaetz comments that it's working.

Comment 21

[benc]

VERIFIED:
Mozilla 0.9 all plats.