91292 - Support directory retrieval in ".tar, .gz, .Z" formats

Status: RESOLVED FIXED

(Core Graveyard :: Networking: FTP, defect)

Description

[robbe]

Many FTP servers will give you a whole directory tree bundled up in a nice
archive when you RETR the directory with an appended ".tar.gz" or similar.

CTAN relies heavily on this, so on the $URL you can e.g. see a link to 
  ftp://ftp.dante.de/tex-archive/macros/latex/contrib/supported/bosisi.tar.gz
This file does not exist on ftp.dante.de, but is generated on the fly from the
bosisi directory.

With 0.9.1, when I click on the ftp link, mozilla will tell me that the file
does not exist. The communication:

[...]
-> SIZE /tex-archive/macr/tex-archive/macros/latex/contrib/supported/bosisi.tar.gz
<- 550 /tex-archive/macros/latex/contrib/supported/bosisi.tar.gz: not a plain file.
-> CWD /tex-archive/macr/tex-archive/macros/latex/contrib/supported/bosisi.tar.gz
<- 550 /tex-archive/macros/latex/contrib/supported/bosisi.tar.gz: No such file
or directory.

Obviously not everything where SIZE fails can not be RETRieved.

Comment 1

[Doug Turner (:dougt)]

Agreed.  Feel like submitting a patch? ;-)

Comment 2

[Francisco León]

Hmmm.... have seen this feature on some ftp servers
However, is this mozilla's fault?
If the server says "just request the directory with an added .tar.gz" and you
do, the server should "understand that"
What could mozilla do to cause the server to not understand this request?

Comment 3

[robbe]

> However, is this mozilla's fault?

Mozilla assumes that because a ftp resource does not have definite size, it must
be a directory, or something that cannot be retrieved at all. That's obviously
wrong for this case, and could be wrong for others. I don't see anything in the
RFC that nurtures this assumption (in fact, I can't find SIZE at all in rfc959).

> Feel like submitting a patch? ;-)

Sure. See next posting.

Comment 4

[robbe]

Attachment: patch that WFM

Comment 5

[Doug Turner (:dougt)]

wow. thank you for the patch.  I will see that it gets in soon.

Gagan, please r= so that I can sr and check this in to the trunk.

Comment 6

[Francisco León]

if (mResponseCode == 550) // File unavailable (e.g., file not found, no access).
-        return FTP_S_CWD;
+        return FTP_S_RETR;

is that really safe? what happens when a file is REALLY not there and the server
does not support tar.gz compressing?

Comment 7

[Gagan]

these were my concerns too. Can we get bbaetz to help verify this? 

Comment 8

[Doug Turner (:dougt)]

if retr fails, then asks to CWD...  

Comment 9

[Francisco León]

It would be cool to get this "feature" as ns4 does not have it, i dont know
about IE, but i think it does not
We could make a test trunk to get this in and let robbe test it :)

Comment 10

[Gagan]

got it. r=gagan

Comment 11

[Doug Turner (:dougt)]

wait a second....

what happens if you want to view the site normally, and the server supports this
tar/gz format?!

Comment 12

[Francisco León]

Well, you are only supposed to "trigger" this "special feature" when manually
typing the filename with the added tar.gz

Comment 13

[Doug Turner (:dougt)]

ah!  I was worrying about this over the weekend but now I understand.  The file
does not exist until you request it.  np.

Comment 14

[Francisco León]

Ok
Who is doing the sr and checking?
This seems safe

Comment 15

[Doug Turner (:dougt)]

sr=me.  

Checking in nsFtpConnectionThread.cpp;
/cvsroot/mozilla/netwerk/protocol/ftp/src/nsFtpConnectionThread.cpp,v  <-- 
nsFtpConnectionThread.cpp
new revision: 1.181; previous revision: 1.180
done

Comment 16

[Doug Turner (:dougt)]

fix in.

Comment 17

[benc]

Do servers that support this on-the fly compression provide any identification
in their sessions?

If so, we might want to offer a "get this directory" change to the UI, maybe a
button or extra link...

-helpwanted b/c doug coded this.

Comment 18

[Doug Turner (:dougt)]

that would be a cool feature - can you file a seperate bug?

Comment 19

[Francisco León]

As far as i know servers only make comments about this feature in their welcome
messages so i dont think a UI option for this would be recommended, because
mozilla would have no way to know when the server supports it or not
Hmm... Unless you would like to store the entire welcome message to a temp space
on memory or disk and look for the string "this server supports .tar.gz
requests" or whatever the server message is.
But before that happens, how about fixing bug 90695 ?
Mozilla does NOT show welcome messages in ftp sites... We can't allow that!
I am increasing severity in bug 90695 to normal...

Comment 20

[robbe]

I've also seen servers that return some information in the error message when
you try
  RETR /some/directory
to the effect that "RETR /some/directory.tar" or similar will work.

But since this is not standardized in any way, I don't think we could detect
this in any reliable way. All you could do is detect it for some servers...

Comment 21

[benc]

forget my suggestion about the UI.

Thanks for the various bits of information on my question.
changed URL from:
http://www.ctan.org/tex-archive/help/Catalogue/entries/quotes.html

Did anyone verify this? I get the same 550 error as reported before, Mozilla
1.0RC1, Win98.

Comment 22

[benc]

I think I got this to work by going to the "/" dir-level, then requesting the
.tar.gz.

Is this how it should work?

Comment 23

[benc]

REOPEN:
Mozilla 1.1b, Win 98.
I'm clicking the URL link, and I get a 550. I've checked the dir to make sure it
exists as:

ftp://ftp.dante.de/tex-archive/macros/latex/contrib/supported/bosisi/

Comment 24

[Doug Turner (:dougt)]

we had this at some point, but it broken some ftp sites.

Comment 25

[Francisco León]

we should contact  [e-mail] the one who wrote the patch. he removed himself from this bug

Comment 26

[Francisco León]

never mind, the reporters are cc automatically. i'm so dumb

i hope that mail is still accurate

Comment 27

[robbe]

I hear ya...

First off, the URL was no longer valid. Corrected. You may want to repeat your
tests.

It WFM with the new URL, but I'm using 1.0.0 still. I would be interested in
results with the trunk. If the only problem was the obsolete path, we can close
this again.

Comment 28

[benc]

VERIFIED: wfm Mozilla 1.1, Mac OS X, Linux, Win32.
-> resolved/fixed (since there was a patch)

Comment 29

[Tracy Walker [:tracy]]

mass remove verifyme requests greater than 4 months old