Closed Bug 55690 Opened 24 years ago Closed 21 years ago

Spool file should be moved once the user picks a filename (saving a file creates a file before location is chosen)

Categories

(Core Graveyard :: File Handling, defect, P2)

Tracking

(Not tracked)

VERIFIED FIXED
mozilla1.7alpha

People

(Reporter: devsin, Assigned: Biesinger)

References

(Blocks 4 open bugs)

Details

(Keywords: arch, Whiteboard: se-radar[adt2] (py8ieh:248) read comment 238, 280 before commenting)

Attachments

(1 file, 3 obsolete files)

Mozilla M18 2000100608, Mac OS 9.0.4 -

1) Try to open a link to an unsupported file type (.tgz, .gz, .zip, etc.)
2) You get a dialog asking what you want to do (save or open)
3) Choose to save the file to disk
4) Navigate to your default downloads directory

A file has already been created (file.ext). If you choose to save the file in
that folder, you get a dialog saying a file already exists in that location and
asking if you want to replace it. If you replace the file, the download proceeds
as expected, but this is unnecessary confusion for the user. Mozilla should not
create any type of file until the user selects a location for said file.
This has been 75% resolved in recent times - temporary files that are created
are given wonderfully non-descriptive titles ("x4as73ggp.ext") so you no longer
get the "Do you want to replace this file?" dialog. My only concern is that
these temp files are created in the user's downloads directory instead of some
temporary folder in the Mozilla data folder or Preferences folder. But the way
downloads work (files saved to disk keep the obscure names until downloading is
complete) this may not be an issue. 
OS: All
See bug 60203 for the "wonderfully non-descriptive titles" given to these
"temporary" files.
Moving to networking per bug 60203
Assignee: don → gagan
Component: XP Apps → Networking
QA Contact: sairuh → tever
Also seeing this on WinNT and Linux [netscape 6.0 official release].
Marking platform = ALL.
Hardware: Macintosh → All
->dougt
Assignee: gagan → dougt
Blocks: 62352
voting for nsbeta1. this is a bug (not a feature, imho) that causes us to need to 
do other crazy things, such as salt filenames.

the solution is to fix this bug, not bend over backwards and confuse the user.
Blocks: 60203
Severity: normal → critical
Keywords: nsbeta1
adjusting milestone
Target Milestone: --- → mozilla0.9
bill, I think that this is your bug.
Assignee: dougt → law
Not going to happen for mozilla0.9, at any rate, since this was reassigned, the
target milestone should be reset.
Target Milestone: mozilla0.9 → ---
The original bug description says "Mozilla should not create any type of file
until the user selects a location for said file."

Unfortunately, that is preceded by description of a scenario that no longer occurs.

I'm left wondering what exactly the bug is.  If it is (was) that saving
sometimes results in a "file already exists" scenario, then that is fixed (by
way of the "salting" of file names).  If the problem is that we download before
the user tells us where, then that's still a bug.

So which is it?  Or is it something else?  BTW, there's not much chance we're
going to change the current behavior.
The bug is that we still start the download *before* the user specifies a 
location to save the file. We do the background spooling thing to a temporary 
file, which then forces us to salt names and stuff because we're downloading 
before the user has accepted the download. We should just not start downloading 
until the user has specified a save location, then not salt the filename.
A couple of reasons why you shouldn't start downloading a file before the user
selects the filename and hits OK:

* It is totally unexpected behavior.  Sure it sounds like a slick PR feature,
but it could potentially be very annoying on a slow (PPP) link.  It also makes
the "Cancel" button in the filename dialog essentially worthless (the file may
have already really been downloaded, like it or not), and the "elapsed time" and
progress indicator in the download dialog box are (sometimes) worthless as well.

* It pretty much requires mangling the filename, which is also fairly
unexpected.  This clobbers the ability (on Linux, at least) to either remotely
monitor the progress of large downloads (since you don't know the filename) or
'mv' the file to a new location while it is downloading (which I used to do
rather frequently with Netscape).
The current behavior is due to a couple of reasons:

1. By the time we've determined that the content can't be displayed in the
browser/mail/etc., we've already downloaded some/all of it (i.e., it's coming
across the wire).  If we don't start saving it to disk (and thereby accept the
data coming in), then we must re-establish the connnection, thereby wasting the
initial connection and/or transfer.

2. In some cases (imap attachments?), it is not possible to re-establish the
connection and get back to the data so we have to do something with it immediately.

Changing the current behavior is unlikely to fit within the resource limits we
have for pre-mozilla1.0.

I'm going to triage this and "minus" it, as we say.
Keywords: nsbeta1nsbeta1-
Why can't we hold the connection open until the user dismisses the dialog? I'm 
sure most servers wouldn't timeout for that period. If they do, then we need a 
way to re-establish the connection, which surely can't be that hard. Just fire 
off the URL again.
re: why not hold connection open

At one point, it was impossible to get Necko to do this (either that, or
impossible to figure out how to get it to do this).  I should be more precise,
though: the problem was in handing off the request from the machinery under the
browser window to another set of observers/listeners/etc.  That's been solved to
a considerable extent (as a byproduct of code related to streaming into the
temporary file) so it may no longer be as difficult a hurdle as it was previously.

re: re-establishing the connection

I'm not sure how hard that would be.  We're not real good at dealing with that
now (in other contexts), from what I can observe.

That still leaves the issue of the content that we can't re-request.  Sorry, but
I'm not familiar with the details of that.
Marking nsbeta1- bugs as future to get off the radar
Target Milestone: --- → Future
This is so catfood, but it's latered. Anyone?
if you want to delay downloading until a user has selected a download location
then you should really delay starting the download.  i don't think we want to 
try to solve the problem of re-establishing broken connections.

it seems that this behavior was put in as an optimization, to reduce the time
the user is waiting for content to download.  but, some users don't like the
fact that we do this.  sounds like justification enough for a preference.
or not.  

Why even bother downloading until a file location is know?  How much 
download time is lost on the average while the user searches for a place to save 
it?  2-5 seconds?  Lets keep it simple.  Ask the users where they want it and 
download it there.  Don't pretend to do something smart.
exactly!! *pinkerton hugs dougt*
Unfuturing because concensus is that this sucks too badly to ignore.
Target Milestone: Future → ---
sounds good to me... any idea how difficult this fix will be?  we probably do
a lot of extra work just to be able to support "Saving a file creates a file
before location is chosen."
Transferring to mscott (I fended these people off as long as I could, Scott :-)
Assignee: law → mscott
marking milestone as 0.9 so that it will appear on mscott's not so short
shortlist. 
Target Milestone: --- → mozilla0.9
moving to 0.9.1
Target Milestone: mozilla0.9 → mozilla0.9.1
*** Bug 74008 has been marked as a duplicate of this bug. ***
Additional Comments From timeless@mac.com 2001-04-01 11:58 in Bug 74008

Um silly question... is it possible for a download session to change where it 
writes while it's downloading?

eg. we d/l 12% to a temp file. the user tells us where to write (and it's on 
another drive), we copy that 12% to the new file. [2% builds up], we copy that 
2% and then update the file writer to start writing to the correct file 
(actually, assuming we have binary seek we could probably start writing the 
incoming data immediately where it belongs and then copy over the start buffer 
when possible). when we're done, we kill the start buffer.

otoh, we could just ask the os to buffer the entire file in memory until the 
user picks a location (wouldn't this make more sense on most operating systems 
w/ decent vm?)
Keywords: arch
No longer blocks: 74008
*** Bug 76718 has been marked as a duplicate of this bug. ***
moving to 0.9.2
Target Milestone: mozilla0.9.1 → mozilla0.9.2
Timeless, I like your ideas, I was thinking more or less the same thing as I
read through this bug.

[I think the suggestion about downloading the file, then copying over the start
buffer is not so good though. It would prevent things like playing audio/video
files whilst they are being downloaded.]
mass move, v2.
qa to me.
QA Contact: tever → benc
*** Bug 74739 has been marked as a duplicate of this bug. ***
moving to 0.9.3
Target Milestone: mozilla0.9.2 → mozilla0.9.3
+mostfreq...

Keywords: mostfreq
No longer blocks: 74739
Little note:

IE also stores the download in a temporary location on my C: drive.  I often
download very large files (Slackware ISOs) to my F: drive, which has, say, 2 GB
free.  If my C: has less than 600 MB available, I cannot download the file.  In
this situation, I use NS4 to download things, as it downloads directly where I
tell it.

I'd very much like for Moz to follow NS4's lead on this issue.  :)

Adding 4xp.
Keywords: 4xp
This could also be related to stopping and resuming of downloads, because if you
know a file's location, it makes it much easier to resume an aborted download.
As has been said before we really must stream the data someplace while the
user is selecting a final location for it, otherwise we'd risk losing our
network connections, etc.  That being said, I think we should move the file
to the correct location ASAP.  It seems like it should be possible to simply
stop writing to the file, close it, and then move it (when it is still small)
and then resume writing to it in the new location.

mscott: does this sound doable?
>As has been said before we really must stream the data someplace while the
>user is selecting a final location for it, otherwise we'd risk losing our
>network connections, etc

So how does 4.x deal with the same problem?
i don't see why we're so worried about losing connections and the like. 4.x does
the right thing and no one has ever complained. why is this suddenly so
difficult to implement?
i'm not sure why it wasn't an issue for 4.x, but i can tell you that when
downloading something large, i have in the past really REALLY appreciated the
fact that we start downloading before a location has been selected.  it means
that i usually get the file that much quicker, and over a fast connections it
can sometimes mean no waiting at all.  so, let's not even consider throwing out
this feature.  i think it should be simple enough to move the file part way
through the download, and hence make it appear as though we are saving it to the
desired location.  how does that sound?
Pink it is hard to implement something like this because our download logic is
keyed off of content type and this usually/always happens after we start
downloading bytes.  

We could stop worring about this, if mozilla just had a default download folder
and all of the saved bits would go there.  If people wanted to explictly save
something to a destination, we could do this via a right click.  With this kind
of explict "SAVE-AS" functionality, we could invoke the download logic
immediately.  my 3 cents.
what if the location you're moving it to is not local (say your iTools disk over
a slow network connection)? Copying the file is not always trivial or
instantaneous. 
my point exactly: the copy is much faster when the file is still small.
Doug, how would having a default download directory help things ? The problem is
with the file name, not with its location. Besides, there is already a default
download location, e.g /tmp on Linux.



What I would like to see is something like this:

1) User clicks to download file.
2) File begins downloading to /tmp as usual.
3) User now inputs filename for saving to.
4) *Mozilla checks to see whether the file name given already exists.*

- If it does exist, and the existing file is *larger* than the downloaded chunk,
the user is asked if they want to continue downloading, or to overwrite the
file. If they select to continue loading, we delete the temporary chunk, and
continue with the existing file.

- If the file does not exist, or is *smaller* than our new chunk, or the user
has chosen to overwrite, copy the downloaded chunk to the selected location, and
continue downloading from there.



And one last thing, I agree with Darin, I *love* the fact that mozilla starts
downloading as soon as it can, if I were _forced_ to choose I'd rather keep this
functionality than have mozilla wait for me to give it a file location before it
started downloading. Navigating around the filepicker can sometimes take a
while.
About the one-download-folder-for-all thing, I don't want that to happen. If we
did that this would become Netscape SmartDownload, and I don't want that.
*Represses memories of SmartDownload and shudders...* People want to save files
where they want to save files, and nowhere else. Current behavior makes more
sense - save it temporarily in one place and move it to destination later.
> About the one-download-folder-for-all thing, I don't want that to happen

That may be the case on your platform, but on Mac, users expect files to go to 
their specified download folder (which is a system-wide preference). Right now, 
they have to work their way through two (2) dialogs to get there. Downloading 
with 4.x, or a dedicated FTP client, requires zero dialog messing.
here here.  Download the file to the same place every time.  If you want to be 
explict about where it goes, the UI should have some way to you can context 
click for a "save-as" dialog.  
yeah, good idea!  but, just to be clear, i don't think any of these dialogs
should hold up the download.  we could simply download to the default download
directory (with a randomized file name), and then, once the user has made the
final selection of where the file should go, we should move the file to the
correct location (maybe just renaming it in the default download directory) and
continue the load.  the critical part to this IMO is to perform the file move
immediately after the user has selected where to put the file.  then there
won't be any confusion about where the file is while it is downloading.

this bug may also want to concern itself with the fact that file downloads get
streamed to the disk cache as well.  we probably don't want to do this.
Ick.  Why have a randomized file name? Why copy the file anywhere?

I see two scenerios:

1. User click on something that winds up being download.  We toss a progress 
dialog and download the thing to the disk.  The dialog remains asking the user 
if they would like us to open up where it was saved.  It was saved to the 
default download directory.  This probably is not /tmp, but it could be.  more 
like ~/moz_downloads or something... On other platforms, this is what most 
clients do.

2. User wants to save the remote files somewhere explictly.  User context clicks 
and selects where to save the file.  We get that info, *then* begin downloading 
directly to where the user specified.  No Copy.
I think the idea that downloading the file in the background while the user 
chooses a save location is a good idea, because it makes things appear quicker, 
is misguided. I think you loose in user confusion what you've gained in time 
saved. And I think that the user won't mind at all if the download is prolonged 
by the amount of time that they spent in navigating the dialogs.

In short, I think not doing background downloading is much better from a user 
experience perspective; it's less confusing, and doesn't really save that much 
time.
I agree with Simon. This behavior seems totally unnecessary, and all of the
proposed solutions seem like too much work for something as simple as
downloading a file. The time it takes to choose a location to save or a helper
application to process the file is negligible. If you run off to tiptoe through
the tulips before performing this task, you deserve to wait. Like the
less-than-30-seconds it takes to deal with the dialog is going to make some big
difference.

With all the suggested solutions, there's going to be a lot of steps for
something so simple. Choose to save it to the default downloads directory (where
the file is already being downloaded), and you just have to rename (really a
solution to bug 60203 and not this one). If you choose to save it to a different
location, the file has to be copied and renamed. What if resumable downloads
ever becomes a feature that works? What happens if the download stalls or the
program hangs before you can perform the copy/rename? You'll end up with a file
possibly in the wrong location and definitely with the wrong name (and, if any
amount of time passes, you might not remember the file you were downloading, and
the filename definitely won't be any help).

Darin says: "the copy is much faster when the file is still small." Uh, isn't
the whole point of the argument so that you can lounge around with the download
dialog up but still have the download begin? What assurance do you have the file
is going to be small? If I have a super-fast connection, routinely download
large files and realize that Mozilla downloads whether you tell it to or not,
why shouldn't I go get another cup of coffee before the dialog even comes up?
What if I end up with a 50-100MB file (let's say I'm a slow walker)? What if I
decide to use the restroom, too, and have to go downstairs and end up with a
150MB file? Then, you have to copy this ridiculously large file, you have to
rename it, and you have to begin writing to the new location and delete the old
file. What if I only have 72MB of HD space left and you try to copy a 150MB file
that's still downloading? With 4.x, it would work fine (I'm downloading a 200MB
file and have 222MB left), but with Mozilla, I'm not sure what you'd do.

If none of these seem to be real (or probable) issues, or it is too difficult to
change the behavior to that of 4.x or IE, then go ahead and do what you want to
do. If you absolutely must continue this behavior, then I'd suggest at least
throwing up the progress dialog before the download dialog comes up. Even if the
progress meter isn't doing anything, the user will be informed that download has
begun and is progressing. Then you can perform the copy/rename voodoo, and hope
for the best.

My major problem with this still remains, though: it makes the behavior in bug
60203 necessary, and it's performing an action that the user has not
specifically requested (I don't want some file on my HD until I have confirmed
that I do indeed want to download that file).
if we hold up the downloading for an indeterminant amount of time, then we risk
loosing the connection to the server.  remember we have to handle the case in 
which the user clicks on a link and we discover an unknown content-type in the
http response headers.  what do we do with the data while the user is deciding
where to put it?  streaming it to a temporary location seems like the correct
thing to do since you wouldn't want to buffer the data in memory.

btw: this temporary location could also be the disk cache, though i would prefer
if it could be in the user's "preferred" or "default" download directory, so
that the cost of renaming the file would in most cases be minimal.

Is it so very hard, then, to re-establish the connection? Yes, this might result
in a delay since you have to reconnect to the server, but why is that such a
terrible thing? And why can't you just handle unknown content-type issues
differently than what you'd normally do (have a differnet method for dealing
with them)?

The behavior in Netscape 4.x is perfectly acceptable, and displays none of the
issues discussed here. Downloads and unknown content-type scenarios are handled
properly and filenames are kept intact, and the file is always where you expect
it to be. I haven't noticed any incredible gains in speed with the way Mozilla
does this. Like I said, in most cases, you're only shaving off 30 seconds or
less of download time. I understand that it's not as simple as I'm making it
sound, but I am not at all impressed with the implied advantages of the current
behavior, and none of the proposed solutions seem like they will ever make the
process better, nor as simple and intuitive, as the behavior in Communicator.
if the move happens when the user selects a location, then from that point forward
we will appear to behave identically to 4x with the exception that we'll have 
downloaded the file sooner.  the perceived benefit of downloading in the
background really depends on your network setup.  over a 56k modem, it really
helps to hit the wire as soon as possible... especially for smaller files.

also, restarting a download is a non-trival thing.  servers don't always support
byte-ranges and the content being downloaded may have changed, etc.

FWIW, IE also downloads in the background to a temporary location until the real
location is selected.
Blocks: 85115
You made your point well. I still agree with Simon Fraser and Mike Pinkerton that 
this is unnecessary and not as good as people want it to appear, but I'm resigned 
to living with the behavior. The only thing I really care about anymore is bug 
60203. That is the biggest pile of stink that's around. Implement the feature so 
that the file is renamed (I don't care about where the file is, because I always 
download to the same directory, and you don't have to worry about stupid things 
like \WINDOWS\TEMP on the Mac) when the user chooses a location/name/helper app 
and it will be acceptable. So, fix that (which will eliminate this and bug 
60203), and I'll shut up. I promise :)

(Note that you have to change the name to what it's supposed to be even if the 
user chooses to open the file with a helper app, otherwise you still get left 
with a junked filename and defeat the whole purpose.)
> Is it so very hard, then, to re-establish the connection?
Sure, it could cost me $1000 if the server decides that I'm trying to cheat it 
out of a second copy of an Ebook, Erecord, Eproduct, or Eserver.

> Yes, this might result in a delay since you have to reconnect to the server,
> but why is that such a terrible thing?
Welcome to Ecommerce, where it could cost me more pretty pennies than I can 
afford.

wrt resumable downloads that really has very little to do w/ this bug. one 
possible implementation is to stick 
'realfilename.realextension.mozresumableextension' into 'realdownloadfolder' 
(not to be confused w/ RealDownload) and have it contain source url and the 
intermediate directory path.
timeless, if reconneting in such a problem with eCommerce, why doesn't anyone
bitch about 4.x? why is this all of a sudden a problem?
*** Bug 85844 has been marked as a duplicate of this bug. ***
I'm using Linux and I *DONT* like it downloading to /tmp and filling up my 125
MB /tmp Directory when I am trying to download the 650 MB ISO image to another
place.

It bugs me immensely to see mozilla downloading this to both the NewCache dir
AND the /tmp directory.  It also bugs me that the tmp file created in the
NewCache dir got to be double the size of the file that used up all my /tmp
space.  And it bugs me even more that the smaller file in /tmp got copied to the
location I specified the file should have been downloaded to.  *IF* mozilla had
either downloaded it *ALL* to the NewCache dir *OR IF* it had not filled up my
/tmp dir *AND* downloaded the file to the *RIGHT* place I would have been one
hell of a lot happier.

Having a default download location is alright *IF* the users can override it. 
This has been a major pain to me.  I do *NOT* want to have a 1 Gig /tmp
directory just because Mozilla wants to download anything to /tmp while I figure
out where it goes.  (means I will have to stuff around in a big way on my work
and home linux boxes)

Plus as I pay for my ADSL connection to a monthly MB limit I am concerned about
the way mozilla just starts downloading in the background... Every MB over the
limit costs extra.
my proposed solution would solve your pains completely.
It would? What if the user dawdles at the save file dialogs, or goes away and 
make a cup of coffee before the save file dialogs come up?
if we just saved the file to a default (user-configurable) location, but allowed
the user to specify an optional location while the file is being downloaded,
wouldn't that work?
So the user now has to specify 2 places to save files:
1. The default location (set in prefs)
2. The location for this particular download

Why do you need the second? If the user specifies a location for saved files
in the prefs, just save to that, without any prompting.

Or are we going to get all up in arms about saving files "quietly" because of 
security issues?
1. The default location (set in prefs)
2. The location for this particular download

Simon Fraser wrote:
> If the user specifies a location for saved files in the prefs, just save
> to that, without any prompting.

The "default location" for download is just a suggestion offered by Mozilla to
the user.  Still, the prompt for "location for this particular download" is
necessary, coz the user need to have final decision on the filename and the
location of the file to be saved.

> Or are we going to get all up in arms about saving files "quietly" because of 
> security issues?

As mentioned, I think it's just a normal attitude to notify the user that Moz is
going to write something new on the user's hard disk.  I don't think this is a
security freak-out...  That's normal way of interacting with the user.
I think adding a pref for a default download folder would be a bad idea, unless
you could specify one folder for each of the file types (i.e. save MP3, MIDI,
and waves to the /media folder, save html, text, and PDF to /docs folder, save
ZIP's, MSI's, and executables in the /download folder, specifying each in the
helper apps window). Unknown/unspecified file types should be downloaded to a
user folder such as users/name/downloads/ and moved to the location the user
initially specifies when the download is finished. Also, files that are set to
run in an application should be run from the user folder, unless the user
specifies a folder (in which case the process would be to download the file to
the specified folder and run it from there). There would be no initial security
risk because there would be no default folders (users who specify these folders
later on acknowledge any risks one might expect from a "quiet download"). If no
location is chosen by the time the download finishes, the finished file would be
left in the user download folder and some notification would be present (e.g.
opening a sidebar with "Complete Downloads" - below this could be "Interrupted
Downloads" ready to be resumed).
Why not have an option in 'helper apps', so that when you define a mime type,
you could also specify a default download location ? But that really belongs to
bug 7840.
Yuck. Double and triple yuck. That has got to be the most confusing way to get
around actually fixing this bug. Mozilla would have to keep track of every
single path to a directory for a certain MIME type. What about users who don't
know how to change settings to their helper apps? What about users who download
everything to a single directory? Are you going to have Mozilla change the
default directory for every single MIME type when you change the global
downloads directory?

I would suggest choosing a default location in the preferences and having a
separate option to "Always download to the default downloads
directory/folder/whatever." Is this a direct copy from the way IE does it? Yes.
Do I care? No. This way, users can simply uncheck that option and choose
different folders to download their MP3s to folder A and JPEGs to folder B and
archives to folder C and I can leave the option checked and everything will
download to folder A (the way I like it).

Remember, if it works out best for Adam, it's usually the best solution. ;)
So, to summarize ...

Reasons to create a file before the user asks you to:
(1) It saves you up to about 20 or 30 seconds of downloading time (Darin's
    argument).
(2) Waiting might lose the connection and require asking the server for the file 
    a second time, which could worry pay-per-download businesses (Timeless's
    argument). But if that's the case, then you're also hosed if you get
    randomly disconnected by your ISP, or if you have a power blackout, or if
    the remote server goes down, or whatever. So any business which relies on
    pay-per-download is asking for trouble already.

Reasons to not create a file before the user asks you to (problems which would
*not* be solved by moving the partial download once the user specifies a location):
(1) For small files, it appears that something has gone wrong with Mozilla,
    since the progress indicator is at 100 % when the progress window first
    appears.
(2) The estimated time remaining and transfer rate shown in the progress window
    become very inaccurate.
(3) If the default download folder is the desktop folder (as it is on Mac OS by
    default), unsightly files are put on your desktop until you specify the
    download location (bug 60203).
(4) The desired destination folder may not be mounted (e.g. iDisk), and mounting 
    it and moving the partial download to it may take long enough that the
    connection gets dropped anyway (Timeless's argument in reverse).
(5) The volume where the default download folder lives may not be large enough
    to store the temporary file, whereas the desired destination folder is.
(6) Mozilla, like the Finder, should refuse to even begin copying a file where
    it knows that the file will not fit. My decision to download something may
    depend on whether it fits on a Zip disk or not; and if it does not, but I am
    still charged by the megabyte for the partial download, I will not be
    amused.
AK: Any user who couldn't set a default path for helper apps doesn't need to be
using Mozilla in the first place. (You don't need me to tell you that this
program is NOT for newbies.) The functionality should be there, but in case
someone doesn't want/know how to set the default folders for each file type, it
would be unspecified and the default download folder would be used (as I
suggested, something in a user folder) - and then the user should be prompted
for save path every time they try to save. I suppose the default save folder
could be changeable too, but I've never known anyone to want to save EVERY file
type in one place. As I said, people will want their music here, their pictures
there, their HTML over here, and their compressed archives over there somewhere.
It might be a little more work getting the support in, but the end result is
much better than just having one download folder.
mpt-6. has a bug. not all file systems can correctly indicate space available.  
:( my comment about that appears to be in another bug. from memory, Netscape4 
will occasionally warn me that it doesn't think there's enough space in 
f:\programf\dev to download a file, it lets me tell it to continue anyways.
this is often how my system looks:
              25 Dir(s)         294,720 bytes free
F:\>
              31 Dir(s)     275,169,280 bytes free
F:\programf\Dev>
               2 Dir(s)   2,286,362,624 bytes free
F:\build\beonex>
If I try to download an iso disk image to f:\build\beonex and my browser 
refuses I'd be very upset.

Various things can come into play, Disk Quotas, AFS partitions, Compression, 
Encryption, Disk Spanning, ...

On occasion, i might even try to download that iso to F:\programf\Dev\ and 
merely race the download against my efforts to make space.  This is kind of 
what mozilla is doing now.

mpt-5 was addressed by some of my earlier recommendations. which means they 
would be solved 'by moving the partial download once the user specifies a 
location'.
mpt-4 doesn't make any sense. I don't see how it even relates to 'problems 
which would *not* be solved by moving the partial download once the user 
specifies a location'. i suspect in the case of a remote mounted device the 
os/driver would be caching the data and sending it out on a schedule 
satisfactory to it.
mpt-3 already includes a bug reference.
mpt-2 is a red herring. there's another bug which he is aware of to revise the 
transfer rate and estimated time remaining.
mpt-1 ignores the fact that we have a bug for rendering pages before 
downloading.  People seem to like the idea.
-> XP Apps.
I've been watching this debate for some time.
This feature is a mess, and this discussion is the worst of it.

Can we make this pref controlled, and send the debate to the netlib newsgroup?
Component: Networking → XP Apps
Okay, first a disclaimer:  I've read through part of this mess, but not all of
it as most of it seems to be ancient history.

Next:  If you feel I should file a new bug with the information I'm about to
provide, then let me know.  I'm adding this here because it seems incredibly
similar to me.  I don't care how a download starts, where it starts, or what
name the file gets when it starts, as long as when I select a location and file
name, that's what it gets and where it goes from that point on.  Using a
mandatory default download location is bad for several reasons, the most
prominent in my case is that there just might not be enough room on that disk
for the download to finish!

On to the current problem:

Trying to download a file with a MIME type (in this case,
application/x-compressed) and change the choice from 'open with <application>'
to 'Save to disk'.  The process of saving a file to disk has been whacked for
the last two nightlies I've used.  (FYI I'm using W95B, and use the w32 talkback
zips.)  

In the previous version (dated 16-07-01 11:29, but I'm afraid that's as close as
I can get to a build ID), it was simply a matter of the program ignoring any
instructions for downloading.  That is, the file continued to download to my
TEMP directory with an obscure filename, rather than download to the
drive/path/filename I chose.

Now, in build 2001072003, an even weirder problem has cropped up... Not only
does the file continue to try to download to my temp directory, but I keep
getting multiple "What do you want to do with it" dialogues which each produce a
'saving file' progress report all downloading different random filenames to the
same place.  :-(  I don't know how long this might continue, because after five
times of giving the same instructions and having them ignored, I canceled them
all and came here.  :-/
I don't know what the plan is for this bug, but a few things to consider
regarding using /tmp on Linux and once UNIXes.

On Linux, /tmp is often on a seperate disk partition then /home. When
downloading a large file (say, 500M), this means:
  a) there quick likely might not be enough room on whatever partition /tmp is
     on. The download will be killed because of this.
  b) we will have to COPY the file across disk when the download is done, since
     you can't do a simple move when /tmp is on a differant partition then
     /home. This causes a lot of wasted IO.

I should note IE5 also suffers from the two above problems. It starts writing to
the temp directory on c:\, but if I specified to save to d:, the problems above
apply. I can't tell you how many large downloads I've lost because C: filled up.

On Solaris (and on some Linux machine), /tmp is a memory file system. This means:
  c) downloading a 500M file USES FIVE-HUNDRED MEGS OF MEMORY! It will use swap,
     if it's available.

I'm not sure if there is a fix for this problem, other then letting the user
decide between the options we have. (start download immediatly, saving to
$LOCATION in the mean-time, or stall download until I pick a place). Both have
their problems.
Whatever the fix is, /tmp needs to NOT be the default download location, though.
The users home directory is probably a better choice.
Perhaps it would be clearer if the open/save dialog appeared as a child of the
download progress dialog so that the user could see that the file was already
downloading while deciding how to open/save it?
Wouldn't the simplest solution be to let the user _set_ a default download
location (if one is really necessary)?
moving to 0.9.4
Target Milestone: mozilla0.9.3 → mozilla0.9.4
*** Bug 93838 has been marked as a duplicate of this bug. ***
*** Bug 93847 has been marked as a duplicate of this bug. ***
-> xp apps qa.
QA Contact: benc → sairuh
*** Bug 62666 has been marked as a duplicate of this bug. ***
*** Bug 95210 has been marked as a duplicate of this bug. ***
Here is some of the info from the recently duped bug. Some of it has been
mentioned here before, but some of it is new.

1. The temp folder is usually on C:\ and for WinNT this is often limited to 2
GB, where much of the space is already used up by other programs that insist on
using C: - I tried downloading a 150 MB file to my F: drive and after 4 hrs!!!
the download failed because C: had become too full - I lost that data =
dataloss!? :(

2. Partiallydownloaded files become lost in temp folder nirvana because most
users never look there. They do however usually know and use the folder THEY
have chosen as the destination folder. We should follow GetRight's lead in this,
as they are very experienced designing good download managers.

3. Again, following GetRight's lead, if the partially downloaded file has the
extension filename.ext.MOZ, then users (A) would know that the file was a
download file that was not finished yet. (B) could doubleclick the file to
resume the download; (C) the MOZ extension would have an accompanying icon that
further identifies it as belonging tio Mozilla. All thiss is most useful if the
partially downloaded file is where the user expects it to be, namely in the
destination directory he chose.

The main reason right now for me is "1." because I am really upset that the
partially downloaded file location is one I have no control over (without
changing my entire system configuration - which is often not possible in an
office environment). Additionally, implementing this has the added bonus of
making points 2 and 3 easier to implement in the (hopefully near) future.

BTW. Shouldn't this be KW "dataloss", since it has caused dataloss and has
wasted up to four hours of PC-time for me.
we'e not going to hold .9.4 for this. 
Target Milestone: mozilla0.9.4 → mozilla0.9.6
Looking through this bug, it seems that somebody needs to specify clearly and
definitively what should happen when a file is downloaded. There are many
suggestions and good ideas contained in here, but apparently no conscensus as yet.
 
*** Bug 97725 has been marked as a duplicate of this bug. ***
In my opinion the default behavior for downloading files should work similiar to
this:

1. After clicking a filelink, Mozilla starts downloading to a (configurable) 
   temporary download directory (e.g.: /tmp).

2. The user eventually picks the final location and filename. 

3. Mozilla should check, if it thinks there is enough room on the harddisk.
   If not, the following dialog should open:
   Warning on <filepath> may not be enough room for <filename>.
   This assumption may not be accurate, depending on your system configuration.
   Should I continue downloading, anyway?

   Continue Cancel

4. Depending whether the destination file already exists, the following   
   scenarios are possible.:

   a.: The file does not exist:
       The temporary file should be moved to its final location.
       Downloaded data is from now on written to the final location.
       
   b.: The file already exists:
       A window showing the follwing text should open.:
       The file <filename> already exists.
       What should I do with existing file?

       Replace Resume Cancel

       Where 'Replace' should move the temporary file to the final location,   
                                                                               
           
       overwriting the old file.
       'Resume' should replace the old file with temporary file, if the        
                                           
       temporary file is bigger than the old file.
       If the old file is bigger, mozilla should try to resume the download (see  
       separate bug). 
       'Cancel' should cancel the download.
       In any case the temporary file should be deleted.



The following (advanced) download preferences could exist:
1. Path to the temporary download directory.
   Default should be the platform specific temporary directory like /tmp or       
   C:\temp or whatever.
   This could be useful for users with limited harddisk space in these  
   directories.

2. Option to turn off data prefetch (background download) before the final 
   location has been choosen.
   This will help people with fast internet connections and/or limited space in
   the temporary download directory.
   With this option turned on, temporary files will not be created for 
   downloads. 

My opinion to some other points mentioned in this bug:
1. Mimetype dependent download directories.
   Not important for me, as I usually download all files to my home directory.
   Might confuse users (where has my download gone?)

2. Partially downloaded files
   Partially downloaded files should be saved at the user picked 
   location and with the selected filename.
   The user will most likely search the file at the location he picked.
   Moreover the user will know, if the download failed, because he got an    
   errormessage, right ? Maybe these errormessages could contain some way 
   (button, checkbox) to delete the partially downloaded file.
   Keeping the correct filename makes life easier for resumeable downloads.

3. If showing a progress dialog with 100% completed, for small downloads is
   really a problem for some users, then do not open it at all, if the file is 
   already completely downloaded. This might however confuse other users... 

One last word... 
  This is how in my opinion a solution for this bug should look like. It is 
  mostly in consensus with other suggestions on this bug.
  
But the most important point is:
  Whichever the final solution for this bug is, fix it soon.
  Downloading is one of the key-functionalities of a browser and imho more     
     
  important than another new skin.
  I can't understand, why an almost one year old CRITICAL bug with the keywords    
  4xp, nsbeta1, nsCatFood (I would also add dataloss (I already lost downloads 
  due to this bug)) and 7 votes, has still no developer assigned...   
mscott:

are you working on this?  or, do you mind if i take this one?  i'd like to try
to implement some portion of the implementation outlined by snfuchs@gmx.de

snfuchs:

i like most of your plan, but i think that implementing the Resume feature will
be difficult, and should probably not be attempted on the first pass.

likewise, the preference to disable the prefetching feature will be troublesome.

IMO giving users a way to configure the temporary download directory is half the
battle.  then, moving the download file from the temporary location to the
choosen location should provide a robust and IMO satisfactory solution.  the
point after all is to prevent dataloss and basically not annoy the user, and i
think this solution would defintely help minimize user frustration.
Hmmm...this seems to have been fixed already. When I download anything it goes
straight to the location I type in, even while it is downloading.

(Linux 2001090408).
*** Bug 98964 has been marked as a duplicate of this bug. ***
IMO, this should just be scrapped. No data should be downloaded except to the
location specified by the user.

The arguments for this behavior:

* It prevents disk-space and other drive-specific issues.
* It avoids a cumbersome pause-resume process (which some servers don't even
support).
* It clears the way for an efficient halted-download storage system (similar to
that of GetRight) - which saves to userpath/filename.ext.moz.
* It cuts down on file system move operations which could cause messy disk
fragmentation and slowdowns on older systems.

The arguments against this behavior:

* It delays downloading by the user's reaction time. The user can avoid this by
choosing to automatically open files of that type (which puts them in the temp
directory anyway).
* It blocks the possibility for a complex auto-save-to system as I described
before. It looks like nobody is willing to put forth the effort to create such a
system anytime in this century (in which there are about 1,190 months left).
* The partially downloaded file could be mistaken for a finished download, and
the user might try to open it, messing up the download. This would be much less
likely to happen with the .moz extension.
Blocks: 18004
why not add a pref for temp files. you could choose between temps going in a 
folder in your profile (what i would want if my profile was local), going in /
tmp (what i would want in any *nix system i set up) or in window's temp 
directory, going in the download directory (which is what it is now), or waiting 
for the user to respond until starting.

this may be a little more work (might not be), but it is the right thing to 
do(tm)--at least in my oppinion.
nnooiissee: The hard part is finding someone who's willing to do that. And
again, that makes things much more complicated when we start a download resuming
system.
skewer: why should that make it any harder? if the preferences define a location
for temp files, then when prematurely ending a download mozilla askes if the
temp file should remain to be resumed later. if the user has chosen not to save
until a location is choosen, then just abort and leave the file.

you are right, finding someone to do it is hard, but with this much debate i
would not touch this bug with a ten foot pole. rather than someone sitting down
and writing the code i think someone should sit down and write a spec something
like what is found at http://www.mozilla.org/projects/ui/communicator/index.html
and then we can review, debate, and finalize that. then finding someone to do it
would probably be alot easier.
For the record, I'm with timeless on this. I *want* Mozilla to be downloading
files while I waste time trying to decide where to stick the file. I pay for my
connection on a per-minute basis, so the less time wasted downloading the better.

Saving the file to either the temporary directory, the cache, or the default
download directory, and once the file name is known renaming/moving the bit
downloaded so far, seems the best idea to me.

I think this bug should be renamed to "Spool file should be moved once the user
picks a filename".
If the 10 seconds it takes to decide on a directory and filename are sooo
important, then I agree with Ian that the spool file should *definetely be
moved* to the user-selected location once the user has selected such a location.
How would we go about moving the file, though? We can't stop transferring from
the server (particularly when the server doesn't support resume), and it would
be very difficult to move a file while it's still downloading.
Everybody is talking about saving the file to a temp directory or something
similar, but why not keep it in memory until a place is chosen? On windows this
would go to the swap file if we run out of memory, and on unix to the swap disk.
Then after a location is chosen we can dump whatever we have in memory to file
and start appending to that.
Memory won't work, what about downloading a 500M file? You could write it to
disk after X seconds, but then you're back to the drawing board where to put
THAT file, and what to do with it.

There's only two solutions here that I can see.

1) pref for spool directory.
2) pref to disable saving until location is chosen (this is how 4.7 works, but
does it do a HEAD or GET first to check the MIME type?)

Let's pick one and get it in. I don't even care which any more.
i agree with hixie.

pausing downloading to move the spool file to the user specified location is not
a problem.  necko automatically handles intermittent delays on the consumer/UI
side, and there would be little concern that the server would drop connections
due to such a brief pause.

modified summary to reflect this solution.
Summary: Saving a file creates a file before location is chosen → Saving a file creates a file before location is chosen; spool file should be moved once the user picks a filename
How long a delay is acceptable, though? I've seen computers that take half a
minute just to move a 1MB file. After that much time a server might timeout.
Pause/resume seems like the better choice in such a case, but there's still the
problem that neither Mozilla nor a percentage of web servers supports it.
if a computer is that slow, then it is certainly better that we move the file as
soon as possible (ie. while it is relatively small) than waiting until the file
is completely downloaded.

tcp/ip applications are expected to allow for latency/delays/etc... the internet
is not always fast, right?

so, i don't think 30 second delays (however likely) are going to be a problem.
now, if it were several minutes worth of delay, then that might be a problem.

overall, i think moving the file early will give a much better user experience,
especially on computers with slow harddrives.
If we're going to move mid-transfer, then I propose the spool directory default
to  $PROFILE/spool or $PROFILE/tmp. On UNIX, the profile is much much much more
likely to be on the same disk as the chosen download location (profile is in
$HOME, users very rarely save outside of $HOME). On Windows, it's not any less
likely to be on the same disk, and probably slightly more ("Documents and
Settings" generally placed on a disk with plentiful free space). Mac filesystems
are alien to me, but it probably can do under $PROFILE there too.
Jeremy, saving the temp file inside the home directory will do terrible horrible
things to people who have disk quotas (fairly common on Unix systems).  It may
make it very difficult to download a file that's bigger than remaining quota. 
/tmp or better yet $TMP/$TEMP (as discussed eslewhere) should be the preferred
default, imo.
And saving to /tmp does "terrible horrible things" on many systems where /tmp is
a memory resident file system. If it takes you a few minutes to pick where to
save it (you walk away from the keyboard for a minute or two), and you're on a
fast connection, you could easily tie up 50-100M of memory. If a user has a disk
quota, they won't be downloading a file larger then it anyway, since they
couldn't do much with it. It should probably be a hidden pref. The default is
debateable.
> If a user has a disk quota, they won't be downloading a file larger then it 
> anyway, since they couldn't do much with it.

You've obviously never tried downloading and using the latest Moz nightly for a
single login session while having a 15MB quota....  Or downloading and printing
a homework assignment that's image-heavy (on the same quota).  :)

> where /tmp is a memory resident file system.

Aren't you asking for trouble from all the apps that actually use /tmp for
temporary files (eg mozilla for files it feeds to helpers).

The point is that I feel that we would be best served by making the location a
non-hidden pref.  At the very least, the pref should be well-documented.
Obviously you don't have 19 partitions, each with only a limited amount of
space, either.  If you ask me, there should be a very obvious and unhidden
option (perhaps even on or along with the first download dialog, with a 'don't
ask me this again' box) to not bother downloading at all until I tell you where
to put the thing.  Also a 'more info' button that will explain what is happening
and the pros and cons of each choice (a link to the relevant help section would
solve this).

Just my $0.02
My parents use Mozilla and they're confused enough without being prompted about
where they want their spool directory. Lets let the general public figure out
what "You are now entering a secure site" means before we prompt for more.
You make this so complicated.  Here's the scoop: you pop up a dialog box.  The
user chooses a directory and filename.  Then you start the download to that
directory and filename.  Done.  If you can check before starting that the file
will fit where the user selected, and it won't, warn the user.

Anything else is too much.  I say this as a user, not a programmer.  Do you ever
want to finish this thing or not?
bob: This isn't about running out of space where we PICK to save (I assume we
*do* check for that), it's about if, how, and where the file is stored during
the before, during, and after the dialog.

Since we've been all over the map here, let's try a proposal:

---

Defaults stay as is. Add a check to the code to verify there is enough space on
both the spool and chosen location, if it's not there already.

Add a download spool directory pref under Advanced->Cache:


[V] Begin downloads before I choose a location to save to
    Spool in: [ C:\WINDOWS\TEMP               ]

Whether or not we move the file from the spool to the chosen location as soon as
it's chosen doesn't matter very much, as long as those prefs are available. If
someone REALLY feels like coding it though, it couldn't hurt. We'd have better
performance then IE (for a change) with that, which would be nice.
---

OK, any complaints? No? Good. Code away.
jeremy: you missed my pint.  My point is: you don't put the file ANYWHERE until
the user tells you where.  After some years as a software user, I've come up
with 2 rules that I expect out of a good user interface:

1) Do what I tell you to do.
2) Don't do stuff I didn't tell you to do.

I'm sure there are more, these are just the 2 which, when violated, aggravate me
the most.
When I first used the download as (previously) implemented, it gave me 2 problems:

b) I thought it wasn't working at all because I couldn't see the file where I
had asked it to be.  Rule 1 violoated.

a) It put a file that would grow to 660M (yes, it was a Mandrake iso) into a
filesystem with only 120M.  Rule 2 violated.

If, instead of trying to save a trivial amount of download time, you had just
waited until the user selected a directory/filename and put the file where the
user told you to put it, you could have saved yourselves this whole bug.  The
time squashing this silly bug could better have been used.
Of course, 0.9.4 seems to do the right thing, so I'm not sure why we're still
disucssing this, unless your'e still creating this spool file, then moving it
when the user selects a name.  If you're doing that, you're still violating the
'rules' I gave above, and sooner or later another grouchy user like me will be
opening a bug on it after taking too long to select a name and filling a filesystem.

Anyhow, thanks.  mozilla looks much better at each milestone.
bob: there is no way to implement what you want... consider dynamically
generated content from a CGI script for which there is no built in viewer.  we
would have to save the document, and we could not refetch it (because perhaps
refetching would mean repeating some credit card transaction, etc).

the solution to move the file to the user specified location ASAP is the best
IMO, because it almost completely addresses both of the rules you outlined.

1) you will notice the file "growing" at the location you specified.

2) you will not have to worry about it "growing" at an inappropriate location,
since we will have moved it to the specified location before the file gets to
large.  <-- this is not exact, and that's why i say that this solution "almost"
completely addresses both of the rules you outlined.
darin:  OK, I see what you mean.  I'm not used to seeing a CGI script that
generates content needing a viewer, but I'll take your word that it happens.

I agree that for any reasonably quick selection of a filename, your approach
nearly satisfies my "rules".  (Though seeing the modem lights continuously
flicker prior to selecting a filename will still puzzle some useres...)

But I still, just out of curiosity, wonder: what happens in this situation for
NS4.whatever? Does it just hold the session up until either the user chooses a
filename or TCP times it out, or what?
(posted w/ 0.9.4 on Linux, which gets used for most of my browsing now.)
I don't see how the save-before method is better. Either

1. After receiving the unknown content-type header, you hold off the saving of
data (which involves a slight timeout risk, but perhaps less of one at this
point) until the save-to location is picked by the user.

or

2. Automatically start saving to a location which might not have enough room for
the whole file (which involves a risk, and might merit hours of extra coding to
add a custom spool directory and disk space checks), then HANG THE DOWNLOAD
(with a shorter, but still present, risk of timeout during filesystem
operations) while the file is moved. This special networking behavior be more
complicated to code.
With 1 the risk is an initial timeout. With 2 the risk is a mid-download timeout
and conflicts with drive space, as well as the need for much more coding. As for
paid transactions, if the download times out before any data is transferred
there might be fewer problems than if it times out after some data is
transferred. I just don't see any reason to do anything before the user chooses
a location to save to.
we have a problem with HTTPS in which the user is required to verify a dialog
before continuing to download a secure page.  if the user does not respond to
this dialog within 1 minute, the connection will be dropped.  for file->saveAs,
i really don't think we want to mess with more of these types of problems.  so,
while normal web servers may be more forgiving in the amount of time they'll
pause, HTTPS servers are usually much more strict (probably because of the cost
of a HTTPS connection).
skewer: the required code will not be as much as you think.
Blocks: 103766
spam: over to File Handling. i have not changed the assigned developer [or the
other fields for that matter], so if anyone realizes that a bug should have a
more appropriate owner, go ahead and change it. :)
Component: XP Apps → File Handling
*** Bug 103856 has been marked as a duplicate of this bug. ***
-> me to try to develop a solution.
Assignee: mscott → darin
Blocks: 104166
Target Milestone: mozilla0.9.6 → mozilla0.9.7
Blocks: 69938
*** Bug 106619 has been marked as a duplicate of this bug. ***
*** Bug 107088 has been marked as a duplicate of this bug. ***
*** Bug 107412 has been marked as a duplicate of this bug. ***
Blocks: 88566
the solution to this bug likely depends on the implementation of the download
manager.  that is, i would imagine the download manager will make some of the
existing "file save as" code obsolete.

anyone know who's working on the download manager?
Severity: critical → major
Target Milestone: mozilla0.9.7 → mozilla0.9.8
ben is working on the download manager.
*** Bug 109105 has been marked as a duplicate of this bug. ***
This is a very interesting issue.  On the one hand, there is potential for
seemingly blazing download speeds.  There is also the potential for
incomprehensible behavior, unnecessary complexity, additional defects, extra
work for developers, and even performance degradation in some scenarios. 
Current users have learned to expect to have to specify a location in a file
system hierarchy to save files, but really, most people could not care less
where they are saved as long as they could be retrieved quickly and easily. Few
AOL users have any idea where their downloaded files are saved, and why should
they? One way around this problem would be to use a default download directory
rather than having to specify a location every time, which I'm sure for many
people is the same location nearly all the time.  Some operating systems (MacOS)
have this already, so our asking where to save the file to is a needless
interrogation of the user. A better way would be to hide the file system from
the user entirely, and provide much more sophisticated means for retrieving
their documents.  That probably should be a separate bug report tho.  ;-)
126 comments and we've gotten nowhere. Solution Proposal:

2 modes, temporarily named auto-download and interactive-download. i-download is
the default. If mac IE actually saves files automatically as trudelle alluded
too, mac defaults to a-download. New pref pane ("Downloading", under
Navigator?), looks something like this:

(o) Interactive download: You will be prompted where to save every time you
    download a file.

( ) Automatic download: Downloads start automatically and are saved to the
    location chosen below. This can save time by getting downloads underway as
    soon as possible, but may end up saving files to drives that are full.
       Automatic download location [ /home/jmd                        ]


I take it there are cases when downloading where we don't have a content length?
Maybe for FTP or gopher... When we do have it, and a-download is selected, we
should check for space, and if there isn't enough room, a dialog should pop up
explaining this, and present a file chooser to manually pick a location.

I suggest i-download as the default because most users expect this (4xp and
IEp). If moz just starting tossing files to $HOME, users might wonder where it
went. I plan to use a-download myself though, so am not against it much. Of
course, with i-download, we DO NOT save to /tmp or whereever before a location
is chosen. That's the whole point of the pref.

Right now we do something inbetween a/i-download, which ends up being what no
one really wants. I think (er, hope) this above proposal would satisfy everyones
needs.
I agree with Dolan, provided interactive download does *NOT* begin transmission 
of the file until I have chosen the location; not to memory, not to the disk, 
not anywhere. Of course it should still wait for a content-type and (if 
available) content-length header from the server.

It would be great for both of these options to integrate with download 
manager/resume when those come around. As requested before (here and in other 
bugs), the file should be saved as-is to the disk as it is downloaded. That's 
another bug, but whoever writes a patch should keep this in mind.

If done correctly such a preference won't threaten stability or performance.
"A better way would be to hide the file system from
the user entirely, and provide much more sophisticated means for retrieving
their documents."  

You say your name is Bill Gates, eh?  <ducks>

Please don't.
"A better way would be to hide the file system from
the user entirely, and provide much more sophisticated means for retrieving
their documents."  

Since you don't know what application would be used for retrieving the
documents, how do you propose to do this?

A plea for comprehensibility and simplicity: the user chooses a path and
filename.  The file is then saved to to that path and filename.  If the user
can't deal with this, they probably shouldn't be using a computer.

It's been pointed out to me that there may be a small number of situations where
dynamic script out put is involved and pokey users may get timed out, and
resubmitting the request is not valid - I suspect that this will be rare.
It will  have to be supported in the OS. Anyone who thinks most users really
want to deal with a hierarchical file system, and understand how documents can
exist there, and in memory, and in various memory/disk caches, is simply
deluding themselves.   Until OS document interfaces evolve in this way, I agree
with Bob that this should be kept as simple as possible, and the edge cases are
inconsequential.  Speeding downloads by a second or two is not worth the added
complexity to the UI, nor the usual regressions.  Adding modes in which
downloads could lead to unexpected errors would be wrong.  Unless someone does
the extensive work required to make this function seamlessly from the users
point of view (which I'm not sure is even possible), or it shouldn't be done at all.
These bugs used to go to networking, so this is still a sensitive point of pain
for me. I think I could put down a coherent design in one or two pages, but the
question is, if I do, will someone consider changes for 1.0? If so, I'll take
the time for this.

Peter?
I'm not sure what you're asking, but anything proposed for 1.0 should be
considered, and performance, correctness and usability are part of the 1.0 roadmap.
Perhaps the solution is to start downloading it to the cache, then when the user
selects a location, a new thread is spawned.

The existing thread calls open(newfile, O_WRONLY|O_CREAT|O_TRUNC)
The new thread calls open(cachefile, O_RDONLY).
The new thread calls open(newfile, O_WRONLY)
The existing thread calls lseek(newfile, tell(cachefile))
The existing thread continues to write the download stream to newfile, starting
at the offset to which it seeked. 
The new thread copies the data from cachefile to newfile
The new thread calls unlink(cachefile)
I don't recall seeing this among these comments, but I don't have time to
re-read them all atm.. Am I the only one seeing this behaviour?  Not only are
the files being saved to cryptically-named temp files, but self-installing .exe
files are being saved as .tgz files, confusing the h*ll out of WinZip (which
shouldn't be called anyway).
trudelle> Speeding downloads by a second or two is not worth the added
trudelle> complexity to the UI, nor the usual regressions.

With regards to my ('the'?) proposal, I mentioned I would be enabling a-download
myself. Not so much so get downloads finished 2 seconds quicker, but simply
because I always always always save every file to $HOME. I've never once not. If
I need it elsewhere (rare), I use mv(1). It's an annoyance issue of clicking
'save' after I already shift-click a link to save it, more then a perf one.
Others, however, (BT internet users, etc) do want it to begin automatically to
save download time and money.

That said, if THAT isn't deemed 1.0 worthy, the half-assed attempt that is the
current default desperatly needs to be removed. This also fixes most of the
blockers as well.
You consider clicking 'Save' to be an annoyance, but it is currently the point
at which users commit to saving the file. I still think the right solution is to
add a way to save a file to a pre-specified location, without any dialog. 
No, there should be a *definable* default download directory which is presented
when selecting "save link". Then one can either simply hit ENTER to confirm that
directory (and filename), or select another directory (it does happen). This is
the ideal combination of speed and flexibility. And don't tell me hitting RETURN
to confirm the default directory is a hastle ;)

PS. Anyone know how I can change the current (nonsensical) "default" directory?
Presenting any alert or dialog is a hassle, because it breaks the users flow,
can cause confusion, and does require action.  Requiring any such action of
users unnecessarily is very bad, we have no reason to continually interrogate
them. In this case it could easily be prevented.  If the download location is
defined, then why should they have to confirm it every time they save a file? 
The whole idea of predefining it is so that stuff just gets saved there.  Note
that it could still be  possible to get the dialog, if you really wanted it
every time.
Oh please, EVERY application on this planet (e.g. WordPerfect) *asks* where the
user wants to store a *new* file - and a downloaded file is a new file! 

Most applications have a default directory - but they still ask! Why break this
useful convention, and why confuse novice users ("I clicked the download link,
but where is the file?"). 

Users might "accidentally" click a download link (e.g., thinking its a
hyperlink), so they will need a confirmation dialogue anyhow, asking them *if*
they want to download the file. This dialogue should at *least* offer to store
to another directory.

I can't believe there is a need to discuss something so obvious.
"Presenting any alert or dialog is a hassle, because it breaks the users flow,
can cause confusion, and does require action.  Requiring any such action of
users unnecessarily is very bad, we have no reason to continually interrogate
them."

Sheesh!  Is M$ taking over development of Mozilla???  Don't you understand that
if I wanted that kind of behaviour I'd just install IE?  I don't -like- programs
that just 'do' without any intervention or direction from -me-.  That's why I
use non-M$ products.  Allowing the user to -define- a default directory if
that's what they -choose- is fine.  But I save to -many- different directories.
 I have a whole drive set aside for downloads, and that drive contains 102
directories, plus I often save to 19 other -drive-partitions- (on -so far, soon
to change- 3 physical drives) depending on the situation.  Obviously I don't
want to be  confined to one location.  I don't pay any extra for bandwidth or
telephone calls so I don't even want the thing to start until I tell it to what
and to where.  I have always been able to do so.  In any other programs I run I
have this choice.  Name a program that doesn't provide a 'save as' option?  Why
is this asking for a filename and location suddenly -not- the intuitive thing to
expect?

Another thing:  Saving to some obscure -filename- doesn't make any sense at all
to me.. especially when even the -extension- is different.  Saving a
self-installing executable file to a .tgz file???  Why?
*** Bug 112175 has been marked as a duplicate of this bug. ***
Peter L., Lythande: Please calm down. This bug is for an OPTIONAL feature that
many people want, to download quicker and not be annoyed by a dialog they NEVER
use or want. It WILL NOT be on by default, that would of course be very
unexpected to a new user. It is an automatic download/background download mode
selectable in prefs and eventually in download manager.

Lythande: Your tgz/exe issue is a seperate one, please stop discussing it here.
Quick-saved files will be named as well as possible, not like the current temp
files.

Darin: any feelings on my proposal? (comment 127)
IMO, confirmation dialogs are, in general, an insulting waste of time; it is
much better to allow some form of undo. In this case, users will get a progress
dialog, which will offer them the chance to cancel the transaction (on the rare
occasion they make a mistake), and offer to 'reveal' the downloaded file in the
OS file system.  I really don't much care what WordPerfect does with new
documents, it has little bearing on people downloading files using a browswer.
I'm not advocating getting rid of save-as, just allowing for a one-click
download for what seems to be the typical case for the vast majority of people
who don't even know what a partition is, let alone have 19 of them for
downloading.  I'm also not very interested in what default behavior anyone in
this bug personally prefers (including myself), since there is little or no
support for building a browser geared primarily for a tiny number of
technophiles.  If what we are building appeals mainly to us, rather than being
adopted widely, then we will fail.
[OT]
Peter trudelle, please stop spamming this bug. Comment #143 already explained
that what you want (non-standard, nonintuitive filesaving) will be an *optional*
feature. So go rejoice with the other 2 people that want this "feature". ;)
[/OT]
I'm with trudelle. After all, he is manager of the browser team, with years of 
software development experience.
I am for the proposal in #143.  Being a technophile myself, my preferences does
not count :-), but I've seen how "normal" people use this feature (mainly family
and friends).  I've seen people download the same thing several times because
they didn't get a save dialog.  I've seen people searching all their hard disks
in search for that file they know they downloaded (they saw the progress
window), but where placed in some strange "default" place.  I've yet to see
anybody using Windows put their stuff in My Documents, they put things in
folders, like "Work", "School", "The car", e.t.c.

Granted, the population studied is not statistically significant, but then
again, who has studies this issue at such length :-)?  Another thing in putting
up a save dialog as the default is because this is how Netscape 4 behaves.
*** Bug 112562 has been marked as a duplicate of this bug. ***
There's a possible security hole related to this bug. If a web page contains
e.g. a 1x1 iframe whose source is a virus, Mozilla downloads the virus to a
temporary file while prompting you were you want the virus saved...
You know, you're funny people. I'm on this list because I reported
a bug that mozilla saves temporary files in a world accessible
directory with world readable permissions, which blasts the whole
idea of privacy. I was told that my bug belongs here. And what
I see is endless discussions about usability vs. awareness and
target milestone is 0.9.8. And during all that time mozilla will
continue to save temporary files with world readable permissions.
Very funny. I realize that there is a problem of cryptic filenames
and user interaction and all that, BUT WILL SOMEBODY WHO KNOWS
THE CODE ADD A SINGLE LINE TO CHANGE PERMISSIONS PLEASEEEEEE!!!!
And then keep on discussing secondary matters. I have a machine
here which is used by 20+ users as an application server and
EVERY file to be opened by Acrobat, GV, mpg123 is saved in /tmp
with world readable permissions and is kept there until mozilla
quits. And if it crashes, they stay there for ages. But the biggest
and at the same time easiest problem to solve is WORLD-READABLE
permissions. Why does it have to wait for 0.9.8? I understand
that it can be viewed as a small part of a bigger problem, but
why wait for the bigger problem when it can be solved now?
permissions have nothing to do with user interaction. Completely
independent.

Thank you
Alex
+cc, is this a security issue?
Blocks: 109594
we currently only save the toplevel document... not the subframes, so i don't
think there is a security hole as neil describes.

alex: the bug you speak of is definitely more severe then this one... it should
not be a dupe of this bug.
*** Bug 113368 has been marked as a duplicate of this bug. ***
Did this mess all start in bug 22796 where I said, "...To answer the 
question of where to download to - a few hundred K (to a Meg) of memory could 
be allocated for the download, and if it got too large, then it would be thrown 
in a temp file. After you clicked ok - it would transfer the data to the other 
directory and continue the download."?

Not copying till the end is not at all what I intended. Many people have
suggested copying the file right after they choose the location. That IS what I
intended.

>Um silly question... is it possible for a download session to change where it 
>writes while it's downloading?

>eg. we d/l 12% to a temp file. the user tells us where to write (and it's on 
>another drive), we copy that 12% to the new file. [2% builds up], we copy that 
>2% and then update the file writer to start writing to the correct file 
>(actually, assuming we have binary seek we could probably start writing the 
>incoming data immediately where it belongs and then copy over the start buffer 
>when possible). when we're done, we kill the start buffer.

>otoh, we could just ask the os to buffer the entire file in memory until the 
>user picks a location (wouldn't this make more sense on most operating systems 
>w/ decent vm?)

Good idea. Couldn't we take advantage of a move for the first 12% if the copy-to
and copy-from are on the same disk? The bytes downloaded during the move could
be to a different tmp file. For instance file.moz.tmp.1 and file.moz.tmp.2, etc.

As for the problem of Linux users having a small tmp directory, Mac's problem of
showing files that hurt the aesthetic appearance of the desktop, and Solaris's
problem of using RAM for the tmp, being able to set the location is a good idea.
The default directory could be a subdirectory of Mozilla, but the user could
change it to any directory they choose.

I also agree with making a pref available.

I don't know what the solution for timeless's "File system returns wrong size
error", but I think that if the temp directory doesn't have enough room, it
should wait until the user selects their directory. We should accept whatever
the operating system says is available for the partition the directory is on. If
this is wrong, and the disk fills before the user selects a location, then a
message should be returned: "The operating system is returning the wrong
available space for disk ____, or the disk size changed while downloading the
file.". An example for the second part: If the user does two downloads in a row
and the second download makes the disk size change for the first download.
Should we also mention that they could try turning off the pref until more space
is available?

The maximum disk size needed would never be more than the file size. Even if the
person clicked OK on the save as dialog when the file was 90% complete and both
the temp location and save as location were on the same disk, the first 90% of
the file could be moved. This would alleviate needing to have double the file
size available on the disk as IE requires.

As for the save-to location being on a network disk in the remote (but highly
unlikely) chance that downloading is faster than copying to a computer on the
network, then the whole tmp file might not get copied over by the end of
download. In that case, copying the rest of the file might be necessary. This is
not really a big deal.
After experiencing this 'optimization' for a while, I'm convinced it is just
plain wrong.  It downloads files before I have authorized, to a hidden location
I haven't authorized, and with a name that is cryptic, so I couldn't find them
even if I was looking for them.  It doesn't show me a download progress dialog
until after I've accepted the file save dialog, so I have no way of knowing it
is sucking up bandwidth, even if I'm paying for it by the byte. It requires that
I have disk space of at least twice the file size, for the copy, and takes up
extra disk and processor time to make the copy after download. It complicates
the code too, leading to errors such as leaving large, hidden, cryptically named
files when the destination I've picked doesn't have enough room.  In this last
case, it doesn't even tell me, but falsely shows the download as completing, but
the file never appears in my chosen destination.  This behavior is just a
collection of defects and potential defects, and should not be allowed in the
product.
Most of Trudell's objections would be addressed if there was a user selectable
default download directory, and if Mozilla used the filename specified by the
webpage until the user selected a filename. Then Mozilla would immediately
replace the webpage filename with the user given filename. This way:

1. files only go where the user defines and can find them.
2. broken downloads are still identifiable by their filename (albeit the name
given by the web page in the short timeperiod where the user has not yet
assigned a name).

Also, GetRight does this excellently by giving the file an extra extension until
the download is complete (e.g., RTCWolfenstein.exe.Mozilla) A simple search for
"*.Mozilla" will reveal all unfinished/broken downloads. :)

Also, most people have much more HD space free than they would ever download
from the web. Even with DSL 100 MB will take over an hour, and everybody has at
least 200 MB free - at VERY least.

This is a cool feature because most downloaded files are in the 50k to 500k
range and these are nearly completely downloaed by the time the user has
selected a directory and filename. Can't this be worked out with the above
suggestions?
[SLIGHTLY OT]
And how about a dropdown list in the "Save As..." dialogues with quick access to
the 10 last used download directories?
[/SLIGHTLY OT]
I just filed a bug on my last idea (dropdown for 10 last used directories) - bug
115574
Blocks: 75364
There still seems to be a persistent misconception about save file dialogs, so
I'll repeat myself: They are not just for selecting where the download goes,
they are the point at which the user commits to the download. Programs have no
business downloading files before the user clicks 'Save' (unless there is a
default download directory that obviates the dialog).  This is fundamental to
the user model; failing to honor it is a breach of trust.
Amen! I thoroughly agree with trudelle's comments.
Hasn't the user *committed to the download* the moment he clicks on the download
link? 

Also, Mozilla (and any browser) "download" all kinds of files from webpages and
put them on the user's hard drive into the cache directory without the user
consenting to it explicitly. I see little harm done by potentially few broken
downloads landing in a user known place (default DL director) - but a greater
benefit on the other side of the coin.

trudelle:

what about when the file is already being downloaded? eg. click on a link... and
that link turns out to be of a content-type that the browser does not know how
to display?  we cannot suspend the download indefinitely.  nor, can we buffer
the download indefinitely in memory.  we must put the downloaded data on disk
*somewhere*.  the question then is: where?

it seems to me that you are suggesting that we should define a default download
directory (make it really easy to change from the preference dialog) and just
put the file there.

ok, but what if the user doesn't want the file to go to the default download
directory?  what are their options?  perhaps the first dialog that comes up (in
which the user can choose a handler app or a save-to location) should
automatically come up displaying the default download directory.  perhaps then
the user could click a "select download location" button to alter the save as
location for this file.  then if the user presses ok and has save-to-disk
selected, the file will simply be downloaded to the location specified in the
dialog.  under the hood, we'd have to be able to move the partial downloaded
file from the default download directory to the new download directory.

there could also be an option to make this new download directory the default
download directory.
>we cannot suspend the download indefinitely.  nor, can we buffer
>the download indefinitely in memory.

why not? that's what every other browser does, right?
It should be able to suspend until TCP/IP disconnects due to timeout, no?
That is what, several minutes?  As a user, I would not be surprised if
there was a "connection lost"-message if I waited too long to commit
the download.
When a user clicks on a link, she expects the browser to load and display the
content.  If the browser is unable to do that, it needs her permission to
download the file. For all we know, she has no other program to view that
content with, so downloading it may be an utter waste of time, disk space and
possibly money.

We should not try to discuss the design of a default download directory in this
bug report, or any other.  How about the UI newsgroup, where everyone can see it
and add their ideas?
looks like 4x reads from the netwerk 1 byte at a time at a rate that diminishes
exponentially or something like that.  eventually, it seems to fall into a
pattern of reading 1 byte every 30 seconds or so.  once the user selects where
they want the file to go, the download continues at full speed.

very interesting... i'll try to see what IE does next.

personally, i much prefer the way mozilla downloads files over 4x (with the
exception of the fact that i can't control the temporary download directory).
i like the fact that mozilla minimizes the amount of time i have to wait for the
file to be downloaded.  i don't understand why it is taboo to download in the
background.
>i don't understand why it is taboo to download in the
>background.

Many users think that the download starts after you seleced the destination.
If the user clicks on a link (he thinks it's a link to another site or a small
File) and it's a 650MB file (Iso-Image) and the user leaves the PC after he
clicked, Mozilla downloads the 650MB file in the background.
This is a problem If you have a transfer limit/month or you must pay each MB.
(many DSL/Kabel user have a transfer limit)

FYI: I like the background download feature...
ok, i just tested IE6 and it appears to behave identically to NS4x.  looks like
it has a much steeper exponential falloff though.

would anyone object to making mozilla behave identically to NS4x/IE6 as a first
cut improvement to the current situation?

eventually, i would love to see an option to always download in the background,
but i guess i can be convinced that it shouldn't be the default mode.
nope. no objection to behaving exactly like IE/4.x here. ;)
No objections here, either.  Let's just get this moving so that the original
point of this issue (not to create a file where a user might not want it) is
resolved first.  THEN we can worry about background downloads and such.
Are we talking about a "Preference" here? If yes, then go ahead to implement
both ideas and the UI to set the preference. I really can't see why it takes so
long to discuss this.
No, we are not talking about a preference, we are talking about correct behavior
versus incorrect behavior, and we don't want it wrong even if a few contributors
would personally prefer it that way.  It took a long time to discuss because no
UI design was done up front, and there should have been UE involvement early on.
The term "Correct" or "Incorrect" is subjective.. If I "prefer" Mozilla to
download file to a predefined directory without asking me again and again,
pop-up a directory selection dialog is "incorrect" in my opinion.
RE comment 169 and comment 170: Yeah, let's take the easy way out and "fix" the
first part of this bug ("Saving a file creates a file before location is
chosen") by breaking what is a really great feature; and not fix the second part
of this bug ("spool file should be moved once the user picks a filename"). I'ts
just much easier that way - NOT.

RE comment 168: 
> would anyone object to making mozilla behave identically to NS4x/IE6 as a
first cut improvement to the current situation?

I object. I definetely do not think it would be an improvement to the current
situation; rather, a very unfortunate degradation. The only real two
disadvantages we have now are: (1) That in the rare case that a download fails,
we don't know where the file is (most people have learned to delete their TEMP
folder periodically) and (2) that some #%&§ might click a download link, see the
"Save As" dialogue and walk away from his computer for a long time (how dumb is
that?) (That *rare* person is only going to do that only *once* anyhow). I just
don't see the big reason for taking away this cool feature. Most users are going
to have the "WOW, Mozilla downloads faaaast" experience. Why take that away?

Suggestion: Make this bug dependent on the "pref for default download directory"
bug (bug#?).

RE comment 173: 
How hard is it really to strech our your right thumb and tap the numeric ENTER
key? I can see this as a personal preference (definetely *not* mine), but
certeinly not "incorrect".
comment 173: adding a predefined download directory to the UI would be a solution.
comment 174: You are ignoring or failing to understand why the current behavior
is wrong.  You might personally prefer the behavior of any number of defects,
but that doesn't mean we should leave them in the product.
[OT]
RE comment 175: Peter T., how can you say I fail to understand the problem, when
in the next sentence you say:

> PT: "adding a predefined download directory to the UI would be a solution."

and in my comment I had already concluded:

> PL: "Suggestion: Make this bug dependent on the "pref for default download
directory"  bug (bug#?)."

Where is the big difference between our two conclusions? I just want to find a
solution *other* than giving up on pre-downloading.

PS. I get the impression that *maybe* you guys are arguing with "Linux logic"
(are you?) and I am thinking with "Windows logic" (I am, although I want to
learn about Linux). If that is true, then please consider that there are far
more windows users, and they should be given consideration too. If this
assumption is incorrect, then just forget this "PS" section.
[/OT]
I personally like this feature. It means I can go to grab a bite to eat, come
back and then choose where to save the file and it is already downloaded. If you
want to do a quick fix (temporarily), then feel free - but bug 22796
 will have to be reopened. I think the proper method for this is having a pref
where the user chooses whether they want to invoke this feature (automatically
off). Have the default download directory be mozilla/download or something. Make
the file filename.ext.moz.tmp or something like that, and make sure it cleans up
broken downloads. To remove a feature permenantly because it is against some
people's perception of how to download a file is incorrect imho.

Step 1 (Quick fix):
1) Use the IE/4.x curve.
2) Use filename.ext.moz.tmp
3) Reopen bug 22796
4) Make it so that when you choose the location, it copies the file over
immediately (which should be rather small because we are using that IE/4.x curve

Step 2 (Re-implement correctly bug 22796):
1) Make a default tmp directory mozilla/tmpdownload or something
2) Make it so that number 4 of previous turns into timeless's idea of copying at
the same time the file downloads.
3) Add UI pref for feature (default off)
4) Add UI pref for default download tmp dir


Note: Don't have a default download directory. It should be chosen every time
imho. It should also be what the user last chose as is the current behavior.
Don't ask what the default temporary download directory is each time. It should
be in prefs.
I have reopened bug 22796 to have this entire 'feature' removed as soon as
possible. We can then consider whether any further work is appropriate to
address the issues this was intended to resolve.  In the future, please keep in
mind that any changes to the UI should involve designers and UI stakeholders up
front, in order to avoid such a waste of time and energy.
Whoops, looks like I have acted too hastily.  Apparently, this was not just a
simple checkin by one of my engineers, as the fixed resolution on bug 22796 led
me to believe.  Obviously, I  need to understand this better, including exactly
which scenarios it was intended to improve, and how much the implementation
issues can be resolved, before we can decide on how to resolve the design
problems it caused.  I retract my implication that this was all a waste of time,
and apologize to those who worked on it to improve performance, this may yet be
salvageable.
*** Bug 116389 has been marked as a duplicate of this bug. ***
Just filed 116753 (downloading a file to a directory w/o permission silently
fails), which would also be fixed (along with probably 30+ other bugs) by
ripping this code out.
Blocks: 22796
Blocks: 65770
Sigh, sorry to keep beating on this bug, but it really actually makes my 
production use difficult at work. Let me explain:
I do some of my work on a Sparc/Solaris 7 development box with only(!) 1024M 
RAM, which is usually full of servlet and database engines. It's on a 100Mbit 
connection to the campus ethernet. Somewhat frequently (maybe twice a month) I 
need to download ISO's -- 650M files. However, unless I'm *very* quick in 
picking a save location, my /tmp (mapping to main memory now) gets all filled 
up and my machine grinds to a halt as processes swap around. In other words, 
clicking on a link can effectively halt my machine (and it'd be less fun if I 
had a piddling 512M ram). My solution is to do this work using IE4 for Solaris.

I've gathered, following this discussion, that it's debatable whether Moz 
should have the ability to download while picking a download location. It's not 
debatable, however, that in some cases this behavior causes serious problems. 
Perhaps this should be a pref, if you assume that Solaris users who download 
big files will know what's up. I doubt that this is a safe assumption, though.

;-)
RE comment #182 (downloading ISO's on Spark's):

This could be easily solved if the partial file went into the *default download
directory* or the *last used directory* (configuable in prefs) and *not into
RAM* , then the user would select a default download directory (or last used)
that had enough space.
-> 0.9.9 unfortunately :-(
Target Milestone: mozilla0.9.8 → mozilla0.9.9
After checking on this behavior a bit more, it seems we did it for parity with
IE5, but have since received mostly complaints about it from customers and
users, with comments such as those supporting it in this bug being the
exception. I also found that IE6 has dropped the behavior, presumably due to
similar complaints.  IE6 still downloads 2-4KB of the file immediately, which I
think yields most of the benefit for dialup users in the typical case where they
quickly dismiss the Save file dialog.
Maybe the user complaits result from this feature not having been advertized and
because the suggested visual feedback has not yet been implemented. The bottom
part of the ASCII art below shows how this visual feedback could look:

Suggested UI:

+- Enter name of file to save to... --------------------+
|                             ____________________      |
| Recently used directories: |____________________|\/|  | <-- bug 115574 :)
| ----------------------------------------------------- |
|           __________________                          |
| Save in: |_MyDownloads______|\/| [^]  [Favorites]     | <-- bug 115981 :)
| +---------------------------------------------------+ |
| | files (or favorites) listed here                  | |
| |                                                   | |
| |                                                   | |
| |                                                   | |
| |                                                   | |
| +---------------------------------------------------+ |
|                      _______________________________  |
| Predownloading File ||||||||____37_%________________| | <-- this bug :)
|                                                       |
| The predownloaded file will be moved to the path and  |
| filename you select once you click SAVE, or deleted   |
| if you click CANCEL.                                  |
|                                                       |
|                              [ SAVE ]    [CANCEL ]    |
+-------------------------------------------------------+

I think once users realize what is going on (and the above UI will do that),
then this will be recognized as an awesome Mozilla feature.
plairo: 2 points
1. that dialog doesn't have a place for me to pick a filename !!
2. for windows we should be using the places bar which includes the recent 
folder. (hint: one of the reasons ms did this was that the dialog would 
otherwise get unnaturally tall by default, just as yours is even without 
critical element #1)
RE timeless' comment #187:
1. (no filename selector): oops, how about this:

+- Enter name of file to save to... --------------------+
|                             ____________________      |
| Recently used directories: |____________________|\/|  | <-- bug 115574 :)
| ----------------------------------------------------- |
|           __________________                          |
| Save in: |_MyDownloads______|\/| [^]  [Favorites]     | <-- bug 115981 :)
| +---------------------------------------------------+ |
| | files (or favorites) listed here                  | |
| |                                                   | |
| |                                                   | |
| |                                                   | |
| |                                                   | |
| +---------------------------------------------------+ |
| File name:    [________________________]   [ SAVE  ]  | <-- fileNAME selector
| Save as type: [_____________________|\/|   [CANCEL ]  | <-- fileTYPE selector
|                                                       |
|                      _______________________________  |
| Predownloading File ||||||||____37_%________________| | <-- bug 22796 :)
|                                                       |
| The predownloaded file will be moved to the path and  |
| filename you select once you click SAVE, or deleted   |
| if you click CANCEL.                                  |
+-------------------------------------------------------+

2a. (places bar, tallness): The favorites button costs ZERO vertical space, so
that cannot be the reason MS added the places bar. BTW, many people have/will
not use WinXP (older versions don't have the places bar). If we limit ouselves
to the places bar, the (very) many win9x users will be left out in the cold.
Also, with current common resolutions of 800 and 1024, the current design is not
too tall, IMO.
In addition to being prohibitively complex for typical users, this dialog has
the same fundamental usability problem, in that it purports to give the user
control over whether the file is downloaded, yet it downloads the file anyway,
to some unknown filename and location without any input from the user at all,
potentially costing her bandwidth, disk space, and time spent trying to figure
out why she has no space on her temporary volume for files that she *does* want
to save.

In bug 22796 comment #30, I have outlined a possible solution involving a
default download directory, and addition of functionality similar to this to the
download manager UI.  I suggest we stop trying to subvert the normal file save
dialog with this "pre-download" concept, and explore such alternate means that
can be done by typical users who retain control over the program's actions.
If we implement the "Favorites" button, then maybe we don't need the "Recently
used Directories" portion. Then all we would be adding is basically a simple
progressbar at the bottom of the dialogue - hardly "prohibitively complex"
(progress meters are very common in a browser).

Also, we download files (to an unknown location) when clicking on links to other
 web pages, so I see no problem with downloading a file when the user explicitly
clicks on a download link (especially, since it can be easily & quickly canceled).

Ideally, the predownlooad would go to either the "default download directory
(bug 7840) or to the "last used directory". The predownloaded file could have
the filename = "URLgivenName.ext.moz". This would hardly be an unknown
location/name.

Fow the few fuddy-duddy "common users" that don't understand the benefits of
predownloading, there could be a pref to toggle it ON or OFF.

Updated UI suggestion:

+- Enter name of file to save to... --------------------+
|           __________________                          |
| Save in: |_MyDownloads______|\/| [^]  [Favorites]     |
| +---------------------------------------------------+ |
| | files (or favorites) listed here                  | |
| |                                                   | |
| |                                                   | |
| |                                                   | |
| |                                                   | |
| +---------------------------------------------------+ |
| File name:    [________________________]   [ SAVE  ]  |
| Save as type: [_____________________|\/|   [CANCEL ]  |
|                                                       |
|                      _______________________________  |
| Predownloading File ||||||||____37_%________________| |
|                                                       |
| The predownloaded file will be moved to the path and  |
| filename you select once you click SAVE, or deleted   |
| if you click CANCEL.                                  |
+-------------------------------------------------------+

PS. I know you all must be very stressed out with all the mozila 1.0 stuff.
Instead of discarding this concept, maybe it should be "futured" to a time when
more resources (and thought) can be put into it. :)
All of these points have been answered before, either here, in the newsgroups,
or in bug 22796.  We're just going round in circles.  Does anyone other than Mr.
Lairo still think that making such choices for the user is valid?
> Does anyone other than Mr. Lairo still think

I (still) think that for a certain small subset of Gecko users, under certain
circumstances, automatically "making such choices" is wanted. But not the way we
currently do it. I am very much in favor of ripping the whole thing out that's
in there currently, and implementing KISS, >>>reliable<<<, non-automatic
downloading for 1.0. We can worry about 'enhancing' it for 2.0, once it actually
works.
Peter Trudelle, I mostly agree with Peter Lairo.  I do think the priority should
be to get downloading working better first before implementing the pre-download
feature.  But the pre-download feature would be quite useful to advance users,
so long as the initial download directory could be specified (to avoid all the
problems with TMP or the default temp drive not having enough room).
Re 192:

>I am very much in favor of ripping the whole thing out that's
>in there currently, and implementing KISS, >>>reliable<<<, non-automatic
>downloading for 1.0. We can worry about 'enhancing' it for 2.0, once it 
>actually works.

Hear, hear! Making Mozilla the cleverest software in the world doesn't seem to 
be part of the 1.0 effort (as far as I've read). Making Mozilla WORK, however, 
does. 

As my teacher used to say, "Perfect is pretty, done is beautiful."
Keywords: nsbeta1
QA Contact: sairuh → benc
Whiteboard: se-radar
i don't expect to have any time to work on this before mozilla 0.9.9 :-(
Keywords: mozilla1.0
Target Milestone: mozilla0.9.9 → mozilla1.0
Status: NEW → ASSIGNED
Priority: P3 → P2
Keywords: nsbeta1nsbeta1+
To clarify what is going on in #182 (because I don't think some people
understood what was really going on...)

/tmp is mapped to memory in Solaris.

If you dump really large files into /tmp, you cause the system to thrash as your
downloaded file starts to cause other processes (including mozilla) to page to
disk, then page back (because mozilla code needs to be executed in memory to
continue the download that is chocking off the system to begin with).

This is why I have argued on occassion that what we need is a single
conversation where a download design is built from the bottom up. And by bottom
up, I mean that we make a couple statements about how we understand memory,
disks, cache, processes and error states would relate, and from there decide
what kinds of ways we would add sensibly add files while browsing. From there we
would proceed to prioritize features and then figure out how to present them to
the user.

It is too late to do a highly-robust, super-cool, frilly download manager. We
probably had time, but nobody really picked up the ball and had the right
conversations at the right time. Mozilla 1.0 is closing, and
trudelle@netscape.com's approach, which is to get something that gets files to
disk in a conservative but working manner is the way to go now. 

I urge individuals reading this (and related bugs) to use some retraint about
commenting in ways that eat up trudelle's bandwith, but do not help us get 1.0 done.

If we could find a place (newsgroup) to start a post-Mozilla 1.0 download
manager conversation, lets do it. If we are going to do this in bugs, lets mark
them clearly as 1.0+, and also people that agree but have no real insights to
add in comments need to learn to use the voting system. 

Mozilla contributors are working code that works on more than the specific
system you have. (If you can't live without a feature that works for you but
would torment other systems, nobody is going to stop you from downloading the
code and building your own features for your own system). Many of these bugs are
features that just don't make sense at the OS level (for example, if I remember
correctly, most systems would not easily be able to continue to pre-download a
file and move it if the destination was a different filesystem). 

I'm sorry that mozilla and bugzilla did not provide a better format for
discussing this feature. Everyone should have a voice, but mozilla is not as
much a place about free expression as it is about free (and implicilty) working
code.

At this point, we know what we can and can't try to do. Please, "do the right
thing" and keep reality in mind as we work on this. 

(In that spirit, I just deleted about 3 ranting paragraphs from this comment.
They were persuasive, somewhat humerous, but possibly upsetting to small
percentage of readers.)
I attempted a 150+M download recently and had a major hassle.... it saved to a
temp location on drive C which had insufficient space.. and after 6Hrs (150+M
download) it ran out of space for the silly temp file and bombed....

If using this terrible temp file kludge is absolutely necessary... then pls
consider...

1) I have my diskcache set to a specific location for temporary data for all
apps, so that I can one shot delete orphaned garbage... mozilla ignores this
setting.. why not use that....

2) check file space available before starting.....

3) why not create the temp file in the same location as the download
destination, then only a rename is needed, instead of taking minutes to do a
full copy....

4) why not ask the user at the very first time he downloads sonething, where
their download location is... ie start up with a blank location, then prompt the
user, and save this for ever more...

But best of all... get rid of the need for the temp file all together....
User selectable...?

Thanks for your work on Mozilla, it is shaping up really well.....
I do appreciate your, and everyone elses efforts...

regards,
Suggestion for changing the downloader completely, so this is no longer an issue.

Why not make a "mini" downloader EXE that mozilla launches after determining
file content...
All it would do is take a URL , ask for a location (if not specified already)
and fetch the URL.

basically, using a "helper" to download means:

a) it's easier to extend the downloader (some other program handling binary
downloads by a pref)
b) mozilla doesn't have to worry about reconnecting as the external prog does
connect and download.
c) the helper could be used by other programs
d) If mozilla crashes on some other content, the download would continue (BIG
BIG FEATURE!)
e) the helper wouldn't start downloading ANYTHING until the location was picked
(this issue)

I think a small program using the Mozilla DLLs could easily prompt for a
location, using the system save dialog, and display a download progress indicator.
> e) the helper wouldn't start downloading ANYTHING until the location was picked
> (this issue)

Clarification: No, this bug aknowledges predownloading as valuable (it's
implied), since it requests that the spool file be moved once the user picks a
filename (see the "summary").
peter, pass me that bong when you're done with it. we've agreed on no such
thing. we're still arguing over the first part ("saving a file creates a file
before location is chosen").
> All it would do is take a URL , ask for a location (if not specified already)
> and fetch the URL.

Totally unacceptable.  At the least, the following information is needed to
properly retreive a generic resource:

1)  URL
2)  postdata
3)  relevant cookies
4)  HTTP Auth information
5)  SSL certificates

Passing all this to an external process (which, note, must handle SSL and so
forth) is hard and likely a security issue.  Oh, and reposting is often _not_ an
acceptable thing to do when using systems that generate documents to be saved in
response to post data...
As a note, the spool file that gets moved solution has one issue with it -- bug
124307...
Since this bug is seemingly not moving, I'm at least going to toss out a 
concrete proposal to try to get in for everyone to pick on.

Predownloading (bad term) is the downloading that is done between the time when 
a user clicks on a link and the time when a user tells Mozilla where to save 
the data.  Not predownloading causes Mozilla to make a connection when the link 
is clicked and to possibly make a new connection when the save location is 
chosen.  As anyone who has worked on View Source can tell you, that's a BAD 
THING.  Predownloading unsupported data types is not in itself a bad behavior 
since if the data had been a supported type, it would have been entirely 
downloaded with no further intervention by the user.  Currently, the solution 
is to spool data to a temp file and move on download completion.  That causes 
performance problems on machines which handle large temp files poorly and data 
loss if the spool area is exhausted or the spool file prevents the actual file 
from being created.  No spool file solution can fully solve these problems 
because they depend on conditions independent of Mozilla's operation.  The 
speed and available space of your spool area can change quickly and without 
notice.  Other problems (security, usability, etc.) with the current solution 
have also been reported.

Proposal:

Adopt a default predownloading behavior similar to IE4.x/IE6.x/NS4.x.

When a link is clicked, determine whether the data is a supported type in 
Mozilla.  If not, begin downloading data into a memory buffer.  The memory 
buffer supports emptying (reading) as well as writing with concurrency control 
as necessary.  The buffer has some way of indicating whether the transfer has 
been completed.  Whenever the accumulated buffer is larger than MAXBYTES in 
size, download 1 byte per THROTTLE seconds.  Whenever the accumulated buffer is 
no more than MAXBYTES in size, download at full speed.  A value of MAXBYTES and 
THROTTLE indicating unlimited should be designated.  If the save is aborted, 
the buffer is immediately freed.

The default buffer reading strategy is to do nothing until a save location is 
selected.  Once a save location is selected, read from the memory buffer when 
bytes are available and write to the save location until the transfer is 
complete.  MAXBYTES of 128K would fully buffer most files and should not cause 
undue delays when downloading larger files.  THROTTLE of 3 would cause memory 
usage to be less than 1M per month and is less than the timeout of any 
reasonable protocol.  

As an option, the buffer reading strategy, MAXBYTES, and THROTTLES could be 
changed.  This may include strategies with potential performance or data loss 
problems.  At least one safe strategy should exist and be chosen as the default 
such as above.  Other strategies could be suggested through separate bugs.

User customizable values are nice but optional.  Even if the save location is 
slow (less than 24 bps for the default!) or the save location box is left open, 
continuing to accumulate the buffer is not a problem as the memory footprint of 
Mozilla would not be significant for years.  Impact on throughput would be 
noticeable only when both the location being read from and written to have 
speeds comparable to a memory-to-memory copy.
Blocks: 129923
*** Bug 129834 has been marked as a duplicate of this bug. ***
blake: does the download manager code have any effect this bug?
At least from my end, doing my own builds, DL manager doesn't work and DL still
doesn't work.

I'm starting to use curl for all my dl needs; copy link and send to curl.

How hard would it be for you guys to just redirect your dl process to curl, anyway?
Please read comment 201 carefully.
Whiteboard: se-radar → se-radar[adt2]
*** Bug 133404 has been marked as a duplicate of this bug. ***
Switching the two parts of the summary around so I don't have a knee-jerk
reaction of coming to this bug and invalidating it each time I see it...
Summary: Saving a file creates a file before location is chosen; spool file should be moved once the user picks a filename → Spool file should be moved once the user picks a filename (saving a file creates a file before location is chosen)
That's really an attempt to change the sense of the original defect report,
which is that a file is saved without the user's knowledge or consent. 
Dismissing the save dialog is not just picking a name and location, it *is* the
Save command in this case.  The difference between supported and unsupported
types is an important distinction when you're filling up someone's temp
directory with data they don't know about and for all we know have no way to
use.  A scheme such as described in comment #203 might be an acceptable
compromise, assuming all the other defects associated with this behavior are
fixed, but it still makes the flawed assumption that a Save command has been
issued when it has not.
For crying out loud. I didn't try to change the meaning of anything. I merely
switched the two independent parts of the summary around, from the format
"cause; solution" to the format "solution (cause)". I have no interest in
rehashing the issues that have already been beaten to death in this bug. If you
want to discuss
this, take it to n.p.m.ui or n.p.m.netlib, where it should have been brought up
months ago.
You took what was originally the entire summary of this bug, and moved it to a
parenthetical comment, prefixed by one proposed 'solution'.  The summary doesn't
even sound like a defect report anymore so much as an enhancement request. 
> The summary doesn't even sound like a defect report anymore so much as an
enhancement request. 

To me (at least), this bug *is* an enhancement request. Please, off to newsgroups.
"Off to newsgroups" is not a valid response.

As one of the users who opened a problem that folded into this bug, I can say
the following:

1) The proposed solution addresses most of the nastiness that caused me to open
my version of this bug.  In particular, downloading ISOs using the badly flawed
implementation in place last time I tried it, when whatever filesystem that
Mozilla chooses to load to can't hold the ISO, is doomed to failure, even if the
destination I chose could hold the ISO.  Also, trying to figure out what's
happening when mozilla uses some random filename, instead of the one I chose, is
more dificult.  The proposed solution does address these issues.

2) I'm not convinced that the proposed solution is wholly correct, especially if
this is to be done soon.  I think starting to download, at least more than a few
frames worth to maintain a connection, possibly even completing the entire
download for small files, before the user has confirmed download, is probably a
mistake, since you are doing something the user hasn't actually finished asking
you to do.  I have previously maintained that the benifit is small in any case,
especioally considering the debating time that has been spent on it.  Anyhow,
thanx for the work on mozilla, it looks better and better (I usually use the
Galeon variant, since it make easier in the user interface a few things I like
to do (disabling popups, disabling/enabling images, scaling text regardless of
brain-dead webmasters' [is there any other kind?] use of the font tag with
absolute size specified...)
*** Bug 140817 has been marked as a duplicate of this bug. ***
no hope of getting this fixed in time for 1.0
Target Milestone: mozilla1.0 → mozilla1.1alpha
*** Bug 144253 has been marked as a duplicate of this bug. ***
Target Milestone: mozilla1.1alpha → ---
mass futuring of untargeted bugs
Target Milestone: --- → Future
Mass removing self from CC list.
Now I feel sumb because I have to add back. Sorry for the spam.
If this bug is really just about *where* the spool file is located (which 
sounds like, in terms of this bug, downloading before the user has chosen what 
to do with the file is a forgone conclusion), I will go ahead and reopen bug 
133404 which is about the fact that downloading is happening before I have 
picked what I want to do with the file.
QA Contact: benc → sairuh
QA Contact: sairuh → benc
What is this bug for? It has two completely different summaries. One says the
spool file should be moved. One describes auto downloading. Those are two very
seperate things. Which is considered the bug? Have meetings occured regarding
how to go about resolving the mass of (auto)download bugs?
as I understand it, the main issue is: after the user picks a filename
(including location, of cause) the retrieved data should be written to this
file, even if the download is not completed entirely. (so the user could watch
the file grow and probably resume a canceled download - but this resuming would
probably be another bug) it's just for that the user can observe what mozilla
does, while it's still in progress. therefore things would be more convinient
for users.

then we're discussing if mozilla should start downloading the file to a
spoolfile *before* the user picked a filename (at this point mozilla does not
know where to save the file). resolving this issue would make things easier. if
you don't have a spoolfile, you don't need to move it but can start downloading
to this file directly.

but hey guys. while i'm on it: i think, we should be implementing the move of
the spoolfile anyway! this is what this bug (or what's left of the issue) is all
about in the first place.

in another bug could track the preference stuff, if the user decides downloads
should start before a filename is picked or not. this second issue really should
be carried to another bug, leaving the primary issue to be fixed more rapidly.
*** Bug 133404 has been marked as a duplicate of this bug. ***
*** Bug 156799 has been marked as a duplicate of this bug. ***
Can't someone just pick an option and implement it?  Then if people don't like
it, they can complain about something new.

But at least in the meanwhile, this two year old bug can be closed.

(IMHO, of course.)

Dan.
Hi,
this bug should be regarded as 'critical' due to blocking e.g. ID:69938 which is
critical itself.
Best regards, Peter
Compare bug 164433. There, I argue that it's a bad idea in the first place to
make any network request before asking for a filename, at least on Unix.
I'm not sure that makes sense.  We don't know if we're going to open or save 
the link until we make a network connection and find out what type the content 
is.  How can we ask for a file name before knowing if it's going to be saved?
We need to determine if we can open it, but the dialog isn't just 'asking for a
filename'; we don't know if it is going to be saved until the user okays it.
> we don't know if it is going to be saved until the user okays it

Why else would a user click on a link to a file, if not to download or view it?
Because they thought it was a link to something else. Happens to me all the
time. I see a link that says, "read more" and I click it. Then it turns out to
be a Word document or something. Also, people click file links to see how big a
file is. I do that occasionally.
Right, they click on links expecting the browser to open them. When it cannot,
it needs the users permission before downloading something that the user may
have no other way to open.  
If you are talking about my comment, please first read the bug I cited. There, I
talk about Save Target As Link and Save As (in the file menu, for displayed
content). In both cases, it is already sure that we want to save to disk.
Ben, in neither of those cases is there a spool file involved.  The spool file
is only created when we go through the helper app "what do I do now?" dialog. 
In other words, that has nothing to do with this bug.
Attached patch step 1 (obsolete) — Splinter Review
This implements step 1 of the "Make this code behave" N-step program.  What
this very rough patch does:

1)  Makes the helper app dialog modal to the browser window involved.
2)  Makes the filepicker modal to the helper app window (if it exists;
otherwise
    it's modal to the browser window as currently).
3)  Moves lots of code from nsHelperAppDlg.js to helperAppDlg.js and has
    helperAppDlg.xul include the latter and call that code (this step is
    semi-optional, but I wanted to separate the component that implements that
    interface from the dialog it pops up.  They were far too intertwined in the

    old world).

Effects of this patch:	Pre-downloading is basically eliminated.  I opened up a
largish remote (http) PDF and waited 20 minutes. The temp file size was 0.  I
then clicked OK in the helper app dialog and the download completed and opened
in my helper app.  Looking in my server log, one single connection was made to
fetch that file (and maintained open for the 20 minutes in question).

Remaining issues:

1)  Decide how much of the infrastructure to support async pre-donwloading we
    want to keep (for embeddors' sake, if nothing else).
2)  Fix nsExternalAppHandler::OnStartRequest to not create the temp file until
    _after_ the dialog has returned (separate the "default filename" code from
    the SetUpTempFile() function).
3)  Fix nsExternalAppHandler::OnStartRequest to always content-decode data
    that's going to a helper.
4)  Clean up the nsHelperAppDlg.js/helperAppDlg.js code. I just moved it
    wholesale for now, commenting out random bits, putting random stuff on the
    window object, etc.
5)  Clean up the way we interface with preferences; hopefully we can pass in
    just the nsHelperAppDlg object and not the window object we're passing
right
    now....
6)  Test like crazy.  I've tested the basic functionality of the helper app
    dialog with the changes I've made so far, but more testing would be
    incredibly welcome.
I forgot remaining issue #1.5.  Now that we know where we're going to be saving
to (if we're saving), we should start spooling to the correct directory.  What
filename should we spool to?  The "final" filename?  The "final" filename with
".moz-download" appended (a la ".getright")?  A salted filename?  Should this
different on different platforms?
Oh. And before we restart the years-long debate, anyone suggesting
pre-downloading had better have a good solution for handling pre-donwloading of
content-encoded content (possibly with multiple encodings).  Such content should
_usually_ be decoded before passing to helper apps and _usually_ not decoded
when being saved.
re comment #237: I dont see a reason why the filename used should not be
the final one (i.e. the one the user specified). After all, that is what one
would expect in the first place. It would also make many things easier:
e.g. checking a large mp3 or movie download to see if it should be continued
or aborted would be easier; the correct name would be remembered in DM,
aborted downloads can be continued using other managers like wget more
easily etc.
Johann, from what I've read somewhere, the idea of a different filename was to let people know when a download is truly finished so they don't try to use a file that is only partially downloaded.
reasons to claim ownership for files we haven't finished downloading:
* parity: (beos)net+, (mac)icab, (mac)ie, (win)getright
* icon handling to show progress
* useful behaviors for properties and opening the document before the download
completes
* resume - if mozilla dies (crash, forcequit, os crash, powerfailure) and the
user tries to double click the partially downloaded file they get mozilla which
tries to find out where the source is and tries to resume it (bug 18004 is the
only reference in the blocking list), instead of the final application yelling
and screaming about a broken file.

for windows, the extension .moz-download would allow us to hook up an icon
handler for the file so that we could provide a progress indicator as part of
the icon and for resume/progress/properties if someone tries to open it too early.

for beos, i'd suggest we write the correct filename with the wrong mime type.
I'm not sure if we can use the right mimetype +";creator=mozilla" or if we
should just use a constant mime type (something like
application/octet-stream;creator=mozilla) but we should look at how NetPositive
does this. again when we finish, we fix up this magic.

for macos, the creator should probably be MOZZ until we're done writing the file
(check MacIE), I think that handles the same issue as .moz-download for windows.

for linux, as usual, I don't really want to think about it, but I'd suggest we
follow windows, the reasons still apply although convincing associations to work
is someone else's problem.

for os/2, i'd imagine some cross between the preceding (iow, ask mkaply)

for photon, i'd imagine some cross between the preceding (iow, i'll ask
irc.joher.com #qnx when the time comes). We can check what QNX Voyager does.
note that QNX users might be more likely to use QNX Voyager w/ mozilla_server
(replacing voyager_server) than to use the mozilla front end, so whatever we do
has to be flexible enough so that an embedding system can choose (not) to use it.

the fun platform is macosx (jaguar), because some of the documentation apple
recently published was basically targetted at mozilla. i think that we'd do the
following:
* if the user hid the extension, we replace the extension with .moz-download
until we finish, at which point we replace .moz-download with the real .ext
* if the user did not hide the extension, then we use .ext.moz-download and hide
the extension until we finish, at which point we remove .moz-download and unhide
the extension

i'd suggest that we address the above in a bug per toolkit and a meta.
I was able to apply the patch and spun a build.  Mozilla appears to do no 
predownloading to disk.  There are problems.

1) Mozilla does appear to still be predownloading something to somewhere.  I 
see 10s of KB being transferred after the helper app dialog appears lasting for 
about 20 seconds.

2) Downloading of executables completely broken.  The helper app dialog appears 
like this:

----------------------------------------------------------
Downloading #1
----------------------------------------------------------
You have chosen to download a file of type: "#1" [#2] from


What should #1 do with this file?

( ) Open using #1                             [Choose...]
( ) Save this file to disk

[ ] Always ask before opening the type of file
-----------------------------------------------------------
[Advanced...]                           [  Ok  ] [Cancel]

With neither option selected.  Selecting save and clicking ok results in no 
file transfer.  Error on console:

JavaScript error:
chrome://global/content/helperAppDlg.js line 12: nsnull is not defined

3) After sitting for a few minutes at the helper dialog, the download fails to 
begin after clicking save.  Download dialog appears but no data is ever 
transferred.

4) After dismissing a download dialog, helper app dialogs for random files I've 
tried to download from that server in this session appear.

5) JavaScript strict warning:
chrome://global/content/helperAppDlg.js line 166: reference to undefined propert
y window.choseApp

6) JavaScript strict warning:
chrome://global/content/helperAppDlg.js line 276: reference to undefined propert
y window.givenDefaultApp

7) Not sure if this is yours (when clicking advanced button in helper dialog):

WARNING: waaah!, file x:/mozilla/content/xul/document/src/nsXULPrototypeDocument
.cpp, line 782
JavaScript strict warning:
chrome://communicator/content/pref/pref-applications-new.js line 120: function o
nOK does not always return a value

WARNING: waaah!, file x:/mozilla/content/xul/document/src/nsXULPrototypeDocument
.cpp, line 782
JavaScript strict warning:
chrome://communicator/content/pref/pref-applications-new.js line 184: function o
nOK does not always return a value

JavaScript strict warning:
chrome://communicator/content/pref/pref-applications-new.js line 51: reference t
o undefined property window.arguments[0].chosenApp

8) JavaScript strict warning:
chrome://global/content/helperAppDlg.js line 276: reference to undefined propert
y window.choseApp
As a user I _want_ the spool file to keep downloading. I have plenty of disk
space (gigabytes) and I often take a long time to decide where to download a
file (sometimes minutes) during which the file has plenty of time to finish
downloading. We've already covered ways to address the problems this raises:
   * Spool. Once a download location has been picked, move the file there and 
     continue downloading to that location.
   * If you get near to running out of room on the download disk, stop spooling
     and wait.
(Another reason we have to keep downloading is that you don't want to let the
connection time out.)
Some do, some don't. And doing background spooling has security and disk space
issues. I don't see why we can't accumulate a fair size chunk in memory (several
megabytes) to avoid the disk spooling issue.
how about this:  let's move forward w/ bz's patch because it does indeed fix a
lot of very serious bugs.  then, let's discuss how best to support
pre-downloading in another bug :-)
OK.  Responding to comments one at a time:

Comment 239: see comment 240.  You don't want to open up a partially-downloaded
Word file...  (again, we may want this to be OS-dependent; on Unix the
expectation is that the filename you chose is the filename used, period).

Comment 241: agreed that separate bugs are a decent idea.

Comment 242: 

1)  We may still be downloading once that dialog comes up, but that's because
    the data is already in the pipe, so to say.  we will only do it for a little
    bit (20s sounds about right) and cache it in memory.  Then we'll suspend the
    socket and let the TCP stack maintain the connection as best it can.

2)  Thank you!  The executable thing is really a Windows-only check, which is
    why I did not catch the typo.  Change the "nsnull" in line 12 to a "null".

3)  Ugh.  That part is bad.  What was the server involved?  I was trying to
    cause that to happen but failed; it's likely very server-dependent...

4)  I won't worry about the strict warnings for now... like I said, this was a
    very preliminary hack.  Most of those variables will get moved around a bit.

Comment 243:  You didn't read comment 238.  Read it.  Then answer my question.

So in summary:  When does this fail to work completely (per comment 242)?  What
can we do about that?
About pre-downloading: why not assume that the file will be saved (more
conservative) until the user decides what he wants, then when the file gets moved
(iow, when a decision is made about saving vs. opening), do any necessary decoding?
Having spoken about this with bz on IRC:

There are two totally separate issues here which an architectural problem is
incorrectly forcing us to consider together: the UI, and the backend.

The two should be separate.

The UI issues involve the parenting of dialogs and windows, and their modality.
The filepicker app should have the helper app window as parent, not the browser.
The helper app window should not have a parent, and thus not be modal -- it
should be a sibling to the browser windows, so that closing a browser window
will not close the spawned helper app windows. The filepicker should be modal to
the helper app window. All of this is UI and should have no bearing on what
happens in the background.

The backend issues involve predownloading and content decoding. The file should
be downloaded to the temporary folder straight away, and continue doing so in
the background while the dialogs are up. This should all happen on a separate
thread from the UI. Once enough of bytes have been downloaded to decide what to
do with the file, if it is found that we need user intervention, then you spring
open a helper app dialog. Note that downloading should continue, at least while
there is plenty of free disk space. Once the user has picked what to do, or once
we know what to do from having looked at the headers/content downloaded, then we
should move/rename the file to the appropriate directory, and if appropriate
apply any content decoders to the file. This may mean moving the content
decoders out of the HTTP code, but that will have to happen anyway since other
protocols have content decoding requirements and we don't want code duplication.

The security issues are moot (the same problem exists with all kinds of
downloads, not just predownloads), the content decoding issues are moot (with
this system we do it all afterwards anyway) and the running-out-of-diskspace
issues are moot (you stop downloading before that becomes an issue). In the
meantime, the people who are paying for their connection by the second have not
wasted entire minutes while trying to decide where to save their ISO.
My proposal above would also fix the following bugs:

   bug 109594, `Reveal Location' button doesn't work while file is downloading
   ...because by the time that button appears, we'll have moved the file to the 
   right place

   bug 103766, Browser hangs after downloading a large file
   ...because that work would happen on a separate thread (and could even be
   given UI like in IE)

   bug 132702, Mozilla doesn't display image in helper app
   ...because we wouldn't apply the content decoders until after we know what we
   plan to do with the file

   bug 124307, Awful permissions for file saving as of recently
   ...because you can just reset the permissions after moving the file

   bug 88566, [RFE] display partially downloaded file in target location
   ...because the file would be moved as soon as we know the target location

...and probably several others for the same reasons.
Comment 247: because I don't have the time, knowledge, or patience to rearrange
necko to expose the content-encoding stuff.  So I'm taking the "correctness
first" approach here. We currently compeletely flub downloads or opening things
in helper apps in many situations.  We should not.

Comment 248: Every Windows user I asked said that their expectation (on Windows)
             is that the helper app dialog is modal to the browser.  My
             expectation (on Linux) is the same.

Comment 249: Would not fix bug 124307, for the reasons described in that bug (it
             would be functionally equivalent to my existing patch in that bug,
             which suffers from a security-related race condition.

I agree that we should have a cleaner separation between UI and back end.  My
proposed fix for this bug will take the first steps in that direction.  Again, I
would rather have us behaving correctly than "conveniently" in this instance. (I
quote "conveniently" because imo being unable to save things to disk or open
things in helper is _incredibly_ inconvenient).

Back to the technical issue at hand.  Under what circumstances does this lead to
dropped connections?  Until that issue is resolved, everything else is moot.
I saw silent server dropping with both ftp://ftp.mozilla.org and 
http://java.sun.com.  It's not something I can consistently get to happen 
though.

Changing nsnull to null does clear up the problem with downloading executables 
on Windows.

Also, to chime in on the modality issue- I could see the helper app dialog 
being modal to the content window that spawned it, but the dialog is globally 
app modal.  I can't, for instance, close the browser or switch tabs with a 
helper app dialog open.  That makes no sense in terms of parenting.
It's modal to the window that spawned it... so you should be able to use other
windows (but yes, you will not be able to use tabs in the same window or close
the window).

I'll do some testing with those sites; thank you.
bz wrote:
> Fix nsExternalAppHandler::OnStartRequest to always content-decode data
> that's going to a helper.

and later:

> content-encoded content (possibly with multiple encodings).  Such content
> should _usually_ be decoded before passing to helper apps and _usually_ not
> decoded when being saved.

(Although I am not sure how this is relevant to this bug:)
That seems to contradict each other. "always" or "usually"?

First, did I understand it correctly that content-encoding is e.g. gzip when a
tar.gz files lies on the server's harddisk and is delivered with HTTP
Content-Encoding: application/gzip or however the syntax is?
What about transfer encoding (the file is being requested as .ps and the server
compresses it with gzip on the fly)?
(Please correct me, if I misunderstood somemthing.)

It is a tough question, if that should be decoded (uncompressed) by Mozilla
before handing it to a helper app. I think that *usually*, content encoding
should be preserved, while transfer encoding should not. There are other cases,
though.
If I request a tar.gz, I most likely don't want it to be touched by Mozilla. If
I want a requested ps.gz (.gz appears in the URL) to be uncompressed depends on
my PS viewer.

There might even be cases where I don't want the transfer encoding to be
touched, but the helper app then of course needs some way to figure out that the
file is encoded(compressed).

bz wrote:
> Every Windows user I asked said that their expectation (on Windows)
> is that the helper app dialog is modal to the browser.  My
> expectation (on Linux) is the same.

Yet, my expectation is also that a download spawn from a certain browser window
will not terminate unless I close the windows directly related to the download,
not the browser window. I often open a browser window, start a download and then
close the browser window. I yelled at Opera Linux when it quit after I closed
the last browser window with the download manager still open and downloading,
i.e. breaking my downloads.

> Under what circumstances does this lead to
> dropped connections?  Until that issue is resolved, everything else is moot.

Not necessarily. If all fails, you can still do a HTTP HEAD, open file picker,
then do the real download, not?
Ben, the theory is that .tar.gz would be delivered with content-encoding and a
PDF compressed on the fly would be delivered with transfer encoding.  The
practice is that no browser or web server supports transfer encoding (vicious
cycle), so both are sent with content-encoding in the real world.  So while your
comment is correct that content-encoding probably should not be decompressed in
practice that will break lots of things.

That's where the "usually" creeps into what I said.. We can't tell which the
site meant without some guessing... In the helper case, we must always decode
because we picked the helper based on the decoded type.  Of course whether we
wanted to pick that helper is  different issue (and a separate bug should be
filed at some point).

HTTP HEAD is so unreliable as to be worthless.  This is even true for GET
requests, and don't even mention POST (which are becomming more and more common
as a method of kicking off downloads).

Also, keep in mind that we can't do an HTTP HEAD on _every_ link click, which is
what you'd need to use HTTP HEAD here.
------- Additional Comments From bzbarsky@mit.edu  2002-09-10 22:20 -------
> The practice is that no browser or web server supports transfer encoding
> (vicious cycle), so both are sent with content-encoding in the real world.

Yes, I know that reality doesn't match the specs here.

Curiosity: Is it possible for Mozilla to support transfer-encoding correctly, so
that we can get out of the cycle as far as Mozilla is concerned?

> That's where the "usually" creeps into what I said.. We can't tell which
> the site meant without some guessing... In the helper case, we must always
> decode because we picked the helper based on the decoded type.

What, if I have ark (that's how it's called, IIRC) sat up to handle |tar.gz|s?
Hopefully, you won't fill up my /tmp with an uncompressed tar. E.g. Mozilla
source: compressed <30 MB, uncompressed >200 MB, not funny.
Based on a 30-second glimpse at the http channel implementation, it should not
be too hard to support transfer-encoding... file a bug and cc me?  ;)

> What, if I have ark (that's how it's called, IIRC) sat up to handle |tar.gz|s?

You can't.  You can have it set up to handle application/x-tar or
application/x-gzip.  In the former case, we need to uncompress, no?  In the
latter case, we would either not uncompress (if that's the content-type the
server sent) or not pick up "ark" as a helper.
> support transfer-encoding... file a bug and cc me?

Bug 167905.

> You can have [Ark] set up to handle application/x-tar or
> application/x-gzip.  In the former case, we need to uncompress, no? In the
> latter case, we would either not uncompress (if that's the content-type the
> server sent) or not pick up "ark" as a helper.

Apache's default config seems to be:
srm.conf:
AddEncoding x-gzip gz
mime.types:
application/x-gtar				gtar tgz taz
application/x-tar				tar
#  Note: Compression schemes like "gzip", "bzip", and "compress" are not
#  actually "mime-types".  They are "encodings" and hence must _not_ have
#  entries in this file to map their extensions.  The "mime-type" of an
#  encoded file refers to the type of data that has been encoded, not the
#  type of the encoding.

which I interpret, that |.tar.gz|s will be sent as mimetype "application/x-tar",
content-encoding "x-gzip". Which means that you would decompress on the fly,
store the uncompressed file in /tmp and open it with ark, which is the default
handler for "application/x-tar" per mailcap (I assume). Meaning that
a) my space on /tmp is wasted
b) I can't easily save the file as tar.gz once opened by the helper app unless I
recompress it manually.
Apache's default config sends:

Content-type: application/x-gzip
Content-encoding: gzip

For .tar.gz.  We have code in http-land to deal with this crap (we basically
ignore the content-encoding in such cases).

There is no default tar handler in mailcap (at least on RedHat 7.2; nothing
stopping people from shipping that I guess).  And this is why we always prompt
the first time a helper from mailcap is used for a given content-type -- the
helper may be wildly wrong (need I mention that "lpr" was the default
application/postscript helper for years?).

In any case, could we take the discussion to the newsgroups if we're not talking
about things directly relevant to this bug or this patch?
Blocks: 152275
Whiteboard: se-radar[adt2] → se-radar[adt2] (py8ieh:248)
Comment 248: Once we know that we cannot handle the file, we should stop
downloading it until the user decides what to do.  We have no business using
their bandwidth and disk space downloading content that we cannot display and
they have not told us to save.  The click was permission to display, not to
download.
comment 260 is incorrect. A click on a "DOWNLOAD" button or .EXE, .ZIP, .MP3,...
link almost always _IS_ permission and intent to download.
Peter Lairo, it is odd to say that the opinion stated in comment #260
is "incorrect". Unless you can prove that it is wrong for all users,
all you can do is disagree. I, for that matter, disagree with *your* view.

There are many reasons why a user might back off from downloading:
the file is already there, the file is not what was expected (oh its 
a dialer.exe and not a cutie.jpg).
I fully support Peter Trudelle's opinion that we should not waste bandwith
or space until the user knows and approves that the download starts. 
(speaking of which, the download manager should show additional info
to make that decision easier, like mime type, and file size, if available).

So,since there are different views possible we should try to compromise
instead of qualifying another person's opinion as "incorrect" - and your
own one as "correct", by the law of "tertium non datur".
OK, the statement is "mostly" incorrect because it claims "The click was
permission to display, not to download", where in reality it *usually* is
permission to DL.

> There are many reasons why a user might back off from downloading:
> the file is already there,

That is a *very rare* case that a user clicks a link to a file he already has.

> the file is not what was expected (oh its a dialer.exe and not a cutie.jpg).

That is yet another *extremely rare* case (NEVER happened to me).

Can't you provide any example(s) that would represent the *majority* of cases,
that would justify removing this useful feature? No? (latinum est arrogantum)
oh, I agree that those cases I mentioned occur in the minority of the
download requests. However even if they never occur it does not prove
that simply clicking on the link already should be regarded as the permission
to start the download. It also depends on personal taste, but usually it is
a request to bring up the save dialog, nothing more. Clicking the OK
button then is the permission to actually start the download. 
Note that this protocol is nearly used everywhere: an ordinary "save" dialog
in practically all applications will do absolutely nothing until the user
presses OK. Practically all other dialogs will not do anything until a
user confirms with the OK button, so having it work differently here is
to say the least, different and unexpected.
I really dont see much usefulness of the feature when compared to the
problems that arise - after all, all you gain in speed is those few
split seconds of time between clicking the link and clicking OK.
It doesnt seem right to me to accept all the trouble this is causing -
from a function not working as expected by the user to the space and file name
issues - just for this.
To comment 261:  I woudln't have guessed that you could tell so much context
when starting your download, such as clicked on a download button.  Just because
the href points to a exe or mp3 or zip doesn't mean that the hyperlink itself
displayed that information.  A lot of times, I click on a link thinking that it
will take me to a page which will have the download listed with a file size when
in fact it is the file to be downloaded.

To comment 263:  I guess you'll have to define *very rare* since this happens to
me quite frequently when I'm trying to download the next version of something. 
In fact, I use the notion that the filename will already be present if I have
already downloaded that version and I let the system match the name for me. 
Maybe it's because I have a bad memory, but I wouldn't classify it as rare at all.

Is this the majority of cases?  No, but I wouldn't call it rare either.  I would
also tend to debate the usefulness of the feature.  What isn't rare is one group
of people finding something to be useful while others call it a bug (Witness the
ordering of the "open in New Tab" .vs "open in New Window").  Personally, I
don't find the preload useful or buggy.  I just didn't like Mozilla filling up
my tmp space when I didn't want the file to go there.
Comment 261:  Do you really think it is possible to divine the user's intent
from the content that they clicked on?  We know our own intent when we click,
but I don't understand how the code could know the user's intent when it doesn't
even know how to display the link target, which is the expected fundamental
behavior of a browser when a link is clicked. 
Peter Lairo, read comment 238 and shut up unless you have a technical solution
to the technical problem I mention there (which absolutely has to be solved).

Could we lay off the flaming and design something that works?  The current
approach to helper apps is why we are, after 3 or 4 years, without a reasonably
functioning helper app system.

Again, I will fix correctness issues first and do nice features later.  If you
want the nice features, code them.
qa to sarah. (you moved it to File Handling, but you've given me two times now...)

If this is in File Handling, then I'm done. I am already tied down in helping
out with some Doc Shell and URL bugs that directly affect Networking, so I'm not
going to be doing other non-Networking work for now.

bz: my suggestion to you is to write a brief 2-3 page design document on
mozilla.org, that addresses these issues everyone keeps "realizing" over and
over, and just post a URL to the document everytime someone repeats a previously
discussed item. 

I had previously thought I had time to help with this, but I don't in the near
future.
QA Contact: benc → sairuh
-> jimmylee for now. i don't have the cycles at present to work on all of the
aspects of this (other than the frontend, perhaps).
QA Contact: sairuh → jimmylee
Blocks: 135736
Could this be causing bug 181374?
Brian: this bug has had no patch checked in. it cannot possibly have caused
another bug.
I am not talking about the patch. I'm talking about the issue that this bug is
meant to fix.
Jimmy, a reminder you should qa assign this...
Yes, I am not the proper QA Contact for File Handling.  Reassigned to default.
QA Contact: jimmylee → petersen
Depends on: 176919
This bug can't be fixed before mozilla 1.0 anymore ;) ?
I suggest reviewing keywords.
You're right. Removing Mozilla1.0 keyword.
Keywords: mozilla1.0
*** Bug 191464 has been marked as a duplicate of this bug. ***
Why is bug 55690 (this one) a different bug than bug 69938 ?

I understand that originally they started for different reasons, but this bug
(saving a file creates file before you choose a location) is just a symptom of
the real problem (downloads are stroed in /tmp regardless of filename picked).

Together they've got 120 votes now, and have each been open for over 2 full
years now. From reading through both of them, it seems like the real problem is
that people disagree about the right solution. One camp says we should wait to
open a file to save into until they pick the file/path, risking timing out the
connection if they're really really slow about it. The other camp says we've got
to keep the data flowing, and need a file to put it into until they choose, even
if it means that this might prevent them from downloading the whole file after
they waste a lot of time filling up their /tmp partition.

Both solutions are valid, both have risks, and different people would rather
take different risks. So make it a pref and be done with it. Both of those
solutions are easy to implement (first one just waits to do something it already
does, second one keeps going like it does now). Making a pref and having both of
those options available would seem to let pretty much everyone get their gripes
resolved.
> Why is bug 55690 (this one) a different bug than bug 69938 ?

It's not, really.  That's why this bug blocks that one.

> From reading through both of them, it seems like the real problem is
> that people disagree about the right solution.

No.  The real problem right this second is that I'm too busy with school to have
time to fix this and that everyone else is having a lot more fun complaining
than trying to follow up on the patch I posted in this bug.

Though at this point, that patch is pointless (it would not come anywhere near
applying, and there may be better ways to do waht needs to be done using the new
necko async I/O stuff).

> Both solutions are valid

No.  See comment 238.  You did read the bug before commenting, right?

> Both of those solutions are easy to implement

Why don't you do it then?

> just waits to do something it already does

There is no way to "wait" to return from a function call without creating a new
event queue to keep the UI painting and all the surrounding shenanigans.  This
is what the patch I attached does, by the way.  The other options are
busy-waiting (freezing up the whole UI in the process, which does not help,
since they you can't pick a place to save to) or rearchitecting the caller to
not assume that once your function has returned everything is set to go.

Don't you think if this were easy it would have been DONE by now by one of the
very helpful people who have ignored the patch and the "helpwanted" keyword for
5 months now?

Whiteboard: se-radar[adt2] (py8ieh:248) → se-radar[adt2] (py8ieh:248) read comment 238, 280 before commenting
Comment on attachment 98567 [details] [diff] [review]
step 1

So to use the async I/O stuff, here's all you have to do:

1)  Change nsExternalHelperAppService::DoContent to take a channel, not just a
URI
2)  Make it suspend the channel and put up the helper app dialog
3)  Make the helper app gandler unsuspend the channel once the user makes a
decision
4)  Sort out the issues involved in setting a proper stream listener in the
URILoader once the user decides what to do (that could wait at first, maybe; we
could start with just setting the helper app handler as the listener to fix
this bug).
5)  Sort our what happens when the user clicks a link in the page while the
dialog is up.  Is the previous load cancelled?	(this may be less of an issue
depending on what happens in item #3).
6)  Change the actual helper app handler code to not use a temp file (since at
this point it will know what's happening to the data _before_ the data starts
flowing).
Attachment #98567 - Attachment is obsolete: true
In my entirely biased and subjective opinion, the only things that need doing
immediately are:

1. Fix bug 7840 (which is now four and a half years old).
2. Go back to the old, effective pre-downloading behavior, except store the temp
file in the user-chosen default download directory with some kind of .moz-temp
extension.
3. Don't move the temp file until the download finishes.
4. Content-decode stuff, if necessary, after it finishes downloading.

If anyone wants a pref to not pre-download, then let them go through the toil of
re-implementing the download system.
Clement, what you're suggesting is a hell of a lot more work to implement than
what I'm suggesting.  Feel free to do it, of course.  I'm willing to review the
patch once you're done.
Blocks: majorbugs
*** Bug 199405 has been marked as a duplicate of this bug. ***
*** Bug 200050 has been marked as a duplicate of this bug. ***
*** Bug 200187 has been marked as a duplicate of this bug. ***
Blocks: 124307
I'm not in the process of developing Mozilla, so I'm not a help in coding. Some
weeks ago, i started the downloading of three files all about the size of 200MB.
To this time I didn't know the files will be first saved to temp which only had
300MB. So each download started and after a half day of downloading I didn't get
any of the files because no space on temp was left and resuming was not
possible. That's annoying!

What about to start the download as usually but with a limited bandwith. So it's
possible to download into memory until mozilla knows what to do with the file.
If you think of a downloadspeed of 1 kb/s it's enough to download small files in
no time as usually. Even after 10 Minutes of thinking where to save the file to,
 not more than 1 MB downloaded. After the location is set, the connection can
speed up. With this logic, it's possible to keep the connection alive, small
files are downloaded immediately. If the user doesn't react within a given time
the partially downloaded file can be saved to a tempfile and continue with the
old behavior.
*** Bug 211650 has been marked as a duplicate of this bug. ***
*** Bug 211886 has been marked as a duplicate of this bug. ***
*** Bug 217143 has been marked as a duplicate of this bug. ***
Blocks: 224692
ok this bug is way too long to read it all

I have a temporary workaround (yes I do intend that this will be temporary),
namely that the file will be moved once the user picks a filename (compare summary).

bz's suggestion is the real solution but requires work in other mozilla areas
that I don't want to do right now.

Fixing this either that way or the right way is required for resuming downloads,
which is why I want this temporary solution.

The effect of this patch: Files are moved to the final directory, with the
additional extension .mdownload.

taking bug, will attach a patch in a second
Assignee: darin → cbiesinger
Status: ASSIGNED → NEW
Target Milestone: Future → mozilla1.7alpha
Attached patch patch (obsolete) — Splinter Review
Attachment #136558 - Flags: review?(bz-vacation)
(I will fix this "the right way" in bug 69938)
Status: NEW → ASSIGNED
Keywords: helpwanted
Comment on attachment 136558 [details] [diff] [review]
patch

found a problem with this patch (it should also Cancel() the channel)
Attachment #136558 - Attachment is obsolete: true
Attachment #136558 - Flags: review?(bz-vacation) → review-
Attached patch patch v2 (obsolete) — Splinter Review
Attachment #136559 - Flags: review?(bz-vacation)
Comment on attachment 136559 [details] [diff] [review]
patch v2

>Index: nsExternalHelperAppService.cpp
>===================================================================
>+      mOutStream->Close();
>+      mOutStream = nsnull;

codesize nit: won't the stream's dtor Close() it for you anyway?
ok, I'll change it to just ->Close and not set to nsnull (to be sure that it
will be closed)
Could we make it .part instead of .mdownload? .mdownload looks kinda silly, and
as far as I can tell there's nothing Mozilla-specific about a partially
downloaded file. Thanks...
why not just make it save to the actual filename immediately?
read up?
to indicate that the file is incomplete, and to allow showing the progress in
the file icon at some point.
(and actually this was slightly simpler to implement)
The filename extention .part looks much better to me than .mdownload for
partially downloaded files.
Plus, .part extention is already a de-facto standard for partially downloaded files.

See http://filext.com/detaillist.php?extdetail=.part&Submit3=Go%21 for a list.

eDonkey and Go!Zilla use it.
hm... I was leaning towards .download because it is a download, while .part can
be anything. I am not sure if it is good to share this extension with other
applications. hence I'd rather use .download. (I will attach no new patch. this
is a trivial change)
If the "format" of a .part file (i.e., it's identical to what the final file will be except truncated, there 
are no extra leading/trailing bits, encoding, etc.) is consistent with other applications, then I think 
it would be a very good idea to use it, so users can begin a download with one application and 
resume it with another (for protocols and applications that allow resuming).

I don't like ".mdownload" due to branding issues (should Netscape use ".ndownload", Beonex use 
".bdownload", etc.?).

I don't like ".download" because I don't feel it really suggests the nature of the file (in progress, 
partially completed).  It's a rather long extension; keep in mind that on some platforms (e.g. Mac 
OS X), the middle part of a filename may be abbreviated "..." to save space on the screen, so having 
a long extension may cause less of the useful part of the filename to be visible.

".part" suggests that the file is partially complete, which is exactly right.  The only way I see the 
meaning could be confused is with archiving programs (ZIP, RAR, StuffIt, etc.) that split an archive 
file into multiple segments; a user seeing a .part extension could think it to be one of these.  
However, presumably the user would only see them in their designated download directory, would 
recognize the filename, would know they started downloading something that hasn't finished yet, 
and would be smart enough to figure out what's going on.
Unlike Christian, I (as with others) see no problem with sharing ".part" with
other applications.  Assuming that, technically, it has the same format - then,
IMO, having Mozilla follow the unofficial "standard" can only be a good thing.

I'll also agree with others that if we are *not* to use ".part", I don't want to
use ".download".  It's too long and just doesn't look right.  Even something
like ".dwn" would be better.
FYI: Getright (windows Downloadmanager) adds .getright to unfinished downloads.
It also registers itself as application for this extension and it will continue
the download if you open this file if it's in the database and it asks for the
URL if the file isn't in the database.
Oh - and, having said that, I should also mention that the naming of the
download is *such* a minor point in comparison to what we're trying to fix here
that I'd be happy with it being called ".somedownloadstillinprogress" if only
that resolved this bug.  So, if it makes it easier to stick with ".download" for
now let's do that.
I like jason's last comment :)
matti's comment is something I want in the future

so whatever. I can use "part". that is a pretty trivial change.
I don't care much. if bz has any preferred wording I'll use that.
The truncating of the extension was a good point; and I really hate .dwn, so if
bz has no opinion I'll use .part.

(I would appreciate it if that discussion wouldn't take place in this bug. yes,
the file will be the final file but truncated.)
My two bits on the file name mangling: (sorry to add to this, but I really do
think my new proposed name has value)

When I first saw 'part' mentioned here, I didn't quite get it. Part of a set?
It's an ambiguous word. Made me think of USENET postings that come in parts. So
I have a part. Is it a complete part? Also, because it's a mere four letters, it
looks like perhaps it should have an application associated with it. (And I
don't mean a download manager). MPEG, JPEG, PART... Probably some new media file
I need to go find a viewer for.

And Andy voiced my feelings on using 'download'. "Yes, I did download this file!
Now why won't it open?".  'downloading' or some more active form of the word
would work better, but is still awkward.

A program called "pyslsk", a linux clone of the SoulSeek file sharing program,
names files in progress as "INCOMPLETE<random_id>filename.mpeg". I forget the
whole rational for the random_id, but I do very much like the "INCOMPLETE" part.

It's at the beginning of the filename. This means if you have 3 or 4 incomplete
downloads in a directory, you'll see this immediately--they're sorted together.

It's at the beginning of the filename. You can't miss it, even if your file
manager truncates the long name.

It still has it's proper extension. If resuming the file isn't possible, you can
still (depending on the format, of course) easily access what you DO have of the
file by opening it as normal. Mp3s, mpegs, jpegs, and most media files will
display/play/whatever with only a partial file.

'INCOMPLETE' is completely unambiguous. It's not a 'part' of something, it's not
something you've 'download'ed already, it's incomplete.
so use .incomplete?
Using a prefix removes the ability to write a shell icon handler to show the
progess in the file icon. I do want that feature.
> So I have a part. Is it a complete part?

No, by definition.  :)

I'm happy with .part or .incomplete (though in the latter case there might be
cause to anally localise) -- but I'd be happy with .pastasurprise and agree with
comment 307.
> Using a prefix removes the ability to write a shell icon handler to show the
> progess in the file icon.

That is a marginally neat feature. But comes at the cost of what I feel are
integral features:
  Sorting
  Noticability (Windows truncates filename display very often)
  Ability to open/play/view file as it downloads

If a fancy schmancy "shell icon handler" is deemed to be more important for
Windows users than the above, can UNIX builds use the INCOMPLETE<filename>
naming scheme? There will be no "shell icon handler" equivilent being added, so
it would be nice if UNIX at least got the benifit of the 3 things listed above,
instead of being completely shafted for the sake of Microsoft users. (as usual)
>instead of being completely shafted for the sake of Microsoft users. (as usual)

osx supports icon handlers as well, and I wouldn't be surprised if nautilus or
konqueror supported it too, which could ideally be supported.
you're too negative :)

note that you can sort by extension.
(I read enough mails -- now I want to say something, too :-) )

There are pros for both having 1.) INCOMPLETE prefix 2.) .incomplete extension

1.) Playing,... incomplete/downloading files is neat (e.g. you don't have to
wait till the file is complete if reader supports that)

However I see more points for 2.)
-- Sorting is possible, too
-- Having a dedicated icon is quite noticeable
-- Extensibility: if we have a special extension, we can make tools that deal
with that kind of files e.g. the shell-icon-animator, even better: a
download-resumer (we'd need to store the URL somewhere...), "data-feeder",
"multi-part downloader"
I don't really like the idea of adding a suffix to partial files, but I think
it's a lot better than a prefix. Prefixed files would be very hard to find
because files are usually sorted alphabetically. I don't see how using a prefix
would partial files any more playable, at least on non-DOS-based OS'.
re "I don't see how using a prefix would partial files any more playable, at least on non-DOS-based 
OS":  in most GUIs (classic Mac OS being the obvious exception but we don't support it anymore), 
the file extension determines which application should be used to open the file by default, in 
addition to which icon to display.  If the filename is "INCOMPLETE-Funny.mpg" it will have a movie 
icon and double-clicking will launch your preferred video player.  If the filename is 
"Funny.mpg.pastasurprise" it will look like a plate of spaghetti and double-clicking could 
potentially launch a download manager to resume it.

Those using a CLI would only be concerned with how the filename displays alphabetically, since 
they have neither icons nor default applications.  These users are in the minority; Mozilla is a GUI 
browser after all.  CLI != UNIX; see comment 313.

.incomplete could be shortened to .inc, but IMHO that's worse than .part.

As mentioned in comment 314, to resume files with a download manager we'd need to store the 
URL; this negates the advantage of sharing a common extension with other applications.  Might 
cause a problem if other apps use the same extension as us and want to use their own download 
managers, since they'll store URLs their own way, and it won't be obvious whose partial files are 
whose...
It doesn't matter what the filename is. It can be changed later. In another bug.
Let's not block this one because of such sillyness.

See http://www.freebsd.org/doc/en_US.ISO8859-1/books/faq/misc.html#BIKESHED-PAINTING
I'll pick .part, mainly because of its shortness.
Comment on attachment 136559 [details] [diff] [review]
patch v2

This looks pretty reasonable, but why all that use of "native" nsIFile methods?
 I'd think you would in fact want the Unicode ones.... (since you are appending
a string, not a byte sequence).
Comment on attachment 136559 [details] [diff] [review]
patch v2

In particular, I like this patch not a bit if "native" encoding is UTF-16.....
Attachment #136559 - Flags: review?(bz-vacation) → review-
One other thing that worries me is that if we download a good chunk of file and
then the user selects the location, we will freeze up for a while as we move,
giving the connection a decent chance to time out.  :(
if native encoding is not ascii compatible, a lot of mozilla code will break...
compare http://lxr.mozilla.org/seamonkey/search?string=appendnative

but ok, will change that
Oh, jeez.  Could we please file some bugs on the most egregious things there?  I
don't care much about extensions, but expat, modules, etc should really not be
screwing up like that.... :(
Attached patch patch v3Splinter Review
now uses the non-native functions.
Attachment #136559 - Attachment is obsolete: true
Comment on attachment 138080 [details] [diff] [review]
patch v3

(bugs on those other issues filed)

as for that freeze mentioned in comment 321: it already happens now (at the end
of the download), but granted, currently the connection has no chance of timing
out. But, for the connection to really time out, you probably need a file that
takes several minutes to copy, and I doubt mozilla is able to download such
large files anyway ;)

(god, comment 321? we should disallow people other than the assignee and
requestee from making comments in a bug once it reaches say 100 comments, imo)
Attachment #138080 - Flags: review?(bz-vacation)
biesi or anyone else interested: 100 comments would be too arbitrary a number.

Bugs that could reduce bugspam:

bug 207090 - Quiz feature for certain bugs before filing a comment
bug 148563 - NOSPAM setting and warning
bug 164310 - "Reason for my vote" feature
bug 148564 - Ability to ignore specific bugs in prefs
bug 48570 - Allow votes against a particular enhancement request
bug 65801 - Allow "pending" votes or weighted votes.
Comment on attachment 138080 [details] [diff] [review]
patch v3

>Index: nsExternalHelperAppService.cpp
>+      rv = NS_NewLocalFileOutputStream(getter_AddRefs(mOutStream), mTempFile,
>+                                         PR_WRONLY | PR_APPEND, 0600);

Fix the indent here.

r=bzbarsky with that.
Attachment #138080 - Flags: review?(bz-vacation) → review+
Attachment #138080 - Flags: superreview?(darin)
Comment on attachment 138080 [details] [diff] [review]
patch v3

>Index: nsExternalHelperAppService.cpp

>+      // Get the old leaf name and append .part to it
>+      nsAutoString name;
>+      mFinalFileDestination->GetLeafName(name);
>+      name.Append(NS_LITERAL_STRING(".part"));
>+      movedFile->SetLeafName(name);
>+
>+      nsCOMPtr<nsIFile> dir;
>+      movedFile->GetParent(getter_AddRefs(dir));
>+
>+      mOutStream->Close();
>+
>+      rv = mTempFile->MoveTo(dir, name);

how about using the "native" versions of the nsIFile methods to avoid extra
native <-> unicode conversions.

sr=darin with that change.
Attachment #138080 - Flags: superreview?(darin) → superreview+
>if native encoding is not ascii compatible, a lot of mozilla code will break...
>compare http://lxr.mozilla.org/seamonkey/search?string=appendnative
>
>but ok, will change that

nsIFile's concept of "native" encoding is assumed to be a superset of ASCII, so
appending an ASCII string is ok.  moreover, we control the definition of the
"native" encoding, so if necessary we could always force the "native" encoding
to be a superset of ASCII (e.g., UTF-8).
i should also add that the "native" encoding is compatible with PR_Open and what
PR_GetEnv returns.  it therefore cannot contain embedded nulls, so UTF-16 is not
a possible "native" encoding.  anyways, none of the STDC library routines would
work with UTF-16, so that really is not something we should worry about.
> nsIFile's concept of "native" encoding is assumed to be a superset of ASCII

Assumed where?  nsIFile.idl certainly doesn't assume it...

There are also other non-ascii-compatible encodings that do not involve embedded
nulls (EBCDIC, to be silly).

If this is in fact a safe assumption to make we need to clearly document that in
nsIFile.idl so nsIFile implementors (anyone can drop in, eg, a ftp-based impl of
non-local nsIFiles) won't break this assumption....
> Assumed where?  nsIFile.idl certainly doesn't assume it...

we assume that the "native" multi-byte encoding is a superset of ASCII.  we
assume this when working with PR_GetEnv/PR_SetEnv, PR_Open, etc.  this same
assumption applies to nsIFile.  how many times have you seen someone call
AppendNative, SetNativeLeafName, etc. passing an ASCII valued string?  in fact,
you see this quite a lot.  while it may limit mozilla from working with some
very obscure multi-byte encodings, that limitation is assumed to be acceptable
for the entire product.

i'm not arguing against documenting this fact in nsIFile.  indeed, nsIFile could
use a lot more documentation.
Sure, I would be fine with that (updating the documentation in nsIFile to
clearly say this).  Would you mind doing that?  Or should I?
ok, I now checked the patch in using the native functions.

I filed Bug 231243 to document that nsIFile's native functions are ascii-compatible
Status: ASSIGNED → RESOLVED
Closed: 21 years ago
Resolution: --- → FIXED
*** Bug 233435 has been marked as a duplicate of this bug. ***
*** Bug 242291 has been marked as a duplicate of this bug. ***
I've been using the Internet since ARPANET days, and this one isn't a real
problem. Download to the default download directory under a generated name, and
rename or delete or copy as needed. If the instantaneous start of download is
taken away, there will be no reason to use Mozilla's download function at all.
As it is, most small files are downloaded before a name is selected, making them
essentially without cost. What could be better. Those who want downloads to wait
to start until OK'd should be able to turn that off. Now, if the download
manager actually displayed URL  and allowed repeated downloads or controllable
restart of download...that is to say, it actually worked, all of us would love it. 
(In reply to comment #338)
> I've been using the Internet since ARPANET days, and this one isn't a real
> problem.

Well, /tmp may not have enough space for the file.

And, when you save to /tmp, you can't really implement resuming, because /tmp
tends to get deleted.

> Now, if the download
> manager actually displayed URL 

column picker -> source

> and allowed repeated downloads or controllable
> restart of download...that is to say, it actually worked, all of us would love
it. 

there are bugs filed on this already.
*** Bug 242679 has been marked as a duplicate of this bug. ***
Just a few observations about the differences in the downloads between Firefox
and Mozilla in OS X.  

I have noticed that Mozilla appears to be downloading into the temp file while I
am going through the process of selecting the target destination of the
download.  Occasionally I will have a completed download of X number of seconds
(say 20 seconds) when I finally click on the destination when using Mozilla.  On
the other hand Firefox does not appear to download in this manner as it only
begins after I click on the final destination of the download.

The other difference that is quite noticeable when comparing downloads between
the two is that Firefox starts out at quite a fast download rate and then
typically tapers off to something lower as a "typical" download rate.  (It is
impressive how fast some of the Firefox downloads are.)  Mozilla, by way of
contrast, starts out rather slower and then builds to some faster "typical"
download rate.  Mozilla has an elapsed time for the download, but Firefox does
not and so I am approximating the total download time for Firefox, but it seems
to be a shorter length of time in most instances.  (It would be nice to have an
elapsed time feature in Firefox as well...actually, there is one browser which I
can not recall that also calculates the average download rate for the completed
download.  It does not improve performance, but is interesting nonetheless.

  
No longer blocks: majorbugs
No longer blocks: 88566
Status: RESOLVED → VERIFIED
Product: Core → Core Graveyard
You need to log in before you can comment on or make changes to this bug.