Closed Bug 1759779 Opened 3 years ago Closed 3 years ago

Add a preference to schedule "opened" download targets for deletion, to compensate for no longer saving temp files

Categories

(Firefox :: Downloads Panel, enhancement)

Product:
Firefox
Component:
Downloads Panel
Platform:
Desktop
Unspecified
Type:
enhancement

Tracking

()

Status:
RESOLVED WONTFIX
Tracking Flags:
Tracking Status
firefox100 --- wontfix

People

(Reporter: aminomancer, Unassigned)

References

Details

Attachments

(1 file)

Default directory greyed out, giving the implication it will not be used when "Always ask you where to save files" is checked
6.68 KB, image/png
Details

Explanation

Previously, the "Open" action caused Firefox to save a download in a tmp directory and (in some environments) schedule it for deletion on app quit. That was removed with bug 1710926, creating a situation where downloads can quickly pile up in the user's chosen downloads directory.

To mitigate that issue, we previously added a new context menu item to allow users to delete download targets from the downloads panel and other views (bug 1745624). And I thought this was a pretty fine solution, and it's been working well for me ever since. But maybe because I was involved in the development, I'm hyper-conscious of the Delete menu item, so I have no trouble remembering to clean up the downloads panel by manually using the context menu.

Today I noticed this thread on r/firefox, in which the majority of users seem to regard the temp folder problem as unresolved. Actually, I didn't see a single person reference the Delete menu item as a possible solution. Instead, the "solution" that is being suggested to users is to toggle the experimental pref behind which all the downloads panel improvements are locked.

From that feedback it sounds like, even with the new menu item, we still haven't met the users' demands for some downloads directory housekeeping functionality. And, rather than deal with this trade-off, many users seem to prefer the temporary solution of disabling all the downloads panel improvements.

Proposal

So, I would like to add a new preference that will automatically call gExternalAppLauncher.deleteTemporaryFileOnExit on download targets that are opened immediately — either as the handler's default action, or as a result of the unknown content type dialog, but not as a result of the regular "open" command called on an already "saved" download.

Let's say my handler action for application/pdf is "Open"; my handler action for application/octet-stream is "Save"; and my handler action for text/plain is "Always Ask".

I download a file of each type. (For now, don't worry about where they're saved) The PDF should be scheduled for deletion, but not the executable. The text file should only be scheduled for deletion if I chose "Open" in the UCT dialog.

Let's say I did it again but this time I chose "Save" in the dialog for the text file. Then, I open the downloads panel and I click the download element, or I right click it and click Open or Open in System Viewer or whatever. In these cases, it should not be scheduled for deletion, since I already chose to permanently save it.

So in simple terms, what I'm suggesting is that (locked behind a pref) if a download target is "opened" because its file handler's action is "Open" or because "Open" was chosen in the UCT dialog, it will be scheduled for deletion on app exit.

Also, I'd like to point out that I think the risk of unexpected data loss is pretty minimal. 2 reasons:

  1. gExternalAppLauncher.deleteTemporaryFileOnExit doesn't do this already, but it's possible to make it set the permission bits in a recognizable way (as it does when making the file read-only), and then when the file is finally going to be deleted, check that the permission bits are still the same.
  2. With bug 1746386 we should be adding file handle tracking, at least for this feature. So when that is done we don't need to delete the file by path alone. I would say, when the deletion method is finally called, it finds the file by handle and checks that it still has the same path.

Options

The next question is where such files should be saved.

Option #1

Technically it would be simplest to just save them exactly where they'd be saved otherwise, i.e., the user's default downloads dir. Because by the time an action has been taken in the UCT dialog, saver has already begun. But it would also be possible to move the download to a temp folder as part of this new method that schedules it for deletion.

The reason moving it to a temp folder might still be appealing, even though a similar feature was intentionally removed, is because users may not expect files in regular folders to be scheduled for deletion in the background. So, the unexpectedness of the behavior would preclude us from ever making this the default. By adding a pref we would be solving the problem for power users, but we couldn't enable it by default without probably causing confusion.

Option #2

Of course, if we added a checkbox for this opt-in pref in about:preferences, the label could conceivably explain this, while making the option discoverable for most users. But I can't think of a concise wording that would really make it clear at the moment.

Option #3

But another option is to just move the download to a temp folder so that deletion is expected. From what I understand, one of the main issues with saving downloads to a temp folder is that their locations are not consistent on Linux distros? It seems like we could resolve that as follows:

  • In about:preferences, underneath the downloads dir selector that says Save files to [Downloads], we add a second path selector that says Save temporary files to [/tmp] so that user can manually choose their preferred temp folder path.
  • Firefox tries to find a temp folder, maybe as an idle task at startup?
  • If it doesn't find one, it toggles some property so that Firefox saves "opened" download targets in the regular downloads directory, and does not schedule them for deletion (basically the current status quo)
  • In this case where a temp folder was not found, Firefox should record the failure on some global component.
  • When the user opens about:preferences, they should now see a little warning card above the Save temporary files to [/tmp] selector. And it says something like "Temporary directory not found. Choose a folder."
  • The user chooses a temp folder, so now that toggled property is flipped back. The next time a download target is "opened", we save it there and schedule it for deletion.

Maybe there are some other considerations I haven't thought of. Let me know and I'll add them to this document. I don't think any of the above options is perfectly ideal, given the OS inconsistencies, but it seems like we have to do something about this.

I do know that Firefox was already unique in this feature. But Firefox users may have grown accustomed to their Downloads directory not ballooning with garbage (I certainly have). For some users, that may have been one of the main reasons they continued to use Firefox instead of Chrome, despite other trade-offs like worse integration with Google services, more Captchas, etc. So by following Chrome and taking it away, we're providing users with one fewer reason to choose Firefox over Chrome.

Since the temp folder changes have been shipped in a release build, we can see many more users grumbling about 1) unwanted download files that users perceive as temporary (PDFs, archives, etc.) piling up in the downloads directory, and 2) part files getting stuck in a permanent download dir (cf. bug 1759670). So if there's a better solution that will have the same effect, please share 🙏

Thank you for considering this and providing a detailed write-up. I have some additional remarks regarding the options considered.

Saving the files in the normal download location

  • If the user considers the Downloads folder to be more of a permanent storage, they may include it in backups (that's what I do at least). Saving opened files there as well can cause them to be unnecessarily included in backups.
  • If firefox is unexpectedly terminated (crash, out of memory, power loss, ...) then these files may persist, although this depends on how exactly this feature is implemented.

Saving in the system temporary directory on linux

I only have experience with linux, so I can't comment on other operating systems.

  • This solves the two remarks I made regarding saving in the normal download location: it will most likely be excluded from backups and on may be automatically cleaned (either on some timer or when rebooting)
  • I think the location of the temp dir is consistent across most distros, and those for which it is not /tmp or /var/tmp, the TMPDIR env var should be set.
  • /tmp might not be backed by storage but by RAM, which would prevent these downloads from hitting the disk at all. However, this also has some disadvantages as the file might be to big to store in RAM. Perhaps firefox could check whether there is enough disk space left, and if not (with some margin possibly) warn the user and default to normal downloads location. In my case /tmp has a limit of 8GB, which is half of my total available memory.

See https://systemd.io/TEMPORARY_DIRECTORIES/ for some more context.

My workflow consists of storing files that I want to keep in the Downloads folder and all other files (mostly PDFs and archives) in some temporary directory. Hence, Option #3 would be my personal preference.

(In reply to Vincent Vanlaer from comment #1)

  • I think the location of the temp dir is consistent across most distros, and those for which it is not /tmp or /var/tmp, the TMPDIR env var should be set.

Thanks for sharing. My understanding is there is a /tmp dir on macOS as well. I think someone previously gave me some reasons why the temp folder feature is problematic but I can no longer remember them, nor where that conversation happened.

I, too, thank you for this detailed write-up. I think, option 3 would be my favorite as well. But I believe that, over at Bug 1738574, :Gijs has already stated that there are not enough capacities to automatically maintain and test such a feature. I am not sure, though.

Regarding the thread on r/firefox, you mention above: You seem to be somewhat surprised that people aren’t considering the new panel item for deleting files a solution.

I noticed this issue. It created a mess in my download directory.

It's unacceptable that files that I don't want to have on my drive are stored on my drive. When I open a file to view it I don't want it to even touch my drive, let alone be forced to keep tracking all the files and keep deleting them.

Previous behaviour was good. Files that I chose to open were put in tmp directory so I didn't have to worry about deleting them manually. Now I'm forced to waste time, space and ssd writes on files I don't even want to keep.

Another observation on the current behaviour - the UI in about:preferences doesn't make clear the fact that the user is setting a default downloads directory even when they choose "Always ask you where to save files".

For example, in the attached screenshot, the "R:\Temp" location (and the whole line that contains it is greyed out). By the normal interpretation of greyed-out UI elements, I would expect this to mean that Firefox will make no use of "R:\Temp". And yet, if I click on a file whose type I've set to the "Always Ask" behaviour, and choose "Open with", that directory is used, without asking me.

That's another factor in favour of adding a second, separate, path selector so that users can choose their preferred "temp" folder path.

Yeah thank you pg_78 that's a very good point.

Firefox tries to find a temp folder, maybe as an idle task at startup?

I'm slightly confused about this. I think Firefox already does this. All the files I open have always been stored in /tmp/mozilla_martin0/

Also, for me the part files have always been put in the destination directory along the destination file (that one with zero size). That's for when I choose to "Save file".

This sound like great step in the right direction, just something I'm not sure about:

When I set "always ask" for everything (since I never want a download to 'permanently' start w/o my interaction), would I still be bothered with a "save as" dialog making me choose a location for the to-be-deleted temporary file? Because for any permanently saved file I'd very much like to specify where it goes (e.g. I pick my media disk for mkvs, my downloads folder for executables, etc.), but for a file where I choose to "open" I don't care about this (and I would not like those temp files to clutter e.g. my last-used download target).

It sounds like your option #3 avoids this, but please correct me if I'm wrong.

(In reply to Martin from comment #9)

Firefox tries to find a temp folder, maybe as an idle task at startup?

I'm slightly confused about this. I think Firefox already does this. All the files I open have always been stored in /tmp/mozilla_martin0/

On most operating systems you can use TmpD to find a temp directory, the directory service will work out its location from the OS. As far as I know this always works because even Snap redirects it to ~/Downloads/firefox.tmp. But in this case, I don't think Firefox checks if that directory is writable. And there are certainly environments I'm unaware of where a temp directory will not even exist.

So it would be wise to go through some extra steps to locate a working temp directory. That way, if we create a temp directory because one couldn't be found, then the one we created can be reused in subsequent sessions. I think the ~/Downloads/firefox.tmp solution works well for Snap so I don't see any reason it couldn't be used in general, if TmpD doesn't exist or isn't writable.

Also, for me the part files have always been put in the destination directory along the destination file (that one with zero size). That's for when I choose to "Save file".

Are you choosing "Save file" in the unknown content type dialog or from the handler list in about:preferences?

(In reply to Adrian from comment #10)

This sound like great step in the right direction, just something I'm not sure about:

When I set "always ask" for everything (since I never want a download to 'permanently' start w/o my interaction), would I still be bothered with a "save as" dialog making me choose a location for the to-be-deleted temporary file? Because for any permanently saved file I'd very much like to specify where it goes (e.g. I pick my media disk for mkvs, my downloads folder for executables, etc.), but for a file where I choose to "open" I don't care about this (and I would not like those temp files to clutter e.g. my last-used download target).

It sounds like your option #3 avoids this, but please correct me if I'm wrong.

The "Save as" dialog depends on the preference browser.download.useDownloadDir which in about:preferences corresponds to the "Downloads" section, which should have 2 radio buttons, one of which says "Always ask you where to save files." That dictates whether "Save file" will always open a "Save as" dialog.

Under the "Applications" section there's a big table with content types in one column, and default actions in the other. The action column has several options for each content type, you can choose "Save file" or "Use x" or "Always ask." In this context, "Always ask" doesn't mean ask you where to save files, it means ask whether you want to open the file in Firefox, open it in an external handler, or just save it and do nothing else. This is a different dialog than "Save as" and it should also have a checkbox that tells Firefox to save your choice and do it again automatically when you download files of the same content type.

That "Always ask" dialog can lead to the "Save as" dialog though. If you choose "Save File" in this dialog, and you have browser.download.useDownloadDir = false, then after clicking OK, the "Save as" dialog will pop up. But if you have useDownloadDir=true, then after clicking OK, it will just save the file in the downloads directory you set in about:preferences.

If you choose "Open" then it doesn't matter what useDownloadDir is set to — right now, it will always save in your default downloads directory. That is, the "Save as" dialog is only opened if you choose "Save file" but not if you choose any of the "Open" options. If we add some pref to reimplement temporary download behavior, then I don't see any reason why this would change.

But I do think it's a problem that, right now, the default download directory is used even though it's grayed out in about:preferences, i.e., when you choose "Always ask you where to save files". It's grayed out here which is supposed to reflect that it's going to prompt a "Save as" dialog every time you save a file. But as mentioned above, when you choose to "Open" a file, there is no "Save as" dialog, so it uses the grayed out default download directory. So the graying out is kind of misleading. I will post a separate bug report about that. (Edit: see bug 1762775)

But yeah, in any case, we would continue only opening the "Save as" dialog when the user chooses "Save file"

(In reply to Shane Hughes [:aminomancer] from comment #11)

If you choose "Open" then it doesn't matter what useDownloadDir is set to — right now, it will always save in your default downloads directory. That is, the "Save as" dialog is only opened if you choose "Save file" but not if you choose any of the "Open" options. If we add some pref to reimplement temporary download behavior, then I don't see any reason why this would change.

Interestingly, as it stands now, the behaviour you get by choosing "Open" for a file whose type you've set to "Always Ask" is different than what you get by setting the file type to "Open In Firefox" (for those types that support that, like PDF).

Case 1: If you set the file type to "Open in Firefox", then opening a file does not write it to your default downloads dir. (Not sure whether it goes to a temp dir, or to some cache location within the FF profile: for example, if you have your PDFs set to "Open In Firefox", then the address bar displays the remote URL for an open PDF, rather than a file:// URL.)

Case 2: But if you set the same file type to "Always Ask", then choosing to "Open" a file instead saves it to the defaults download dir. I'd vote for changing this behaviour to match Case 1.

(In reply to Shane Hughes [:aminomancer] from comment #11)

Also, for me the part files have always been put in the destination directory along the destination file (that one with zero size). That's for when I choose to "Save file".

Are you choosing "Save file" in the unknown content type dialog or from the handler list in about:preferences?

I've just tested by downloading linux kernel from kernel.org (tar.xz). One one system I don't have such type defined in preferences. I clicked the link, chose "Save file" from the dialogue, picked location where to save. In that directory the part file was created. On another system, where there is xz in file types it seems to be working the same. I tried both when "always ask" was set for xz files and "save file" were set in preferences. Both time it created the part file in the destination directory. The second system I'm checking is a VM with pretty much out of the box installation (Fedora, Firefox 97).
It's worked this way since I can remember.

See Also: → 1762775

(In reply to Martin from comment #13)

It's worked this way since I can remember.

Well you can easily set it to "Always ask" and then download a giant file and when the dialog comes up asking if you want to open or save, just do nothing for 20 minutes. In the meantime you can confirm there is no part file in your ~/Downloads directory. At least for me on Windows 10, during this time there is a .part file in the Temp folder corresponding to the download stream. It's 0 bytes during the download and then, when it's finally finished downloading, you'll see it reach the full file size. It won't move to ~/Downloads unless I choose "Save file" from this dialog.

When you finally choose "Open with x" or "Save file" you'll see in Firefox's downloads panel it immediately finishes the download, because it's been downloading the stream in the background while the dialog was open. I'm not a C++ expert so I don't know all the nuances of this background download thing, but I trust what others have told me. If you want to investigate for yourself, this is the source code.

Edit: I just tested it with this file which is about a gigabyte. You don't need to use an old version of Firefox to test this, just disabling the pref browser.download.improvements_to_download_panel will enable the old code path. You can see where it's referenced in the source code I linked above. Another difference is that there's no way to avoid the unknown content type dialog in old versions.

But anyway, if the file's content type (7z archive) is set to "Always ask" then Firefox will open an unknown content type dialog as soon as the download begins. So at this moment you can check the Temp folder and see there's a .part file there, and there isn't a .part file in your ~/Downloads directory. And if you just wait around (leaving the dialog open) for a while, eventually you'll see the download finished because the .part file's size will jump from 0B to 1GB. Then you know the file has been written to the Temp directory. And you can then choose "Save file" in the dialog, and you'll see the file disappear and reappear in ~/Downloads. And now you can understand why it's a problem if ~/Downloads is on a volume with less storage capacity.

(In reply to pg_78 from comment #12)

Interestingly, as it stands now, the behaviour you get by choosing "Open" for a file whose type you've set to "Always Ask" is different than what you get by setting the file type to "Open In Firefox" (for those types that support that, like PDF).

That is unrelated to this bug, but you should report it under Firefox > File Handling. I think it's a special file handler for specific internally viewable file types.

I don't know why the "Open in Firefox" action in the UCT dialog does something different than the "Open in Firefox" action in about:preferences. Probably this internally-viewably handler was just added as a handler, and wasn't factored into the UCT dialog because the UCT dialog was to be removed from the normal downloads flow. That is, it used to open for every download but now only if the mime type is set to "Always ask," which isn't the default. The goal was mainly to eliminate the number of steps so I guess the UCT dialog has taken a back seat. But that definitely sounds like a bug that should be resolved.

(In reply to Shane Hughes [:aminomancer] from comment #0)

Thanks for the productive way forward - Option 3 sounds good, or option 1, as long as it's easy to tell temporary downloads from others (ie. a separate dir). Otherwise, if the browser is open for a long time it may be a surprise when many files suddenly disappear from "Downloads".

(I don't think ff needs to track these files necessarily, if they are in a separate dir then it's easy to manually clean it out every now and then, which is usually what happens with /tmp)

(In reply to Shane Hughes [:aminomancer] from comment #15)

(In reply to pg_78 from comment #12)

I don't know why the "Open in Firefox" action in the UCT dialog does something different than the "Open in Firefox" action in about:preferences. Probably this internally-viewably handler was just added as a handler, and wasn't factored into the UCT dialog because the UCT dialog was to be removed from the normal downloads flow. That is, it used to open for every download but now only if the mime type is set to "Always ask," which isn't the default. The goal was mainly to eliminate the number of steps so I guess the UCT dialog has taken a back seat. But that definitely sounds like a bug that should be resolved.

Thanks! It's https://bugzilla.mozilla.org/show_bug.cgi?id=1762955.

I think Option 1 but making it a ".tmp" or similarly named subfolder in the download location and adding something like also cleaning it when firefox starts (so on a crash files don't just litter forever) makes the most sense, because to me it seems to solve the flatpak/snap problem while otherwise getting back the closest to the previous behavior, in particular that it doesn't clutter these temporary files together with all the ones users may want to keep and that it won't permanently use up disk space for them either.

While Option 3 also seems nice, I feel like it will make the UI needlessly complicated for something I don't see much users wanting: from what I've gathered from all the comments, users don't want to specify a specific temporary folder, they just want it to be actually TEMPORARY and separate. So I don't think the main need is to be able to store where it is, but rather that the folder and its contents behave right and not like the regular download space. And for those who really want, something like ~/Downloads/.tmp could still be mounted as a tmpfs, so it doesn't need to be inside the regular system temp folder just to make that possible as far as I can tell.

My apologies if I misunderstood some of the details, and I hope this comment is helpful.

I'm closing this now since work on this can be handled in bug 1738574. Thanks for sharing your ideas everyone

Status: NEW → RESOLVED
Closed: 3 years ago
status-firefox100: affected → wontfix
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: