1461618 - Feature request: Avoid duplicate downloads via hashsum

Status: RESOLVED WONTFIX

(Toolkit :: Downloads API, enhancement)

Description

[JohannesD]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.9) Gecko/20100101 Goanna/3.4 Firefox/52.9 PaleMoon/27.8.1
Build ID: 20180306202754

Steps to reproduce:

Clicking the same link several times downloads the file and numbers them.


Actual results:

Multiple downloaded files appear on disk and in the download manager.


Expected results:

It would be very easy to calculate a hashsum of the downloaded file and store that with the list of past downloads to avoid the numbering and save space on disk. While the calculation of the hash sum probably requires to download the file, again, one could easily notify the user that this file was already downloaded or similar.

Comment 1

[JohannesD]

There is no way to request features other than by free text via feedback, that's not really useful!

Comment 2

[Gingerbread Man]

(In reply to JohannesD from comment #0)
> While the calculation of the hash sum probably requires to download the file, again

Yes, this request isn't possible. When downloading, the file name is known, and typically the file size as well. So at best maybe there could be some file manager-style prompt that lets you compare files and decide whether to overwrite, rename, or cancel.

> one could easily notify the user that this file was already downloaded or similar.

Bug 485776 for a lack of prompt with default settings, when downloading a file with the same name.
You can set your preference to "Always ask you where to save files" and then you'll be prompted if a file with that name already exists.

(In reply to JohannesD from comment #1)
> There is no way to request features other than by free text via feedback,
> that's not really useful!

You can also file a bug report and set the Severity to "enhancement".

Comment 3

[JohannesD]

I guess the new proposal here is to deduplicate files _after_ download because otherwise you can only distinguish by name, url and size. Say you retrieve the file, saving it to a temporary and pipe the data stream through a hash calculation function, you can notify the user once the download is completed or just omit the file in the download folder and link the download to the previous file in the download manager. Only makes sense if not downloaded via "save as". It's most useful when you happen to download for instance large ISO files. Implementation-wise, this should be cheap and would help you to keep your downloads smaller.

Comment 4

[:Paolo Amadini]

This is an interesting idea, but it's quite complex to implement and requires some user interface design, so I don't see it happening any time soon. The benefit is also small since the file has already been downloaded entirely by the time we can check, making this purely a disk space saving feature.

There may also be a few edge cases like multiple downloads may be intentional, for example downloading a template file that is then edited in the file system, which may mean that we'd have not only to store the hash of previously downloaded files along with the hash, but when a duplicate file is actually found, we'd have to verify again that this is actually the same file. We may need a special state at the end of the download during the waiting time required to do this for large files.

This also wouldn't work reliably for old downloads that expired from history.

We already compute the hash of some downloads for the Application Reputation service, but we'd have to extend this to all downloads, which requires more CPU. Also, storing the additional data in the history database may by itself have a performance impact.

These problems are all solvable, and I was initially considering marking this bug as something low priority for which we could accept user experience design mockups and patches, but given the effort involved in solving and evaluating all these issues, and the small benefit, I'd say we just shouldn't do this.

Comment 5

[JohannesD]

Fair enough, I see this might be out of scope of a web browser application which should concentrate on basic functionalities. Given the mentioned edge cases, however, maybe a light version summarized as "notify possible duplicate downloads based on metadata" might be possible and useful, since the metadata is easy to collect and keep track of. Metadata is for instance download link, filesize and hash after download. I would think that calculating a simple hash based on a stream of data does not impact CPU heavily and could also be disabled in the settings.

Otherwise, IMO this should go into an advanced download manager.