Closed Bug 1572888 Opened 1 year ago Closed 4 months ago

Programmatically handle downloads

Categories

(Firefox :: Downloads Panel, enhancement)

70 Branch
enhancement
Not set

Tracking

()

RESOLVED DUPLICATE of bug 1306339

People

(Reporter: traverse.da, Unassigned)

Details

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0

Steps to reproduce:

I'm using selenium to create a web-archiving tool.

Actual results:

When the web archiver "clicks" on a download link, the entire piece of software freezes up, waiting for user input.

Expected results:

There should be some way to automatically handle all downloads, either stopping them from downloading or downloading them all anyway.

There are partial solutions, but they require you to list every individual mime type you want to handle.

Allowing browser.helperApps.neverAsk.saveToDisk to accept wildcards could help.

It sounds like you may want to write a WebExtension add-on for your case.
How did you set downloads in preferences, did you set it to download all files in a given folder?

Flags: needinfo?(traverse.da)

So as near as I can tell every time the browser encounters a new mimetype it prompts the user for action, regardless of your auto-download settings. If you click "do this automatically for files like this from now on" it works from that point on.

Unfortunately that means that in order to be sure firefox doesn't block, I need to add a download handler for each and every mimetype that the browser might encounter.

My understanding is that webextentions have similar issues, if there is a way to solve that problem via webextention I haven't figured it out yet.

https://github.com/ugetdm/uget-extension is probably the closest equivalent, and it requires user interaction before interrupting the download as well.

As far as I can tell there is literally no way to handle download of arbitrary mimetyped files without requiring user-interaction. If there is a way to do it, please let me know, but I've put a fair amount of time trying to figure out a workaround. I have a solution for chromium, but I'd really rather use firefox for my web archiving.

Flags: needinfo?(traverse.da)

Of course what I can do is try to guess every mimetype it might possibly come across, and create an entry in handlers.json, but that is not really robust enough for a serious project.

An add-on could likely intercept the traffic and initiate downloads, you may try asking the DownThemAll! author for hints.

What's the Chromium solution you are using?

I'm using python-cef, which has support for handling downloads and uses the chromium-embedded-framework. But my understanding is that the chromium-webdriver solutions works just fine.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_experimental_option("prefs", {
  "download.default_directory": r"C:\Users\xxx\downloads\Test",
  "download.prompt_for_download": False,
  "download.directory_upgrade": True,
  "safebrowsing.enabled": True
})

An add-on could likely intercept the traffic and initiate downloads, you may try asking the DownThemAll! author for hints.

DownThemAll does not intercept traffic. It gets around this issue by never "clicking" on the download links, instead it adds them to an internal queue. If you click a link with DownThemAll (or any other extension I've tested) it still prompts for blocking user-input. It serves a completely different function and operates in a completely different way. Remember, the problem isn't actually downloading the content, it's the browser entering an uninterruptible blocking state when you try to use it for automation.

If you're intent on not fixing this I do have a solution working using chrome. For various reasons I'd prefer to use firefox, but I understand it's a fair amount of work to fix a niche problem. Still, as you can see from the various stack-overflow posts, it's not that niche of a problem.

I probably could hack something together by intercepting HTTP-requets, modifying headers and all that, but where I'm using this for web-archiving the archive would record any changes I make, causing WARC-playback to fail when users tried to download that same file over again. I note that none of the existing download-manager plugins use that technique, so I'm skeptical that it would even work, and it's a lot of work for an exploratory solution to a problem that chromium doesn't have.

Alternatively there could be a webextension api for pragmatically handling download links without blocking user-prompts, or perhaps a flag equivalent to chrome's "download.prompt_for_download", but that actually works for mimetypes that aren't explicitly set in handlers.json?

Gijs, thoughts?

Flags: needinfo?(gijskruitbosch+bugs)

I'm confused about a few things.

(In reply to traverse.da from comment #5)

For various reasons I'd prefer to use firefox, but I understand it's a fair amount of work to fix a niche problem. Still, as you can see from the various stack-overflow posts, it's not that niche of a problem.

Sorry, which? Were there links that got eaten by markdown or something?

But also, I'm a bit surprised about the Chromium thing. I have mostly tested things on Chrome when I needed to check for webcompat issues, so it's possible that Chromium is different, but IME Chrome always just downloads whatever you click on, and they will do their warning / asking when you actually click on an item in their downloads bar (or whatever it's called). Is that not what you're seeing? Or am I missing some other prompt that you're having to deal with?

I'm asking because we already have bug 1306339. My understanding is that would align us with Chrome, so I'm confused about why that wouldn't be enough, effectively. That said, AIUI there's no resources to work on bug 1306339 at the moment.

Alternatively there could be a webextension api for pragmatically handling download links without blocking user-prompts, or perhaps a flag equivalent to chrome's "download.prompt_for_download", but that actually works for mimetypes that aren't explicitly set in handlers.json?

This is probably a question for the webextension team. Shane, do you know if this is something that you'd consider and/or that should already be possible with the existing APIs?

Flags: needinfo?(traverse.da)
Flags: needinfo?(mixedpuppy)
Flags: needinfo?(gijskruitbosch+bugs)

Sorry, which? Were there links that got eaten by markdown or something?

You can see a few of those thread here:

There are literally dozens of us.

Is that not what you're seeing? Or am I missing some other prompt that you're having to deal with?

That's what I'm seeing.

My understanding is that would align us with Chrome, so I'm confused about why that wouldn't be enough, effectively.

That would solve this nicely. I'm on board for that solution. I'll close this as a duplicate and chime in over there.

Status: UNCONFIRMED → RESOLVED
Closed: 4 months ago
Flags: needinfo?(traverse.da)
Resolution: --- → DUPLICATE
Duplicate of bug: 1306339

(In reply to :Gijs (he/him) from comment #7)

Alternatively there could be a webextension api for pragmatically handling download links without blocking user-prompts, or perhaps a flag equivalent to chrome's "download.prompt_for_download", but that actually works for mimetypes that aren't explicitly set in handlers.json?

This is probably a question for the webextension team. Shane, do you know if this is something that you'd consider and/or that should already be possible with the existing APIs?

Based on a rough understanding of the situation, which is to allow for download automation without user input, it seems like supporting a "download.prompt_for_download" pref in firefox is the simplest solution.

Even if this were done via webextension, we'd probably want a way to disable that prompt to make this work well.

Flags: needinfo?(mixedpuppy)
You need to log in before you can comment on or make changes to this bug.