Prevent auto-resumption of downloads at startup
Categories
(Firefox :: File Handling, enhancement)
Tracking
()
People
(Reporter: from-mozilla, Unassigned)
Details
Attachments
(3 files)
User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:88.0) Gecko/20100101 Firefox/88.0
Steps to reproduce:
Start downloading any large file (10 GB or larger) from Google Takeout
browser crashes / OOM killed / system powers off / etc
Check for a ".part" file, note that a large portion of the download has been completed
Restart browser
(the remaining steps are my intent; they have no bearing on the outcome, because by this point the fault has already occurred)
Log into Google Takeout
Attempt to re-start download
Change browser.download.manager.resumeOnWakeDelay to a much larger number, and try again next time it fails.
Actual results:
Background: Firefox crashes on average once every 8-10 GiB during downloads, but the same thing happens however Firefox gets restarted. Other activity influences this, but even just leaving the computer completely untouched it rarely gets past about 12GB.
Before I even get to log into Google Takeout, Firefox has attempted to resume the download, but without the necessary credentials.
This results in Takeout sending a 68-byte file that says "domain expired", and the previous partial download is discarded.
This results in repeated attempts to download the same file, often resulting in Google refusing saying "too many download attempts".
Setting browser.download.manager.resumeOnWakeDelay to 1000000 does not seem to have any effect. I would expect that to make it wait about a quarter of an hour before resuming downloads, but they seem to resume anyway.
My current work-around is:
- before restarting Firefox, move the .part files so that Firefox can't remove them
- start Firefox, wait 10 seconds for it to auto-resume
- delete any 68-byte files
- request download again
- wait until it appears in the download drop-down menu, then hit the "×" to pause the download
- wait 5 seconds; move the previous .part file so that it replaces the much smaller, new .part file (this works even if you use a different "save as" name)
- in the download drop-down menu, hit the circular arrow to resume
The delay makes this more reliable; I'm not sure why.
Expected results:
This is both a bug report and a feature request:
bug report: Setting browser.download.manager.resumeOnWakeDelay to a huge number should effectively prevent downloads from automatically resuming.
feature request: please provide some UI managing partial downloads upon restart; I suggest making it part of about:sessionrestore; in any case, don't auto-resume until after the user responds to about:sessionrestore.
I don't even care about having to mess about renaming the .part files; what's important is that I don't lose them simply by restarting firefox before I realise that they're still pending.
This is important because I need to download dozens of files that take in excess of an hour each, within a six-day limit.
The six-day limit is important because a Takeout image comprises multiple tarballs or zip files, and the grouping of images, videos, emails, etc within those files is essentially random each time (it's a bin-packing problem to bring each file as close to but not exceeding the requested size limit, in my case 10 GB), so it's not simply a matter of making a new Takeout image and starting from the part number where I left off.
The "domain expired" is because the URL for each part of the Takeout image changes each time it's requested, as a security measure to prevent replay attacks.
Comment 1•3 years ago
|
||
The Bugbug bot thinks this bug should belong to the 'Firefox::File Handling' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.
I mentioned a 68-byte file; this takes the form Locked Domain Expired: Not valid after ????-??-??T??:??:??.???-0?:00
(where the last part is -07:00
or -08:00
for the US Pacific Timezone, or "Mountain View Standard Time"). These bytes are inside a file whose name ends with .zip
or .tgz
.
Log into Google Takeout
Attempt tore-startresume download
I should have said "resume" rather than "re-start", as the difference is quite important in this context.
Each part takes 70-80 minutes if Firefox isn't also doing anything else, and will usually complete without interruption, while two simultaneously will take about 120-130 minutes, but increases the likelihood of Firefox crashing before either can complete.
There is a limit on the number of attempts that can be made to download each part, about six though I'm not certain as I only ever encounter it when other things are already going wrong. As a result downloading more than 4 parts concurrently pushes up the number of restarts required - even using my pseudo-resume method - to the point where exceeding the per-part limit on download attempts becomes likely or inevitable.
A complete download takes of the order 2 DAYS when it goes WELL, or 3-4 if I have to manage interruptions and resume downloads.
There is a limit of six days to download all parts of the Takeout image, so there isn't much leeway for "problems".
If I exceed either limit that drastically reduces the utility of the entire download as it fails to be a complete backup.
Whilst in theory I could elect to take the download in smaller pieces, to reduce the loss if parts have to be restarted, that means I effectively had a full-time job for 3 days just hitting "download next" every few minutes. Me taking a coffee break results in lost download time.
While the "mean time between crashes" is about an hour, or about 8GB, I've seen it abort as early as 25 minutes or 3GB, and - just once - managed to download two full 10GB parts consecutively without crashing on either.
I'm confused why the User-Agent
says x86_64
when this is the 32-bit version:
$ file -L /usr/bin/firefox
/usr/bin/firefox: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=cb9f59d390f9a888e55fcbc2ea2fa2ec69183094, stripped
$ dpkg -l libc6:i386
...
ii libc6:i386 2.23-0ubuntu11.2 i386 GNU C Library: Shared libraries
$ uname -ioprsv
Linux 4.4.0-140-generic #166~14.04.1-Ubuntu SMP Sat Nov 17 01:52:43 UTC 2018 x86_64 x86_64 GNU/Linux
$ free
total used free shared buffers cached
Mem: 8165492 7909328 256164 200820 465720 4708768
-/+ buffers/cache: 2734840 5430652
Swap: 5001212 309040 4692172
$ grep model /proc/cpuinfo
model : 15
model name : Intel(R) Core(TM)2 Duo CPU T7100 @ 1.80GHz
model : 15
model name : Intel(R) Core(TM)2 Duo CPU T7100 @ 1.80GHz
Oh, and I should probably mention: I have a Linux cgroup
that restricts Firefox to 4GiB of memory (across all instances combined), to ensure that it can't go into a swapping frenzy, or starve Xorg or other critical processes.
And no, I can't simply upgrade the OS, because I've yet to find a newer release that supports my graphics card.
Comment 6•3 years ago
|
||
Hi Martin, I will change the type of this Bug bug to Enhancement/Request based on:
- bug report: Setting browser.download.manager.resumeOnWakeDelay to a huge number should effectively prevent downloads from automatically resuming.
- feature request: please provide some UI managing partial downloads upon restart; I suggest making it part of about:sessionrestore; in any case, don't auto-resume until after the user responds to about:sessionrestore.
Can you please also share the about:support info? Unfortunately, I don't have the needed setup to check this out. Please also share the crash report, even if it is an OOM crash, it might have some useful info that could fix this crash in the first place. You can find the crash report in about:crashes, please make sure to submit it and send over that link.
Thanks!
(Oh great, "upload as attachment" ALSO submits my half-written comment, with no way to go back and edit it.
I'm afraid github is now setting a high bar as the minimum standard for all bug trackers.)
I was going to say, I finally finished my 3-day download a few days ago, and Firefox has since auto-updated from 88.0.1 to 89.0, so I'm not certain whether the about:support will be useful, but I've attached it anyway.
If I run Firefox from a shell, I get random reports about widgets, and usually the last thing before the shutdown is something about a channel closing, rather than termination by a signal or exception.
As soon as I've finished writing this, I will restart Firefox from a terminal, and the next time it stops I will add another report here.
uploaded crash report bp-e74945c5-aa88-402f-89a2-ab80c0210607
Reporter | ||
Comment 10•3 years ago
|
||
That crash report was from 3 May (5 weeks ago), which now I think about it, corresponds to (shortly before) when I upgraded the RAM from 2.5 GiB to 8 GiB, and increased the Firefox cgroup limit from 1.5GiB to 4GiB.
So those crashes were very likely related to memory pressure, but (because of the cgroup) would have been malloc/sbrk/mmap(ANON) failing, rather than the kernel's OOM-killer on the loose.
Although it's no longer "crashing" (merely quitting without me asking it to) I suspect it's still encountering the (revised) memory limit.
I will try to add some instrumentation to periodically check the "available memory" in that cgroup.
Reporter | ||
Comment 11•3 years ago
|
||
Reporter | ||
Comment 12•3 years ago
|
||
Comment 13•1 year ago
|
||
(In reply to Timea Cernea [:tbabos][inactive] from comment #6)
Can you please also share the about:support info? Unfortunately, I don't have the needed setup to check this out. Please also share the crash report, even if it is an OOM crash, it might have some useful info that could fix this crash in the first place. You can find the crash report in about:crashes, please make sure to submit it and send over that link.
See my comments above.
Until version 115 things had been quite stable, but since version 116 I'm seeing a lot more crashes and abrupt shutdowns due to memory pressure. I save stdout/stderr from Firefox into a log file, and since 2023-08-03_11:32:52+1000 the term [GFX1-]: wr_renderer_render: OutOfMemory
has appeared 1200 times.
These are my submitted crash reports; I note that they are only a small fraction of incidents, as often Firefox dies (or has to be killed) without getting a chance to generate a crash report:
bp-2faff6a6-ffe1-4278-9a78-d5a2e0230622 6/22/23, 12:10
bp-a5ca0d1c-f4f3-4cb2-b847-4acbc0230617 6/17/23, 17:26
bp-3d872912-2d77-44bb-93b3-e581a0230617 6/17/23, 17:20
bp-dfe343dc-9d1f-4378-9e35-f49520230617 6/17/23, 16:18
bp-e4ca1cd0-2841-4a83-aae4-6145e0230617 6/17/23, 16:17
bp-d8a766bd-5a0d-4029-b2a5-a7e830230617 6/17/23, 16:14
bp-f669651f-3e6e-440e-bc77-a857b0230617 6/17/23, 16:14
bp-2deb8e28-d944-400f-b322-715940230606 6/6/23, 13:20
bp-ad4e85bc-7062-4716-9d83-70c950230601 6/2/23, 04:24
bp-576f160a-50a8-4951-bd68-901eb0230601 6/2/23, 04:21
bp-fea5ba55-7275-4e78-b470-277450230601 6/2/23, 04:21
bp-09f1f355-849f-4a53-a183-3b1110230309 3/9/23, 15:19
bp-cd85dd1f-c6ca-44d4-8768-6c7ec0220810 8/10/22, 10:49
bp-214e01a0-4604-4398-accd-1a6e80210928 9/28/21, 10:13
bp-335440c1-52ad-40c1-9743-ccd8a0210928 9/28/21, 10:13
bp-f221bf6f-c35b-4207-bf1a-abc650210928 9/28/21, 10:13
bp-8d897fc7-5d48-4729-842a-cb77a0210928 9/28/21, 10:13
bp-61cd04ea-62c8-4243-a5d7-01c090210928 9/28/21, 10:13
bp-0884c738-5679-40ec-913b-1b67b0210928 9/28/21, 10:13
bp-43289927-5ae5-47b9-853c-3c5680210928 9/28/21, 10:13
bp-cf48d420-6592-438f-b3b6-768fb0210928 9/28/21, 10:13
bp-9c231e85-e248-4f44-ad19-d54890210928 9/28/21, 10:13
bp-bbd5fe69-423c-4e5e-8b13-38a460210928 9/28/21, 10:13
bp-e66f8421-538f-4393-bbb0-982350210819 8/19/21, 11:15
bp-6a7bf943-8cad-4321-b6f8-7bb610210813 8/13/21, 10:32
bp-e74945c5-aa88-402f-89a2-ab80c0210607 6/8/21, 00:29
bp-1a031998-33c5-4997-9673-d4a800210505 5/5/21, 15:41
bp-8735ab2a-7c80-4036-b9cf-74d020210412 4/13/21, 09:55
bp-b7403e8d-225a-4d4f-a696-c86b50210103 1/3/21, 11:08
bp-abc69664-9542-4971-aced-594c80201231 1/1/21, 09:57
bp-f6ae73b4-64cd-4c93-95d5-c2dac0201228 12/28/20, 12:10
bp-5c401f46-ea82-4c1d-91c1-718a50201214 12/14/20, 10:53
bp-bd6e583f-eaf1-4dd8-8103-aaa620201012 10/12/20, 14:51
bp-a1998c2b-3d02-4560-8650-4f0710201012 10/12/20, 14:44
bp-4833428b-df50-4eef-b8d6-e81c00201012 10/12/20, 14:44
bp-5d57accd-aa03-4f74-84ec-57d230201012 10/12/20, 14:44
bp-c9ff636c-12b6-424f-84e6-1d4950200917 9/18/20, 07:45
bp-ea255756-692d-436c-9dd8-730090200906 9/6/20, 10:41
bp-e7b8913c-203d-4864-9ff2-a02ce0200629 6/29/20, 11:43
bp-31b5abfa-1d87-406a-87b2-c8c2d0230809 8/9/23, 14:48
bp-55409764-cb90-4546-b333-2620e0230809 8/9/23, 14:48
bp-6980546e-9313-46bd-b53b-ef61f0230809 8/9/23, 14:48
bp-b242d8f8-41e1-4823-a7d5-f49970230809 8/9/23, 14:48
bp-b7098dd5-39e7-4565-991e-1f6590230809 8/9/23, 14:48
bp-a107ca8f-c730-409c-9d65-a48470230809 8/9/23, 14:48
bp-274947d6-1fc0-4c38-8301-2fe340230809 8/9/23, 14:48
bp-cc5f5da1-d1bd-438a-a74a-96a7e0230809 8/9/23, 14:48
bp-221db55f-803f-4839-a285-149d50230809 8/9/23, 14:48
bp-89d378fc-a1f2-4b7d-b4db-76b980230809 8/9/23, 14:48
bp-84b267b1-686b-4349-afc2-6dd930230809 8/9/23, 14:48
bp-aefb1101-c213-41f4-909a-e82fb0230809 8/9/23, 14:48
bp-6c72b08b-a04e-4918-9f20-6e6820230809 8/9/23, 14:48
bp-13ef43ce-aac1-44ce-a8c2-0aa1e0230809 8/9/23, 14:48
bp-40a4c536-ba3b-40af-a275-1bba80230809 8/9/23, 14:48
bp-e2d0e8ef-e716-4cb0-a556-a13d30230809 8/9/23, 14:48
bp-040db3a9-8b05-4953-8e66-54ee80230809 8/9/23, 14:48
Description
•