Closed Bug 32741 Opened 26 years ago Closed 26 years ago

Downloading big files over FTP causes huge memory consumptions

Categories

(Core :: Networking, defect, P3)

x86
Linux
defect

Tracking

()

VERIFIED WORKSFORME

People

(Reporter: graf, Assigned: gagan)

References

Details

(Whiteboard: [dogfood-][nsbeta2+])

I was downloading a 50Mb file over the FTP and I noticed lots of swapping gooing on (this is a 128Mb RAM system). Running "top" revealed the horrors: -- PID USER PRI NI SIZE RSS SHARE STAT LIB %CPU %MEM TIME COMMAND 15204 graf 0 0 108M 55M 6804 S 0 0.0 44.2 24:39 mozilla-bin 15205 graf 0 0 108M 55M 6804 S 0 0.0 44.2 0:00 mozilla-bin 15206 graf 0 0 108M 55M 6804 S 0 0.0 44.2 0:18 mozilla-bin 15207 graf 0 0 108M 55M 6804 S 0 0.0 44.2 0:00 mozilla-bin 15223 graf 0 0 108M 55M 6804 S 0 0.0 44.2 0:00 mozilla-bin 14473 root 0 0 61480 52M 2104 S 0 5.9 41.8 5:26 X 4577 graf 20 19 13804 13M 248 R N 0 92.3 10.6 6751m setiathome ... --- Don't tell me it sticks all downloaded data into RAM and only writes it to the disk when the download is complete? Here's "top" right after Mozilla launches: --- ¹ PID USER PRI NI SIZE RSS SHARE STAT LIB %CPU %MEM TIME COMMAND 14473 root 0 0 63788 59M 2492 S 0 2.5 47.6 5:40 X 15278 graf 0 0 19744 19M 8952 S 0 0.0 15.4 0:15 netscape 15298 graf 0 0 16756 16M 9336 S 0 0.0 13.1 0:04 mozilla-bin 15299 graf 0 0 16756 16M 9336 S 0 0.0 13.1 0:00 mozilla-bin 15300 graf 0 0 16756 16M 9336 S 0 0.0 13.1 0:00 mozilla-bin 15301 graf 0 0 16756 16M 9336 S 0 0.0 13.1 0:00 mozilla-bin 4577 graf 20 19 13804 13M 252 R N 0 96.1 10.6 6759m setiathome ... --- This is Linux 2.2.13, glibc-2.1.2 (Slackware-7.0), Mozilla build 2000.03.20.18.
Networking - is this one for you? Gerv
Assignee: cbegle → gagan
Component: Browser-General → Networking
QA Contact: asadotzler → tever
*** Bug 33854 has been marked as a duplicate of this bug. ***
I've noticed the same thing. But it doesn't save all in RAM and then write to disk. It saves to disk in chunks of exactly 64k (64*1024 bytes). However, the "leak" or what this is, is huge. Just do a "free" while downloading. When i d/l Mozilla at least 20 MB free ram has vanished before it's half finished.
*** Bug 34580 has been marked as a duplicate of this bug. ***
also applies to Windows NT 4.0, Mozilla M14, Netscape 6.0 PR1 Downloaded file size was 30 megs, virtual memory usage according to Windows NT task manager was about 220 megs.
Status: UNCONFIRMED → NEW
Ever confirmed: true
=> valeski
Assignee: gagan → valeski
Target Milestone: --- → M18
and, over to law... ftp just opens a stream.
Assignee: valeski → law
*** Bug 36059 has been marked as a duplicate of this bug. ***
Same thing happens with downloads over HTTP.
is bug 33808, "Strange continuing ftp download." related to this one?
*** Bug 36672 has been marked as a duplicate of this bug. ***
Copied from bug 36672 "Virtual memory consuption during download": I saw that everytime you download a file, the memory used to temporary locate the data stream is not released, so if you open the Control Task window, and look at performance section, you can verify that there is this never-ending encrease of memory needed by the system ... and after some big file downloads (it also depends on memory avalaibility of your system) you receive a message that your system is out of virtual memory!!!
Move to M21 target milestone.
Target Milestone: M18 → M21
*** Bug 41566 has been marked as a duplicate of this bug. ***
Does setting this bug to M21 indicate that the new Netscape isn't supposed to be used for downloads?
No, we will straighten this out when we get downloading straightened out in general. I'm marking this [dogfood]. It should be M18.
Status: NEW → ASSIGNED
Keywords: dogfood
dogfood-, since it doesn't prevent developer's use of the product. Nominating for nsbeta2. Need more info on the problem, is this just a large unnecessary use of VM, or is it a leak, and is it cumulative across FTP operations?
Keywords: nsbeta2
Whiteboard: [dogfood-][need info]
*** Bug 39006 has been marked as a duplicate of this bug. ***
Reassigning to Networking component. The front-end code doesn't allocate any storage so it must be coming from Necko. Question: Does it consume the same amount of if you right-click on a link to the file and do "Save as..."? If not, then the problem may lie where we detect the "unknown content type" (and don't properly close the original connection).
Assignee: law → gagan
Status: ASSIGNED → NEW
cc'ing relevant people.
Is QA able to reproduce this? Could someone characterize it?
Yeah I can reproduce this using top on linux. Size of ram usage really increases during download as original reporter stated. I downloaded the jdk for a test and watched size (code, data, stack space) increase to about 80M. Compared to 4.7, size went to 16 M and stayed constant during the process.
Do you see the same result if you right click on the link and choose "Save Link As..." as you do if you click the link and get the unknown content dialog first?
nsbeta2+
Whiteboard: [dogfood-][need info] → [dogfood-][nsbeta2+]
right clicking, 'save link as' doesn't appear to be working right now. I can also see this in NT task manager, physical mem available continues to decrease for every download.
Chris: Can you run this through the gc leak detector?
I just ran FTP through purify using TestProtocols (mozilla/netwerk/test/) and a 10Mb file, and found one minor string leak (which I just fixed). That leak was during init and would have leaked 128 bytes at the most. So FTP is not the culprit here.
bug 43695 is about the same thing, but states that on linux the last week it now also crashes, somewhere after 30-50% d/l.
Whiteboard: [dogfood-][nsbeta2+] → [dogfood-][nsbeta2 ]
Adding the + to [nsbeta2 ] oops :-)
Whiteboard: [dogfood-][nsbeta2 ] → [dogfood-][nsbeta2+]
This appears to be working now. top is showing memory usage remains constant throughout process. WFM
Status: NEW → RESOLVED
Closed: 26 years ago
Resolution: --- → WORKSFORME
verif. Linus 2000071308
Status: RESOLVED → VERIFIED
*** Bug 45833 has been marked as a duplicate of this bug. ***
You need to log in before you can comment on or make changes to this bug.