Closed Bug 91232 Opened 23 years ago Closed 18 years ago

Mozilla eats CPU time like candy during downloading.

Categories

(Core Graveyard :: File Handling, defect)

Product:
Core Graveyard
Component:
File Handling
Platform:
x86
Linux
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
VERIFIED INVALID

People

(Reporter: dylang, Assigned: mscott)

References

(
URL
)

Details

(Keywords: perf, qawanted) 

Downloading a 150mb file, mozilla-bin is listed as taking 95% of all CPU time. 
In fact, I can type faster than this box can show text because of the weird CPU
suckage.  The only reason Mozilla-bin is not eating the entire 100% is that
dnetc merits some time.

 3350 dylang    16   0  136M 136M 13272 R    83.8 36.4  13:57 mozilla-bin
  161 root      19  19   488  488   412 R N  14.7  0.1 997:39 dnetc

On a 1.1Ghz Athlon, this should not be happening.  Please find the code in the
dowloading dialog to 2001070521 which does this.  I'm only downloading at
126k/s.  I hate to think what'd happen if I went and grabbed a larger file from
a server connected to me via 100Mbps switched ethernet.
Update: enter in the URL bar for going places ceased functioning, and transfers
in general slowed to a stall.

Upon canceling the dialog, some memory was freed.  But overall, the mozilla-bin
image is up to (wait for it!) 154MB.

 3350 dylang     9   0  154M 154M 13284 S     0.5 41.0  19:23 mozilla-bin

Ye gods!  wget handles the download at a constant 350k/s no problem.
Severity: normal → major
Keywords: perf
I don't see the memory growth.  I _do_ see about 30% cpu usage on a pIII 733.

The amount of data downloaded is updated about every 90K.  I'm on a fairly fast
connection (I was getting about 200K/sec from the site in question).  Twice a
second seems like a little too often to update...

Over to jrgm for perf testing...
Assignee: asa → jrgm
Status: UNCONFIRMED → NEW
Ever confirmed: true
(Heh, I hit "Commit" like 12 hours ago, and just noticed that this browser 
window was still waiting for me to log in. Oops!)

It sounds more like something went wrong with Dylan's download session. I've 
tried this on linux and win2k, both at full available bandwidth (LAN->Internet)
and through a "throttling" proxy server that does 56K. On Linux, I could see 
two mozilla-bin threads, one at ~4% and the other at ~33% during the download
at high-speed. For the "56K" download, only one thread at ~1%. On windows, it
was ~10% and ~1% (I didn't look at the individual threads on Windows). [Testing
with current trunk builds for today].

But by comparison, 'cat' and 'cp' will peak out at ~40% CPU on linux rh 6.2
when streaming/copying a ~20MB file, and a command line perl getter will take
about 15%.

Does this happen everytime you do this download (both the 95% CPU, and the 
excessive RAM usage). Anything unusual about your setup?
What OS were you using?

I was using 2001070521 under Slackware 8 with its Gnome 1.4.
John, I'll try to give some more info.  wget on this download (with constant
redraws, etc) in a gnome-terminal does:
15376 dylang     9   0   804  804   692 R     1.7  0.2   0:00 wget 

Even though it's spewing:
 9650K .......... .......... .......... .......... ..........  6% @ 344.83 KB/s
 9700K .......... .......... .......... .......... ..........  6% @ 333.33 KB/s
 9750K .......... .......... .......... .......... ..........  6% @ 344.83 KB/s
 9800K .......... .......... .......... .......... ..........  6% @ 344.83 KB/s

almost constantly.
1.7 % CPU time, 800kb ram :)

Now I load Mozilla, with one blank window open:
15392 dylang     9   0 19776  19M 10976 S     0.0  5.1   0:02 mozilla-bin
15393 dylang     9   0 19776  19M 10976 S     0.0  5.1   0:00 mozilla-bin
15394 dylang     9   0 19776  19M 10976 S     0.0  5.1   0:00 mozilla-bin
15395 dylang     9   0 19776  19M 10976 S     0.0  5.1   0:00 mozilla-bin
15396 dylang     9   0 19776  19M 10976 S     0.0  5.1   0:00 mozilla-bin

Now I go to download the file... but I can't seem to browse to /tmp (file dialog
bug?)..  I manually enter a location to download to.


15392 dylang     9   0 33432  32M 11964 R    53.3  8.6   0:22 mozilla-bin
15393 dylang     8   0 33432  32M 11964 S     0.0  8.6   0:00 mozilla-bin
15394 dylang     9   0 33432  32M 11964 R     2.5  8.6   0:01 mozilla-bin
15395 dylang     9   0 33432  32M 11964 S     0.0  8.6   0:00 mozilla-bin
15402 dylang     9   0 33432  32M 11964 S     0.0  8.6   0:00 mozilla-bin

The longer I let it run, the more memory and CPU time it takes...

15392 dylang    15   0 38892  37M 11964 R    75.3 10.1   0:39 mozilla-bin
15393 dylang     8   0 38892  37M 11964 S     0.0 10.1   0:00 mozilla-bin
15394 dylang     9   0 38892  37M 11964 R     3.9 10.1   0:02 mozilla-bin
15395 dylang     9   0 38892  37M 11964 S     0.0 10.1   0:00 mozilla-bin
15402 dylang     9   0 38892  37M 11964 S     0.0 10.1   0:00 mozilla-bin
The speed has dropped to 200k/s..

15392 dylang    12   0 48456  47M 11964 R    79.2 12.5   1:19 mozilla-bin
15393 dylang     8   0 48456  47M 11964 S     0.0 12.5   0:00 mozilla-bin
15394 dylang     9   0 48456  47M 11964 S     0.7 12.5   0:02 mozilla-bin
15395 dylang     9   0 48456  47M 11964 S     0.0 12.5   0:00 mozilla-bin
15402 dylang     9   0 48456  47M 11964 S     0.0 12.5   0:00 mozilla-bin
The longer it downloads, the longer it takes.  It's an evil problem.
3:39 into the download:

15392 dylang    15   0 60152  58M 11964 R    80.7 15.6   2:29 mozilla-bin
15393 dylang     8   0 60152  58M 11964 S     0.0 15.6   0:00 mozilla-bin
15394 dylang     9   0 60152  58M 11964 S     0.0 15.6   0:02 mozilla-bin
15395 dylang     9   0 60152  58M 11964 S     0.0 15.6   0:00 mozilla-bin
15402 dylang     9   0 60152  58M 11964 S     0.0 15.6   0:00 mozilla-bin

More ram, but no more CPU time because now it's hitting against dnetc, X, etc in
the scheduler.

Finally, just before I canceled the dowload:

15392 dylang    19   0 68144  66M 11964 R    80.6 17.7   3:31 mozilla-bin
15393 dylang     8   0 68144  66M 11964 S     0.0 17.7   0:00 mozilla-bin
15394 dylang     9   0 68144  66M 11964 S     0.3 17.7   0:02 mozilla-bin
15395 dylang     9   0 68144  66M 11964 S     0.0 17.7   0:00 mozilla-bin
15402 dylang     9   0 68144  66M 11964 S     0.0 17.7   0:00 mozilla-bin

Immediately after canceling:

15392 dylang     9   0 69288  67M 11964 S     0.0 18.0   3:40 mozilla-bin
15393 dylang     8   0 69288  67M 11964 S     0.0 18.0   0:00 mozilla-bin
15394 dylang     9   0 69288  67M 11964 S     0.0 18.0   0:03 mozilla-bin
15395 dylang     9   0 69288  67M 11964 S     0.0 18.0   0:00 mozilla-bin
15402 dylang     9   0 69288  67M 11964 S     0.0 18.0   0:00 mozilla-bin

No CPU time, but lots of wasted ram.  Leak :p

So there you have it.  Moz leaks somewhere it is download stack, and it acts as
its own throttling tool for some reason -- eating CPU time and RAM like a
naughty app.
Assignee: jrgm → mscott
I am using redhat 6.1 (despite above, where I slipped and said 6.2).
At any rate, on my system, as before, I see about 35% CPU usage, most of 
that apparently on the main UI thread, and only about 2 or 3% on the necko
thread. I do not see memory usage climbing.

So:
1) yes that dialog updates 2 times a second (although it may get many more 
   onProgressChange updates than that). But I think that 2 per second is 
   right and shouldn't be a problem. 

2) we're not particularly efficient with cpu resources while downloading, but
   as long as we stay in the 30% range [on linux] I don't see this as a major
   concern.

3) ... but 85%+ usage, and a growing heap, are major concerns [so that's what
   this bug "is"]. However, we don't yet know why you see this, but bz and I 
   don't. 

I don't know. What is the kernel you are running (2.2 or 2.4; slackware 8.0
comes with both, no?).

It looks from the PIDs that it is the UI thread that is screaming away, but
aside from that, there's not much else I can say about what's happening. And,
given that, I'm kicking this over to mscott (don't thank me too much) who 
knows more about the helperAppDldProgress.js. Any ideas about this?
> as long as we stay in the 30% range [on linux] I don't see this as a major

I'm not saying I _like_ that; just that it is not the tallest poppy out there
(first thing to cut down). 
I'm using 2.4.6 because of the VM issues in 2.4.5.

I couldn't reproduce this.  I let the download run for 6 minutes (about 50%) and
Moz had used 42 seconds of CPU without leaking memory.  I am using a proxy on
localhost.
Seen on RH7.1/Rawhide with a self-built 2.4.5 kernel as well, with 20010727
build.  It sucks it down just fine speed-wise (a 13M MP3 of an hour-long SANS
lecture just came down at 850K/sec on a 10BaseT) but is chowing CPU (80%+ on a
700mz Dell).

This almost smells like the old lack-of-optimization bugs *very* old FTP clients
had - in ASCII mode, they'd do basically "while (isascii(getchar()) putchar();"
rather than doing reasonable-lenght reads.  I wonder if there's different
handling based on whether the content-length is known before hand or not (the
high-CPU downloads all seem to display 'nnnK of ??').
*** Bug 87249 has been marked as a duplicate of this bug. ***
I'm copying over this insightful comment from bug 87249 (courtesy of Yaron Minsky):


When I download a big file fast on Linux, I also see ridiculous CPU usage.  An
example is downloading an OpenOffice build.  I'm running a ~700MhZ Pentium III,
and on the same machine, wget uses about .9% of CPU, whereas Mozilla uses 80%. 
This is with download speeds of roughly 500KB/s.  

I think part of this is that the display dialog gets updated faster when
downloading is faster, so that with a really fast download the numbers on the
dialog are spinning at a frantic pace.  I suspect (indeed I hope) that that's
the source of the high CPU usage.
Summary: Mozilla eats CPU time like candy during downdloading. → Mozilla eats CPU time like candy during downloading.
*** Bug 103544 has been marked as a duplicate of this bug. ***
Coping darin's comment from the dupe (Darin and I looked at this a couple of
months ago):

"this problem only occurs after choosing a location for the downloaded file, but
necko is busy downloading the file in the background well before this happens.
so, this must be a problem with the download progress dialog.

-> mscott (mr. download dialog)"
Basically, we're spending all our time updating the progress bar....
can the progress bar be made to render more like IE download dialogs on a
WinBox? It seems like it is rendering a lot of little progress lines which (IE
browser does and Mozilla browser -Transfering Data Progress Bar, and Throbber)..
as much as I like this incrimentally correct download item.  I would like maybe
see an implementation using block sizes.  like 1 block = 16K?  I think that
sometimes the it does eat to many cpu cycles for rendering this content, but
have not really looked deeply on the perf of this.
->file handling, but retriage if needed.
Component: Browser-General → File Handling
QA Contact: doronr → sairuh
Do people see the same cpu usage characteristics if they download via
right-click and "Save Link As..."?  That progress dialog only updates half as
often (once every second).
Some evaluation of such a download session:
The load seems to increase during the download from an average of 15% at the
beginning of the download to an average of 50% with average peak values of 98 %
CPU load at the end of the download. At download time I have an average of
~80kb/sec download volume. The latency of the connection to www.eff.org is quite
high in comparison to a mainstream site here in germany (www.heise.de). The time
of a ping request is about 5 times higher than to heise.de
Mozilla.org has a resident set size of 94MB when downloading a 64MB MP3 file
from www.eff.org

System: SuSE Linux 7.2 with 2.4.4 Kernel, Athlon 1.4 Ghz, 512 MB DDR-RAM, 2 MBit
DSL connection
the problem with that is that CPU is trying to manage the memory or VM in
windows for files before it writes it out to disk it stores the whole file in
memory and that is another bug I've seen someone file about 100% cpu load at end
of downloading 100MB file while total memory Mozilla was using was more than his
RAM size and Vm allocation I think.  So that is part of the problem, I dont
think IE stores the file in Memory before writing it out.. Mozilla does though
and I dont know if this can be changed give that mozilla is a XPapp.  The ideal
solution is like ie/aimster/napster does it writes the file to disk as it is
downloading.. because you can start to play the file as soon as it is written to
disk, just like how images can start to be displayed even though they are not
fully downloaded yet..  and even if it stops downloading..

Bill, maybe we should take the later approach to downloads write to disk like we
display images right away.. this alone adds the most CPU usage to Mozilla
downloading.. progress bar takes away the UI update refreshing cpu/interupts the
writing to disk - unless it can be done in parallel?  some stuff to think about.. 

BTW, check the newsgroup discussion for some suggestions on way to cut back on
rendering during the download.. 

-dman84
check out bug 113163 for possible patch that may fix some of the problem here.
Depends on: 113163
*** Bug 122355 has been marked as a duplicate of this bug. ***
*** Bug 131449 has been marked as a duplicate of this bug. ***
QA Contact: sairuh → petersen
Mozilla tracks what it downloads in a file called downloads.rdf. The more
entries in this file, the slower Mozilla appears to download a file. 

I deleted the file download.rdf and then as a small file test, downloaded the
file from http://antwrp.gsfc.nasa.gov/apod/image/9707/eagle_cfa_big.gif - size
346KB; dimension 848 x 848. It downloaded nearly instantly. The cpu load was low
with only a momentary spike to 81% as Mozilla opened the download manager and
created a new downloads.rdf file.

For the big file test, I downloaded the lastest build of
mozilla-win32-installer-sea.exe (11.4MB) from
http://ftp.mozilla.org/pub/mozilla/nightly/latest/
Download time was about 56 seconds and the cpu load never rose 24% on my PIII
550 MHz w/256MB RAM.

Prior to this test, cpu load would peg to 100% and remain there for upwards of a
minute even for the tiniest download.

My current version is
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.4b) Gecko/20030508

actually, issues related to downloads.rdf is bug 161783
the single testcase URL prior to comment 23 is dead. no useful info in the dupes (one was MAC). there are many old and open bugs this could be duped to but no way to verify without one of the commentors to test. so closing this INVALID
Status: NEW → RESOLVED
Closed: 18 years ago
Resolution: --- → INVALID
verify. This bug is from 2001, no activity since 2003.

I have seen Mozilla eat candy while downloading bugzilla bug lists, but that candy is not CPU time.
Status: RESOLVED → VERIFIED
Product: Core → Core Graveyard
You need to log in before you can comment on or make changes to this bug.