Last Comment Bug 291643 - high cpu and 100-1 ratio memory opening large xml file
: high cpu and 100-1 ratio memory opening large xml file
Status: NEW
[MemShrink:P3]
: perf
Product: Core
Classification: Components
Component: XML (show other bugs)
: Trunk
: x86 Windows XP
-- major with 10 votes (vote)
: ---
Assigned To: Nobody; OK to take it and work on it
:
: Andrew Overholt [:overholt]
Mentors:
http://www.w3.org/2000/10/swap/test/p...
: 106357 378330 436150 526825 678849 (view as bug list)
Depends on: 197956
Blocks: 378330 mlk-fx4
  Show dependency treegraph
 
Reported: 2005-04-23 15:14 PDT by Carl Antaki
Modified: 2014-06-30 10:34 PDT (History)
26 users (show)
jonas: blocking1.9-
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
The file in question, zipped up (76.77 KB, application/zip)
2007-03-06 15:51 PST, Boris Zbarsky [:bz] (still a bit busy)
no flags Details
This is the current file where I could see the problem. (149.84 KB, application/xml)
2008-10-24 02:20 PDT, Gerald
no flags Details
This file causes 100 % CPU load on my FireFox 3.0.3 (149.84 KB, application/zip)
2008-10-24 02:23 PDT, Gerald
no flags Details
about:memory results (16.03 KB, text/plain)
2011-07-15 20:43 PDT, Zlip792
no flags Details
about:memory?verbose (latest nightly) (46.50 KB, text/plain)
2011-08-06 09:39 PDT, Zlip792
no flags Details
DMD output (82.93 KB, text/x-vhdl)
2014-05-27 17:00 PDT, Nicholas Nethercote [:njn]
no flags Details

Description User image Carl Antaki 2005-04-23 15:14:07 PDT
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.7) Gecko/20050414 Firefox/1.0.3 (ax)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.7) Gecko/20050414 Firefox/1.0.3 (ax)

Firefox 1.0.3 takes too much memory when opening iTunes Music library.xml, my
file size is 2.46MB. Firefox memory consumption goes as high as 300MB of RAM and
300MB of virtual memory as seen on Windows task manager. It takes around 5min
for the page to load which is unacceptable (I have and old P3 1Ghz PC with 512MB
of RAM)

Reproducible: Always

Steps to Reproduce:
1.Open the file with Firefox
2.
3.

Actual Results:  
Firefox consumes 300MB of RAM and 300MB of virtual memory

Expected Results:  
The software should have consumed less memory

Windows XP using the old Windows 2k theme. 
Pentium 3 1Ghz PC with 512MB of RAM
Comment 1 User image Matthias Versen [:Matti] 2005-04-23 16:35:46 PDT
Reporter:
Can you pelase retest a current trunk build because the gecko in FF1.0.3 is very
old.
Thanks
Comment 2 User image Carl Antaki 2005-04-23 16:55:32 PDT
I tried it on the most recent trunk build and it gives the same results.
Comment 3 User image Matthias Versen [:Matti] 2005-04-23 17:42:57 PDT
Do you have a URL for a testcase ?
Comment 4 User image Carl Antaki 2005-04-23 19:46:34 PDT
This is a test case:
http://www.w3.org/2000/10/swap/test/plist/iTunesMusicLibrary.xml

In this test Firefox doesn't take 300MB because the file is smaller, it takes
169MB of RAM and 169MB of virtual memory.
Comment 5 User image Boris Zbarsky [:bz] (still a bit busy) 2005-05-24 10:56:51 PDT
This is a little odd.  I'd expect a 10-1 size increase (a la view-source), but
not 100-1....  sicking, any idea what's up here?
Comment 6 User image Wayne Mery (:wsmwk, NI for questions) 2007-03-06 15:43:49 PST
I get roughly 100-1 ratio also.  xml file size 1.6MB, firefox took something over 120MB. High CPU ~50% for the duration of loading

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a3pre) Gecko/20070216 Minefield/3.0a3pre 

Comment 7 User image Boris Zbarsky [:bz] (still a bit busy) 2007-03-06 15:51:47 PST
Created attachment 257578 [details]
The file in question, zipped up
Comment 8 User image Jonas Sicking (:sicking) No longer reading bugmail consistently 2007-03-06 16:18:04 PST
That it's taking more than view-source doesn't surprise me at all given that we're creating both more elements, and more complex layout trees (that includes tables).
Comment 9 User image Erik den Houter 2007-04-22 13:49:36 PDT
Don't know what you're talking about, but maybe
have a look at "my" bug 378330, and maybe you can help me
explane what's going on.

I'm just new in wwwonderland, and would love to know whats
eating my memory and time...
Comment 10 User image Erik den Houter 2007-04-23 13:42:38 PDT
*** Bug 378330 has been marked as a duplicate of this bug. ***
Comment 11 User image Jonas Sicking (:sicking) No longer reading bugmail consistently 2008-09-11 21:34:14 PDT
Is this still happening? Things have changed a lot in how we prettyprint xml so we might be in a better state. Also, if this was caused by leaks we might be in a better state due to the cycle collector.
Comment 12 User image Erik den Houter 2008-09-12 05:18:08 PDT
I do not use that machine any more for daily use, so I hat to boot it just to check what's up.
As I reported at "my" bug 378330 where this URL caused the problem:
http://www.ndfr.net/sitemap.xml
on a Win 98SE P3 450 Mhz with 284 MB memory and FF 2.0.0.3,  I cannot reproduce it for the fact that that page has completely changed, so there's no comparing.

Trying the URL from "this" bug-report:
http://www.w3.org/2000/10/swap/test/plist/iTunesMusicLibrary.xml
on the same machine, but now with FF 2.0.0.14, it takes about four minutes to load, and during that time the PC still cannot be used, an hourglass for four minutes, and all of the free memory is still consumed, but it loads The page was loaded direct after boot.

Trying the URL on a faster machine with FF 3.01  (C2D 6600, 2GB) the page loads within eight seconds, and it seems that I can switch to other activities while loading, so the handling of that pages is much better.
( Still 147 MB RAM is consumed according to Freeram XP Pro, while saving the page is taking 1,54 MB harddisk space.)
Comment 13 User image Boris Zbarsky [:bz] (still a bit busy) 2008-09-12 08:44:21 PDT
So with a current tip build, starting the browser takes about 43MB of RAM.  Opening this XML file peaks at about 226MB RAM, and then drops down to 193MB.

So about 150MB for the rendering.  All this on Mac.  I wonder how much bug 231925's suggestion to use XML would help.  I'll test.
Comment 14 User image Boris Zbarsky [:bz] (still a bit busy) 2008-09-12 10:29:45 PDT
I tried that (on the assumption that it would shave off those 4 bytes for the class attribute), but it wasn't much of a change, if any.

Fundamentally, when rendering something like:

  <myTag>
    ...
  </myTag>

where the "myTag" is N characters long, we have the following objects here (most of these numbers also involve malloc overhead, but I'm ignoring it for now):

1 pointer from the parent: 4 bytes
1 nsXMLElement (the original element): 32 bytes
1 nsHTMLDivElement wrapping the whole thing: 32 bytes
1 nsHTMLDivElement for the expander: 32 bytes
1 nsTextNode for the expander text: 36 bytes + 1 byte for the text
1 nsTextNode for the text '<': 36 bytes + 1 byte for the text
1 nsHTMLSpanElement with one attribute: 40 bytes
1 nsTextNode for the "myTag": 36 bytes + N bytes for the text
1 nsTextNode for the text '>': 36 + 1 bytes
1 nsHTMLDivElement with one attribute for the stuff this contains: 40 bytes
1 nsTextNode for the text "</": 36 + 2 bytes
1 nsHTMLSpanElement with one attribute: 40 bytes
1 nsTextNode for the text '>': 36 + 1 bytes
6 nsTextFrames: 6 * 64 bytes
2 nsInlineFrames: 2 * 56 bytes
3 nsBlockFrames: 3 * 88 bytes
3 nsLineBoxes: 3 * 40 bytes

Total (minimal estimate): 1322 + 2N bytes.  I think I'm not counting the linebox in the parent that contains all that stuff, etc, but let's go with this number.

Initial memory usage in the file was 2N + 5 bytes.   If N is about 10, then we have 1342 bytes vs 25 bytes, which is a 50x increase.  If N is about 5, then we have 1332 bytes vs 15 bytes, which is closer to 80x.

This file has tagnames (more or less in order of how common they are): key, integer, string, dict, date.

For "key" the factor is 1328/11 or about 130x.
For "dict" and "date" the factor is 1330/13 or about 100x
For "string" the factor is 1334/17 or about 80x
For "integer" the factor is 1336/19 or about 65x

So an average of 100x is not surprising at all.  In fact, if the page were using one-character tag names, the factor would be 1324/7, or about 200x.

So what can we do to reduce memory usage here?  One obvious thing to do would be to not create separate textnodes for the '>' and '<'.  In this case, that would save us 400 bytes per node with kids, or about 1/3 to 1/4 of the memory we're using.  It might not look as nice, but I doubt it'd look that much less nice, honestly.
Comment 15 User image Boris Zbarsky [:bz] (still a bit busy) 2008-09-12 11:17:25 PDT
I poked at the XSLT, and I'm not sure I can actually do separate textnodes here.... I assume that if an xsl:text and an xsl:valueof are next to each other they don't get combined into a single textnode, or anything?
Comment 16 User image Jonas Sicking (:sicking) No longer reading bugmail consistently 2008-09-12 14:23:34 PDT
XSLT should never produce two sibling textnodes. A <xsl:text> next to an <xsl:value-of> should get combined into a single node in the result DOM.
Comment 17 User image Boris Zbarsky [:bz] (still a bit busy) 2008-09-12 18:01:20 PDT
OK.  So I did test getting rid of some of the extra textnodes and as expected the memory used to do the prettyprinting drops from about 150MB to about 110-120MB.  Performance increases a bit (from 9.5 seconds drops to 7.7 or so).

Not sure whether we care enough to sacrifice looks here for that.
Comment 18 User image Gerald 2008-10-24 02:20:30 PDT
Created attachment 344602 [details]
This is the current file where I could see the problem.

My FireFox 3.0.3 has 100 % CPU load when opening this file. It also happens when trying to update a RSS feed, referring to this file!
Comment 19 User image Gerald 2008-10-24 02:22:21 PDT
Comment on attachment 344602 [details]
This is the current file where I could see the problem.

Sorry, I will re-upload the file - I needed to zip it...
Comment 20 User image Gerald 2008-10-24 02:23:25 PDT
Created attachment 344603 [details]
This file causes 100 % CPU load on my FireFox 3.0.3

My FireFox 3.0.3 has 100 % CPU load when opening this file. It also happens when trying to update a RSS feed, referring to this file!
Comment 21 User image Boris Zbarsky [:bz] (still a bit busy) 2008-10-24 10:36:02 PDT
Gerald, I don't see any problems loading the file you attached.  Loads in about 3 seconds.

On the other hand, that file refers to an XSL file that you did NOT attach.  Care to just point to the original URI where the problem shows up?  Probably in a separate bug, since I doubt it has anything to do with this one.
Comment 22 User image Gerald 2008-10-24 16:05:45 PDT
Thanks a lot for the analysis, Boris!

Here is the original Link I downloaded the file from:
<A FEEDURL="http://www.zid.tugraz.at/rss.xml" HREF="http://www.tugraz.at/">TU Graz (ZID)</A>
This is the bookmark I had to delete (indluding the places.sqlite file) to avoid my FireFox freezing (100 % CPU load). I could see that the reason was the RSS feed update. Then I tried to download the file in the browser -- with the same effect. As far as I remember, this already happened to me in an earlier version of FF.

Sorry when this is not the correct thread -- if I get some more detailed information, I will open a new bug!
Comment 23 User image Boris Zbarsky [:bz] (still a bit busy) 2008-10-24 17:24:47 PDT
I still get that loading in just a few seconds...  It really sounds like you want a separate bug no matter what; adding irrelevant comments to this bug just makes it harder to figure out what the state of this bug is.
Comment 24 User image Erik den Houter 2008-10-25 01:49:02 PDT
<A FEEDURL="http://www.zid.tugraz.at/rss.xml" HREF="http://www.tugraz.at/">TU
Graz (ZID)</A>

On both my machines:
Loading well in 30 Seconds on Win 98SE P3 450 Mhz with 284 MB memory and FF 2.0.0.3
Loading well in two seconds on Win XP FF C2D 6600, 2GB and FF3.03
Comment 25 User image Erik den Houter 2008-10-25 01:52:15 PDT
(sorry, small mistake, the FF on the old machine was version 2,0.0.14)
Comment 26 User image Alexander Fefer 2009-11-05 10:02:06 PST
Is this bug still unresolved? 

I can reproduce with this XML file:
http://www.cmegroup.com/wrappedpages/clearing/pbrates/MarginRates.xml

I get 100% CPU usage and the entire machine becomes unresponsive. Occasionally there's a crash as well - is this the correct place to post this, or is there another thread out there?
Comment 27 User image Alexander Fefer 2009-11-05 10:07:04 PST
Sorry, forgot to note the bug is reproduced with FF 3.5.4 on Windows 7, C2D 9300/4 GB RAM.
Comment 28 User image Erik den Houter 2009-11-05 11:04:04 PST
Cannot be of more help than this report:
C2D E6600 Windows XP Prof up-to-date,  FF 3.5.4  4GB installed.

Tried that URL http://www.cmegroup.com/wrappedpages/clearing/pbrates/MarginRates.xml

Loading took long (more than 30 sec), with periods of an onresponding machine.
First time load ended OK. 
Closed it again.

Second time loading of the same page the machine was again sometimes not responding, and i tried to switch to another FF tab (one of about ten) and this worked after a while, but when going back to the slow tab again, and while freezing, Firefox closed with this error:

Windows Error box:
"Microsoft Visual C++  Runtime library

(red cross)  
Runtime error!
Program C:\Program Files\Mozilla Firefox\Firefox.exe

This application has requested the runtime to terminate it in an unusual way.
Please contact the application's supoport team for more information.
OK"
Comment 29 User image Jonas Sicking (:sicking) No longer reading bugmail consistently 2009-11-05 11:08:35 PST
(In reply to comment #26)
> I get 100% CPU usage and the entire machine becomes unresponsive. Occasionally
> there's a crash as well - is this the correct place to post this, or is there
> another thread out there?

Please file a separate bug on the crash
Comment 30 User image Matěj Cepl 2010-07-23 08:13:49 PDT
Wonder whether this isn't duplicate of bug 526825
Comment 31 User image Boris Zbarsky [:bz] (still a bit busy) 2010-07-23 10:31:26 PDT
The other way around, but yes, seems to be.
Comment 32 User image Jonas Sicking (:sicking) No longer reading bugmail consistently 2010-07-23 13:37:47 PDT
*** Bug 526825 has been marked as a duplicate of this bug. ***
Comment 33 User image Nicholas Nethercote [:njn] 2011-04-13 17:30:38 PDT
*** Bug 106357 has been marked as a duplicate of this bug. ***
Comment 34 User image Nicholas Nethercote [:njn] 2011-04-13 17:30:53 PDT
*** Bug 436150 has been marked as a duplicate of this bug. ***
Comment 35 User image Nicholas Nethercote [:njn] 2011-06-28 17:08:44 PDT
Marking this as MemShrink:P3 because XML files aren't opened that often.
Comment 36 User image Zlip792 2011-07-15 20:43:38 PDT
Created attachment 546276 [details]
about:memory results

Nicholas, this what I did, if I should follow any other methodology, please explain to me. I will follow and help you. If I messed up with something. Correct me as well.
Comment 37 User image Nicholas Nethercote [:njn] 2011-07-16 14:45:59 PDT
Comment on attachment 546276 [details]
about:memory results

Thanks for the careful measurements, that's very helpful.

The important number here is from the second about:memory snapshot:

├──213.76 MB (63.39%) -- heap-unclassified

I guess memory reporters for the DOM (bug 663271 and its children) will be needed to get more information here.
Comment 38 User image Zlip792 2011-07-17 17:11:06 PDT
Thanks a lot for appreciation. Don`t know whether it is well mentioning but I noticed one think when testing and even verified later. When URL is loading, memory usage don`t go beyond maximum level (127 MB usage) but once its completely load all of a sudden it reached 334 MB usage. I think it was well mentioning so I am mentioning it. Maybe of any help.
Comment 39 User image Zlip792 2011-08-06 09:39:54 PDT
Created attachment 551259 [details]
about:memory?verbose (latest nightly)

After landings of many memory reporters and patches, I thought of providing another more test result of mine for this bug. I wish they will be helpful for developers.
In this, heap classified looks more strange, 70%.
Comment 40 User image Mats Palmgren (:mats) 2011-08-14 18:55:59 PDT
*** Bug 678849 has been marked as a duplicate of this bug. ***
Comment 41 User image Nicholas Nethercote [:njn] 2014-05-27 17:00:41 PDT
Created attachment 8429672 [details]
DMD output

I remeasured. about:memory says that "heap-unclassified" is 46%. I've attached the top 20 records from DMD's output.

420.97 MB (100.0%) -- explicit
├──193.44 MB (45.95%) ── heap-unclassified
├──158.93 MB (37.75%) -- window-objects
│  ├──149.02 MB (35.40%) -- top(file:///home/njn/iTunesMusicLibrary.xml, id=8)
│  │  ├──147.90 MB (35.13%) -- active/window(file:///home/njn/iTunesMusicLibrary
.xml)
│  │  │  ├──119.30 MB (28.34%) -- layout
│  │  │  │  ├───69.83 MB (16.59%) -- frames
│  │  │  │  │   ├──41.81 MB (09.93%) ── nsTextFrame
│  │  │  │  │   ├──18.68 MB (04.44%) ── nsInlineFrame
│  │  │  │  │   ├───9.24 MB (02.20%) ── nsBlockFrame
│  │  │  │  │   └───0.11 MB (00.03%) ++ (2 tiny)
│  │  │  │  ├───38.48 MB (09.14%) ── text-runs
│  │  │  │  ├────9.59 MB (02.28%) ── line-boxes
│  │  │  │  └────1.41 MB (00.33%) ++ (5 tiny)
│  │  │  ├───28.26 MB (06.71%) -- dom
│  │  │  │   ├──16.08 MB (03.82%) ── text-nodes
│  │  │  │   ├──12.18 MB (02.89%) ── element-nodes
│  │  │  │   └───0.01 MB (00.00%) ++ (2 tiny)
│  │  │  └────0.34 MB (00.08%) ++ (3 tiny)
│  │  └────1.12 MB (00.27%) ++ (2 tiny)
│  ├────6.16 MB (01.46%) -- top(chrome://browser/content/browser.xul, id=3)
│  │    ├──5.55 MB (01.32%) -- active
│  │    │  ├──5.39 MB (01.28%) ++ window(chrome://browser/content/browser.xul)
│  │    │  └──0.16 MB (00.04%) ++ window(about:blank)
│  │    └──0.61 MB (00.14%) ++ js-zone(0x7f741cfaa000)
│  └────3.75 MB (00.89%) ++ (4 tiny)
├───30.94 MB (07.35%) -- js-non-window
│   ├──22.91 MB (05.44%) -- zones
│   │  ├──19.97 MB (04.74%) ++ zone(0x7f7431854800)
│   │  └───2.93 MB (00.70%) ++ (3 tiny)
│   ├───7.59 MB (01.80%) ++ runtime
│   └───0.45 MB (00.11%) ++ gc-heap
├───14.19 MB (03.37%) -- heap-overhead
│   ├───7.84 MB (01.86%) ── bookkeeping
│   └───6.35 MB (01.51%) ++ (3 tiny)
├────7.52 MB (01.79%) ++ (19 tiny)
├────6.38 MB (01.52%) -- startup-cache
│    ├──6.38 MB (01.52%) ── data
│    └──0.00 MB (00.00%) ── mapping
├────4.83 MB (01.15%) ++ storage
└────4.74 MB (01.13%) ++ workers/workers()

Note You need to log in before you can comment on or make changes to this bug.