The default bug view has changed. See this FAQ.

high cpu and 100-1 ratio memory opening large xml file

NEW
Unassigned

Status

()

Core
XML
--
major
12 years ago
3 years ago

People

(Reporter: Carl Antaki, Unassigned)

Tracking

(Depends on: 1 bug, {perf})

Trunk
x86
Windows XP
Points:
---
Dependency tree / graph
Bug Flags:
blocking1.9 -

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [MemShrink:P3], URL)

Attachments

(6 attachments)

(Reporter)

Description

12 years ago
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.7) Gecko/20050414 Firefox/1.0.3 (ax)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.7) Gecko/20050414 Firefox/1.0.3 (ax)

Firefox 1.0.3 takes too much memory when opening iTunes Music library.xml, my
file size is 2.46MB. Firefox memory consumption goes as high as 300MB of RAM and
300MB of virtual memory as seen on Windows task manager. It takes around 5min
for the page to load which is unacceptable (I have and old P3 1Ghz PC with 512MB
of RAM)

Reproducible: Always

Steps to Reproduce:
1.Open the file with Firefox
2.
3.

Actual Results:  
Firefox consumes 300MB of RAM and 300MB of virtual memory

Expected Results:  
The software should have consumed less memory

Windows XP using the old Windows 2k theme. 
Pentium 3 1Ghz PC with 512MB of RAM
Reporter:
Can you pelase retest a current trunk build because the gecko in FF1.0.3 is very
old.
Thanks
(Reporter)

Comment 2

12 years ago
I tried it on the most recent trunk build and it gives the same results.
Version: unspecified → Trunk
Do you have a URL for a testcase ?
(Reporter)

Comment 4

12 years ago
This is a test case:
http://www.w3.org/2000/10/swap/test/plist/iTunesMusicLibrary.xml

In this test Firefox doesn't take 300MB because the file is smaller, it takes
169MB of RAM and 169MB of virtual memory.
Assignee: firefox → xml
Status: UNCONFIRMED → NEW
Component: General → XML
Ever confirmed: true
Product: Firefox → Core
QA Contact: general → ashshbhatt
This is a little odd.  I'd expect a 10-1 size increase (a la view-source), but
not 100-1....  sicking, any idea what's up here?
I get roughly 100-1 ratio also.  xml file size 1.6MB, firefox took something over 120MB. High CPU ~50% for the duration of loading

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a3pre) Gecko/20070216 Minefield/3.0a3pre 

Keywords: perf
Summary: Firefox 1.0.3 takes too much memory when opening iTunes Music library.xml → high cpu and 100-1 ratio memory opening large xml file
Flags: blocking1.9?
Created attachment 257578 [details]
The file in question, zipped up
That it's taking more than view-source doesn't surprise me at all given that we're creating both more elements, and more complex layout trees (that includes tables).

Comment 9

10 years ago
Don't know what you're talking about, but maybe
have a look at "my" bug 378330, and maybe you can help me
explane what's going on.

I'm just new in wwwonderland, and would love to know whats
eating my memory and time...
Blocks: 378330

Updated

10 years ago
Duplicate of this bug: 378330
Flags: blocking1.9? → blocking1.9-
Depends on: 197956
Is this still happening? Things have changed a lot in how we prettyprint xml so we might be in a better state. Also, if this was caused by leaks we might be in a better state due to the cycle collector.

Comment 12

9 years ago
I do not use that machine any more for daily use, so I hat to boot it just to check what's up.
As I reported at "my" bug 378330 where this URL caused the problem:
http://www.ndfr.net/sitemap.xml
on a Win 98SE P3 450 Mhz with 284 MB memory and FF 2.0.0.3,  I cannot reproduce it for the fact that that page has completely changed, so there's no comparing.

Trying the URL from "this" bug-report:
http://www.w3.org/2000/10/swap/test/plist/iTunesMusicLibrary.xml
on the same machine, but now with FF 2.0.0.14, it takes about four minutes to load, and during that time the PC still cannot be used, an hourglass for four minutes, and all of the free memory is still consumed, but it loads The page was loaded direct after boot.

Trying the URL on a faster machine with FF 3.01  (C2D 6600, 2GB) the page loads within eight seconds, and it seems that I can switch to other activities while loading, so the handling of that pages is much better.
( Still 147 MB RAM is consumed according to Freeram XP Pro, while saving the page is taking 1,54 MB harddisk space.)
So with a current tip build, starting the browser takes about 43MB of RAM.  Opening this XML file peaks at about 226MB RAM, and then drops down to 193MB.

So about 150MB for the rendering.  All this on Mac.  I wonder how much bug 231925's suggestion to use XML would help.  I'll test.
I tried that (on the assumption that it would shave off those 4 bytes for the class attribute), but it wasn't much of a change, if any.

Fundamentally, when rendering something like:

  <myTag>
    ...
  </myTag>

where the "myTag" is N characters long, we have the following objects here (most of these numbers also involve malloc overhead, but I'm ignoring it for now):

1 pointer from the parent: 4 bytes
1 nsXMLElement (the original element): 32 bytes
1 nsHTMLDivElement wrapping the whole thing: 32 bytes
1 nsHTMLDivElement for the expander: 32 bytes
1 nsTextNode for the expander text: 36 bytes + 1 byte for the text
1 nsTextNode for the text '<': 36 bytes + 1 byte for the text
1 nsHTMLSpanElement with one attribute: 40 bytes
1 nsTextNode for the "myTag": 36 bytes + N bytes for the text
1 nsTextNode for the text '>': 36 + 1 bytes
1 nsHTMLDivElement with one attribute for the stuff this contains: 40 bytes
1 nsTextNode for the text "</": 36 + 2 bytes
1 nsHTMLSpanElement with one attribute: 40 bytes
1 nsTextNode for the text '>': 36 + 1 bytes
6 nsTextFrames: 6 * 64 bytes
2 nsInlineFrames: 2 * 56 bytes
3 nsBlockFrames: 3 * 88 bytes
3 nsLineBoxes: 3 * 40 bytes

Total (minimal estimate): 1322 + 2N bytes.  I think I'm not counting the linebox in the parent that contains all that stuff, etc, but let's go with this number.

Initial memory usage in the file was 2N + 5 bytes.   If N is about 10, then we have 1342 bytes vs 25 bytes, which is a 50x increase.  If N is about 5, then we have 1332 bytes vs 15 bytes, which is closer to 80x.

This file has tagnames (more or less in order of how common they are): key, integer, string, dict, date.

For "key" the factor is 1328/11 or about 130x.
For "dict" and "date" the factor is 1330/13 or about 100x
For "string" the factor is 1334/17 or about 80x
For "integer" the factor is 1336/19 or about 65x

So an average of 100x is not surprising at all.  In fact, if the page were using one-character tag names, the factor would be 1324/7, or about 200x.

So what can we do to reduce memory usage here?  One obvious thing to do would be to not create separate textnodes for the '>' and '<'.  In this case, that would save us 400 bytes per node with kids, or about 1/3 to 1/4 of the memory we're using.  It might not look as nice, but I doubt it'd look that much less nice, honestly.
I poked at the XSLT, and I'm not sure I can actually do separate textnodes here.... I assume that if an xsl:text and an xsl:valueof are next to each other they don't get combined into a single textnode, or anything?
XSLT should never produce two sibling textnodes. A <xsl:text> next to an <xsl:value-of> should get combined into a single node in the result DOM.
OK.  So I did test getting rid of some of the extra textnodes and as expected the memory used to do the prettyprinting drops from about 150MB to about 110-120MB.  Performance increases a bit (from 9.5 seconds drops to 7.7 or so).

Not sure whether we care enough to sacrifice looks here for that.

Comment 18

9 years ago
Created attachment 344602 [details]
This is the current file where I could see the problem.

My FireFox 3.0.3 has 100 % CPU load when opening this file. It also happens when trying to update a RSS feed, referring to this file!

Comment 19

9 years ago
Comment on attachment 344602 [details]
This is the current file where I could see the problem.

Sorry, I will re-upload the file - I needed to zip it...

Comment 20

9 years ago
Created attachment 344603 [details]
This file causes 100 % CPU load on my FireFox 3.0.3

My FireFox 3.0.3 has 100 % CPU load when opening this file. It also happens when trying to update a RSS feed, referring to this file!
Gerald, I don't see any problems loading the file you attached.  Loads in about 3 seconds.

On the other hand, that file refers to an XSL file that you did NOT attach.  Care to just point to the original URI where the problem shows up?  Probably in a separate bug, since I doubt it has anything to do with this one.

Comment 22

9 years ago
Thanks a lot for the analysis, Boris!

Here is the original Link I downloaded the file from:
<A FEEDURL="http://www.zid.tugraz.at/rss.xml" HREF="http://www.tugraz.at/">TU Graz (ZID)</A>
This is the bookmark I had to delete (indluding the places.sqlite file) to avoid my FireFox freezing (100 % CPU load). I could see that the reason was the RSS feed update. Then I tried to download the file in the browser -- with the same effect. As far as I remember, this already happened to me in an earlier version of FF.

Sorry when this is not the correct thread -- if I get some more detailed information, I will open a new bug!
I still get that loading in just a few seconds...  It really sounds like you want a separate bug no matter what; adding irrelevant comments to this bug just makes it harder to figure out what the state of this bug is.

Comment 24

9 years ago
<A FEEDURL="http://www.zid.tugraz.at/rss.xml" HREF="http://www.tugraz.at/">TU
Graz (ZID)</A>

On both my machines:
Loading well in 30 Seconds on Win 98SE P3 450 Mhz with 284 MB memory and FF 2.0.0.3
Loading well in two seconds on Win XP FF C2D 6600, 2GB and FF3.03

Comment 25

9 years ago
(sorry, small mistake, the FF on the old machine was version 2,0.0.14)
Assignee: xml → nobody
QA Contact: ashshbhatt → xml

Comment 26

8 years ago
Is this bug still unresolved? 

I can reproduce with this XML file:
http://www.cmegroup.com/wrappedpages/clearing/pbrates/MarginRates.xml

I get 100% CPU usage and the entire machine becomes unresponsive. Occasionally there's a crash as well - is this the correct place to post this, or is there another thread out there?

Comment 27

8 years ago
Sorry, forgot to note the bug is reproduced with FF 3.5.4 on Windows 7, C2D 9300/4 GB RAM.

Comment 28

8 years ago
Cannot be of more help than this report:
C2D E6600 Windows XP Prof up-to-date,  FF 3.5.4  4GB installed.

Tried that URL http://www.cmegroup.com/wrappedpages/clearing/pbrates/MarginRates.xml

Loading took long (more than 30 sec), with periods of an onresponding machine.
First time load ended OK. 
Closed it again.

Second time loading of the same page the machine was again sometimes not responding, and i tried to switch to another FF tab (one of about ten) and this worked after a while, but when going back to the slow tab again, and while freezing, Firefox closed with this error:

Windows Error box:
"Microsoft Visual C++  Runtime library

(red cross)  
Runtime error!
Program C:\Program Files\Mozilla Firefox\Firefox.exe

This application has requested the runtime to terminate it in an unusual way.
Please contact the application's supoport team for more information.
OK"
(In reply to comment #26)
> I get 100% CPU usage and the entire machine becomes unresponsive. Occasionally
> there's a crash as well - is this the correct place to post this, or is there
> another thread out there?

Please file a separate bug on the crash

Comment 30

7 years ago
Wonder whether this isn't duplicate of bug 526825
The other way around, but yes, seems to be.
Duplicate of this bug: 526825

Updated

7 years ago
Blocks: 640457
Duplicate of this bug: 106357
Duplicate of this bug: 436150
Blocks: 659856
No longer blocks: 640457
Whiteboard: [MemShrink:P2]
Whiteboard: [MemShrink:P2] → [MemShrink:P3]
Marking this as MemShrink:P3 because XML files aren't opened that often.

Comment 36

6 years ago
Created attachment 546276 [details]
about:memory results

Nicholas, this what I did, if I should follow any other methodology, please explain to me. I will follow and help you. If I messed up with something. Correct me as well.
Comment on attachment 546276 [details]
about:memory results

Thanks for the careful measurements, that's very helpful.

The important number here is from the second about:memory snapshot:

├──213.76 MB (63.39%) -- heap-unclassified

I guess memory reporters for the DOM (bug 663271 and its children) will be needed to get more information here.

Comment 38

6 years ago
Thanks a lot for appreciation. Don`t know whether it is well mentioning but I noticed one think when testing and even verified later. When URL is loading, memory usage don`t go beyond maximum level (127 MB usage) but once its completely load all of a sudden it reached 334 MB usage. I think it was well mentioning so I am mentioning it. Maybe of any help.

Comment 39

6 years ago
Created attachment 551259 [details]
about:memory?verbose (latest nightly)

After landings of many memory reporters and patches, I thought of providing another more test result of mine for this bug. I wish they will be helpful for developers.
In this, heap classified looks more strange, 70%.

Updated

6 years ago
Duplicate of this bug: 678849
Created attachment 8429672 [details]
DMD output

I remeasured. about:memory says that "heap-unclassified" is 46%. I've attached the top 20 records from DMD's output.

420.97 MB (100.0%) -- explicit
├──193.44 MB (45.95%) ── heap-unclassified
├──158.93 MB (37.75%) -- window-objects
│  ├──149.02 MB (35.40%) -- top(file:///home/njn/iTunesMusicLibrary.xml, id=8)
│  │  ├──147.90 MB (35.13%) -- active/window(file:///home/njn/iTunesMusicLibrary
.xml)
│  │  │  ├──119.30 MB (28.34%) -- layout
│  │  │  │  ├───69.83 MB (16.59%) -- frames
│  │  │  │  │   ├──41.81 MB (09.93%) ── nsTextFrame
│  │  │  │  │   ├──18.68 MB (04.44%) ── nsInlineFrame
│  │  │  │  │   ├───9.24 MB (02.20%) ── nsBlockFrame
│  │  │  │  │   └───0.11 MB (00.03%) ++ (2 tiny)
│  │  │  │  ├───38.48 MB (09.14%) ── text-runs
│  │  │  │  ├────9.59 MB (02.28%) ── line-boxes
│  │  │  │  └────1.41 MB (00.33%) ++ (5 tiny)
│  │  │  ├───28.26 MB (06.71%) -- dom
│  │  │  │   ├──16.08 MB (03.82%) ── text-nodes
│  │  │  │   ├──12.18 MB (02.89%) ── element-nodes
│  │  │  │   └───0.01 MB (00.00%) ++ (2 tiny)
│  │  │  └────0.34 MB (00.08%) ++ (3 tiny)
│  │  └────1.12 MB (00.27%) ++ (2 tiny)
│  ├────6.16 MB (01.46%) -- top(chrome://browser/content/browser.xul, id=3)
│  │    ├──5.55 MB (01.32%) -- active
│  │    │  ├──5.39 MB (01.28%) ++ window(chrome://browser/content/browser.xul)
│  │    │  └──0.16 MB (00.04%) ++ window(about:blank)
│  │    └──0.61 MB (00.14%) ++ js-zone(0x7f741cfaa000)
│  └────3.75 MB (00.89%) ++ (4 tiny)
├───30.94 MB (07.35%) -- js-non-window
│   ├──22.91 MB (05.44%) -- zones
│   │  ├──19.97 MB (04.74%) ++ zone(0x7f7431854800)
│   │  └───2.93 MB (00.70%) ++ (3 tiny)
│   ├───7.59 MB (01.80%) ++ runtime
│   └───0.45 MB (00.11%) ++ gc-heap
├───14.19 MB (03.37%) -- heap-overhead
│   ├───7.84 MB (01.86%) ── bookkeeping
│   └───6.35 MB (01.51%) ++ (3 tiny)
├────7.52 MB (01.79%) ++ (19 tiny)
├────6.38 MB (01.52%) -- startup-cache
│    ├──6.38 MB (01.52%) ── data
│    └──0.00 MB (00.00%) ── mapping
├────4.83 MB (01.15%) ++ storage
└────4.74 MB (01.13%) ++ workers/workers()
Depends on: 1032291
No longer depends on: 1032291
You need to log in before you can comment on or make changes to this bug.