291643 - high cpu and 100-1 ratio memory opening large xml file

Status: NEW

(Core :: XML, defect)

Description

[Carl Antaki]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.7) Gecko/20050414 Firefox/1.0.3 (ax)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.7) Gecko/20050414 Firefox/1.0.3 (ax)

Firefox 1.0.3 takes too much memory when opening iTunes Music library.xml, my
file size is 2.46MB. Firefox memory consumption goes as high as 300MB of RAM and
300MB of virtual memory as seen on Windows task manager. It takes around 5min
for the page to load which is unacceptable (I have and old P3 1Ghz PC with 512MB
of RAM)

Reproducible: Always

Steps to Reproduce:
1.Open the file with Firefox
2.
3.

Actual Results:  
Firefox consumes 300MB of RAM and 300MB of virtual memory

Expected Results:  
The software should have consumed less memory

Windows XP using the old Windows 2k theme. 
Pentium 3 1Ghz PC with 512MB of RAM

Comment 1

[Matthias Versen [:Matti]]

Reporter:
Can you pelase retest a current trunk build because the gecko in FF1.0.3 is very
old.
Thanks

Comment 2

[Carl Antaki]

I tried it on the most recent trunk build and it gives the same results.

Comment 3

[Matthias Versen [:Matti]]

Do you have a URL for a testcase ?

Comment 4

[Carl Antaki]

This is a test case:
http://www.w3.org/2000/10/swap/test/plist/iTunesMusicLibrary.xml

In this test Firefox doesn't take 300MB because the file is smaller, it takes
169MB of RAM and 169MB of virtual memory.

Comment 5

[Boris Zbarsky [:bzbarsky]]

This is a little odd.  I'd expect a 10-1 size increase (a la view-source), but
not 100-1....  sicking, any idea what's up here?

Comment 6

[Wayne Mery (:wsmwk)]

I get roughly 100-1 ratio also.  xml file size 1.6MB, firefox took something over 120MB. High CPU ~50% for the duration of loading

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a3pre) Gecko/20070216 Minefield/3.0a3pre 

Comment 7

[Boris Zbarsky [:bzbarsky]]

Attachment: The file in question, zipped up

Comment 8

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

That it's taking more than view-source doesn't surprise me at all given that we're creating both more elements, and more complex layout trees (that includes tables).

Comment 9

[Erik den Houter]

Don't know what you're talking about, but maybe
have a look at "my" bug 378330, and maybe you can help me
explane what's going on.

I'm just new in wwwonderland, and would love to know whats
eating my memory and time...

Comment 11

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Is this still happening? Things have changed a lot in how we prettyprint xml so we might be in a better state. Also, if this was caused by leaks we might be in a better state due to the cycle collector.

Comment 12

[Erik den Houter]

I do not use that machine any more for daily use, so I hat to boot it just to check what's up.
As I reported at "my" bug 378330 where this URL caused the problem:
http://www.ndfr.net/sitemap.xml
on a Win 98SE P3 450 Mhz with 284 MB memory and FF 2.0.0.3,  I cannot reproduce it for the fact that that page has completely changed, so there's no comparing.

Trying the URL from "this" bug-report:
http://www.w3.org/2000/10/swap/test/plist/iTunesMusicLibrary.xml
on the same machine, but now with FF 2.0.0.14, it takes about four minutes to load, and during that time the PC still cannot be used, an hourglass for four minutes, and all of the free memory is still consumed, but it loads The page was loaded direct after boot.

Trying the URL on a faster machine with FF 3.01  (C2D 6600, 2GB) the page loads within eight seconds, and it seems that I can switch to other activities while loading, so the handling of that pages is much better.
( Still 147 MB RAM is consumed according to Freeram XP Pro, while saving the page is taking 1,54 MB harddisk space.)

Comment 13

[Boris Zbarsky [:bzbarsky]]

So with a current tip build, starting the browser takes about 43MB of RAM.  Opening this XML file peaks at about 226MB RAM, and then drops down to 193MB.

So about 150MB for the rendering.  All this on Mac.  I wonder how much bug 231925's suggestion to use XML would help.  I'll test.

Comment 14

[Boris Zbarsky [:bzbarsky]]

I tried that (on the assumption that it would shave off those 4 bytes for the class attribute), but it wasn't much of a change, if any.

Fundamentally, when rendering something like:

  <myTag>
    ...
  </myTag>

where the "myTag" is N characters long, we have the following objects here (most of these numbers also involve malloc overhead, but I'm ignoring it for now):

1 pointer from the parent: 4 bytes
1 nsXMLElement (the original element): 32 bytes
1 nsHTMLDivElement wrapping the whole thing: 32 bytes
1 nsHTMLDivElement for the expander: 32 bytes
1 nsTextNode for the expander text: 36 bytes + 1 byte for the text
1 nsTextNode for the text '<': 36 bytes + 1 byte for the text
1 nsHTMLSpanElement with one attribute: 40 bytes
1 nsTextNode for the "myTag": 36 bytes + N bytes for the text
1 nsTextNode for the text '>': 36 + 1 bytes
1 nsHTMLDivElement with one attribute for the stuff this contains: 40 bytes
1 nsTextNode for the text "</": 36 + 2 bytes
1 nsHTMLSpanElement with one attribute: 40 bytes
1 nsTextNode for the text '>': 36 + 1 bytes
6 nsTextFrames: 6 * 64 bytes
2 nsInlineFrames: 2 * 56 bytes
3 nsBlockFrames: 3 * 88 bytes
3 nsLineBoxes: 3 * 40 bytes

Total (minimal estimate): 1322 + 2N bytes.  I think I'm not counting the linebox in the parent that contains all that stuff, etc, but let's go with this number.

Initial memory usage in the file was 2N + 5 bytes.   If N is about 10, then we have 1342 bytes vs 25 bytes, which is a 50x increase.  If N is about 5, then we have 1332 bytes vs 15 bytes, which is closer to 80x.

This file has tagnames (more or less in order of how common they are): key, integer, string, dict, date.

For "key" the factor is 1328/11 or about 130x.
For "dict" and "date" the factor is 1330/13 or about 100x
For "string" the factor is 1334/17 or about 80x
For "integer" the factor is 1336/19 or about 65x

So an average of 100x is not surprising at all.  In fact, if the page were using one-character tag names, the factor would be 1324/7, or about 200x.

So what can we do to reduce memory usage here?  One obvious thing to do would be to not create separate textnodes for the '>' and '<'.  In this case, that would save us 400 bytes per node with kids, or about 1/3 to 1/4 of the memory we're using.  It might not look as nice, but I doubt it'd look that much less nice, honestly.

Comment 15

[Boris Zbarsky [:bzbarsky]]

I poked at the XSLT, and I'm not sure I can actually do separate textnodes here.... I assume that if an xsl:text and an xsl:valueof are next to each other they don't get combined into a single textnode, or anything?

Comment 16

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

XSLT should never produce two sibling textnodes. A <xsl:text> next to an <xsl:value-of> should get combined into a single node in the result DOM.

Comment 17

[Boris Zbarsky [:bzbarsky]]

OK.  So I did test getting rid of some of the extra textnodes and as expected the memory used to do the prettyprinting drops from about 150MB to about 110-120MB.  Performance increases a bit (from 9.5 seconds drops to 7.7 or so).

Not sure whether we care enough to sacrifice looks here for that.

Comment 18

[Gerald]

Attachment: This is the current file where I could see the problem.

My FireFox 3.0.3 has 100 % CPU load when opening this file. It also happens when trying to update a RSS feed, referring to this file!

Comment 19

[Gerald]

Comment on attachment 344602 [details]
This is the current file where I could see the problem.

Sorry, I will re-upload the file - I needed to zip it...

Comment 20

[Gerald]

Attachment: This file causes 100 % CPU load on my FireFox 3.0.3

My FireFox 3.0.3 has 100 % CPU load when opening this file. It also happens when trying to update a RSS feed, referring to this file!

Comment 21

[Boris Zbarsky [:bzbarsky]]

Gerald, I don't see any problems loading the file you attached.  Loads in about 3 seconds.

On the other hand, that file refers to an XSL file that you did NOT attach.  Care to just point to the original URI where the problem shows up?  Probably in a separate bug, since I doubt it has anything to do with this one.

Comment 22

[Gerald]

Thanks a lot for the analysis, Boris!

Here is the original Link I downloaded the file from:
<A FEEDURL="http://www.zid.tugraz.at/rss.xml" HREF="http://www.tugraz.at/">TU Graz (ZID)</A>
This is the bookmark I had to delete (indluding the places.sqlite file) to avoid my FireFox freezing (100 % CPU load). I could see that the reason was the RSS feed update. Then I tried to download the file in the browser -- with the same effect. As far as I remember, this already happened to me in an earlier version of FF.

Sorry when this is not the correct thread -- if I get some more detailed information, I will open a new bug!

Comment 23

[Boris Zbarsky [:bzbarsky]]

I still get that loading in just a few seconds...  It really sounds like you want a separate bug no matter what; adding irrelevant comments to this bug just makes it harder to figure out what the state of this bug is.

Comment 24

[Erik den Houter]

<A FEEDURL="http://www.zid.tugraz.at/rss.xml" HREF="http://www.tugraz.at/">TU
Graz (ZID)</A>

On both my machines:
Loading well in 30 Seconds on Win 98SE P3 450 Mhz with 284 MB memory and FF 2.0.0.3
Loading well in two seconds on Win XP FF C2D 6600, 2GB and FF3.03

Comment 25

[Erik den Houter]

(sorry, small mistake, the FF on the old machine was version 2,0.0.14)

Comment 26

[Alexander Fefer]

Is this bug still unresolved? 

I can reproduce with this XML file:
http://www.cmegroup.com/wrappedpages/clearing/pbrates/MarginRates.xml

I get 100% CPU usage and the entire machine becomes unresponsive. Occasionally there's a crash as well - is this the correct place to post this, or is there another thread out there?

Comment 27

[Alexander Fefer]

Sorry, forgot to note the bug is reproduced with FF 3.5.4 on Windows 7, C2D 9300/4 GB RAM.

Comment 28

[Erik den Houter]

Cannot be of more help than this report:
C2D E6600 Windows XP Prof up-to-date,  FF 3.5.4  4GB installed.

Tried that URL http://www.cmegroup.com/wrappedpages/clearing/pbrates/MarginRates.xml

Loading took long (more than 30 sec), with periods of an onresponding machine.
First time load ended OK. 
Closed it again.

Second time loading of the same page the machine was again sometimes not responding, and i tried to switch to another FF tab (one of about ten) and this worked after a while, but when going back to the slow tab again, and while freezing, Firefox closed with this error:

Windows Error box:
"Microsoft Visual C++  Runtime library

(red cross)  
Runtime error!
Program C:\Program Files\Mozilla Firefox\Firefox.exe

This application has requested the runtime to terminate it in an unusual way.
Please contact the application's supoport team for more information.
OK"

Comment 29

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

(In reply to comment #26)
> I get 100% CPU usage and the entire machine becomes unresponsive. Occasionally
> there's a crash as well - is this the correct place to post this, or is there
> another thread out there?

Please file a separate bug on the crash

Comment 30

[Matěj Cepl]

Wonder whether this isn't duplicate of bug 526825

Comment 31

[Boris Zbarsky [:bzbarsky]]

The other way around, but yes, seems to be.

Comment 35

[Nicholas Nethercote [inactive]]

Marking this as MemShrink:P3 because XML files aren't opened that often.

Comment 36

[Ahmad Saleem]

Attachment: about:memory results

Nicholas, this what I did, if I should follow any other methodology, please explain to me. I will follow and help you. If I messed up with something. Correct me as well.

Comment 37

[Nicholas Nethercote [inactive]]

Comment on attachment 546276 [details]
about:memory results

Thanks for the careful measurements, that's very helpful.

The important number here is from the second about:memory snapshot:

├──213.76 MB (63.39%) -- heap-unclassified

I guess memory reporters for the DOM (bug 663271 and its children) will be needed to get more information here.

Comment 38

[Ahmad Saleem]

Thanks a lot for appreciation. Don`t know whether it is well mentioning but I noticed one think when testing and even verified later. When URL is loading, memory usage don`t go beyond maximum level (127 MB usage) but once its completely load all of a sudden it reached 334 MB usage. I think it was well mentioning so I am mentioning it. Maybe of any help.

Comment 39

[Ahmad Saleem]

Attachment: about:memory?verbose (latest nightly)

After landings of many memory reporters and patches, I thought of providing another more test result of mine for this bug. I wish they will be helpful for developers.
In this, heap classified looks more strange, 70%.

Comment 41

[Nicholas Nethercote [inactive]]

Attachment: DMD output

I remeasured. about:memory says that "heap-unclassified" is 46%. I've attached the top 20 records from DMD's output.

420.97 MB (100.0%) -- explicit
├──193.44 MB (45.95%) ── heap-unclassified
├──158.93 MB (37.75%) -- window-objects
│  ├──149.02 MB (35.40%) -- top(file:///home/njn/iTunesMusicLibrary.xml, id=8)
│  │  ├──147.90 MB (35.13%) -- active/window(file:///home/njn/iTunesMusicLibrary
.xml)
│  │  │  ├──119.30 MB (28.34%) -- layout
│  │  │  │  ├───69.83 MB (16.59%) -- frames
│  │  │  │  │   ├──41.81 MB (09.93%) ── nsTextFrame
│  │  │  │  │   ├──18.68 MB (04.44%) ── nsInlineFrame
│  │  │  │  │   ├───9.24 MB (02.20%) ── nsBlockFrame
│  │  │  │  │   └───0.11 MB (00.03%) ++ (2 tiny)
│  │  │  │  ├───38.48 MB (09.14%) ── text-runs
│  │  │  │  ├────9.59 MB (02.28%) ── line-boxes
│  │  │  │  └────1.41 MB (00.33%) ++ (5 tiny)
│  │  │  ├───28.26 MB (06.71%) -- dom
│  │  │  │   ├──16.08 MB (03.82%) ── text-nodes
│  │  │  │   ├──12.18 MB (02.89%) ── element-nodes
│  │  │  │   └───0.01 MB (00.00%) ++ (2 tiny)
│  │  │  └────0.34 MB (00.08%) ++ (3 tiny)
│  │  └────1.12 MB (00.27%) ++ (2 tiny)
│  ├────6.16 MB (01.46%) -- top(chrome://browser/content/browser.xul, id=3)
│  │    ├──5.55 MB (01.32%) -- active
│  │    │  ├──5.39 MB (01.28%) ++ window(chrome://browser/content/browser.xul)
│  │    │  └──0.16 MB (00.04%) ++ window(about:blank)
│  │    └──0.61 MB (00.14%) ++ js-zone(0x7f741cfaa000)
│  └────3.75 MB (00.89%) ++ (4 tiny)
├───30.94 MB (07.35%) -- js-non-window
│   ├──22.91 MB (05.44%) -- zones
│   │  ├──19.97 MB (04.74%) ++ zone(0x7f7431854800)
│   │  └───2.93 MB (00.70%) ++ (3 tiny)
│   ├───7.59 MB (01.80%) ++ runtime
│   └───0.45 MB (00.11%) ++ gc-heap
├───14.19 MB (03.37%) -- heap-overhead
│   ├───7.84 MB (01.86%) ── bookkeeping
│   └───6.35 MB (01.51%) ++ (3 tiny)
├────7.52 MB (01.79%) ++ (19 tiny)
├────6.38 MB (01.52%) -- startup-cache
│    ├──6.38 MB (01.52%) ── data
│    └──0.00 MB (00.00%) ── mapping
├────4.83 MB (01.15%) ++ storage
└────4.74 MB (01.13%) ++ workers/workers()

Comment 42

[Andrei Purice]

Still reproducible on the latest versions of Firefox Nightly 95.0a1 (2021-10-19), beta 94.0b7 or release 93.0 on Windows 10.

Comment 43

[BugBot [:suhaib / :marco/ :calixte]]

In the process of migrating remaining bugs to the new severity system, the severity for this bug cannot be automatically determined. Please retriage this bug using the new severity system.

Comment 44

[Henri Sivonen (:hsivonen)]

Still relevant.