Closed Bug 49539 Opened 25 years ago Closed 23 years ago

Does a >3.2MB file (bunch of tables, >7975 rows), really need 19MB of memory?

Categories

(Core Graveyard :: Tracking, defect, P2)

Product:
Core Graveyard
Component:
Tracking
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED
Milestone:
mozilla1.1beta

People

(Reporter: jag+mozbugs, Assigned: attinasi)

References

(
URL
)

Details

(Keywords: memory-footprint, perf)

For comparison, testing the 2000-08-18-08-M18 nightly on WinNT box with 128MB physical RAM: 1. After start (displaying http://www.mozilla.org/): Mem: 18752k VM: 12876k (55MB of physical RAM free at this point) 2. After displaying URL above: Mem: 50972k VM: 45000k (34MB of physical RAM free at this point) 3. After clicking on [Mozilla.org] button on Perrsonal Toolbar: Mem: 36384k VM: 30400k (49MB of physical RAM free at this point) (no other action taken between steps 2 and 3, other than recording these numbers in a NN 4.74 window) Early in the pageload (before the first </TABLE>), Mozilla was increasing memory usage ~80k at a time, so the ballooning memory usage must be nonlinear.
adding perf keyword
Keywords: perf
nav triage team: Does this bloat still occur?
Keywords: nsbeta1-
Target Milestone: --- → Future
yes, this bloat still exists. on w2k, mem goes up 50M when I load this page, and cpu gets pegged. I had to kill mozilla. I was clicking on the url in the first report of this bug. over to cathleen for look/owner.
Assignee: mcafee → cathleen
Verified with my win2K box. 1) run taskinfo2000 2) launch mozilla and load this bug page, memory used 60,768Kb 3) click on the quary link in the bug (a long while till it's done), memory used 191,720Kb. The highest run I saw was 202,584Kb. Eventually (within a minute or two) mozilla will freeze and die. I did the same test with 4.77, memory went from 53,908Kb to 115,348Kb. 4.77 still standing.
OS: Linux → All
Hardware: PC → All
Let's use bloat-blame to figure out where the memory is going! 1. Make a Linux build (using egcs-1.1.2) with the following flags in a partition with a _ton_ of disk space. --enable-trace-malloc --enable-cpp-rtti --enable-debug 2. Run gtkEmbed from mozilla/dist/bin as follows: LD_LIBRARY_PATH=`pwd` MOZILLA_FIVE_HOME=`pwd` \ gtkEmbed --trace-malloc /dev/null "(Insert URL Here)" 3. Once the page finishes loading, press the ``dump memory usage button''. This will create a monster file in mozilla/dist/bin called ``allocations.log'' 4. Check out the trace-malloc tools. From the mozilla directory, cvs co tools/trace-malloc 5. Process the log: tools/trace-malloc/histogram.pl dist/bin/allocations.log > hist tools/trace-malloc/histogram-pretty.sh hist 6. Paste the output into the bug More details on how to use the trace-malloc tools at http://www.mozilla.org/projects/footprint/live-bloat.html Good luck!
*** Bug 78142 has been marked as a duplicate of this bug. ***
Updating the summary, noting that the query is for any bug reopened ever, which means the list will grow month by month. See bug 78142 for a slightly more sane query (about 3000 bugs, which will not change over time).
Summary: Does a 2.1MB file (bunch of tables, 5114 rows), really need 25MB of memory? → Does a >3.2MB file (bunch of tables, >7975 rows), really need 25MB of memory?
new test query, using jrgm's bug 78142 http://bugzilla.mozilla.org/buglist.cgi?bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=RESOLVED&bug_status=VERIFIED&bug_status=CLOSED&resolution=FIXED&resolution=INVALID&resolution=WONTFIX&resolution=LATER&resolution=REMIND&resolution=DUPLICATE&resolution=WORKSFORME&resolution=MOVED&rep_platform=All&rep_platform=DEC&rep_platform=HP&rep_platform=Macintosh&rep_platform=PC&rep_platform=SGI&rep_platform=Sun&rep_platform=Other&op_sys=All&op_sys=Windows+3.1&op_sys=Windows+95&op_sys=Windows+98&op_sys=Windows+ME&op_sys=Windows+2000&op_sys=Windows+NT&op_sys=Mac+System+7&op_sys=Mac+System+7.5&op_sys=Mac+System+7.6.1&op_sys=Mac+System+8.0&op_sys=Mac+System+8.5&op_sys=Mac+System+8.6&op_sys=Mac+System+9.x&op_sys=MacOS+X&op_sys=Linux&op_sys=BSDI&op_sys=FreeBSD&op_sys=NetBSD&op_sys=OpenBSD&op_sys=AIX&op_sys=BeOS&op_sys=HP-UX&op_sys=IRIX&op_sys=Neutrino&op_sys=OpenVMS&op_sys=OS%2F2&op_sys=OSF%2F1&op_sys=Solaris&op_sys=SunOS&op_sys=other&priority=P1&priority=P2&priority=P3&priority=P4&priority=P5&bug_severity=blocker&bug_severity=critical&bug_severity=major&bug_severity=normal&bug_severity=minor&bug_severity=trivial&bug_severity=enhancement&email1=&emailtype1=substring&emailassigned_to1=1&email2=&emailtype2=substring&emailreporter2=1&bugidtype=include&bug_id=&changedin=&votes=&chfieldfrom=&chfieldto=Now&chfieldvalue=&short_desc=&short_desc_type=substring&long_desc=&long_desc_type=substring&bug_file_loc=&bug_file_loc_type=substring&status_whiteboard=&status_whiteboard_type=substring&keywords=&keywords_type=anywords&field0-0-0=noop&type0-0-0=noop&value0-0-0=&cmdtype=doit&order=Reuse+same+sort+as+last+time though this query results 54427 bugs trace malloc output file coming up...
trace malloc output ================================ large test url (8151 total query bugs) http://bugzilla.mozilla.org/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=RESOLVED&bug_status=VERIFIED&email1=&emailtype1=substring&email2=&emailtype2=substring&bugidtype=include&bug_id=&changedin=&votes=&chfield=bug_status&chfieldfrom=01%2F01%2F1970&chfieldto=Now&chfieldvalue=reopened&short_desc=&short_desc_type=substring&long_desc=&long_desc_type=substring&bug_file_loc=&bug_file_loc_type=substring&status_whiteboard=&status_whiteboard_type=substring&keywords=&keywords_type=anywords&field0-0-0=noop&type0-0-0=noop&value0-0-0=&namedcmd=allmybugs&newqueryname=&order=bugs.bug_id Type Count Bytes %Total TOTAL 510395 37463975 100.00 FrameArena 3906 16073190 42.90 nsScanner 291 4482698 11.97 HTMLAttributesImpl 51141 3067160 8.19 void* 117690 2424841 6.47 nsTextNode 50154 2407392 6.43 nsHTMLTableCellElement 45384 2359968 6.30 nsHTMLSpanElement 33854 1489576 3.98 nsLineBox 48292 965840 2.58 nsVoidArray 17756 613580 1.64 nsTextFragment 50121 546808 1.46 nsRegistry 1 512000 1.37 nsHTMLAnchorElement 5887 329672 0.88 nsHTMLTableRowElement 5655 271440 0.72 nsCellMap 50997 226636 0.60 nsTokenAllocator 40 194840 0.52 gtk-unclassified 647 130057 0.35 xpti-unclassified 97 110336 0.29 nsImageGTK 11 105676 0.28 nsFontMetricsGTK 1067 104295 0.28 OTHER 27404 1047970 2.80 jrgm's url (3039 total query bugs) **ignore my previous comment and url** http://bugzilla.mozilla.org/buglist.cgi?email1=&emailtype1=substring&emailassigned_to1=1&email2=&emailtype2=substring&emailreporter2=1&bugidtype=include&bug_id=&changedin=&votes=&chfield=%5BBug+creation%5D&chfieldfrom=2001-02-23&chfieldto=2001-03-22&chfieldvalue=&short_desc=&short_desc_type=substring&long_desc=&long_desc_type=substring&bug_file_loc=&bug_file_loc_type=substring&status_whiteboard=&status_whiteboard_type=substring&keywords=&keywords_type=anywords&field0-0-0=noop&type0-0-0=noop&value0-0-0=&cmdtype=doit&namedcmd=asdf&newqueryname=&order=Reuse+same+sort+as+last+time Type Count Bytes %Total TOTAL 102749 6975755 100.00 FrameArena 716 2946340 42.24 nsScanner 51 590784 8.47 HTMLAttributesImpl 9341 560080 8.03 void* 22855 518142 7.43 nsHTMLTableCellElement 8371 435292 6.24 nsTextNode 8963 430224 6.17 nsHTMLSpanElement 6270 275880 3.95 nsLineBox 8632 172640 2.47 nsVoidArray 3463 125336 1.80 nsTextFragment 9088 108748 1.56 nsTokenAllocator 15 73065 1.05 nsHTMLAnchorElement 1116 62496 0.90 nsHTMLTableRowElement 1046 50208 0.72 X-unclassified 1235 45490 0.65 xpti-unclassified 44 45056 0.65 nsCellMap 9362 41696 0.60 nsComponentManagerImpl 1599 38928 0.56 nsZipArchive 691 30187 0.43 unclassified-string 1009 29581 0.42 OTHER 8882 395582 5.67
The above information indicates the requested size of objects that are live at the time the snapshot was taken. The snapshot was taken after the page finished loading. So...I would've thought that the objects attributed to nsScanner (4MB worth!) would go away once the document load completes. Here are the stack traces that are being ``counted'' as nsScanner... <nsScanner> PR_Malloc nsMemoryImpl::Alloc(unsigned int) nsMemory::Alloc(unsigned int) nsScanner::Append(char const *, unsigned int) <nsScanner> __builtin_new nsSlidingString::AppendBuffer(unsigned short *, unsigned short *, unsigned short *) nsScanner::AppendToBuffer(unsigned short *, unsigned short *, unsigned short *) <nsScanner> __builtin_new nsScanner::AppendToBuffer(unsigned short *, unsigned short *, unsigned short *) <nsScanner> PR_Malloc nsMemoryImpl::Alloc(unsigned int) nsMemory::Alloc(unsigned int) unsigned short * AllocateStringCopy<unsigned short, unsigned short>(basic_nsAReadableString<unsigned short> const &, unsigned short *) ToNewUnicode(basic_nsAReadableString<unsigned short> const &) nsScanner::Append(basic_nsAReadableString<unsigned short> const &) <nsScanner> PR_Malloc nsMemoryImpl::Alloc(unsigned int) nsMemory::Alloc(unsigned int) nsStr::Alloc(nsStr &, unsigned int) nsStr::Realloc(nsStr &, unsigned int) nsStr::EnsureCapacity(nsStr &, unsigned int) nsStr::GrowCapacity(nsStr &, unsigned int) nsString::SetCapacity(unsigned int) nsString::SetLength(unsigned int) CopyUnicodeTo(nsReadingIterator<unsigned short> const &, nsReadingIterator<unsigned short> const &, basic_nsAWritableString<unsigned short> &) nsScanner::CopyUnusedData(nsString &) <nsScanner> PR_Malloc nsMemoryImpl::Alloc(unsigned int) nsMemory::Alloc(unsigned int) nsStr::Alloc(nsStr &, unsigned int) nsStr::Realloc(nsStr &, unsigned int) nsStr::EnsureCapacity(nsStr &, unsigned int) nsStr::GrowCapacity(nsStr &, unsigned int) nsStr::StrAppend(nsStr &, nsStr const &, unsigned int, int) nsStr::StrAssign(nsStr &, nsStr const &, unsigned int, int) nsString::nsString(nsString const &) nsScanner::nsScanner(nsString &, int, nsString const &, nsCharsetSource)
Component: Browser-General → Tracking
QA Contact: doronr → chofmann
The problem sounds very similar to bug 75641. Waterson, which build are you using? I'll try this on my machine.
On the 3000 bugs URL, I'm seeing about 3.1MB allocated at the stack: PR_Malloc nsMemory::Alloc(unsigned int) nsScanner::Append(const char*, unsigned int) nsByteArrayInputStream::ReadSegments(PFjP14nsIInputStreamPvPKcjjPjES3_jS6_) nsParser::OnDataAvailable(nsIRequest*, nsIRequest*, nsIInputStream, unsigned int, unsigned int) (I did this in mozilla, so it could be related to the search stuff, but I don't think so. The demangling of the parameters to |ReadSegments| is left as an exercise to the reader :-) What uses nsByteArrayInputStream?
data was generated based on a debug build from Monday 5/14/01
new trace-malloc results from 5/22 linux debug build: large test url (8092 bugs) http://bugzilla.mozilla.org/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=RESOLVED&bug_status=VERIFIED&email1=&emailtype1=substring&email2=&emailtype2=substring&bugidtype=include&bug_id=&changedin=&votes=&chfield=bug_status&chfieldfrom=01%2F01%2F1970&chfieldto=Now&chfieldvalue=reopened&short_desc=&short_desc_type=substring&long_desc=&long_desc_type=substring&bug_file_loc=&bug_file_loc_type=substring&status_whiteboard=&status_whiteboard_type=substring&keywords=&keywords_type=anywords&field0-0-0=noop&type0-0-0=noop&value0-0-0=&namedcmd=allmybugs&newqueryname=&order=bugs.bug_id Type Count Bytes %Total TOTAL 520512 38578000 100.00 FrameArena 3971 16340665 42.36 nsScanner 295 4768896 12.36 void* 119911 3161235 8.19 HTMLAttributesImpl 51960 3116472 8.08 nsTextNode 51100 2452800 6.36 nsHTMLTableCellElement 46071 2395692 6.21 nsHTMLSpanElement 34399 1513556 3.92 nsLineBox 48996 979920 2.54 nsVoidArray 18118 631452 1.64 nsTextFragment 51075 557600 1.45 nsHTMLAnchorElement 5975 334600 0.87 nsHTMLTableRowElement 5799 278352 0.72 nsCellMap 51827 230248 0.60 nsTokenAllocator 46 224066 0.58 gtk-unclassified 664 123578 0.32 nsFontMetricsGTK 1032 118215 0.31 xpti-unclassified 101 117912 0.31 nsImageGTK 6 105116 0.27 nsComponentManagerImpl 3362 77364 0.20 OTHER 25804 1050261 2.72 small test url (3031 bugs) http://bugzilla.mozilla.org/buglist.cgi?email1=&emailtype1=substring&emailassigned_to1=1&email2=&emailtype2=substring&emailreporter2=1&bugidtype=include&bug_id=&changedin=&votes=&chfield=%5BBug+creation%5D&chfieldfrom=2001-02-23&chfieldto=2001-03-22&chfieldvalue=&short_desc=&short_desc_type=substring&long_desc=&long_desc_type=substring&bug_file_loc=&bug_file_loc_type=substring&status_whiteboard=&status_whiteboard_type=substring&keywords=&keywords_type=anywords&field0-0-0=noop&type0-0-0=noop&value0-0-0=&cmdtype=doit&namedcmd=asdf&newqueryname=&order=Reuse+same+sort+as+last+time Type Count Bytes %Total TOTAL 147879 10422436 100.00 FrameArena 1027 4226105 40.55 nsScanner 73 1270018 12.19 HTMLAttributesImpl 13412 803856 7.71 void* 32961 746625 7.16 nsTextNode 13000 624000 5.99 nsHTMLTableCellElement 11940 620880 5.96 nsHTMLSpanElement 8883 390852 3.75 nsLineBox 12281 245620 2.36 nsVoidArray 4953 173536 1.67 nsTextFragment 12997 131109 1.26 xpti-unclassified 76 92312 0.89 nsHTMLAnchorElement 1571 87976 0.84 nsHTMLTableRowElement 1518 72864 0.70 nsTokenAllocator 14 68194 0.65 nsCellMap 13375 59336 0.57 gtk-unclassified 438 55390 0.53 nsComponentManagerImpl 2322 54708 0.52 unclassified-string 1506 45828 0.44 nsZipArchive 982 42922 0.41 OTHER 14550 610305 5.86
Let me investigate this bug. CCing cathleen.
Status: NEW → ASSIGNED
Target Milestone: Future → mozilla0.9.2
Assignee: cathleen → harishd
Status: ASSIGNED → NEW
Moving to my plate.
Priority: P3 → P1
*** Bug 84820 has been marked as a duplicate of this bug. ***
*** Bug 50411 has been marked as a duplicate of this bug. ***
*** Bug 84944 has been marked as a duplicate of this bug. ***
Probably this is related to bug 87370.
Status: NEW → ASSIGNED
There is always an expansion factor when you read a file and display the contents, because you need so much more for display. The expansion factor will naturally depend on the structures in the document and the complexity of the display (like tables) that is needed. What is a reasonable expansion factor is a different thing. I did some testing a 2-3 years ago with Mozilla and Multidoc Pro (an SGML browser I was working on at that time). Multidoc Pro's expansion factor was about 5. This meant that if I loaded a file of size 1MB, it took about 5MB of RAM to load and display this document. This is extremely good expansion factor. The code had been optimized for size and speed over several years. Multidoc Pro did not support Unicode when the testing was done. At that time Mozilla's expansion factor was 12-16 (it varied, and I can't remember exactly). That was obviously too much. I thought an expansion factor of 10 might be attainable at that time. If 3.2MB file grows into 25MB in memory, we get an expansion factor of about 8! Somebody might want to try and calculate some kind of theoretical lower limit to the expansion factor (supposing we load the whole document and lay it out completely). Because our internal representation is 32-bit Unicode, this means the expansion factor must be over 2 for non-Unicode documents. I also checked this file with IE 5.0, the expansion factor was 12. Just to be sure, I checked what it was for NS6.1b1: 17. I used the first URL in the first comment to this bug, assuming that was 3.8 megs. So it seems I was measuring something else than what you are talking about...
6/25 debug trace-malloc build, running small test url (3030 bugs) http://bugzilla.mozilla.org/buglist.cgi?email1=&emailtype1=substring&emailassigned_to1=1&email2=&emailtype2=substring&emailreporter2=1&bugidtype=include&bug_id=&changedin=&votes=&chfield=%5BBug+creation%5D&chfieldfrom=2001-02-23&chfieldto=2001-03-22&chfieldvalue=&short_desc=&short_desc_type=substring&long_desc=&long_desc_type=substring&bug_file_loc=&bug_file_loc_type=substring&status_whiteboard=&status_whiteboard_type=substring&keywords=&keywords_type=anywords&field0-0-0=noop&type0-0-0=noop&value0-0-0=&cmdtype=doit&namedcmd=asdf&newqueryname=&order=Reuse+same+sort+as+last+time Type Count Bytes %Total TOTAL 254480 18750891 100.00 FrameArena 1756 7225940 38.54 nsScanner 135 2254934 12.03 void* 57598 1884127 10.05 HTMLAttributesImpl 23183 1390060 7.41 nsTextNode 22516 1080768 5.76 nsHTMLTableCellElement 20531 1067612 5.69 nsHTMLSpanElement 15371 676324 3.61 nsLineBox 21436 428720 2.29 nsVoidArray 8543 299268 1.60 nsTextFragment 22505 242637 1.29 nsTokenAllocator 43 209453 1.12 gtk-unclassified 777 155744 0.83 nsHTMLAnchorElement 2693 150808 0.80 xpti-unclassified 126 141432 0.75 nsHTMLTableRowElement 2549 122352 0.65 nsComponentManagerImpl 4022 116560 0.62 nsImageGTK 14 105888 0.56 nsCellMap 23058 102540 0.55 nsZipArchive 1820 79312 0.42 OTHER 25804 1016412 5.42
Target Milestone: mozilla0.9.2 → mozilla0.9.3
new build from 6/26, pulled trunk after dougt checked in fix for bug 70398 trace-malloc result from running small url test (3030 bugs) http://bugzilla.mozilla.org/buglist.cgi?email1=&emailtype1=substring&emailassigned_to1=1&email2=&emailtype2=substring&emailreporter2=1&bugidtype=include&bug_id=&changedin=&votes=&chfield=%5BBug+creation%5D&chfieldfrom=2001-02-23&chfieldto=2001-03-22&chfieldvalue=&short_desc=&short_desc_type=substring&long_desc=&long_desc_type=substring&bug_file_loc=&bug_file_loc_type=substring&status_whiteboard=&status_whiteboard_type=substring&keywords=&keywords_type=anywords&field0-0-0=noop&type0-0-0=noop&value0-0-0=&cmdtype=doit&namedcmd=asdf&newqueryname=&order=Reuse+same+sort+as+last+time Type Count Bytes %Total TOTAL 165738 10404470 100.00 FrameArena 1127 4637605 44.57 void* 38201 1007977 9.69 HTMLAttributesImpl 14542 872052 8.38 nsTextNode 14115 677520 6.51 nsHTMLTableCellElement 12843 667836 6.42 nsHTMLSpanElement 9680 425920 4.09 nsLineBox 13528 270560 2.60 nsVoidArray 5381 187492 1.80 nsTextFragment 13981 137522 1.32 nsZipArchive 2634 113998 1.10 xpti-unclassified 91 107696 1.04 nsImageGTK 9 105264 1.01 gtk-unclassified 479 97875 0.94 nsHTMLAnchorElement 1675 93800 0.90 nsHTMLTableRowElement 1570 75360 0.72 nsComponentManagerImpl 2072 64536 0.62 nsCellMap 14463 64224 0.62 X-unclassified 1676 49980 0.48 JS-unclassified 1056 43572 0.42 OTHER 16615 703681 6.76
voila! nsScanner bloat disappeared :-)
Actually, dougt checked in a patch for bug 87370. The patches in bug 70398 & in bug 87370 are identical.
Sweet Justice!
large url (8551 bugs) trace-malloc results: http://bugzilla.mozilla.org/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=RESOLVED&bug_status=VERIFIED&email1=&emailtype1=substring&email2=&emailtype2=substring&bugidtype=include&bug_id=&changedin=&votes=&chfield=bug_status&chfieldfrom=01%2F01%2F1970&chfieldto=Now&chfieldvalue=reopened&short_desc=&short_desc_type=substring&long_desc=&long_desc_type=substring&bug_file_loc=&bug_file_loc_type=substring&status_whiteboard=&status_whiteboard_type=substring&keywords=&keywords_type=anywords&field0-0-0=noop&type0-0-0=noop&value0-0-0=&namedcmd=allmybugs&newqueryname=&order=bugs.bug_id Type Count Bytes %Total TOTAL 304043 19191342 100.00 FrameArena 2323 9559145 49.81 HTMLAttributesImpl 30432 1825700 9.51 void* 70469 1514253 7.89 nsTextNode 29949 1437552 7.49 nsHTMLTableCellElement 26942 1400984 7.30 nsHTMLSpanElement 20207 889108 4.63 nsLineBox 28495 569900 2.97 nsVoidArray 10528 365740 1.91 nsTextFragment 29896 345132 1.80 nsHTMLAnchorElement 3508 196448 1.02 nsHTMLTableRowElement 3371 161808 0.84 nsCellMap 30303 134664 0.70 xpti-unclassified 58 70600 0.37 nsFontMetricsGTK 419 54154 0.28 gtk-unclassified 352 50490 0.26 nsComponentManagerImpl 1870 50224 0.26 nsPersistentProperties 2016 40178 0.21 nsZipArchive 851 37062 0.19 FrameManager 67 35577 0.19 OTHER 11987 452623 2.36
nice work dougt!
This rocks!
So, what do we do with this bug now :-/ ? Pass it on?
What expansion factor are we aiming for? If we're at 6 now I guess we could aim for 3-4. Otherwise i think we could mark this bug as fixed. As I know very little about the data, the only stuff that interests me: FrameArena 2323 9559145 49.81 <- this is real data, right? void* 70469 1514253 7.89 <- can we find out who's using this nsVoidArray 10528 365740 1.91 <- can we find out who's using this OTHER 11987 452623 2.36 <- should we spend any time looking at this? but this probably isn't interesting...
Assignee: harishd → cathleen
Status: ASSIGNED → NEW
Since the scanner issue has been resolved giving bug back to cathleen to find appropriate owners for the remaining problems ;-)
Doesn't look like this is getting fixed before the freeze tomorrow night. Pushing out a milestone. Please correct if I'm mistaken.
Target Milestone: mozilla0.9.3 → mozilla0.9.4
Summary: Does a >3.2MB file (bunch of tables, >7975 rows), really need 25MB of memory? → Does a >3.2MB file (bunch of tables, >7975 rows), really need 19MB of memory?
Target Milestone: mozilla0.9.4 → mozilla0.9.5
marc, can you help figure out the bloat in FrameArena?
Assignee: cathleen → attinasi
Blocks: 71874
Sure, happy to help - as soon as I finish fixing my 5 topembed bugs ;)
Status: NEW → ASSIGNED
FrameArena holds frames, as you might guess. I loaded the query (which results in more than 9000 hits now) and dumped the frame size stats from Viewer. Here is the result: Frame Type Count TotalSize MinSize MaxSize AvgSize ---------- ----- --------- ------- ------- ------- AreaFrame 5 380 76 76 76 BRFrame 2 88 44 44 44 BlockFrame 73787 5607812 76 76 76 CanvasFrame 1 72 72 72 72 HRFrame 2 128 64 64 64 ImageFrame 1 0 0 0 0 InlineFrame 64627 3619112 56 56 56 LineBox:block,big 1 60 60 60 60 LineBox:block,small 59 2360 40 40 40 LineBox:inline,big 11 660 60 60 60 LineBox:inline,smal 74202 2968080 40 40 40 ScrollFrame 4 224 56 56 56 TableCellFrame 73779 6787668 92 92 92 TableCellMap 52 2912 56 56 56 TableColFrame 379 37900 100 100 100 TableColGroupFrame 52 3536 68 68 68 TableFrame 52 7488 144 144 144 TableLayoutStrategy 52 1248 24 24 24 TableOuterFrame 52 3952 76 76 76 TableRowFrame 9227 701252 76 76 76 TableRowGroupFrame 52 3120 60 60 60 TextFrame 81994 5249248 64 68 64 TextInputFrame 1 56 56 56 56 ViewportFrame 1 60 60 60 60 gfxButtonControlFra 3 168 56 56 56 *** Total *** 378420 24998744 378 thousand frames taking up almost 25MB. Wow. 81 thousand text frames alone take up 5MB, and 73 thousand table cells take up over 6 and a half MB. That is a seriously large page or pretty complicated HTML (the file I loaded is 3.7 MB in size, when saved from Mozilla). I don't know just where to go with this data: could we try to shrink the size of frames (tableCell frames are pretty big)? Could we try to eliminate some frames (like, there is a block for every table cell - is that really necessary)? Should we look at the LineBox 'frames' and see if they can be shrunk or eliminated? To reduce this much, I think we need to seriously rethink the layout architecture, and that scares me plenty. Other ideas?
Cc: vidur and dp 'cos I think they would be interested in this...
*** Bug 99903 has been marked as a duplicate of this bug. ***
I would like to mention that I reported the bug 99903 that has been said duplicate of that one. My problem is that, as I said in the report for that bug, the browser then hangs for 30 seconds after the page is downloaded. I have a good machine (Pentium 1Ghz, 256 Mo) and not that many applications opened. I see that if I open the same page with IE 5.x, IE doesn't hang as Mozilla does. I forgot to say in my report that even if Mozilla is unresponsive, other applications are still responsive. So it is not a memory only problem. Furthermore, in the report for the bug 49539, the expansion factor is said to be bigger in IE. Why then should Mozilla hang? Are we sure the two bugs are identical?
end of milestone shuffle...
Target Milestone: mozilla0.9.5 → mozilla0.9.6
Blocks: 92580
Severity: normal → major
Priority: P1 → P2
Target Milestone: mozilla0.9.6 → mozilla1.0
See http://bugzilla.mozilla.org/process_bug.cgi#c37
Keywords: footprint
Target Milestone: mozilla1.0 → mozilla1.1
*** Bug 126861 has been marked as a duplicate of this bug. ***
*** Bug 124441 has been marked as a duplicate of this bug. ***
Build from 2/14 on Win98/400MHz/128MB: - The list of open bugs I reported (~300) takes 7 seconds. - The list of bugs I reported (~800) takes 25 seconds. A 2/20 build seems to be about 1/3 faster. On the 2/20 build, the UNCO list (~2000) effectively hangs Mozilla for a minute while it allocates >100MB. The strange thing is that although it hangs for several periods of ~7 seconds as it loads the page, the one-minute hang appears to happen just after the throbber *stops* spinning. If this is unrelated to the FrameArena stuff, please reopen one of the dups...
This could be partially fixed (although the memory issue still remains) by not rendering the entire page at once. There is no need to render the entire page when only a small portion is visible on screen. See bug 122581.
I don't know what you mean by "render", but the whole page has to be parsed and analyzed to display only a small subset of it. That is beacuse the page can contain elements that affect the displayed area even if they are written late in the file. That could be absolute positioned elemenents, elements that requires a table to grow, elements that contain scripts or event handlers that must be run... But if there are anything specific you think we do now that don't have to done, please tell us. We could really need more optimizations for big pages.
Right, but that doesn't mean it has to be all in memory at once. If you are parsing a page that is extremely huge, you could sort of "page" parts of it in and out of memory from disk. Doing so would definately be more efficient than bogging down the system memory and causing the operating system to scrap for more space. For instance, images that are off screen don't need to be stored in memory, and even sections of tables, etc don't need to be if they are off screen.
no no no no no! mozilla has reinvented lots of things, but i'm pretty sure shaver and brendan would tell us that we should not reinvent paging. That's the os's job, and it should do it well enough for our purposes. if it doesn't, then the user should take steps to fix it as having bad paging would hurt all apps. any memory not recently used (and even memory recently used) is a candidate for the os to page out, let the os do its job. mind you i'm not saying we shouldn't release memory.
I'm not talking about reinventing paging. I am talking about only loading so much of the page as so it doesn't overburden memory if it is a huge page. There is no reason on a page that is 1MB of tables or whatever that the whole page needs to stored in memory at once. This isn't reinventing paging, this is not being a memory hog so that the os doesn't have to page. That is a big difference. When you have a web page that is 10000 lines and only 100 are visible in the window, why have all 10000 lines in memory? Wouldn't 500 suffice?
Paging is inevitably a slow operation, and avoiding it by proper use of the program's memory would be more efficient than sucking up gobs of system memory so that the operating system has to start paging. Timeless, I agree we shouldn't handle paging, but it almost sounds as if you are saying that we should just suck up memory in loading a huge page and blame it on the os because its not fast at paging. What about when the system is low on hard disk space? Bug 122581 talks about a page with a lot of images. Is it really necessary to have all the images in memory at once? Is it necessary to have an entire page of text in memory at once? Couldn't Mozilla tag the things that other parts of the page might depend on and keep them in memory? What I'm basically asking is if there is any way to more efficiently (memory-wise) parse a page. Is there any way to not parse it all at once and not have it all in memory at one time?
Where do you suppose the other 9500 lines should be stored? We can't just forget about them, because soon the user will scroll down to see it. I don't see much of a difference between us writing down the information to disk and letting the OS do it. The main difference is that us doing it will always kill performance while letting the OS do will let us run from memory when there is enough memory. Image handling is another thing though, storing them in memory compressed/uncompressed, in main memory/graphic card memory... There are lots of bugs about that and I suggest you see if you can contribute something there if you have ideas.
Daniel: Of course if there is enough _physical_ memory, then by all means the entire page should be loaded in memory. Otherwise, there should be some cut-off where part of the page is left un-parsed until the user enters that location of the page. First of all, paging continuously on a system will kill performance. When the user gets partway down in the page and a new part has to be loaded will only be a temporary jolt to the system. You can even start loading the second part before the user gets to it so that there is no delay. I would much rather see Mozilla be kinder on memory than try to increase performance on an already doomed situation of a huge page. Worry about speed on the average page and worry about memory on these huge pages. Its not worth bogging down a user's system memory in order to get a little more speed.
When the page exceeds a pre-determined (based on system memory, etc) amount of memory, the page should be scanned to determine if there is anything (such as javascript) that should be parsed ahead of time because the part of the page currently parse depends on it. Everything else should be left until the user gets to it.
my programming skills are very minimalistic, im more like a hardware guy. But HOW can it take so much memory at all ? the unconfirmed buglist is approx 850k and takes 400megs to parse/render. sounds like a more basic problem than when render what and when have it in memory or not.
Yes, that do sound excessive. If 850 KB simple HTML code uses 400 MB of RAM to render, then thats a enormous regression. As can be seen in this bug's summary the know memory usage for such data is much, much lower (19 MB for ~8000 rows / 3.2 MB HTML)
a funny thing i recognized during the page is rendered the win2k taskmanager shows ~80megs, but once its DONE rendering the mem usagesa goes over 400megs. is just taskmanager too slow to show the info just in time, or aint this a parsing problem after all ?
yes, I'm seeing this after the fact increase in memory consumption too
I doubt Task Manager is missing it unless its frozen. Could this be some kind of memory leak?
No the task manager is not frozen. And I always see this when loading various of these image heavy pages.
im still using the unconfirmed bugs list from bugzilla as my personal test. only image is the head and it still aquires so much memory(over 400megs) AFTER the rendering is complete it seems
Do you have the Search sidebar tab enabled? (Is that still active if it's not open?)
hmm, right completly turning off the sidebar takes me down to 110 megs mem usage. so the side bar swallows over 300megs ???? yikes
I do not have the sidebar showing.
Just to add my comments here. Using the BugZilla link my memory usage increases steadily till around 110 MB, where the page is now fully parsed. It the decreases till its back to 24MB. HOWEVER, Mozilla is now non-responsive. Clicking anywhere in the Mozilla window kills it. Will check with sidebar showing.
Correction on my last post. Got different things this time. As usual, page starts loading, memory rises. It peaked at 127MB RAM Usage, plus 80%CPU (Pentium 4 1.4GHZ). However, the page was NOT finished loading. While the page was still loading, my mem usage went all over the place. It ended up jumping between 19MB (Which is lower than Mozilla normally users), and 30MB, while the page was still being parsed. At the end of parsing, Mozilla is again unresponsive. Dont know what that tells you, but the fact that it decreased hugely while still rendering, I would imagine tells you something :)
anyone try with a nightly today ? i just had a very special happening. via the bugzilla page i requested all bugs containing "mail", this made mozilla user 600 megs swap and 80 megs ram, with sidebar turned off !!! then i opened one bug in a new window, real ram usage jumped to 94 megs. scoll a bit up and down in the newly created window and you can watch mozilla grabbing more and more memory, about 120 k for each full scroll. close mozilla, leave quicklaunch on, mozilla drops down to 64megs ram/530megs swap. during freeing memory windows warned twice that the app is not responding. cant that be optimized in any way ?
on low spec systems this also results in crash and dataloss because of the freezing ... and the bug would get more attention ;P
*** Bug 147115 has been marked as a duplicate of this bug. ***
*** Bug 149861 has been marked as a duplicate of this bug. ***
the behavior here starting at comment 45 is bug 120712, caused by checkin for bug 77460. this seems to be FIXED (by 87370) unless the FrameArena memory usage still needs work (comment 37)
retargeting as 1.1a is already out
Target Milestone: mozilla1.1alpha → mozilla1.1beta
marking FIXED by bug 87370. It seems unlikely that Attinasi is going to work on this anymore If you are "seeing" this bug, it's bug 120712, which needs sr=
Status: ASSIGNED → RESOLVED
Closed: 23 years ago
Resolution: --- → FIXED
phoenix stats about:blank 18944k url: 39876k file: 1444k that appears to be an expansion factor of 14. it's now a <1.5mb file that takes 20mb of memory.
sorry that was (phoenix) Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.1b) Gecko/20020910 i just tested w/ Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.2b) Gecko/20020921 Phoenix/0.1 current phoenix stats about:blank 16852k url: 43468k file: 1444k so it's now a <1.5mb file that takes 26mb of memory. that appears to be an expansion factor of 18. note that the query results in 3033 bugs and since i saved it using phoenix, there was some random (ok, not random, because of multipart/replace) garbage (the please wait page) at the top of the file: --thisrandomstring&#9689;Content-Type: text/html&#9689;&#9689;<!-- 1.0@bugzilla.or g -->&#9689;&#9689;&#9689;&#9689; &#9689;<html>&#9689; <head>&#9689; <title>Bugzilla is pondering you r query</title>&#9689; </head>&#9689; <body>&#9689; <h1 style="margin-top: 20 %; text-align: center;">Please stand by ...</h1>&#9689;&#9689;&#9689; </body>&#9689;</h tml>&#9689;--thisrandomstring&#9689;Content-Type: text/html&#9689;Set-Cookie: LAST ORDER=bugs.bug_id ; path=/; expires=Sun, 30-Jun-2029 00:00:00 GM T&#9689;Set-Cookie: BUGLIST= ; path=/; expires=Sun, 30-Jun-2029 00:00: 00 GMT&#9689;&#9689;<!-- 1.0@bugzilla.org -->&#9689;&#9689;&#9689;&#9689;&#9689;&#9689;&#9689;&#9689;&#9689;&#9689;&#9689;&#9689;<!-- 1.0@bugzilla.o rg -->&#9689;&#9689;&#9689;&#9689;&#9689;&#9689;&#9689;&#9689;<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Trans steps: 1. run phoenix 2. load data:text/html,<a href=about:blank>a 3. right click a and select open in new window 4. close original window 5. checkpoint memory usage (taskmanager) 6. file>open file 7. checkpoint memory usage (taskmanager)
as for 'seeing' that other bug, phoenix doesn't have search, so i'm not affected by that.
Product: Core → Core Graveyard
You need to log in before you can comment on or make changes to this bug.