Closed Bug 49539 Opened 24 years ago Closed 22 years ago

Does a >3.2MB file (bunch of tables, >7975 rows), really need 19MB of memory?

Categories

(Core Graveyard :: Tracking, defect, P2)

Tracking

(Not tracked)

RESOLVED FIXED
mozilla1.1beta

People

(Reporter: jag+mozbugs, Assigned: attinasi)

References

()

Details

(Keywords: memory-footprint, perf)

For comparison, testing the 2000-08-18-08-M18 nightly on WinNT box with 128MB
physical RAM:

1. After start (displaying http://www.mozilla.org/): 
   Mem: 18752k  VM: 12876k (55MB of physical RAM free at this point)
2. After displaying URL above:
   Mem: 50972k  VM: 45000k (34MB of physical RAM free at this point)
3. After clicking on [Mozilla.org] button on Perrsonal Toolbar:
   Mem: 36384k  VM: 30400k (49MB of physical RAM free at this point)
(no other action taken between steps 2 and 3, other than recording these
 numbers in a NN 4.74 window)

Early in the pageload (before the first </TABLE>), Mozilla was increasing
memory usage ~80k at a time, so the ballooning memory usage must be nonlinear.
adding perf keyword
Keywords: perf
nav triage team:

Does this bloat still occur?
Keywords: nsbeta1-
Target Milestone: --- → Future
yes, this bloat still exists.  on w2k, mem goes up 50M when I load this page,
and cpu gets pegged.  I had to kill mozilla.  I was clicking on the url in the
first report of this bug.  over to cathleen for look/owner.
Assignee: mcafee → cathleen
Verified with my win2K box.

1) run taskinfo2000
2) launch mozilla and load this bug page, memory used 60,768Kb
3) click on the quary link in the bug (a long while till it's done), memory used
   191,720Kb.  The highest run I saw was 202,584Kb.  

Eventually (within a minute or two) mozilla will freeze and die.

I did the same test with 4.77, memory went from 53,908Kb to 115,348Kb.  4.77 
still standing.

OS: Linux → All
Hardware: PC → All
Let's use bloat-blame to figure out where the memory is going!

1. Make a Linux build (using egcs-1.1.2) with the following flags in a
   partition with a _ton_ of disk space.

  --enable-trace-malloc
  --enable-cpp-rtti
  --enable-debug

2. Run gtkEmbed from mozilla/dist/bin as follows:

   LD_LIBRARY_PATH=`pwd` MOZILLA_FIVE_HOME=`pwd` \
   gtkEmbed --trace-malloc /dev/null "(Insert URL Here)"

3. Once the page finishes loading, press the ``dump memory usage button''. This
   will create a monster file in mozilla/dist/bin called ``allocations.log''

4. Check out the trace-malloc tools. From the mozilla directory,

   cvs co tools/trace-malloc

5. Process the log:

   tools/trace-malloc/histogram.pl dist/bin/allocations.log > hist
   tools/trace-malloc/histogram-pretty.sh hist

6. Paste the output into the bug

More details on how to use the trace-malloc tools at

  http://www.mozilla.org/projects/footprint/live-bloat.html

Good luck!
*** Bug 78142 has been marked as a duplicate of this bug. ***
Updating the summary, noting that the query is for any bug reopened ever, 
which means the list will grow month by month. See bug 78142 for a slightly
more sane query (about 3000 bugs, which will not change over time).
Summary: Does a 2.1MB file (bunch of tables, 5114 rows), really need 25MB of memory? → Does a >3.2MB file (bunch of tables, >7975 rows), really need 25MB of memory?
new test query, using jrgm's bug 78142 

http://bugzilla.mozilla.org/buglist.cgi?bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=RESOLVED&bug_status=VERIFIED&bug_status=CLOSED&resolution=FIXED&resolution=INVALID&resolution=WONTFIX&resolution=LATER&resolution=REMIND&resolution=DUPLICATE&resolution=WORKSFORME&resolution=MOVED&rep_platform=All&rep_platform=DEC&rep_platform=HP&rep_platform=Macintosh&rep_platform=PC&rep_platform=SGI&rep_platform=Sun&rep_platform=Other&op_sys=All&op_sys=Windows+3.1&op_sys=Windows+95&op_sys=Windows+98&op_sys=Windows+ME&op_sys=Windows+2000&op_sys=Windows+NT&op_sys=Mac+System+7&op_sys=Mac+System+7.5&op_sys=Mac+System+7.6.1&op_sys=Mac+System+8.0&op_sys=Mac+System+8.5&op_sys=Mac+System+8.6&op_sys=Mac+System+9.x&op_sys=MacOS+X&op_sys=Linux&op_sys=BSDI&op_sys=FreeBSD&op_sys=NetBSD&op_sys=OpenBSD&op_sys=AIX&op_sys=BeOS&op_sys=HP-UX&op_sys=IRIX&op_sys=Neutrino&op_sys=OpenVMS&op_sys=OS%2F2&op_sys=OSF%2F1&op_sys=Solaris&op_sys=SunOS&op_sys=other&priority=P1&priority=P2&priority=P3&priority=P4&priority=P5&bug_severity=blocker&bug_severity=critical&bug_severity=major&bug_severity=normal&bug_severity=minor&bug_severity=trivial&bug_severity=enhancement&email1=&emailtype1=substring&emailassigned_to1=1&email2=&emailtype2=substring&emailreporter2=1&bugidtype=include&bug_id=&changedin=&votes=&chfieldfrom=&chfieldto=Now&chfieldvalue=&short_desc=&short_desc_type=substring&long_desc=&long_desc_type=substring&bug_file_loc=&bug_file_loc_type=substring&status_whiteboard=&status_whiteboard_type=substring&keywords=&keywords_type=anywords&field0-0-0=noop&type0-0-0=noop&value0-0-0=&cmdtype=doit&order=Reuse+same+sort+as+last+time

though this query results 54427 bugs
trace malloc output file coming up...
trace malloc output
================================

large test url (8151 total query bugs)
http://bugzilla.mozilla.org/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=RESOLVED&bug_status=VERIFIED&email1=&emailtype1=substring&email2=&emailtype2=substring&bugidtype=include&bug_id=&changedin=&votes=&chfield=bug_status&chfieldfrom=01%2F01%2F1970&chfieldto=Now&chfieldvalue=reopened&short_desc=&short_desc_type=substring&long_desc=&long_desc_type=substring&bug_file_loc=&bug_file_loc_type=substring&status_whiteboard=&status_whiteboard_type=substring&keywords=&keywords_type=anywords&field0-0-0=noop&type0-0-0=noop&value0-0-0=&namedcmd=allmybugs&newqueryname=&order=bugs.bug_id

Type                    Count    Bytes %Total
TOTAL                  510395 37463975 100.00
FrameArena               3906 16073190  42.90
nsScanner                 291  4482698  11.97
HTMLAttributesImpl      51141  3067160   8.19
void*                  117690  2424841   6.47
nsTextNode              50154  2407392   6.43
nsHTMLTableCellElement  45384  2359968   6.30
nsHTMLSpanElement       33854  1489576   3.98
nsLineBox               48292   965840   2.58
nsVoidArray             17756   613580   1.64
nsTextFragment          50121   546808   1.46
nsRegistry                  1   512000   1.37
nsHTMLAnchorElement      5887   329672   0.88
nsHTMLTableRowElement    5655   271440   0.72
nsCellMap               50997   226636   0.60
nsTokenAllocator           40   194840   0.52
gtk-unclassified          647   130057   0.35
xpti-unclassified          97   110336   0.29
nsImageGTK                 11   105676   0.28
nsFontMetricsGTK         1067   104295   0.28
OTHER                   27404  1047970   2.80       

jrgm's url (3039 total query bugs) **ignore my previous comment and url**
http://bugzilla.mozilla.org/buglist.cgi?email1=&emailtype1=substring&emailassigned_to1=1&email2=&emailtype2=substring&emailreporter2=1&bugidtype=include&bug_id=&changedin=&votes=&chfield=%5BBug+creation%5D&chfieldfrom=2001-02-23&chfieldto=2001-03-22&chfieldvalue=&short_desc=&short_desc_type=substring&long_desc=&long_desc_type=substring&bug_file_loc=&bug_file_loc_type=substring&status_whiteboard=&status_whiteboard_type=substring&keywords=&keywords_type=anywords&field0-0-0=noop&type0-0-0=noop&value0-0-0=&cmdtype=doit&namedcmd=asdf&newqueryname=&order=Reuse+same+sort+as+last+time

Type                    Count    Bytes %Total
TOTAL                  102749  6975755 100.00
FrameArena                716  2946340  42.24
nsScanner                  51   590784   8.47
HTMLAttributesImpl       9341   560080   8.03
void*                   22855   518142   7.43
nsHTMLTableCellElement   8371   435292   6.24
nsTextNode               8963   430224   6.17
nsHTMLSpanElement        6270   275880   3.95
nsLineBox                8632   172640   2.47
nsVoidArray              3463   125336   1.80
nsTextFragment           9088   108748   1.56
nsTokenAllocator           15    73065   1.05
nsHTMLAnchorElement      1116    62496   0.90
nsHTMLTableRowElement    1046    50208   0.72
X-unclassified           1235    45490   0.65
xpti-unclassified          44    45056   0.65
nsCellMap                9362    41696   0.60
nsComponentManagerImpl   1599    38928   0.56
nsZipArchive              691    30187   0.43
unclassified-string      1009    29581   0.42
OTHER                    8882   395582   5.67       
The above information indicates the requested size of objects that are live at 
the time the snapshot was taken. The snapshot was taken after the page finished 
loading. So...I would've thought that the objects attributed to nsScanner (4MB 
worth!) would go away once the document load completes. Here are the stack 
traces that are being ``counted'' as nsScanner...

<nsScanner>
PR_Malloc
nsMemoryImpl::Alloc(unsigned int)
nsMemory::Alloc(unsigned int)
nsScanner::Append(char const *, unsigned int)

<nsScanner>
__builtin_new
nsSlidingString::AppendBuffer(unsigned short *, unsigned short *, unsigned short 
*)
nsScanner::AppendToBuffer(unsigned short *, unsigned short *, unsigned short *)

<nsScanner>
__builtin_new
nsScanner::AppendToBuffer(unsigned short *, unsigned short *, unsigned short *)

<nsScanner>
PR_Malloc
nsMemoryImpl::Alloc(unsigned int)
nsMemory::Alloc(unsigned int)
unsigned short * AllocateStringCopy<unsigned short, unsigned 
short>(basic_nsAReadableString<unsigned short> const &, unsigned short *)
ToNewUnicode(basic_nsAReadableString<unsigned short> const &)
nsScanner::Append(basic_nsAReadableString<unsigned short> const &)

<nsScanner>
PR_Malloc
nsMemoryImpl::Alloc(unsigned int)
nsMemory::Alloc(unsigned int)
nsStr::Alloc(nsStr &, unsigned int)
nsStr::Realloc(nsStr &, unsigned int)
nsStr::EnsureCapacity(nsStr &, unsigned int)
nsStr::GrowCapacity(nsStr &, unsigned int)
nsString::SetCapacity(unsigned int)
nsString::SetLength(unsigned int)
CopyUnicodeTo(nsReadingIterator<unsigned short> const &, 
nsReadingIterator<unsigned short> const &, basic_nsAWritableString<unsigned 
short> &)
nsScanner::CopyUnusedData(nsString &)

<nsScanner>
PR_Malloc
nsMemoryImpl::Alloc(unsigned int)
nsMemory::Alloc(unsigned int)
nsStr::Alloc(nsStr &, unsigned int)
nsStr::Realloc(nsStr &, unsigned int)
nsStr::EnsureCapacity(nsStr &, unsigned int)
nsStr::GrowCapacity(nsStr &, unsigned int)
nsStr::StrAppend(nsStr &, nsStr const &, unsigned int, int)
nsStr::StrAssign(nsStr &, nsStr const &, unsigned int, int)
nsString::nsString(nsString const &)
nsScanner::nsScanner(nsString &, int, nsString const &, nsCharsetSource)

Component: Browser-General → Tracking
QA Contact: doronr → chofmann
The problem sounds very similar to bug 75641. Waterson, which build are you 
using? I'll try this on my machine.
On the 3000 bugs URL, I'm seeing about 3.1MB allocated at the stack:

PR_Malloc
nsMemory::Alloc(unsigned int)
nsScanner::Append(const char*, unsigned int)
nsByteArrayInputStream::ReadSegments(PFjP14nsIInputStreamPvPKcjjPjES3_jS6_)
nsParser::OnDataAvailable(nsIRequest*, nsIRequest*, nsIInputStream,
                          unsigned int, unsigned int)

(I did this in mozilla, so it could be related to the search stuff, but I don't
think so.  The demangling of the parameters to |ReadSegments| is left as an
exercise to the reader :-)

What uses nsByteArrayInputStream?
data was generated based on a debug build from Monday 5/14/01

new trace-malloc results from 5/22 linux debug build:

large test url (8092 bugs)
http://bugzilla.mozilla.org/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=RESOLVED&bug_status=VERIFIED&email1=&emailtype1=substring&email2=&emailtype2=substring&bugidtype=include&bug_id=&changedin=&votes=&chfield=bug_status&chfieldfrom=01%2F01%2F1970&chfieldto=Now&chfieldvalue=reopened&short_desc=&short_desc_type=substring&long_desc=&long_desc_type=substring&bug_file_loc=&bug_file_loc_type=substring&status_whiteboard=&status_whiteboard_type=substring&keywords=&keywords_type=anywords&field0-0-0=noop&type0-0-0=noop&value0-0-0=&namedcmd=allmybugs&newqueryname=&order=bugs.bug_id

Type                    Count    Bytes %Total
TOTAL                  520512 38578000 100.00
FrameArena               3971 16340665  42.36
nsScanner                 295  4768896  12.36
void*                  119911  3161235   8.19
HTMLAttributesImpl      51960  3116472   8.08
nsTextNode              51100  2452800   6.36
nsHTMLTableCellElement  46071  2395692   6.21
nsHTMLSpanElement       34399  1513556   3.92
nsLineBox               48996   979920   2.54
nsVoidArray             18118   631452   1.64
nsTextFragment          51075   557600   1.45
nsHTMLAnchorElement      5975   334600   0.87
nsHTMLTableRowElement    5799   278352   0.72
nsCellMap               51827   230248   0.60
nsTokenAllocator           46   224066   0.58
gtk-unclassified          664   123578   0.32
nsFontMetricsGTK         1032   118215   0.31
xpti-unclassified         101   117912   0.31
nsImageGTK                  6   105116   0.27
nsComponentManagerImpl   3362    77364   0.20
OTHER                   25804  1050261   2.72

small test url (3031 bugs)
http://bugzilla.mozilla.org/buglist.cgi?email1=&emailtype1=substring&emailassigned_to1=1&email2=&emailtype2=substring&emailreporter2=1&bugidtype=include&bug_id=&changedin=&votes=&chfield=%5BBug+creation%5D&chfieldfrom=2001-02-23&chfieldto=2001-03-22&chfieldvalue=&short_desc=&short_desc_type=substring&long_desc=&long_desc_type=substring&bug_file_loc=&bug_file_loc_type=substring&status_whiteboard=&status_whiteboard_type=substring&keywords=&keywords_type=anywords&field0-0-0=noop&type0-0-0=noop&value0-0-0=&cmdtype=doit&namedcmd=asdf&newqueryname=&order=Reuse+same+sort+as+last+time

Type                    Count    Bytes %Total
TOTAL                  147879 10422436 100.00
FrameArena               1027  4226105  40.55
nsScanner                  73  1270018  12.19
HTMLAttributesImpl      13412   803856   7.71
void*                   32961   746625   7.16
nsTextNode              13000   624000   5.99
nsHTMLTableCellElement  11940   620880   5.96
nsHTMLSpanElement        8883   390852   3.75
nsLineBox               12281   245620   2.36
nsVoidArray              4953   173536   1.67
nsTextFragment          12997   131109   1.26
xpti-unclassified          76    92312   0.89
nsHTMLAnchorElement      1571    87976   0.84
nsHTMLTableRowElement    1518    72864   0.70
nsTokenAllocator           14    68194   0.65
nsCellMap               13375    59336   0.57
gtk-unclassified          438    55390   0.53
nsComponentManagerImpl   2322    54708   0.52
unclassified-string      1506    45828   0.44
nsZipArchive              982    42922   0.41
OTHER                   14550   610305   5.86
Let me investigate this bug. 

CCing cathleen.
Status: NEW → ASSIGNED
Target Milestone: Future → mozilla0.9.2
Assignee: cathleen → harishd
Status: ASSIGNED → NEW
Moving to my plate.
Priority: P3 → P1
*** Bug 84820 has been marked as a duplicate of this bug. ***
*** Bug 50411 has been marked as a duplicate of this bug. ***
*** Bug 84944 has been marked as a duplicate of this bug. ***
Probably this is related to bug 87370.

Status: NEW → ASSIGNED
There is always an expansion factor when you read a file and display the
contents, because you need so much more for display. The expansion factor will
naturally depend on the structures in the document and the complexity of the
display (like tables) that is needed. What is a reasonable expansion factor is a
different thing. I did some testing a 2-3 years ago with Mozilla and Multidoc
Pro (an SGML browser I was working on at that time).

Multidoc Pro's expansion factor was about 5. This meant that if I loaded a file
of size 1MB, it took about 5MB of RAM to load and display this document. This is
extremely good expansion factor. The code had been optimized for size and speed
over several years. Multidoc Pro did not support Unicode when the testing was done.

At that time Mozilla's expansion factor was 12-16 (it varied, and I can't
remember exactly). That was obviously too much. I thought an expansion factor of
10 might be attainable at that time.

If 3.2MB file grows into 25MB in memory, we get an expansion factor of about 8!

Somebody might want to try and calculate some kind of theoretical lower limit to
the expansion factor (supposing we load the whole document and lay it out
completely). Because our internal representation is 32-bit Unicode, this means
the expansion factor must be over 2 for non-Unicode documents.

I also checked this file with IE 5.0, the expansion factor was 12. Just to be
sure, I checked what it was for NS6.1b1: 17. I used the first URL in the first
comment to this bug, assuming that was 3.8 megs. So it seems I was measuring
something else than what you are talking about...
6/25 debug trace-malloc build, running small test url (3030 bugs)
http://bugzilla.mozilla.org/buglist.cgi?email1=&emailtype1=substring&emailassigned_to1=1&email2=&emailtype2=substring&emailreporter2=1&bugidtype=include&bug_id=&changedin=&votes=&chfield=%5BBug+creation%5D&chfieldfrom=2001-02-23&chfieldto=2001-03-22&chfieldvalue=&short_desc=&short_desc_type=substring&long_desc=&long_desc_type=substring&bug_file_loc=&bug_file_loc_type=substring&status_whiteboard=&status_whiteboard_type=substring&keywords=&keywords_type=anywords&field0-0-0=noop&type0-0-0=noop&value0-0-0=&cmdtype=doit&namedcmd=asdf&newqueryname=&order=Reuse+same+sort+as+last+time

Type                    Count    Bytes %Total
TOTAL                  254480 18750891 100.00
FrameArena               1756  7225940  38.54
nsScanner                 135  2254934  12.03
void*                   57598  1884127  10.05
HTMLAttributesImpl      23183  1390060   7.41
nsTextNode              22516  1080768   5.76
nsHTMLTableCellElement  20531  1067612   5.69
nsHTMLSpanElement       15371   676324   3.61
nsLineBox               21436   428720   2.29
nsVoidArray              8543   299268   1.60
nsTextFragment          22505   242637   1.29
nsTokenAllocator           43   209453   1.12
gtk-unclassified          777   155744   0.83
nsHTMLAnchorElement      2693   150808   0.80
xpti-unclassified         126   141432   0.75
nsHTMLTableRowElement    2549   122352   0.65
nsComponentManagerImpl   4022   116560   0.62
nsImageGTK                 14   105888   0.56
nsCellMap               23058   102540   0.55
nsZipArchive             1820    79312   0.42
OTHER                   25804  1016412   5.42  
Target Milestone: mozilla0.9.2 → mozilla0.9.3
new build from 6/26, pulled trunk after dougt checked in fix for bug 70398
trace-malloc result from running small url test (3030 bugs)
http://bugzilla.mozilla.org/buglist.cgi?email1=&emailtype1=substring&emailassigned_to1=1&email2=&emailtype2=substring&emailreporter2=1&bugidtype=include&bug_id=&changedin=&votes=&chfield=%5BBug+creation%5D&chfieldfrom=2001-02-23&chfieldto=2001-03-22&chfieldvalue=&short_desc=&short_desc_type=substring&long_desc=&long_desc_type=substring&bug_file_loc=&bug_file_loc_type=substring&status_whiteboard=&status_whiteboard_type=substring&keywords=&keywords_type=anywords&field0-0-0=noop&type0-0-0=noop&value0-0-0=&cmdtype=doit&namedcmd=asdf&newqueryname=&order=Reuse+same+sort+as+last+time

Type                    Count    Bytes %Total
TOTAL                  165738 10404470 100.00
FrameArena               1127  4637605  44.57
void*                   38201  1007977   9.69
HTMLAttributesImpl      14542   872052   8.38
nsTextNode              14115   677520   6.51
nsHTMLTableCellElement  12843   667836   6.42
nsHTMLSpanElement        9680   425920   4.09
nsLineBox               13528   270560   2.60
nsVoidArray              5381   187492   1.80
nsTextFragment          13981   137522   1.32
nsZipArchive             2634   113998   1.10
xpti-unclassified          91   107696   1.04
nsImageGTK                  9   105264   1.01
gtk-unclassified          479    97875   0.94
nsHTMLAnchorElement      1675    93800   0.90
nsHTMLTableRowElement    1570    75360   0.72
nsComponentManagerImpl   2072    64536   0.62
nsCellMap               14463    64224   0.62
X-unclassified           1676    49980   0.48
JS-unclassified          1056    43572   0.42
OTHER                   16615   703681   6.76                   
voila! nsScanner bloat disappeared :-)
Actually, dougt checked in a patch for bug 87370. The patches in bug 70398 & in
bug 87370 are identical.
Sweet Justice!
large url (8551 bugs) trace-malloc results:
http://bugzilla.mozilla.org/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=RESOLVED&bug_status=VERIFIED&email1=&emailtype1=substring&email2=&emailtype2=substring&bugidtype=include&bug_id=&changedin=&votes=&chfield=bug_status&chfieldfrom=01%2F01%2F1970&chfieldto=Now&chfieldvalue=reopened&short_desc=&short_desc_type=substring&long_desc=&long_desc_type=substring&bug_file_loc=&bug_file_loc_type=substring&status_whiteboard=&status_whiteboard_type=substring&keywords=&keywords_type=anywords&field0-0-0=noop&type0-0-0=noop&value0-0-0=&namedcmd=allmybugs&newqueryname=&order=bugs.bug_id


Type                    Count    Bytes %Total
TOTAL                  304043 19191342 100.00
FrameArena               2323  9559145  49.81
HTMLAttributesImpl      30432  1825700   9.51
void*                   70469  1514253   7.89
nsTextNode              29949  1437552   7.49
nsHTMLTableCellElement  26942  1400984   7.30
nsHTMLSpanElement       20207   889108   4.63
nsLineBox               28495   569900   2.97
nsVoidArray             10528   365740   1.91
nsTextFragment          29896   345132   1.80
nsHTMLAnchorElement      3508   196448   1.02
nsHTMLTableRowElement    3371   161808   0.84
nsCellMap               30303   134664   0.70
xpti-unclassified          58    70600   0.37
nsFontMetricsGTK          419    54154   0.28
gtk-unclassified          352    50490   0.26
nsComponentManagerImpl   1870    50224   0.26
nsPersistentProperties   2016    40178   0.21
nsZipArchive              851    37062   0.19
FrameManager               67    35577   0.19
OTHER                   11987   452623   2.36
nice work dougt!
This rocks!
So, what do we do with this bug now :-/ ? Pass it on?
What expansion factor are we aiming for? If we're at 6 now I guess we could aim 
for 3-4. Otherwise i think we could mark this bug as fixed.

As I know very little about the data, the only stuff that interests me:
FrameArena    2323  9559145  49.81 <- this is real data, right?
void*        70469  1514253   7.89 <- can we find out who's using this
nsVoidArray  10528   365740   1.91 <- can we find out who's using this
OTHER        11987   452623   2.36 <- should we spend any time looking at this?
but this probably isn't interesting...
Assignee: harishd → cathleen
Status: ASSIGNED → NEW
Since the scanner issue has been resolved giving bug back to cathleen to find
appropriate owners for the remaining problems ;-)
Doesn't look like this is getting fixed before the freeze tomorrow night.
Pushing out a milestone.  Please correct if I'm mistaken.
Target Milestone: mozilla0.9.3 → mozilla0.9.4
Summary: Does a >3.2MB file (bunch of tables, >7975 rows), really need 25MB of memory? → Does a >3.2MB file (bunch of tables, >7975 rows), really need 19MB of memory?
Target Milestone: mozilla0.9.4 → mozilla0.9.5
marc, can you help figure out the bloat in FrameArena?
Assignee: cathleen → attinasi
Blocks: 71874
Sure, happy to help - as soon as I finish fixing my 5 topembed bugs ;)
Status: NEW → ASSIGNED
FrameArena holds frames, as you might guess. I loaded the query (which results
in more than 9000 hits now) and dumped the frame size stats from Viewer. Here is
the result:

Frame Type           Count TotalSize MinSize MaxSize AvgSize
----------           ----- --------- ------- ------- -------
AreaFrame                5       380      76      76      76
BRFrame                  2        88      44      44      44
BlockFrame           73787   5607812      76      76      76
CanvasFrame              1        72      72      72      72
HRFrame                  2       128      64      64      64
ImageFrame               1         0       0       0       0
InlineFrame          64627   3619112      56      56      56
LineBox:block,big        1        60      60      60      60
LineBox:block,small     59      2360      40      40      40
LineBox:inline,big      11       660      60      60      60
LineBox:inline,smal  74202   2968080      40      40      40
ScrollFrame              4       224      56      56      56
TableCellFrame       73779   6787668      92      92      92
TableCellMap            52      2912      56      56      56
TableColFrame          379     37900     100     100     100
TableColGroupFrame      52      3536      68      68      68
TableFrame              52      7488     144     144     144
TableLayoutStrategy     52      1248      24      24      24
TableOuterFrame         52      3952      76      76      76
TableRowFrame         9227    701252      76      76      76
TableRowGroupFrame      52      3120      60      60      60
TextFrame            81994   5249248      64      68      64
TextInputFrame           1        56      56      56      56
ViewportFrame            1        60      60      60      60
gfxButtonControlFra      3       168      56      56      56
*** Total ***        378420  24998744

378 thousand frames taking up almost 25MB. Wow. 81 thousand text frames alone
take up 5MB, and 73 thousand table cells take up over 6 and a half MB. That is a
seriously large page or pretty complicated HTML (the file I loaded is 3.7 MB in
size, when saved from Mozilla).

I don't know just where to go with this data: could we try to shrink the size of
frames (tableCell frames are pretty big)? Could we try to eliminate some frames
(like, there is a block for every table cell - is that really necessary)? Should
we look at the LineBox 'frames' and see if they can be shrunk or eliminated? To
reduce this much, I think we need to seriously rethink the layout architecture,
and that scares me plenty.

Other ideas?
Cc: vidur and dp 'cos I think they would be interested in this...
*** Bug 99903 has been marked as a duplicate of this bug. ***
I would like to mention that I reported the bug 99903 that has been said
duplicate of that one.
My problem is that, as I said in the report for that bug, the browser then hangs
for 30 seconds after the page is downloaded.
I have a good machine (Pentium 1Ghz, 256 Mo) and not that many applications opened.
I see that if I open the same page with IE 5.x, IE doesn't hang as Mozilla does. 
I forgot to say in my report that even if Mozilla is unresponsive, other
applications are still responsive. So it is not a memory only problem.

Furthermore, in the report for the bug 49539, the expansion factor is said to be
bigger in IE. Why then should Mozilla hang?

Are we sure the two bugs are identical?
end of milestone shuffle...
Target Milestone: mozilla0.9.5 → mozilla0.9.6
Blocks: 92580
Severity: normal → major
Priority: P1 → P2
Target Milestone: mozilla0.9.6 → mozilla1.0
See http://bugzilla.mozilla.org/process_bug.cgi#c37
Keywords: footprint
Target Milestone: mozilla1.0 → mozilla1.1
*** Bug 126861 has been marked as a duplicate of this bug. ***
*** Bug 124441 has been marked as a duplicate of this bug. ***
Build from 2/14 on Win98/400MHz/128MB:
- The list of open bugs I reported (~300) takes 7 seconds.
- The list of bugs I reported (~800) takes 25 seconds.
A 2/20 build seems to be about 1/3 faster.

On the 2/20 build, the UNCO list (~2000) effectively hangs Mozilla for a minute 
while it allocates >100MB.  The strange thing is that although it hangs for 
several periods of ~7 seconds as it loads the page, the one-minute hang appears 
to happen just after the throbber *stops* spinning.

If this is unrelated to the FrameArena stuff, please reopen one of the dups...
This could be partially fixed (although the memory issue still remains) by not
rendering the entire page at once. There is no need to render the entire page
when only a small portion is visible on screen. See bug 122581.

I don't know what you mean by "render", but the whole page has to be parsed and
analyzed to display only a small subset of it. That is beacuse the page can
contain elements that affect the displayed area even if they are written late in
the file. That could be absolute positioned elemenents, elements that requires a
table to grow, elements that contain scripts or event handlers that must be run...

But if there are anything specific you think we do now that don't have to done,
please tell us. We could really need more optimizations for big pages.
Right, but that doesn't mean it has to be all in memory at once. If you are
parsing a page that is extremely huge, you could sort of "page" parts of it in
and out of memory from disk. Doing so would definately be more efficient than
bogging down the system memory and causing the operating system to scrap for
more space. For instance, images that are off screen don't need to be stored in
memory, and even sections of tables, etc don't need to be if they are off screen.
no no no no no! mozilla has reinvented lots of things, but i'm pretty sure 
shaver and brendan would tell us that we should not reinvent paging.  That's 
the os's job, and it should do it well enough for our purposes. if it doesn't, 
then the user should take steps to fix it as having bad paging would hurt all 
apps. any memory not recently used (and even memory recently used) is a 
candidate for the os to page out, let the os do its job.

mind you i'm not saying we shouldn't release memory.
I'm not talking about reinventing paging. I am talking about only loading so
much of the page as so it doesn't overburden memory if it is a huge page. There
is no reason on a page that is 1MB of tables or whatever that the whole page
needs to stored in memory at once. This isn't reinventing paging, this is not
being a memory hog so that the os doesn't have to page. That is a big
difference. When you have a web page that is 10000 lines and only 100 are
visible in the window, why have all 10000 lines in memory? Wouldn't 500 suffice?
Paging is inevitably a slow operation, and avoiding it by proper use of the
program's memory would be more efficient than sucking up gobs of system memory
so that the operating system has to start paging. Timeless, I agree we shouldn't
handle paging, but it almost sounds as if you are saying that we should just
suck up memory in loading a huge page and blame it on the os because its not
fast at paging. What about when the system is low on hard disk space?

Bug 122581 talks about a page with a lot of images. Is it really necessary to
have all the images in memory at once? Is it necessary to have an entire page of
text in memory at once? Couldn't Mozilla tag the things that other parts of the
page might depend on and keep them in memory?

What I'm basically asking is if there is any way to more efficiently
(memory-wise) parse a page. Is there any way to not parse it all at once and not
have it all in memory at one time?
Where do you suppose the other 9500 lines should be stored? We can't just forget
about them, because soon the user will scroll down to see it. I don't see much
of a difference between us writing down the information to disk and letting the
OS do it. The main difference is that us doing it will always kill performance
while letting the OS do will let us run from memory when there is enough memory. 

Image handling is another thing though, storing them in memory
compressed/uncompressed, in main memory/graphic card memory... There are lots of
bugs about that and I suggest you see if you can contribute something there if
you have ideas. 
Daniel: Of course if there is enough _physical_ memory, then by all means the
entire page should be loaded in memory. Otherwise, there should be some cut-off
where part of the page is left un-parsed until the user enters that location of
the page. First of all, paging continuously on a system will kill performance.
When the user gets partway down in the page and a new part has to be loaded will
only be a temporary jolt to the system. You can even start loading the second
part before the user gets to it so that there is no delay. I would much rather
see Mozilla be kinder on memory than try to increase performance on an already
doomed situation of a huge page. Worry about speed on the average page and worry
about memory on these huge pages. Its not worth bogging down a user's system
memory in order to get a little more speed.
When the page exceeds a pre-determined (based on system memory, etc) amount of
memory, the page should be scanned to determine if there is anything (such as
javascript) that should be parsed ahead of time because the part of the page
currently parse depends on it. Everything else should be left until the user
gets to it.
my programming skills are very minimalistic, im more like a hardware guy. But 
HOW can it take so much memory at all ? the unconfirmed buglist is approx 850k 
and takes 400megs to parse/render. sounds like a more basic problem than when 
render what and when have it in memory or not. 
Yes, that do sound excessive. If 850 KB simple HTML code uses 400 MB of RAM to
render, then thats a enormous regression. As can be seen in this bug's summary
the know memory usage for such data is much, much lower (19 MB for ~8000 rows /
3.2 MB HTML)
a funny thing i recognized during the page is rendered the win2k taskmanager 
shows ~80megs, but once its DONE rendering the mem usagesa goes over 400megs. is 
just taskmanager too slow to show the info just in time, or aint this a parsing 
problem after all ?
yes, I'm seeing this after the fact increase in memory consumption too
I doubt Task Manager is missing it unless its frozen. Could this be some kind of
memory leak?
No the task manager is not frozen. And I always see this when loading various of
these image heavy pages.
im still using the unconfirmed bugs list from bugzilla as my personal test. only 
image is the head and it still aquires so much memory(over 400megs) AFTER the 
rendering is complete it seems
Do you have the Search sidebar tab enabled?  (Is that still active if it's not
open?)
hmm, right completly turning off the sidebar takes me down to 110 megs mem
usage. so the side bar swallows over 300megs ???? yikes
I do not have the sidebar showing.
Just to add my comments here.

Using the BugZilla link my memory usage increases steadily till around 110 MB,
where the page is now fully parsed. It the decreases till its back to 24MB.

HOWEVER, Mozilla is now non-responsive. Clicking anywhere in the Mozilla window
kills it.

Will check with sidebar showing.
Correction on my last post.

Got different things this time.

As usual, page starts loading, memory rises. It peaked at 127MB RAM Usage, plus
80%CPU (Pentium 4 1.4GHZ). However, the page was NOT finished loading.

While the page was still loading, my mem usage went all over the place. It ended
up jumping between 19MB (Which is lower than Mozilla normally users), and 30MB,
while the page was still being parsed.

At the end of parsing, Mozilla is again unresponsive.

Dont know what that tells you, but the fact that it decreased hugely while still
rendering, I would imagine tells you something :)
anyone try with a nightly today ?
i just had a very special happening. via the bugzilla page i requested all bugs 
containing "mail", this made mozilla user 600 megs swap and 80 megs ram, with 
sidebar turned off !!!
then i opened one bug in a new window, real ram usage jumped to 94 megs.
scoll a bit up and down in the newly created window  and you can watch mozilla 
grabbing more and more memory, about 120 k for each full scroll.
close mozilla, leave quicklaunch on, mozilla drops down to 64megs ram/530megs 
swap. during freeing memory windows warned twice that the app is not responding. 
cant that be optimized in any way ?
on low spec systems this also results in crash and dataloss because of the
freezing ... and the bug would get more attention ;P
*** Bug 147115 has been marked as a duplicate of this bug. ***
*** Bug 149861 has been marked as a duplicate of this bug. ***
the behavior here starting at comment 45 is bug 120712, caused by checkin for
bug 77460.

this seems to be FIXED (by 87370) unless the FrameArena memory usage still needs
work (comment 37)
retargeting as 1.1a is already out
Target Milestone: mozilla1.1alpha → mozilla1.1beta
marking FIXED by bug 87370.
It seems unlikely that Attinasi is going to work on this anymore

If you are "seeing" this bug, it's bug 120712, which needs sr=
Status: ASSIGNED → RESOLVED
Closed: 22 years ago
Resolution: --- → FIXED
phoenix stats
about:blank 18944k
       url: 39876k

      file:  1444k
that appears to be an expansion factor of 14.

it's now a <1.5mb file that takes 20mb of memory.
sorry that was (phoenix) Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US;
rv:1.1b) Gecko/20020910

i just tested w/
Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.2b) Gecko/20020921 Phoenix/0.1

current phoenix stats
about:blank 16852k
       url: 43468k

      file:  1444k

so it's now a <1.5mb file that takes 26mb of memory.
that appears to be an expansion factor of 18.
note that the query results in 3033 bugs and since i saved it using phoenix,
there was some random (ok, not random, because of multipart/replace) garbage
(the please wait page) at the top of the file:

--thisrandomstring&#9689;Content-Type: text/html&#9689;&#9689;<!-- 1.0@bugzilla.or
g -->&#9689;&#9689;&#9689;&#9689;  &#9689;<html>&#9689;  <head>&#9689;    <title>Bugzilla is pondering you
r query</title>&#9689;  </head>&#9689;  <body>&#9689;    <h1 style="margin-top: 20
%; text-align: center;">Please stand by ...</h1>&#9689;&#9689;&#9689;  </body>&#9689;</h
tml>&#9689;--thisrandomstring&#9689;Content-Type: text/html&#9689;Set-Cookie: LAST
ORDER=bugs.bug_id ; path=/; expires=Sun, 30-Jun-2029 00:00:00 GM
T&#9689;Set-Cookie: BUGLIST= ; path=/; expires=Sun, 30-Jun-2029 00:00:
00 GMT&#9689;&#9689;<!-- 1.0@bugzilla.org -->&#9689;&#9689;&#9689;&#9689;&#9689;&#9689;&#9689;&#9689;&#9689;&#9689;&#9689;&#9689;<!-- 1.0@bugzilla.o
rg -->&#9689;&#9689;&#9689;&#9689;&#9689;&#9689;&#9689;&#9689;<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Trans

steps:
1. run phoenix
2. load data:text/html,<a href=about:blank>a
3. right click a and select open in new window
4. close original window
5. checkpoint memory usage (taskmanager)
6. file>open file
7. checkpoint memory usage (taskmanager)
as for 'seeing' that other bug, phoenix doesn't have search, so i'm not affected
by that.
Product: Core → Core Graveyard
You need to log in before you can comment on or make changes to this bug.