Closed Bug 21637 Opened 25 years ago Closed 24 years ago

Large/long pages won't load

Categories

(Core :: Layout, defect, P3)

Product:
Core
Component:
Layout
Type:
defect

Tracking

()

Status:
VERIFIED DUPLICATE of bug 29641
Milestone:
Future

People

(Reporter: colin, Assigned: nisheeth_mozilla)

References

Details

(Keywords: arch, Whiteboard: [nsbeta2-] [nsbeta3-]) 

I have a test file that is 100,000 lines of

This is line 1 of 100000<br>
This is line 2 of 100000<br>
etc

Its loads fine using Nav 4.6 on Windows (takes about 40 seconds).

On Mozilla Linux (M12), it loads in about 200 seconds, but when you scroll to
the bottom of the window there's only about 30,000 lines present.

On Mozilla OpenVMS (M12), it loads in about 140 seconds, but when I try to
scroll to the bottom of the windows it stack dumps on me (its dying in GTK).

Haven't tried it on Windows.

I can send you the HTML file if you wish, but you can probably make your own
just as quick.
Assignee: leger → rickg
Component: Browser-General → Layout
QA Contact: leger → gerardok
Moving over to layout...
Assignee: rickg → nisheeth
Nisheeth -- can you please take a look.
Setting milestone to M13 and accepting bug...
Status: NEW → ASSIGNED
Target Milestone: M13 → M14
Moving the last two M13 bugs to M14 because they aren't M13 show stoppers.
I tried this on windows 98 and it hung up the computer.  I had to reboot.  Does 
this qualify as a beta blocker?  Marking it beta1 so that the PDT team can 
decide...

CCing troy and vidur...
Keywords: beta1
I tried this on NT and the mem usage went up to around 123 MB and the cpu usage 
was 100%.  The browser took close to 200 seconds to display the page.  During 
those 200 seconds, the UI was frozen.  Navigator 4.7 loads up the same page from 
disk in around 10 seconds.  We definitely need to become more incremental, 
responsive and fast for large pages like these.
Attaching the test case...
The lines in the example vary from about 24 characters per line, to roughly 30
characters per line. With 100,000 characters, the file size is roughly 3 megs.

Due to either overhead, or fragmentation, or both, Nisheeth comments that we use
123MB.  IF we are generous, and assume a footprint of 23 before the load, then
roughly 100MB are taken by this 3 meg file. This means roughly a factor of 30
overhead.  (or maybe fragmentation cost has pushed us a bit higher).

First off: on Win 95, with restricted virtual memory, we'd surely hit a wall in
a Bad Way.  At best we could hope for a graceful termination.  Worst case, the
OS may come to its knees (permanently) before we get an out of memory condition :-(

The fact that scrolling induces a stack dump would suggest (to me) that the
overhead rises as display lines are (probably lazily) constructed (and hopefully
destroyed).  This is sad.. but believable.

Can this be fixed by beta1/M14 freeze (2 weeks)?  I'd say no way, don't even
dream about it.  I believe that a lot of the modularity and flexibility that we
bought/installed came at the (current) expense of objects with overhead for
small units of data.  We might help this a little (factor of 2 :-/ ), but then a
200,000 line file will kill us.

We need a spec for a "reasonable" size file.  We should try not to be subject to
attacks courtesy of a "large" file.  We could even say (at the front end, while
loading html files), that "some size" is too darn large.  That we could do for
beta... but it would probably be worse for the product than helpful.  Maybe we
can track memory used for a single page of display... or even total memory that
we're willing to allocate for the app... and be restrictive there.

Of the many bugs I've seen for beta that will impact many customers... I'm not
convinced that this should be a "Beta stopper" (re: beta1 candidate).  PDT will
make the final decision... but I personally wouldn't hold up a beta ship for
this bug.

Putting on PDT- radar for beta1.  Won't hold beta (would release note) for beta.
Whiteboard: [PDT-]
Moving to M15 because this has been marked PDT-
Target Milestone: M14 → M15
Looks like a dup of 25509.

I think this is a really important bug to fix. My time spent with Quantify in 
25509 suggests that the layout algorithm is O(n^2) for the number of lines. 
Consequently, if this bug is affecting large files (rendering them impossible to 
lay out), then this bug is actually affecting _all_ layout performance. Could 
explain why we look so bad relative to 4.x performance. 

I suggesting making this the highest priority M14 bug, PDT+. Performance is 
presumably the second highest priority next to the ship date.
Warren, please don't make blanket statements like "the layout algorithm is 
O(n^2) for the number of lines". It's a meaningless and misleading statement. 
Maybe Kipp has a specific algorithm in this case that is not optimal, but that 
certainly doesn't generalize to all of layout...

It's really frustrating to always have everyone assume that every peformamnce 
issue is because of layout...
Two things worth noting:
1. this test case is not representative of normal Web pages, because all of the 
content is directly a child of the BODY tag. Typically there are child elements 
like P tags or DIV tags that contain a majority of the content

2. there is a known issue of how the block code computes text runs and numbers 
counters in a very non-incremental way. This is the code from 
nsBlockFrame::Reflow()

  if (eReflowReason_Resize != aReflowState.reason) {
    RenumberLists(aPresContext);
    ComputeTextRuns(aPresContext);
  }

That's especially a problem in cases like this where all of the running matter 
is a child of the BODY
Troy: Let me revise my statement. Layout is O(n^2) for laying out text files, 
and probably for any documents that call nsTextNode::AppendData.
Stop saying "Layout is O(n^2)". It may be true that "the block frame code has a 
O(n^2) algorithm in its reflow code when laying out text", but it certainly is 
not true that all of the layout algorithms in all of the frame classes are 
O(n^2). And it certainly isn't true that the basic reflow process is O(n^2)...
nsTextNode::AppendData is a potential problem because of the way it's used by 
the content sink - the content sink appends to a text node for every netlib 
chunk of data within that node. Kipp and I spent time discussing this and my 
impression was that he cut down the redundancy in subequent layout. 
Specifically, we are no longer laying out the entire text run for each append.  
We still do a realloc() (and hence a copy) in nsTextNode::AppendData() that is 
O(n*m) where n is the number of bytes in the node and m is the number of netlib 
chunks the data spans.

I suppose we could restrict the size of text nodes in the content sink. This 
would not be correct accoding to the DOM spec, but I could live with it for now.
N bytes times m network chunks would certainly explain it. My cpu pegs seemingly 
indefinitely for the tinderbox logs.

Can we agree that this needs to be fixed for M14?
Let me take that back. Because of the <BR>s in the text case, there should be no 
long text nodes and, hence, no calls to nsTextNode::AppendData. Did you really 
see that method on in your Quantify data?
I managed to get Quantify to give me the data one time. I think the data for 
this run is growing too large and overflowing what my machine can handle. I did 
see nsTextNode::AppendData as significant though after factoring out all the 
wait and i/o time.

You can see what happens to your cpu by bringing up the task manager while 
loading a tinderbox log. Just break anytime after the cpu pegs. You'll usually 
have that call in the stack.
I don't know if someone fixed this, but in my M14 code I can now load really big 
pages. Its not fast, but I don't expect it to be.
Nisheeth improved things in nsTextNode::AppendData a while back, but they're 
not totally fixed yet.
Actually, it was Vidur who did that!  But, hey, I'll take the credit for it... 
:-)
Summary: Large pages won't load → Large/long pages won't load
*** Bug 30650 has been marked as a duplicate of this bug. ***
The performance on large pages is much better now.  Is there consensus among us 
to mark this bug fixed?
I've gone to this page with the latest build (on linux) and it never
finishes.  I have to kill it.. It does update the title, but never repaints
anthing.  It's a 2mb file. netscape 4.72 takes about 16 seconds to render it
with my 128mb pentium 233mhz

 http://www.informix.com/answers/english/ierrors.htm
Moving bugs out by one milestone...
Target Milestone: M15 → M16
Scheduling for M17...
Target Milestone: M16 → M17
*** Bug 37409 has been marked as a duplicate of this bug. ***
*** Bug 36285 has been marked as a duplicate of this bug. ***
*** Bug 37824 has been marked as a duplicate of this bug. ***
can't load this one at all (1.9 mb - only text)
http://bugs.gnome.org/db/pa/lgnome-core.html
It stops rendering the page pretty quick, and after having downloaded 95% of it
it freezes. Tried several times, deleted cache and all - no go. Up to 100% CPU
usage and full hang. Had to kill it in the end - nothing more was happening.
M16 Linux 2000-052809
Same build: another one..no freeze, just never loads, while spinning "forever"

http://www.rpmfind.net/linux/RPM/GByName.html

BTW: seems bug 29461 with dups is about same thing?


Cc'ing Waterson.
Keywords: arch, nsbeta2
[nsbeta2-], nominating for nsbeta3, when we are scheduled to tackle performance 
problems.
Keywords: nsbeta3
Whiteboard: [PDT-] → [nsbeta2-]
typo above: it seems bug 29641 is a dup (not 29461)

*** Bug 43648 has been marked as a duplicate of this bug. ***
Sorry: Bug 43648 is not a dup - it's a crasher. Reopened it.
Seems much better now, thanks to Waterson's changes. nsbeta3-, ->future

*** This bug has been marked as a duplicate of 29641 ***
Status: ASSIGNED → RESOLVED
Closed: 24 years ago
Resolution: --- → DUPLICATE
Whiteboard: [nsbeta2-] → [nsbeta2-] [nsbeta3-]
Target Milestone: M17 → Future
Verified duplicate.
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.