Closed Bug 21637 Opened 25 years ago Closed 25 years ago

Large/long pages won't load

Categories

(Core :: Layout, defect, P3)

Product:
Core
Component:
Layout
Type:
defect

Tracking

()

Status:
VERIFIED DUPLICATE of bug 29641
Milestone:
Future

People

(Reporter: colin, Assigned: nisheeth_mozilla)

References

Details

(Keywords: arch, Whiteboard: [nsbeta2-] [nsbeta3-])

I have a test file that is 100,000 lines of This is line 1 of 100000<br> This is line 2 of 100000<br> etc Its loads fine using Nav 4.6 on Windows (takes about 40 seconds). On Mozilla Linux (M12), it loads in about 200 seconds, but when you scroll to the bottom of the window there's only about 30,000 lines present. On Mozilla OpenVMS (M12), it loads in about 140 seconds, but when I try to scroll to the bottom of the windows it stack dumps on me (its dying in GTK). Haven't tried it on Windows. I can send you the HTML file if you wish, but you can probably make your own just as quick.
Assignee: leger → rickg
Component: Browser-General → Layout
QA Contact: leger → gerardok
Moving over to layout...
Assignee: rickg → nisheeth
Nisheeth -- can you please take a look.
Setting milestone to M13 and accepting bug...
Status: NEW → ASSIGNED
Target Milestone: M13 → M14
Moving the last two M13 bugs to M14 because they aren't M13 show stoppers.
I tried this on windows 98 and it hung up the computer. I had to reboot. Does this qualify as a beta blocker? Marking it beta1 so that the PDT team can decide... CCing troy and vidur...
Keywords: beta1
I tried this on NT and the mem usage went up to around 123 MB and the cpu usage was 100%. The browser took close to 200 seconds to display the page. During those 200 seconds, the UI was frozen. Navigator 4.7 loads up the same page from disk in around 10 seconds. We definitely need to become more incremental, responsive and fast for large pages like these.
Attaching the test case...
The lines in the example vary from about 24 characters per line, to roughly 30 characters per line. With 100,000 characters, the file size is roughly 3 megs. Due to either overhead, or fragmentation, or both, Nisheeth comments that we use 123MB. IF we are generous, and assume a footprint of 23 before the load, then roughly 100MB are taken by this 3 meg file. This means roughly a factor of 30 overhead. (or maybe fragmentation cost has pushed us a bit higher). First off: on Win 95, with restricted virtual memory, we'd surely hit a wall in a Bad Way. At best we could hope for a graceful termination. Worst case, the OS may come to its knees (permanently) before we get an out of memory condition :-( The fact that scrolling induces a stack dump would suggest (to me) that the overhead rises as display lines are (probably lazily) constructed (and hopefully destroyed). This is sad.. but believable. Can this be fixed by beta1/M14 freeze (2 weeks)? I'd say no way, don't even dream about it. I believe that a lot of the modularity and flexibility that we bought/installed came at the (current) expense of objects with overhead for small units of data. We might help this a little (factor of 2 :-/ ), but then a 200,000 line file will kill us. We need a spec for a "reasonable" size file. We should try not to be subject to attacks courtesy of a "large" file. We could even say (at the front end, while loading html files), that "some size" is too darn large. That we could do for beta... but it would probably be worse for the product than helpful. Maybe we can track memory used for a single page of display... or even total memory that we're willing to allocate for the app... and be restrictive there. Of the many bugs I've seen for beta that will impact many customers... I'm not convinced that this should be a "Beta stopper" (re: beta1 candidate). PDT will make the final decision... but I personally wouldn't hold up a beta ship for this bug.
Putting on PDT- radar for beta1. Won't hold beta (would release note) for beta.
Whiteboard: [PDT-]
Moving to M15 because this has been marked PDT-
Target Milestone: M14 → M15
Looks like a dup of 25509. I think this is a really important bug to fix. My time spent with Quantify in 25509 suggests that the layout algorithm is O(n^2) for the number of lines. Consequently, if this bug is affecting large files (rendering them impossible to lay out), then this bug is actually affecting _all_ layout performance. Could explain why we look so bad relative to 4.x performance. I suggesting making this the highest priority M14 bug, PDT+. Performance is presumably the second highest priority next to the ship date.
Warren, please don't make blanket statements like "the layout algorithm is O(n^2) for the number of lines". It's a meaningless and misleading statement. Maybe Kipp has a specific algorithm in this case that is not optimal, but that certainly doesn't generalize to all of layout... It's really frustrating to always have everyone assume that every peformamnce issue is because of layout...
Two things worth noting: 1. this test case is not representative of normal Web pages, because all of the content is directly a child of the BODY tag. Typically there are child elements like P tags or DIV tags that contain a majority of the content 2. there is a known issue of how the block code computes text runs and numbers counters in a very non-incremental way. This is the code from nsBlockFrame::Reflow() if (eReflowReason_Resize != aReflowState.reason) { RenumberLists(aPresContext); ComputeTextRuns(aPresContext); } That's especially a problem in cases like this where all of the running matter is a child of the BODY
Troy: Let me revise my statement. Layout is O(n^2) for laying out text files, and probably for any documents that call nsTextNode::AppendData.
Stop saying "Layout is O(n^2)". It may be true that "the block frame code has a O(n^2) algorithm in its reflow code when laying out text", but it certainly is not true that all of the layout algorithms in all of the frame classes are O(n^2). And it certainly isn't true that the basic reflow process is O(n^2)...
nsTextNode::AppendData is a potential problem because of the way it's used by the content sink - the content sink appends to a text node for every netlib chunk of data within that node. Kipp and I spent time discussing this and my impression was that he cut down the redundancy in subequent layout. Specifically, we are no longer laying out the entire text run for each append. We still do a realloc() (and hence a copy) in nsTextNode::AppendData() that is O(n*m) where n is the number of bytes in the node and m is the number of netlib chunks the data spans. I suppose we could restrict the size of text nodes in the content sink. This would not be correct accoding to the DOM spec, but I could live with it for now.
N bytes times m network chunks would certainly explain it. My cpu pegs seemingly indefinitely for the tinderbox logs. Can we agree that this needs to be fixed for M14?
Let me take that back. Because of the <BR>s in the text case, there should be no long text nodes and, hence, no calls to nsTextNode::AppendData. Did you really see that method on in your Quantify data?
I managed to get Quantify to give me the data one time. I think the data for this run is growing too large and overflowing what my machine can handle. I did see nsTextNode::AppendData as significant though after factoring out all the wait and i/o time. You can see what happens to your cpu by bringing up the task manager while loading a tinderbox log. Just break anytime after the cpu pegs. You'll usually have that call in the stack.
I don't know if someone fixed this, but in my M14 code I can now load really big pages. Its not fast, but I don't expect it to be.
Nisheeth improved things in nsTextNode::AppendData a while back, but they're not totally fixed yet.
Actually, it was Vidur who did that! But, hey, I'll take the credit for it... :-)
Summary: Large pages won't load → Large/long pages won't load
*** Bug 30650 has been marked as a duplicate of this bug. ***
The performance on large pages is much better now. Is there consensus among us to mark this bug fixed?
I've gone to this page with the latest build (on linux) and it never finishes. I have to kill it.. It does update the title, but never repaints anthing. It's a 2mb file. netscape 4.72 takes about 16 seconds to render it with my 128mb pentium 233mhz http://www.informix.com/answers/english/ierrors.htm
Moving bugs out by one milestone...
Target Milestone: M15 → M16
Scheduling for M17...
Target Milestone: M16 → M17
*** Bug 37409 has been marked as a duplicate of this bug. ***
*** Bug 36285 has been marked as a duplicate of this bug. ***
*** Bug 37824 has been marked as a duplicate of this bug. ***
can't load this one at all (1.9 mb - only text) http://bugs.gnome.org/db/pa/lgnome-core.html It stops rendering the page pretty quick, and after having downloaded 95% of it it freezes. Tried several times, deleted cache and all - no go. Up to 100% CPU usage and full hang. Had to kill it in the end - nothing more was happening. M16 Linux 2000-052809
Same build: another one..no freeze, just never loads, while spinning "forever" http://www.rpmfind.net/linux/RPM/GByName.html BTW: seems bug 29461 with dups is about same thing?
Cc'ing Waterson.
Keywords: arch, nsbeta2
[nsbeta2-], nominating for nsbeta3, when we are scheduled to tackle performance problems.
Keywords: nsbeta3
Whiteboard: [PDT-] → [nsbeta2-]
typo above: it seems bug 29641 is a dup (not 29461)
*** Bug 43648 has been marked as a duplicate of this bug. ***
Sorry: Bug 43648 is not a dup - it's a crasher. Reopened it.
Seems much better now, thanks to Waterson's changes. nsbeta3-, ->future *** This bug has been marked as a duplicate of 29641 ***
Status: ASSIGNED → RESOLVED
Closed: 25 years ago
Resolution: --- → DUPLICATE
Whiteboard: [nsbeta2-] → [nsbeta2-] [nsbeta3-]
Target Milestone: M17 → Future
Verified duplicate.
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.