Slow scrolling performance on http://www.chicago-l.org/stations/damen-ohare.html
Categories
(Core :: Graphics: WebRender, defect)
Tracking
()
People
(Reporter: sam, Unassigned)
References
(Blocks 2 open bugs)
Details
(Keywords: perf, Whiteboard: wr-planning)
Attachments
(4 files)
User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:79.0) Gecko/20100101 Firefox/79.0
Steps to reproduce:
- Enable WebRender on a macOS machine
- Visit http://www.chicago-l.org/stations/damen-ohare.html
- Scroll up and down and observe poor performance.
Actual results:
Scrolling was incredibly laggy. Profile: https://share.firefox.dev/30FyuyG
Restarting Firefox resolved the issue, but I am unsure as to why since I only had this one tab open, no settings had been modified during that session, and there was no new build installed.
Expected results:
Page should scroll smoothly if it is the only thing open.
|
||
Comment 1•6 years ago
|
||
Bugbug thinks this bug should belong to this component, but please revert this change in case of error.
|
||
Comment 2•6 years ago
|
||
From the profile, I see lots of long "WR OS Compositor Frames", during which glBufferData takes up all of the time allocating VBOs. I also see some very long "CompositorScreenshot"s.
Sam, If you are able to reproduce again, could you please try taking a profile with the screenshots feature disabled? It can interfere with the profile sometimes. (In the profiler toolbar menu, change the settings to "custom", click "edit settings", then make sure "screenshots" is not ticked. Also ensure the "Renderer" thread remains ticked.)
I see that gfx.webrender.compositor is enabled. Could that be causing this, Markus? FWIW I cannot reproduce on Windows, the page scrolls very smoothly for me.
|
||
Comment 3•6 years ago
|
||
Allocating VBOs takes a long time when there are a lot of VBOs. That would also explain why restarting the browser solved it. Are we leaking VBOs maybe?
|
|
||
Comment 4•6 years ago
|
||
Jeff, can you reproduce this? Do we use a lot of VBOs on this page on Mac?
|
|
||
Comment 5•6 years ago
|
||
I don't think it's about this page, otherwise Sam would have been able to reproduce it right after the restart. But no:
(In reply to Sam Johnson from comment #0)
Restarting Firefox resolved the issue, but I am unsure as to why since I only had this one tab open
So I think this bug is more about "even simple pages become slow after a while of browsing".
|
|
||
Comment 6•6 years ago
•
|
||
That makes sense.
(In reply to Markus Stange [:mstange] from comment #3)
Allocating VBOs takes a long time when there are a lot of VBOs. That would also explain why restarting the browser solved it. Are we leaking VBOs maybe?
We should be allocating a new buffer and orphaning the old one for every upload. So I guess the driver might be doing a very bad job of reusing orphaned buffers, especially since our buffer size will keep changing. I think Jeff might have a way of seeing what buffers are allocated?
If this is the cause, then I guess bug 1640952 or bug 1602550 would help fix it.
|
Reporter | |
Comment 7•6 years ago
|
||
Here's another profile with screenshots disabled. The issue is not yet nearly as pronounced today as it was yesterday, but scrolling this page has noticeably deteriorated so far throughout the day. If this profile isn't as helpful, let me know and I can keep this browser session open until it gets as bad as it was in the previous profile. https://share.firefox.dev/2UGR4Tt
|
||
Comment 8•6 years ago
|
||
That profile is somewhat valuable, but I'd be very interested to see how it compares to a latter browser session where it has gotten worse.
|
|
Reporter | |
Comment 9•6 years ago
|
||
It has worsened, here's another profile: https://share.firefox.dev/30K9tCB
|
|
||
Comment 10•6 years ago
|
||
This is fascinating.
|
|
||
Comment 11•6 years ago
•
|
||
I looked into the orphan node mechanisms in GLEngine a little bit.
When a buffer object is orphaned, an OrphanNode is created for it. Based on the buffer object size, a hash is computed: (((roundedSize & 0xffffffff) >> 0xc) & 0x3ff) ^ size >> 0x16) % 113.
There is one global linked list of all orphan nodes, and 113 linked lists of orphan nodes for each size hash. Every node is present in both the global list and in the list for its size hash.
Orphan nodes are created in gleOrphanBufferObject and freed in gleBufferObjectAdoptOrphan or gleFreeOrphan.
The orphan node lists are shared between GPUs. Each node also contains device-specific user data for each GPU.
gleGetFreeOrphanNode finds a node for the requested size and two other params: the usage hint and the writeCombined.
It only iterates the size-hash based list, not the global list.
If a node was found, it is taken out of both lists and returned. Otherwise gleGetFreeOrphanNode returns null.
If gleGetFreeOrphanNode is slow, this probably means that there are a lot of orphan nodes of the same size hash. So if we can't find a matching node, that's either because we're looking for a different size that computed to the same hash, or because one of the other two requested parameters (usage hint or writeCombined) didn't match. We recently landed a patch in bug 1642495 comment 10 that makes the usage hint match more often.
There doesn't seem to be a global limit on the number of orphan nodes. Instead, there is a limit of total bytes of orphaned buffer objects, at around 67MB. This limit is enforced by gleCleanupOrphans, which is called on every call to gleOrphanBufferObject. So if we have lots of orphan nodes sticking around, this may mean that those orphan nodes are wrapping very small buffer objects.
|
||
Updated•6 years ago
|
|
|
||
Comment 12•6 years ago
|
||
|
|
||
Comment 13•6 years ago
|
||
Here's the output I get from the debugging patch just scrolling up and down the page: https://gist.github.com/mstange/26559e2808e96464fc0b46618f1682d4
|
|
||
Comment 14•6 years ago
|
||
And here's a snapshot of all nodes at one given time: https://gist.github.com/mstange/47a1124a6cc9975b75bea39aae8ad4ba
|
|
||
Comment 16•6 years ago
|
||
We can clean up the nodes by calling CGLSetParameter with kCGLCPReclaimResources.
|
|
||
Comment 17•6 years ago
|
||
kCGLCPReclaimResources will clean up a bunch of other things too. We can you use the private value of 0x142 instead of kCGLCPReclaimResources if we just want to clean up the orphans.
|
|
||
Comment 18•6 years ago
|
||
In my testing, the node count usually tops out at around 3000 to 3500 nodes. The memory limit for orphaned buffers on my machine is around 67MB. This implies an average buffer size of 19KB.
I wasn't able to consistently get into a state where gleGetFreeOrphanNode was slow. Even when it did show up in the profile, individual calls to glBufferData still only took 50 microseconds (0.05ms) at most; the vast majority of them took less than 2 microseconds.
I was trying to find out whether we had a bug that caused us to create too many orphan nodes. I don't believe that to be the case. Orphaned buffers are no longer under our control and it is the driver's responsibility to manage their count.
On the linked page, I've been able to get the number of draw calls up to 386 by scrolling up and down aggressively. Lots of draw calls mean lots of calls to glBufferData.
I guess the outcome of this investigation is what you would have expected from the start: To avoid long times in gleGetFreeOrphanNode, create a small amount of large buffers rather than a large amount of small buffers, and always try to reduce the number of draw calls.
|
|
||
Comment 19•6 years ago
|
||
Depends on D79864
|
|
||
Updated•6 years ago
|
|
|
Reporter | |
Comment 20•5 years ago
|
||
I've found another page that scrolls incredibly slowly in WebRender, and it looks like the slowness is happening in gleGetFreeOrphanNode like the other one. This profile is from a different system, but it occurs on the system from the above profiles as well. If this should be a new bug, let me know and I can move it!
Page: https://hackernoon.com/leaky-by-design-7b423142ece0
Profile: https://share.firefox.dev/3iIokDu
|
|
Reporter | |
Comment 21•5 years ago
|
||
|
|
||
Comment 22•5 years ago
|
||
Another bug might help keep things a bit more organized, thanks! It does indeed have the same profile, and is also caused by too many draw calls. But the reason for there being too many draw calls may be different.
Is the performance on the original website still poor? I believe bug 1645665 should have helped.
|
|
Reporter | |
Comment 23•5 years ago
|
||
No problem, I'll split that one off!
The original website still scrolls poorly, here is another profile from today's Nightly: https://share.firefox.dev/3iPAZoc
|
|
Reporter | |
Comment 24•4 years ago
|
||
I haven't seen this issue for quite some time. Is it plausible that changes to WebRender over the past ~1.5 years have solved this? If so, I'm OK with closing.
|
||
Updated•2 years ago
|
Description
•