Open Bug 1400534 Opened 7 years ago Updated 2 years ago

Scrolling fps drops when a tab with big amount of thumbnails is loading in the background with uMatrix installed

Categories

(WebExtensions :: Request Handling, defect, P5)

57 Branch
defect

Tracking

(Not tracked)

People

(Reporter: ajfhajf, Unassigned)

References

(Depends on 1 open bug, Blocks 1 open bug)

Details

(Keywords: perf)

User Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0
Build ID: 20170915220136

Steps to reproduce:

Scrolling fps drops when a tab with big amount of thumbnails is loading in the background with uMatrix installed. In other words, scrolling animation is not smooth. Latest performance profile that I recorded: https://perfht.ml/2h8HjIX (I've already posted it in bug 1396856).


Create new profile, launch it, go to https://addons.mozilla.org/en-US/firefox/addon/umatrix/versions/beta and install  1.0.1rc1
Click on uMatrix icon, click on the black bar on top of uMatrix dashboard
Click on "Hosts files" tab,  click on "Update now"
Once all hosts are updated, close uMatrix settings

Go to https://perf-html.io/
Click with left mouse button in the places marked on the screenshots (1st image is step 1, image below it is step 2, etc): https://imgur.com/a/du2X5
This will block all types of content except for .css and images on all websites. The only exception is https://perf-html.io/, uMatrix is basically disabled there in order to be able to upload performance profiles.

Enable bookmarks toolbar, go to https://2ch.hk/hw/res/2173210.html and drag this tab to the bookmarks toolbar in order to make a bookmark
Go to the link from this pastebin https://pastebin.com/TbQRs04q and drag this tab to the bookmarks toolbar in order to make a bookmark. Site is NSFW but, unfortunately, I couldn't find an alternative. 
Close Nightly, launch Nightly
Go to Tools - Options - "Privacy and Security" and clear cache

Make sure there is one tab open, go to https://2ch.hk/hw/res/2173210.html 
Start recording the profile,  hold "Arrow down" key on keyboard, left-click on bookmark from pastebin, hold left mouse button and drag the bookmark to the tab bar, release left mouse button

It makes sense to hold "Arrow down" key until the pastebin tab finishes loading. If you reach the bottom of the page, hold "Arrow Up" key. If the amount of fps drops is  not enough, try to repeat the STR. Experience shows you don't need to restart the browser, just close pastebin tab, clear cache and open pastebin tab again. Also, the amount of fps drops is different in different runs so it makes sense to repeat STR a couple of times.



Actual results:

Constant scrolling fps drops, scrolling animation is not smooth even though CPU utilization is ~30%, you can see that in the profile from the link above: https://perfht.ml/2h8HjIX


Expected results:

Scrolling frame rate stays at 60 fps all the time
Please report the issue to the extension author. Thank you!
Status: UNCONFIRMED → RESOLVED
Closed: 7 years ago
Resolution: --- → INVALID
Kris Maglione told me to file this bug in this comment: https://bugzilla.mozilla.org/show_bug.cgi?id=1396856#c29
Status: RESOLVED → UNCONFIRMED
Resolution: INVALID → ---
Component: Untriaged → WebExtensions: General
Product: Firefox → Toolkit
Which platform are you on when testing this? Also the perf.html URL does not load for me.
Component: WebExtensions: General → WebExtensions: Compatibility
Priority: -- → P5
I am on Windows 8.1 x64 with latest updates, Intel i5 2300 integrated graphics. I've just loaded this url without problems: https://perfht.ml/2h8HjIX
Maybe try this link, that's what I get after I go to https://perfht.ml/2h8HjIX and click on "Permalink": https://perf-html.io/public/986400e8d9d74bca3ec45fc3edcc849d7beb8b7f/calltree/?hiddenThreads=&thread=2&threadOrder=0-2-3-4-5-6-1-7
Thanks, and that perf.html link works for me.
I've recorded one more perf profile in new Nightly profile with one change: latest uBO beta installed in addition to uMatrix. Same STR from the description. Pay attention to the first ~5 seconds: as soon as background tab starts loading, scrolling fps in the active tab drops heavily. You can see the same thing in my other profiles, actually. Maybe that's the reason why I constantly come across jank in my regular Nightly profile with browser.sessionstore.restore_on_demand false.
https://perfht.ml/2wEjG0A
One more performance profile with uBlock origin and uMatrix enabled: https://perfht.ml/2xgSibb
Lots of fps drops at the beginning and near the end of the profile when CPU utilization is very low (~20%).
Keywords: perf
Moving this back to WebExtensions: General, since I think this bug is about a general performance problem that any extension that uses webRequest may run into.  The previous iteration of this was bug 1396856.
Blocks: webext-perf
Status: UNCONFIRMED → NEW
Component: WebExtensions: Compatibility → WebExtensions: General
Ever confirmed: true
See Also: → 1396856
Thanks for the detailed steps to reproduce.

Unfortunately, it looks like a large part of the problem is that the site triggers so many requests (and the extension adds so many listeners) that they're still able to saturate the event loop even after major performance improvements. The requests just happen faster. In fact, when I noticed and fixed one major bottleneck in the extension process, the jank actually got worse in the main process.

So at some point, I think we're going to have to do something clever with task scheduling so that we can avoid blocking painting or user interaction as much as possible without adding too much latency to requests.
Component: WebExtensions: General → WebExtensions: Request Handling
Depends on: 1402944
So, even with the webRequest handling about as optimized as we're likely to get it in the near future, the test site still generates enough concurrent requests to completely saturate the event queue. It loads somewhere between 2,000 and 6,000 images at once, and the webRequest traffic is made much worse by the fact that each image request redirects twice (once to upgrade to https, and again to redirect to a CDN). (I don't remember whether there were 2,000 requests that redirected twice to become 6,000, or 6,000 requests before redirects... sorry; and after my umpteenth reload test, the CDN throttled me to almost nothing, so I don't really want to try again quite so soon)

At the necko layer, we limit the number of concurrent active requests to (currently) 900. But at the DOM/imgLoader, we don't appear to have any limits to the number of pending loads we attempt to initiate at once, and all of that initialization traffic gets processed by the main thread of the main process (with or without the webExtension framework's involvement). Then the actual opening of the sockets is throttled after the initial stages have been processed.

So I think we need to do something higher up the stack to try to limit the impact of this problem. For this particular case, we could try to limit the number of concurrent image loads we process for a single page.

But there might be other scenarios where we run into this problem that don't involve the image loader. So it may make more sense to throttle AsyncOpen handling at the necko layer, so we don't pump all of that traffic through the main process UI thread in performance-killing chunks.

I'm not sure which is the better approach.

Ehsan, do you have any thoughts on this?
Flags: needinfo?(ehsan)
Depends on: 1404650
Depends on: 1404652
Hmm, I can see scenarios where there is so much Necko IPC load triggered by content that the UI thread just can't keep up.  I think throttling things at the level of img loader is a mistake since that will give us a partial view over the current load (as there may be other connections from other resources or from other pages at the same time.)  Throttling in the parent process based on the global knowledge we have on all of the AsyncOpen requests being services seems much better.  Perhaps we can even take the existence of http-on-modify-request/http-on-before-connect handlers into account for throttling decisions and throttle more aggressively when those exist (an indication of extensions existing)?

Also CCing Olli as he may have thoughts on this as well.
Flags: needinfo?(ehsan)
See Also: → 1404950
The more I think about it, the more I agree that imgLoader isn't the place to do this, mostly because it would only really address the case where most of the traffic comes from a single page.

I'll file a necko bug to try to address it there. We may even be able to use the Quantum DOM/Quantum Necko priority queues to prioritize requests for active sites. Or theoretically even to just queue all events and process them at a lower priority.
(I think I'm a bit lost with the issue here. Comment 10 seems to hint we do too much main thread stuff even if Necko was throttling, but then comments 11 and 12 contradict that.)
(In reply to Olli Pettay [:smaug] from comment #13)
> (I think I'm a bit lost with the issue here. Comment 10 seems to hint we do
> too much main thread stuff even if Necko was throttling, but then comments
> 11 and 12 contradict that.)

Necko currently limits the number of active connections, but not the number of open channels. If a content process opens thousands of channels at once, we still pump all of the notifications for the channel creations through the main process IO thread more or less immediately. If we have cached redirects, or do HSTS upgrades, we also pump the events for the redirect through the main thread more or less immediately. Throttling only happens when we get to the point of actually opening the connection.

So what we're talking about is adding another level of throttling, where we don't process more requests through the main thread than it can handle without causing jank.
I was watching Pixel 2 XL live stream in my regular profile and realized scrolling fps was very low so I created a new profile, installed uBlock and uMatrix and recorded these two profiles. They are very short, around 5 seconds, CPU usage was very low when I was recording them:
https://perfht.ml/2xSsLI6 
https://perfht.ml/2xSWn84
I don't know if they are related to this bug or not but I will post them here anyway.
Oops, profiles turned out to be a bit longer - 7.5s and 10.0s but scrolling fps starts to drop from the very beginning so it is enough to analyze the first 3-4 seconds of the profile.
I don't think this is the same issue, I filed bug 1405801 about the profiles in comment 15.
Product: Toolkit → WebExtensions
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.