1100840 - Tweetdeck will hog resources until Firefox become unusable

Status: NEW

(Core :: General, defect)

Description

[:gerard-majax]

Reported by users. After keeping Tweetdeck loaded for a certain amount of time (may vary, ranging from a couple of hours to days), Firefox will eventually gets killed.

This is getting reproduced consistently on 33 release, running on several different systems (FreeBSD, NetBSD and Debian).

Comment 1

[Sylvestre Ledru [:Sylvestre]]

It would be nice to have a crash report.
I am using Tweetdeck but I didn't experience this issue.

Comment 2

[:gerard-majax]

Ok, after investigations with new profiles, the issue was getting reproduced consistently.
Yet, with release 34.0.5, it looks like it's getting better for now. I'll wait more feedback to confirm.

Comment 3

[iMil]

A bit more information on that one. Running 34.0.5 on FreeBSD 10.1/amd64 (same behavior on Debian Jessie/i386 with 34.0):

FF started with a separate profile for https://tweetdeck.twitter.com, only one tab, that one.
Started at 6PM yesterday night, around 1% CPU used
This morning, 15 hours later, around 5% CPU used
Ever since, the CPU used has grown exponentially
As we speak, FF/tweetdeck is eating 60% CPU

Unfortunately I have not saved memory measurements, but I've watched it and its memory consumption seems right (around 400MB since yesterday).

I'm witnessing this behavior since about a year, it is not related to 34.0.

Comment 4

[iMil]

Weirdest thing: I've saved GC & CC logs, CPU consumption has dropped to 4%...

Comment 5

[iMil]

And weirder, I just closed the about:memory page and CPU usage rose up to 50% and stays like this.

Comment 6

[iMil]

Ok, this is not related to the about:memory page, but only to tweetdeck receiving or losing the focus.

Comment 7

[:gerard-majax]

iMil, as we discussed on IRC, I think the best at this point is to make use of the Profiler in Devtools on the Tweetdeck page, maybe it will give us a nice clue on what's going on.

Comment 8

[User Dderss]

Attachment: Безымянный.png

Window's header indicates that browser in e10s mode became 100% to the point where GUI became totally unresponsive; this lasted for about 30 seconds.

Comment 9

[User Dderss]

Attachment: Безымянный1.png

"About:preformance" indicates right after this freeze that Twitdeck page was the culprit -- notice the bright red colour of the tag.

Since Twitter has stopped supporting TwitDeck as separate application, it is now has to be used directly in FireFox itself, and in Nightly with e10s it sometimes causes 100% busy, 100% GUI freeze situations even while working in background.

Comment 10

[User Dderss]

This should be an oxymoron for e10s mode -- no matter what a tab does in the background, it's tag should never go bright red in "about:performance", it should never freeze the GUI.

Tracy, sorry for bothering, but could you please advice on how to deal with that? How to force absolute priority of GUI over Javascript, garbage collection, or whatever else is causing those unpredictable freezes by TweetDeck (or any other) page? I thought e10s in principle should have solved that, but, apparently, not completely. Maybe some projects on this regard are going on to totally mitigate GUI freezes, but I am just not aware of them?

Comment 11

[User Dderss]

Attachment: Безымянный.png

In certain periods TweetDeck reaches 25-40% CPU usage in the background even when it does not completely freeze the GUI.

Comment 12

[Tracy Walker [:tracy]]

Dderss, those are questions for developers.  I'll flag this bug for e10s bug triage, tomorrow, 4/19.

Comment 13

[Mike Conley (:mconley) (:⚙️)]

Dderss - I'm interested in trying to figure out what is bogging down your content process, and causing it to bog down your main process.

Have you ever gathered a performance profile before? If not, here's a video demonstrating how to do it: https://www.youtube.com/watch?v=kGBs0BQsoQg

If you can reproduce the problem and gather a profile, can you post a link to it here in the bug? I'll happily read it and try to figure out what's going on here.

Comment 14

[User Dderss]

I did run a profiler once, but I forgot how to do it; I will use your video for reference, thanks.

There are few issues:
1) most critical, but not methodically reproducible is 100% GUI freeze, generated by TweetDeck, as show in the first pair of screenshots. Running profiler to wait for when it happens would require a terabytes of memory since data output is giant. So for this, profiling is useless, alas.
2) third screenshot; high CPU usage (up to 40% as per "about:performance" statistics) of TweetDeck in the background. I can try to catch it again with profiler running, and I will write back.

What do you think can be done on #1? This is, IMHO, is more important issue as with e10s mode on such situation should be simply impossible, and yet it is possible, as turns out. GUI should be always absolute priority, but, alas, e10s can not provide that, and I wonder, why, and how it should be redesigned to avoid that.

Comment 15

[:gerard-majax]

(In reply to Mike Conley (:mconley) - Needinfo me! from comment #13)
> Dderss - I'm interested in trying to figure out what is bogging down your
> content process, and causing it to bog down your main process.
> 
> Have you ever gathered a performance profile before? If not, here's a video
> demonstrating how to do it: https://www.youtube.com/watch?v=kGBs0BQsoQg
> 
> If you can reproduce the problem and gather a profile, can you post a link
> to it here in the bug? I'll happily read it and try to figure out what's
> going on here.

Mike, I am not the one experiencing the issue (even if I filed the bug) :). Emile, I know you might not have time, but who knows I might be lucky :)

Comment 16

[Mike Conley (:mconley) (:⚙️)]

(In reply to Alexandre LISSY :gerard-majax from comment #15)
> Mike, I am not the one experiencing the issue (even if I filed the bug) :).
> Emile, I know you might not have time, but who knows I might be lucky :)

I apologize - yes, I should actually have ni?'d User Dderss.

(In reply to User Dderss from comment #14)
> I did run a profiler once, but I forgot how to do it; I will use your video
> for reference, thanks.
> 
> There are few issues:
> 1) most critical, but not methodically reproducible is 100% GUI freeze,
> generated by TweetDeck, as show in the first pair of screenshots. Running
> profiler to wait for when it happens would require a terabytes of memory
> since data output is giant. So for this, profiling is useless, alas.

Not sure where this terabytes of memory thing is coming from. The Profiler uses a circular buffer, kind of like an airplane blackbox, so it won't just accumulate more and more data; it'll start to just erase the beginning as more profiler data comes in. There's a fixed limit as to how big the buffer is, and that can be configured by entering the settings of the Gecko Profiler add-on.

That profile is key to understanding why there is a "100% GUI freeze". If I had to guess, however, based on your screenshots, I would say that there is a 95% likelihood that this hang is caused by an add-on that is synchronously accessing the Tweetdeck content over the add-on shims. Synchronous access to web content is great for add-on compatibility, but _terrible_ for main process performance.

If a profile can't be gathered, perhaps just try using Tweetdeck without your add-ons enabled. I have a high degree of certainty that you will not see the same lock-ups. If you do still see it without your add-ons, that's cause for some alarm[1].

You can do that add-on disabling experiment, or you can provide a profiler. I'm reasonably certain both will point to the fact that an add-on is causing this Tweetdeck issue that is blocking up the main process.

[1] Because, as you say, e10s is supposed to prevent this sort of thing. Note, however, that e10s is not immune from limited system resources. If, for example, there is "memory pressure" due to low memory availability on your machine, then both the main and content processes might start getting bogged down as they start doing more aggressive garbage and cycle collection in attempts to free up memory. This can hamper performance with multi-second pauses, e10s or not.

Comment 17

[:gerard-majax]

FYI, one year ago when we started looking into this with iMil, he already tried (on my suggestion) to enable e10s and the tests were done on a new empty profile, no addon installed.

Comment 18

[:gerard-majax]

(In reply to Alexandre LISSY :gerard-majax from comment #17)
> FYI, one year ago when we started looking into this with iMil, he already
> tried (on my suggestion) to enable e10s and the tests were done on a new
> empty profile, no addon installed.

last test was about 2-3 weeks ago. This is on debian and freebsd 10, systems are Core i5 with 8GB of RAM. And they are mostly used only for this.

Comment 19

[Mike Conley (:mconley) (:⚙️)]

An about:memory report might also be useful again. If gerard-majax is right, then this could be what I mentioned in my footnote for comment 16 - a memory leak that is causing the main process and content process to thrash, and GC / CC a lot.

Comment 20

[Jim Mathies [:jimm]]

This was originally filed against 33 but has morphed over to an e10s bug with the same site.

Comment 21

[User Dderss]

Attachment: profile.json

Michael, I have recorded a snippet from profiler, thanks.


It was recorded about the time when TweetDeck shows as red almost constantly in "about:performance", an it absolutely murders usability. Profile data shows GC at work mostly. I am still unable how and why GC works in synchronous mode, why it can not be made in a way that it would not kill the GUI.

As per TweetDeck, I suppose it always generates a lot of garbage as it has columns that always load and unload a lot of tweets (that include media).

I also do not understand why FireFox (even in e10s mode) can not provide such fundamental feature as "suspend all tabs but active" or allow users choose to suspend tabs manually. Wladimir Palant tried to come up with an extension that would do that few years ago, but it has worked only partly, did not use recommended APIs, so he has withdrawn it.

Comment 22

[Mike Conley (:mconley) (:⚙️)]

Strangely, that profile only has samples from the content process.

However, it seems to bolster my suspicion that long GC's are what's causing the performance problem.

What'd be more interesting at this point, perhaps, is an about:memory report when the browser is showing this behaviour on Tweetdeck.

Comment 23

[User Dderss]

Attachment: memory-report.json.gz

This is memory JSON (anonymized) during the time when TweetDeck in red in "about:performance".

Comment 24

[User Dderss]

The first explicit 220 MB-sized tab is TweetDeck:

│  ├────220,240,756 B (04.38%) -- top(https://tweetdeck.twitter.com/, id=4294967297)
│  │    ├──160,160,972 B (03.19%) -- active
│  │    │  ├───92,002,872 B (01.83%) ++ window(https://tweetdeck.twitter.com/)
│  │    │  └───68,158,100 B (01.36%) ++ window(https://tweetdeck.twitter.com/#)
│  │    └───60,079,784 B (01.20%) ++ js-zone(0xee37000)

Comment 25

[Mike Conley (:mconley) (:⚙️)]

220MB is quite large, but looking at the rest of this report, it looks like in general, a lot of memory is being consumed by other tabs. Like, TweetDeck apparently only accounts for 7% of the total ~3GB memory usage in your content process.

Looking at this memory report, it looks to me like TweetDeck is probably not throwing away DOM nodes or JS classes - it's just accumulating them.

So, at least for User Dderss, what I'd recommend for an immediate fix is trying to close some other background tabs to reclaim some system memory. Specifically, the ones listed as siblings of TweetDeck that are consuming ~100MB each.

In the meantime, I'm not sure if there's enough information here to help us determine if there's much we can do decrease memory usage, but I'll see what erahm thinks.

Comment 26

[Eric Rahm [:erahm]]

At least from that profile it looks like more than half the time is spent in GC, triggered by:

> TD.net.StreamRequest/f() @ bundle.7c2ded1e21.js:48 (ton.twimg.com)

So presumably some image loading code from tweetdeck is causing pathological behavior in GC/CC?

The memory report itself doesn't look too strange, there are 81 tabs loaded, 4GiB of explicit, so roughly averaging 50MiB/tab. That's not great, but it's not unexpected.

Comment 27

[User Dderss]

Yes, I see no issues with the memory in this case. The issue here is that TweetDeck is making FF slow as hell even in e10s mode.

Is there a way to re-make e10s FF core engine so it would do GC in sort of "asynchronous" way at least for background tabs, so it would not kill UI responsiveness? Or a way to "suspend" background tabs altogether -- manually, or by smart algorithm that would see that tabs like TweetDeck are in "red" zone (in "about:performance" colour code scale) and "suspend" them automatically, or offer user a choice on that?

FireFox already tries to detect certain extensions that are "making FireFox slow" and offers to stop them. Why the same can not be done with parasitic tabs like TweetDeck? As far as I understand, all underlying measurement instruments are already there.

Eric, Michael, could you please raise this question (if it was not raised yet) in a meeting or something? Implementation of such feature would be fundamental and universal usability issues solver.

Thank you in advance.

Comment 28

[User Dderss]

Michael, I am sorry for bothering you, but maybe you have heard if some one could be assigned for this bug? Because it really kills FF's usability. I suppose if underlying causes such synchronous garbage collection will be solved, then at least the GUI will not get so lame, and it will solve many bugs that affect the GUI. Adding possibility to (auto)(smart)suspend all operations of background operations would be also solver.

Comment 29

[Mike Conley (:mconley) (:⚙️)]

Hey Dderss,

It looks like nobody is currently assigned to this bug. I'm not sure there's anything actionable in here just yet - and actionable data is what is needed before progress can be made.

Here's what we know:

1) Your profile in comment 21 shows a ~13s window of time where you report the issue. During that time, the content process was blocked up doing garbage collection.

2) You mention that the GUI became unusable. I assume you mean the GUI of TweetDeck? Because your profile is showing (only) a Content process, it suggests that you have e10s enabled, so normally that'd mean that only the content process should be blocked when doing such a heavy GC. The parent process, which manages the browser UI should still be responsive during that. The only cases that I can think of  where that'd not be true is if your profile has some add-ons in it that are causing sync messages to be sent to the child, or if the parent also started GC'ing around the same time due to global memory pressure.

I'm going to use TweetDeck for the rest of the day to see if I can reproduce what you're seeing - though clarity on #2 would be helpful here.

Comment 30

[Eric Rahm [:erahm]]

I looked a bit closer at the code from comment 26. It appeared to be an ajaxy thing doing a "gimme the bytes you got" via polling. So maybe someone more familiar with XHR could tag in here. I'll try to get a cleaned up snippet in the meantime.

Comment 31

[User Dderss]

Thanks for reply, Michael.

I mean GUI of whole FireFox, not TweetDeck (this is why I was writing about suspending background tabs as a way to solve this). I may never visit the tab, but it still moves so much around in the background that it causes those garbage collection spikes and red coloured peaks in "about:performance".

There are no extensions that do anything with the tabs in the background, let alone specifically with TweetDeck tab (I know there are extensions that try, for example, track some events that might happen in the background to inflict some action upon them, but I have no anything like that). I do use advertisement blocker extension (uBlock Origin, not AdBlock Plus, as the former is lighter on CPU); theoretically they can affect such things.

As to your test with TweetDeck, it might not work in your case, if you have few columns, not many people to follow. I have 16 columns, most of which are updated slowly, but my main stream is moving fast, and I also subscribed to three lists that are also updated fast.

Normally, at the start, this configuration does not affect the GUI at all, and it works fine in the background, too. However, after sometime, the garbage starts to get accumulated and GC starts to take a toll. This is when I get GUI freezes where I can click on things or try to type things out in text fields, but my input's processing gets delayed.

So, if there is some extensions that can cause this, why it only make this affect after a while? Degradation of the performance is not immediate, it is accumulated. Restarting TweetDeck via "about:performance" helps the GUI for some time. Full list of my extensions:

Доступ к Рутрекеру	1.1.0	true	[e-mail]
About sessionstore	0.32.1-signed.1-signed	true	aboutsessionstore@dt
Contextual Google Image Search	2.1.1.1-signed.1-signed	true	{D46504D3-4959-4351-AED6-C7EA276DBB93}
Firefox Hello	1.2.6	true	[e-mail]
Flagfox	5.1.11	true	{1018e4d6-728f-4b20-ad56-37578a4de76b}
Flash and Video Download	1.82	true	{bee6eb20-01e0-ebd1-da83-080329fb9a3a}
Google search link fix	1.5.3	true	jid0-XWJxt5VvCXkKzQK99PhZqAn7Xbg@jetpack
Greasemonkey	3.8	true	{e4a8a97b-f2ed-450b-b12d-ee082ba24781}
HighlightAll	1.8	true	{26FD1F83-A45B-4c74-AF5A-F2EE0EE4D691}
Multi-process staged rollout	1.0	true	[e-mail]
Open Link In Pinned Tab	0.1.2.1-signed.1-signed	true	jid1-btKbvVtcxyAJ6Q@jetpack
Page Title in URL Bar	5.2.4.1-signed	true	[e-mail]
Pocket	1.0.2	true	[e-mail]
Quick Context Search	1.5.3	true	quickcontextsearch@pf
Save Image in Folder	1.3.18	true	{5e594888-3e8e-47da-b2c6-b0b545112f84}
SQLite Manager	0.8.3.1-signed.1-signed	true	[e-mail]
stealthy	3.0.1.1-signed	true	[e-mail]
Tab Mix Plus	0.4.2.2	true	{dc572301-7619-498c-a57d-39143191b318}
Tab Scope	1.6.2.1-signed.1-signed	true	[e-mail]
TinEye Reverse Image Search	1.2.1	true	[e-mail]
uBlock Origin	1.7.0	true	[e-mail]
Youtube's Autoplay No More	0.3	true	jid1-XQEcUtyD5PwB8w@jetpack

Comment 32

[Mike Conley (:mconley) (:⚙️)]

(In reply to User Dderss from comment #31)
> Thanks for reply, Michael.

Mike, please. :)

> 
> I mean GUI of whole FireFox, not TweetDeck (this is why I was writing about
> suspending background tabs as a way to solve this). I may never visit the
> tab, but it still moves so much around in the background that it causes
> those garbage collection spikes and red coloured peaks in
> "about:performance".

I'd be interested in hearing about your experience with TweetDeck when those add-ons are disabled.

Comment 33

[User Dderss]

(In reply to Mike Conley (:mconley) - (needinfo me!) from comment #32)
> (In reply to User Dderss from comment #31)
> > Thanks for reply, Michael.
> 
> Mike, please. :)

Sorry, Mike; I thought that using full name is more respectable.

> I'd be interested in hearing about your experience with TweetDeck when those
> add-ons are disabled.

I have already tried that few months ago when I did not yet knew the the culprit for GUI freezes was TweetDeck tab (I discovered it only recently when I used "about:performance" for the first time):

Please see this message: https://bugzilla.mozilla.org/show_bug.cgi?id=1135719#c30

And second part of this message:
https://bugzilla.mozilla.org/show_bug.cgi?id=1135719#c31

Even in safe mode, the GUI freezes are still reproducible.

Comment 34

[Eric Rahm [:erahm]]

Attachment: Deminified code snippet

This is the |TD.net.StreamRequest| snippet of the library deminified, and run through jsnice.org.

The function (previously |f|) is mapped to |success| here. You can see it is essentially be called every 100ms until the XHR completes and it grabs the delta of the amount of bytes available and the amount of bytes it previously grabbed.

Comment 35

[Chris Lord [:cwiiis]]

For what it's worth, I've also been suffering with this for a really long time (years?). I've just gotten into the habit of having to close and restart Firefox every now and again. My add-ons:

*Activity Stream
DuckDuckGo Plus
Mass Password Reset
Open Tabs Next to Current
*Stylish
*Tab Center
*Tab Groups
*Test Pilot
uBlock Origin

Ones marked with a star are ones I've only been using recently, the problem has existed for a lot longer than I've been using those add-ons.

The slow-down I experience causes long pauses in UI interactions in Firefox (possibly not with the main UI, but many primary interactions seem to block on content) - specifically, switching tabs, opening new tabs, scrolling page content, clicking links, typing in input fields, right-clicking on links, choosing options in pop-up menus, etc.

Comment 36

[sam.mantha.katz]

For what it is worth - The problem with Tweetdeck hanging the browser at intervals after Tweetdeck is in use for a while is not exclusive to Firefox. We also experience it in Chrome but Firefox recovers faster.

Comment 37

[znmeb]

Jumping into the fray - I'm currently seeing this on Nightly [52.0a1 (2016-09-23) (64-bit), Fedora 24 x86_64 Workstation with GNOME 3 desktop]. With e10s enabled it happens fairly quickly. With e10s disabled, not at all. I've got the machine set up to capture profiles via operf / oprofile if it will help. 

With e10s enabled it freezes; the only operating control once it freezes is the "X" in the upper right corner. When I click that, it waits a while before offering to shut down, but eventually does shut down.

I'll try to reproduce this in "Safe Mode" - I've hit some other bugs recently that went away in safe mode even though I had no extensions installed.

Comment 38

[znmeb]

So far (half an hour) on Fedora 24 Nightly 52 e10s enabled and safe mode is stable - there are occasional slow spots but no freezes.

Comment 39

[znmeb]

This may be connected with LastPass - see bug https://bugzilla.mozilla.org/show_bug.cgi?id=1305831

Comment 40

[Ludovic Hirlimann [:Usul]]

(In reply to znmeb from comment #39)
> This may be connected with LastPass - see bug
> https://bugzilla.mozilla.org/show_bug.cgi?id=1305831

I don't have last pass and see the issue.