1277206 - Performance degradation in heavy CSS / Javascript interactive text.

Status: RESOLVED WORKSFORME

(Core :: Graphics, defect, P4)

Description

[Knowguy]

Attachment: Performance Profile while using slow Network monitoring web page environment. Science Logic EM7.

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:49.0) Gecko/20100101 Firefox/49.0
Build ID: 20160531030258

Steps to reproduce:

1) Login
2) Performance is decent.
3) Begin manipulating rows in interactive event table.
4) Double clicking to open pop windows and edit drop down menus / saving input.
5) The longer I do this the longer it takes Firefox to do the same task again.
6) Continues until Firefox is too slow to get work done.



Actual results:

I get heavy page stuttering and freezing.  When in other tabs in same browser session even typing text into search bar will pause and then catch up.  Performance on this same page in Blink based browser does not seem effected nearly as much.  


Expected results:

Things should flow smoothly when hovering over each row.  Clicking and opening pop windows should take the same amount of time each time. No stuttering when scrolling through hundreds of table rows.  And saving of changed input should not take progressively longer.

Comment 1

[Knowguy]

Attachment: 2nd performance profile while using the page.

Comment 2

[Knowguy]

Attachment: Example layout of rows in monitoring environment.

Comment 3

[Mike Conley (:mconley) (:⚙️)]

Looking at the profile posted in comment 1, a good chunk of time is being spent by the uBlock add-on. It's sending sync IPC messages to the parent process.

If you disable uBlock, does performance seem to improve?

Comment 4

[Knowguy]

I am in the process of testing with no add-ons.

Comment 5

[Knowguy]

Functionality of the events page is still laggy on scrolling, and selecting rows.  Although there may be a slight improvement.

The tickets page has improved quite a bit since disabling ublock. But there still is slowness when saving changes and on rows updating with changed information.

Comment 6

[Knowguy]

Attaching performance profiles for using both pages without addons.

Comment 7

[Knowguy]

Added to Cleopatra instead:
Tickets page profile:https://people.mozilla.org/~bgirard/cleopatra/#report=7626999af84a1b6afdd7109fd588158a4bec5d15

Events page profile:https://people.mozilla.org/~bgirard/cleopatra/#report=8f1f92c66000130891f4ac4383a0f8c2587f0b57

Comment 8

[Jeff Muizelaar [:jrmuizel]]

The page is taking around 130ms to paint.

Is it possible for you to save a copy of the page using "Save Page As" and attach it to the bug?

Comment 9

[Knowguy]

Attaching both saved pages (Events and Tickets). Didn't know if I needed them separately since they probably are functioning with the same resources.  I see a lot of JavaScript!

Comment 10

[Knowguy]

Attachment: Websites (events, tickets). 7zip.

Comment 11

[Mike Conley (:mconley) (:⚙️)]

Adding to jrmuizel's radar - the slow-painting page has been attached.

Comment 12

[Knowguy]

I also failed to mention that  this site is also on 15 second refresh timer which is very important for seeing new events happening in our monitoring environment.  Randomly it seems that the events page will get stuck at the 15 second mark and never refresh.  The page does not seem to lock up or have a script error and outside of having the timer stopped at 15, the page is still responsive.  

It just makes us have to reload the page entirely to get it working again, and really defeats the purpose.  Other browsers appear to not have this issue according to other employees so they switched from Firefox.

Not sure how I would catch this issue in the act since it is random as far as I can tell.

Comment 13

[Pascal Chevrel:pascalc]

(In reply to Mike Conley (:mconley) - (needinfo me!) from comment #11)
> Adding to jrmuizel's radar - the slow-painting page has been attached.

Should that bug be moved from the UNCONFIRMED status to the NEW status?

Comment 14

[Knowguy]

Submitting proper performance profile from Gecko Profiler addon instead of nightly dev tools. This profile is for the page with the most issues (also note near the end, the page was refusing to do its 15 second refresh of events): https://cleopatra.io/#report=a9ae4a3e9288e2def8af1daabeb56a44b0d3e3ce

Comment 15

[Knowguy]

Submitting proper performance profile for tickets page as well (this time performance was very spotty again): https://cleopatra.io/#report=e58634bdd1732e60a0534caaf62fa8b4ff1b8c8b

Comment 16

[Jeff Muizelaar [:jrmuizel]]

Those profiles don't have proper symbols. Was this done with an official Nightly build?

Comment 17

[Knowguy]

Yes, official Nightly channel.

Comment 18

[Jeff Muizelaar [:jrmuizel]]

It looks like our symbol server was broken and so that's why the profiles weren't getting symbols. It has now been fixed. Can you get a new profile?

Comment 19

[Knowguy]

New profile for events page: https://cleopatra.io/#report=13edc1251355284b996c3242d60d47760653075f

Comment 20

[Knowguy]

New profile for tickets page: https://cleopatra.io/#report=5c7940ceece93fc578936fd574d163919b18396e

Comment 21

[Mike Conley (:mconley) (:⚙️)]

These are much, much better! Thank you Jesse!

Comment 22

[Jeff Muizelaar [:jrmuizel]]

(In reply to Jesse Roland from comment #19)
> New profile for events page:
> https://cleopatra.io/#report=13edc1251355284b996c3242d60d47760653075f

This one doesn't seem to have worked. Can you try again. Sorry about the unreliability here...

Comment 23

[Knowguy]

Events page profile that should be good: https://cleopatra.io/#report=5b0dbf0af23a649a0720833e5c3ce55665e8a010

a note: this page froze and would not do its normal 15 second refresh so I had to reload the page.  So it is a good example.

Comment 24

[Mike Conley (:mconley) (:⚙️)]

(In reply to Jesse Roland from comment #23)
> Events page profile that should be good:
> https://cleopatra.io/#report=5b0dbf0af23a649a0720833e5c3ce55665e8a010
> 
> a note: this page froze and would not do its normal 15 second refresh so I
> had to reload the page.  So it is a good example.

For this one, it looks like uBlock's content policy code is causing sync messages to be sent to the parent, which is blocking the content process.

If you disable uBlock, do things improve?

Comment 25

[Knowguy]

Tried this with you up in comment 3 and it does improve ever so slightly and more so with the tickets page.  The performance issue is still completely noticeable.

Comment 26

[Mike Conley (:mconley) (:⚙️)]

(In reply to Jesse Roland from comment #25)
> Tried this with you up in comment 3 and it does improve ever so slightly and
> more so with the tickets page.  The performance issue is still completely
> noticeable.

Bah, sorry - it's easy to lose context moving between bugs. :/ I'll take another look at the profiles and see if anything else pops out.

Comment 27

[Mike Conley (:mconley) (:⚙️)]

(In reply to Mike Conley (:mconley) - (Needinfo me!) from comment #26)
> (In reply to Jesse Roland from comment #25)
> > Tried this with you up in comment 3 and it does improve ever so slightly and
> > more so with the tickets page.  The performance issue is still completely
> > noticeable.
> 
> Bah, sorry - it's easy to lose context moving between bugs. :/ I'll take
> another look at the profiles and see if anything else pops out.

Wow, sorry for not getting back for about a month. Yikes. :(

So nothing else really jumps out from these profiles when I ignore the uBlock stuff. The uBlock samples are completely dominating your profiles.

You gathered profiles back in Comment 7 with your add-ons disabled, but it looks to me like you were using the built-in DevTools Performance Tab.

What I need is for you to gather a profile by using the Gecko Profiler Add-on with the rest of your add-ons disabled.

I'm so sorry to keep making you get data for us, but really this is the only way we're going to figure out what's going on here.

Comment 28

[Knowguy]

Back on shift tonight and will collect some better profiles. I have switched to 64 bit Firefox and it does not like gecko profiler at all.  I may have to switch back to 32 in the mean time.

Comment 29

[Knowguy]

Well I tried, getting this:
http://imgur.com/zx8a2nv
"Symbolication failed"

Comment 30

[Knowguy]

Finally got a profile through hopefully this helps.  Had to install 32 bit and I have Ublock disabled.
https://cleopatra.io/#report=4ded13f522c711ed91c913e5687be841fabfe5d3

Comment 31

[Mike Conley (:mconley) (:⚙️)]

Oh, that's much better, yes! Thank you!

This looks to me like we're spending a lot of time painting. We're also cycle-collecting a lot towards the beginning of the profile.

Hey jrmuizel - we've got the site saved to this bug, and a pretty clear profile here. Is this enough information to go on?

Comment 32

[Jeff Muizelaar [:jrmuizel]]

It looks like a lot of the painting slowdown is because of widget backgrounds. We could probably do much better.

Comment 33

[Jeff Muizelaar [:jrmuizel]]

Can you try creating a new boolean pref for "mozilla.widget.disable-native-theme" and setting the pref to true to see if that improves performance?

Comment 34

[Knowguy]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #33)
> Can you try creating a new boolean pref for
> "mozilla.widget.disable-native-theme" and setting the pref to true to see if
> that improves performance?

Didn't realize this was pointed at me, wish I would have tested this sooner.  This indeed does improve performance on this site in my limited testing.  I would generate another profile but gecko profiler addon crashes 64bit nightly right now.  I have submitted a ticket on github.  Thanks for helping out on this!

Although this does make Firefox look a little weird!

Comment 35

[Knowguy]

(In reply to Jesse Roland from comment #34)
> (In reply to Jeff Muizelaar [:jrmuizel] from comment #33)
> > Can you try creating a new boolean pref for
> > "mozilla.widget.disable-native-theme" and setting the pref to true to see if
> > that improves performance?
> 
> Didn't realize this was pointed at me, wish I would have tested this sooner.
> This indeed does improve performance on this site in my limited testing.  I
> would generate another profile but gecko profiler addon crashes 64bit
> nightly right now.  I have submitted a ticket on github.  Thanks for helping
> out on this!
> 
> Although this does make Firefox look a little weird!

This fix was all but a mask, it continued to degrade in performance even after this initial performance boost.

Comment 36

[Knowguy]

Attachment: GIF Example of slow down of hover.

Comment 37

[Jet Villegas (inactive)]

94% of this profile is in widget::WinUtils::WaitForMessage().

Comment 38

[(no longer active)]

(In reply to Jet Villegas (:jet) from comment #37)
> 94% of this profile is in widget::WinUtils::WaitForMessage().

That's just waiting for messages.  This profile is really heavy on display list construction and painting.  Because painting seems to be heavier I'm moving it to graphics.  There's a good amount of pixman region manipulation going on here (search for pixman_region).  Also some expensive rasterization.

Comment 39

[Milan Sreckovic [:milan] (needinfo for best results)]

Bas, because this is a split of graphics and layout, lets see if you can get to the graphics part soon, to give layout a chance at it after we're done.  Unless retained display lists ends up being required.

Comment 40

[Bas Schouten (:bas.schouten)]

Note that at least some part of this profile (the expensive use of SystemParametersInfo) was addressed in bug 1325743. Other than that the long rasterizes seem to be caused by the fact that we're drawing lots of text, and with a significant additional increase in cost because we're using Component Alpha layers. These are not costs that we can do anything in graphics to avoid, so I'm not so sure there -is- much of a a graphics part. If on the layout side we can avoid using Component Alpha layers here that will be the biggest improvement we can get here.

There is a small bit of improvement we may get by avoiding calling SetTextAntialiasingMode, I'll have a look at that but it's not going to help much, component alpha layers are what is increasing our cost by over 2x.

There's also a fair amount of context manipulation overhead because there is a lot of nsDisplayText display items doing small draws looking at the profile. Likely this is very hard to avoid and in any case the win would in no way be comparable to the win from not doing component alpha layers.

Comment 41

[Jet Villegas (inactive)]

(In reply to Away for a while from comment #38)
> This profile is really heavy on display list construction and painting. 

Now that we have Retained Display Lists enabled in Nightly, it would be good to give this another look.

Comment 42

[Knowguy]

(In reply to Jet Villegas (:jet) from comment #41)
> (In reply to Away for a while from comment #38)
> > This profile is really heavy on display list construction and painting. 
> 
> Now that we have Retained Display Lists enabled in Nightly, it would be good
> to give this another look.

It appears to work better than I before. I do not have access to this live environment anymore though. The archive of the site above does work better but there is some delay when moving fast over the rows with the hover taking place.

Comment 43

[Gregory Pappas [:gregp]]

Both test cases perform well now.