1167667 - Shims affect performance in non-e10s user profiles

Status: RESOLVED WORKSFORME

(Firefox :: Extension Compatibility, defect)

Description

[Justin Wood (:Callek)]

From :mconley:

> So I think we've identified LastPass as the culprit. What's not great is that
> LastPass is causing lots of shim traffic _despite_ e10s being disabled. If
> there's been a serious performance regression with _non-e10s_ with LastPass
> due to the shims, that's probably something we should fix ASAP, since the
> shims are now on Aurora.

Profile that shows the issue: http://people.mozilla.org/~bgirard/cleopatra/#report=0dba7dc7010f57d636717c24519b7d8be714c691

Mike did ask me to describe how I'm hitting it, but the c#0 was the best way to describe it.

I do have lots of tabs, and a few app tabs. But most of them are not even loaded into memory.

Following is a slight excerpt from the bug where we found this.

+++ This bug was initially created as a clone of Bug #1167615 +++

So, when I switch tabs I get a 2 to 3 full second delay before the active tab UI switches and before the content area switches.

This is with today's Dev-Edition (5-22) and I updated from at least Monday 5-11 as the oldest possible, where I didn't experience this issue. (If you can't reproduce a similar issue, I'm happy to help bisect)

In between that last build and this one I did an update (that was staged earlier) and then got the doorhanger prompt about e10s "is coming soon" I selected "NOT NOW". incase that matters.

If you need more info from me, it is best to use a needinfo for a fast turnaround.

Comment 1

[Justin Wood (:Callek)]

[Tracking Requested - why for this release]: from c#0 "that's probably something we should fix ASAP, since the shims are now on Aurora."

Mike can we track this down further, even on latest aurora I'm hitting this *constantly*

Comment 2

[Mike Conley (:mconley) (:⚙️)]

(In reply to Justin Wood (:Callek) from comment #1)
> [Tracking Requested - why for this release]: from c#0 "that's probably
> something we should fix ASAP, since the shims are now on Aurora."
> 
> Mike can we track this down further, even on latest aurora I'm hitting this
> *constantly*

Hey gabor - do you suspect that the shim optimizations in bug 1164014 will help Callek out here?

Are the shims riding up the trains even though e10s is currently halted on Aurora? Or are the shims only being enabled on the releases that have e10s as an option?

Comment 3

[Gabor Krizsanits (INACTIVE)]

(In reply to Mike Conley (:mconley) - Needinfo me! from comment #2)
> (In reply to Justin Wood (:Callek) from comment #1)
> > [Tracking Requested - why for this release]: from c#0 "that's probably
> > something we should fix ASAP, since the shims are now on Aurora."
> > 
> > Mike can we track this down further, even on latest aurora I'm hitting this
> > *constantly*
> 
> Hey gabor - do you suspect that the shim optimizations in bug 1164014 will
> help Callek out here?
> 

Maybe... But 2-3 seconds seem like a LOT for shim traffic. Are we sure that e10s
is not enabled? Because CPOW traffic could explain something like that especially
because Bug 1164011. We have a LastPass specific CPOW optimization in the shim layer
that got turned off. The patch in the mentioned bug fixes it, and I'm trying to
uplift to aurora but there is not much movement around it. Do you know how can
I speed up the aurora uplift request process? Anyway I will try to look into this.

> Are the shims riding up the trains even though e10s is currently halted on
> Aurora? Or are the shims only being enabled on the releases that have e10s
> as an option?

I think the intention was to let them ride the train and last time we talked
about this I was told that the flags are set. Checking it though I'm not sure
http://mxr.mozilla.org/mozilla-aurora/source/browser/app/profile/firefox.js#1877
http://mxr.mozilla.org/mozilla-aurora/source/configure.in#3538
We should ask someone who knows more about these flags, flagging Bill, since I also
wanted to ask him if we really need the multiprocess shim layer for non-e10s enabled
mode. We will at some point, but for now on aurora I don't think so... Maybe we could
just turn them off if e10s is off?

Comment 4

[Mike Conley (:mconley) (:⚙️)]

(In reply to Gabor Krizsanits [:krizsa :gabor] from comment #3)
> (In reply to Mike Conley (:mconley) - Needinfo me! from comment #2)
> > (In reply to Justin Wood (:Callek) from comment #1)
> > > [Tracking Requested - why for this release]: from c#0 "that's probably
> > > something we should fix ASAP, since the shims are now on Aurora."
> > > 
> > > Mike can we track this down further, even on latest aurora I'm hitting this
> > > *constantly*
> > 
> > Hey gabor - do you suspect that the shim optimizations in bug 1164014 will
> > help Callek out here?
> > 
> 
> Maybe... But 2-3 seconds seem like a LOT for shim traffic. Are we sure that
> e10s
> is not enabled? Because CPOW traffic could explain something like that
> especially
> because Bug 1164011. We have a LastPass specific CPOW optimization in the
> shim layer
> that got turned off. The patch in the mentioned bug fixes it, and I'm trying
> to
> uplift to aurora but there is not much movement around it. Do you know how
> can
> I speed up the aurora uplift request process? Anyway I will try to look into
> this.

The best way to speed up the uplift request process, as far as I know, is to find the release manager responsible for 40, and ask them directly. Maybe ping lmandel, sledru or lizzard in #relman?

Comment 5

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

The shims are enabled in Aurora for e10s and non-e10s but they won't ride the trains. I think that's probably how we want it. Otherwise we wouldn't get bugs like this.

Is there any way to reproduce this bug starting from a clean profile? I'd be happy to look at it if there's a specific STR.

Comment 6

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Justin, Is there a reason why this bug is not a duplicate of bug 1167615? Thanks!

Comment 7

[Justin Wood (:Callek)]

(In reply to Ritu Kothari (:ritu) from comment #6)
> Justin, Is there a reason why this bug is not a duplicate of bug 1167615?
> Thanks!

Because of https://bugzilla.mozilla.org/show_bug.cgi?id=1167615#c7 only.

Comment 8

[Justin Wood (:Callek)]

Ok, I'm pretty adamant that the regression I'm hitting is *not* ship-worthy. Considering I disabled e10s, and I'm now in the "45 seconds on an update restart before I can do *anything* with firefox, including opening the menu "hamburger", and 3-5 seconds on tab switches for a while thereafter.

We are *already* advertising this to dev-edition users. And I'm happy to try and take this through very specific STR and help narrow down, however I have my own deliverables to hit that are currently AT RISK, among very little personal time to add to things atm, so would like very very specific direction/hints as to this.

Without some serious eyes on this issue, I don't see how this *won't* completely hurt our overall brand. And I have certainly provided what I can so far (the profiles being one thing -- Happy to provide more/new ones if necessary).

But lets get some eyes on this please.

Comment 9

[Mike Conley (:mconley) (:⚙️)]

Hey Callek,

(In reply to Justin Wood (:Callek) from comment #8)
> Ok, I'm pretty adamant that the regression I'm hitting is *not* ship-worthy.

e10s is currently holding on Aurora and will not uplift. The shims are also holding on Aurora, and will not uplift.

> Considering I disabled e10s, and I'm now in the "45 seconds on an update
> restart before I can do *anything* with firefox, including opening the menu
> "hamburger", and 3-5 seconds on tab switches for a while thereafter.
> 
> We are *already* advertising this to dev-edition users. And I'm happy to try
> and take this through very specific STR and help narrow down, however I have
> my own deliverables to hit that are currently AT RISK, among very little
> personal time to add to things atm, so would like very very specific
> direction/hints as to this.

Please list for me your enabled add-ons.

Comment 10

[Justin Wood (:Callek)]

(In reply to Mike Conley (:mconley) - Needinfo me! from comment #9)
> Hey Callek,
> 
> (In reply to Justin Wood (:Callek) from comment #8)
> > Ok, I'm pretty adamant that the regression I'm hitting is *not* ship-worthy.
> 
> e10s is currently holding on Aurora and will not uplift. The shims are also
> holding on Aurora, and will not uplift.
>

Sure, but by 'release' in this case I *also* mean for Developer Edition, which is already released, and we have this behavior for the userbase we want to target of tech-saavy users. And ones who will be quick to say "Firefox is too slow again, lets use Chrome for my web dev". :/

> Please list for me your enabled add-ons.

Sure, already done in Bug 1167615

Comment 11

[Mike Conley (:mconley) (:⚙️)]

How much does disabling LastPass help?

Comment 12

[Justin Wood (:Callek)]

(In reply to Mike Conley (:mconley) - Needinfo me! from comment #11)
> How much does disabling LastPass help?

https://bugzilla.mozilla.org/show_bug.cgi?id=1167615#c6

Comment 13

[Mike Conley (:mconley) (:⚙️)]

(In reply to Justin Wood (:Callek) from comment #12)
> (In reply to Mike Conley (:mconley) - Needinfo me! from comment #11)
> > How much does disabling LastPass help?
> 
> https://bugzilla.mozilla.org/show_bug.cgi?id=1167615#c6

Ok, thank you.

The conclusion that I draw from this is that you have some add-ons that are bogging your profile down. We need to identify those add-ons.

Can you please try disabling all of your add-ons (or use safe-mode, since you've got e10s disabled already), to see how that changes Firefox's behaviour?

Assuming performance goes back to somewhere reasonable, the next step would be to identify and rank the add-ons that are causing your performance to go down the tubes.

Comment 14

[Sylvestre Ledru [:Sylvestre]]

(In reply to Justin Wood (:Callek) from comment #8)
> But lets get some eyes on this please.
Indeed. Tracking.

Do you know if 39 is affected?

Comment 15

[Justin Wood (:Callek)]

(In reply to Mike Conley (:mconley) - Needinfo me! from comment #13)
> (In reply to Justin Wood (:Callek) from comment #12)
> > (In reply to Mike Conley (:mconley) - Needinfo me! from comment #11)
> > > How much does disabling LastPass help?
> > 
> > https://bugzilla.mozilla.org/show_bug.cgi?id=1167615#c6
> 
> Ok, thank you.
> 
> The conclusion that I draw from this is that you have some add-ons that are
> bogging your profile down. We need to identify those add-ons.

Ok Mike, let me backup since I'm a bit confused here.

Per IRC at the time (of filing) you had me run the profile I put into Bug 1167615

I tested with and without Safe Mode, and With lastpass specifically disabled.

LastPass was the culprit, but at the same time it wasn't.

Without lastpass the tab switch performance was ~half a second. With lastpass it was 2-5 seconds.

The Profile pointed at the shims, and you pointed at a bug to improve those performances and told me to file this bug so it can get looked at and fixed ASAP for aurora users.

The referenced bug for performance is not yet fixed, though has at least 2 landed patches that have not yet had approval requested.

And I have e10s disabled entirely.

Am I misunderstanding something along the way, since I feel these past few bugzilla comments from us is repeat...

(In reply to Sylvestre Ledru [:sylvestre] from comment #14)
> (In reply to Justin Wood (:Callek) from comment #8)
> > But lets get some eyes on this please.
> Indeed. Tracking.
> 
> Do you know if 39 is affected?

Nope, not affected in 39, and As I Understand the above, is not *intended* to exist in 40's beta cycle (since the shims are not uplifting, whatever that will look like in relatity dunno)

Comment 16

[Mike Conley (:mconley) (:⚙️)]

(In reply to Justin Wood (:Callek) from comment #15)
> (In reply to Mike Conley (:mconley) - Needinfo me! from comment #13)
> > (In reply to Justin Wood (:Callek) from comment #12)
> > > (In reply to Mike Conley (:mconley) - Needinfo me! from comment #11)
> > > > How much does disabling LastPass help?
> > > 
> > > https://bugzilla.mozilla.org/show_bug.cgi?id=1167615#c6
> > 
> > Ok, thank you.
> > 
> > The conclusion that I draw from this is that you have some add-ons that are
> > bogging your profile down. We need to identify those add-ons.
> 
> Ok Mike, let me backup since I'm a bit confused here.
> 
> Per IRC at the time (of filing) you had me run the profile I put into Bug
> 1167615
> 
> I tested with and without Safe Mode, and With lastpass specifically disabled.
> 
> LastPass was the culprit, but at the same time it wasn't.
> 
> Without lastpass the tab switch performance was ~half a second. With
> lastpass it was 2-5 seconds.
> 
> The Profile pointed at the shims, and you pointed at a bug to improve those
> performances and told me to file this bug so it can get looked at and fixed
> ASAP for aurora users.
> 
> The referenced bug for performance is not yet fixed, though has at least 2
> landed patches that have not yet had approval requested.
> 
> And I have e10s disabled entirely.

Yep, understood.

> 
> Am I misunderstanding something along the way, since I feel these past few
> bugzilla comments from us is repeat...

Heh, my apologies. I'd forgotten the history of this bug. Disabling add-ons and getting profiles are the automatic response for performance problems, and I'd forgotten we'd already gone through some of those steps. Sorry about that.

In comment 5, billm was curious to know if there was a way to reproduce the issue you're hitting with a clean profile. Are you able to find STR?

Comment 17

[Gabor Krizsanits (INACTIVE)]

I looked into this today. All the profiler data show that we spend most of the time in some google docs related scripts during the peeks (something something ritz_waffle_i18n_core.js) it looks some Google Web Toolkit stuff. Anyway, I could get similar named js files loaded when I opened up some spreadsheets in google docs (only for spreadsheets for some reason). And things seemed to get worse when I installed lastpass and added my google profile to it.

So Bill, I think if you want to investigate it that's a good start.

Justin, could you please try and close the spreadsheets, or actually all google docs related tabs and see if the problem is going away or not? (just to confirm that my observations are correct)

Comment 18

[Justin Wood (:Callek)]

(In reply to Gabor Krizsanits [:krizsa :gabor] from comment #17)
> I looked into this today. All the profiler data show that we spend most of
> the time in some google docs related scripts during the peeks (something
> something ritz_waffle_i18n_core.js) it looks some Google Web Toolkit stuff.
> Anyway, I could get similar named js files loaded when I opened up some
> spreadsheets in google docs (only for spreadsheets for some reason). And
> things seemed to get worse when I installed lastpass and added my google
> profile to it.
> 
> So Bill, I think if you want to investigate it that's a good start.
> 
> Justin, could you please try and close the spreadsheets, or actually all
> google docs related tabs and see if the problem is going away or not? (just
> to confirm that my observations are correct)

The trick there is I have ~400 tabs, many of which are not "loaded" (session restart delay-load stuff) but still have sessionstore-stored data that I need/want to retain.

I also have 3 app-tabs all google, (2 gmail, and one google calendar).

I have 3 tied google accounts, (also all tied in lastpass) for this e-mail (bugspam) my personal one (remove bugspam. ) and my moco one (jwood@) So those are all exposed as potential switch-to-accounts in each of those google tabs.

I don't presently have any google docs "loaded" in my session and still experience lengthy delays in tab switching.

Comment 19

[Justin Wood (:Callek)]

Bah submitted before finished:

IFF you really want me to close all google docs anyway, I can. Per about:tabs (glandium's extension) I have:

 docs.google.com (6 tabs) [Close]

Which isn't really that much, based on how many tabs I have open. Related:

 www.google.com (11 tabs) [Close]
 mail.google.com (2 tabs) [Close]
..
 54 other hosts

Comment 20

[Mike Conley (:mconley) (:⚙️)]

Callek emailed me a copy of his sessionstore.js and prefs.js

I have restored his session in the latest version of Firefox Dev Edition with e10s disabled. I am not seeing long pauses between tab switches, I'm afraid. :/

Callek - is it possible that your machine is swapping to disk a lot? How much free memory do you have?

Comment 21

[Mike Conley (:mconley) (:⚙️)]

Ok, talked a bit with Callek over IRC.

When he set extensions.interposition.enabled to false, performance reverted back to what he was experiencing pre-40.

So that sounds like pretty compelling evidence that non-e10s performance suffers when the shims are enabled.

I feel like I should know this - but why are the shims enabled when e10s is disabled? What does that give us? I guess that's directed at either billm or gabor.

Comment 22

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

There are two reasons to have shims enabled without e10s: better test coverage and supporting "New e10s window".

I tried running LastPass in Aurora with the shims enabled and I didn't see any performance problems. Perhaps we should do a custom try build of Aurora with the shim optimizations (bug 1164014) backported and see if it helps Callek. I'm not sure what else we can do here since, as far as I can tell, no one else has managed to reproduce the problem.

Gabor, would you mind doing the try build?

Comment 23

[Mike Conley (:mconley) (:⚙️)]

(In reply to Bill McCloskey (:billm) from comment #22)
> There are two reasons to have shims enabled without e10s: better test
> coverage and supporting "New e10s window".
> 
> I tried running LastPass in Aurora with the shims enabled and I didn't see
> any performance problems. Perhaps we should do a custom try build of Aurora
> with the shim optimizations (bug 1164014) backported and see if it helps
> Callek. I'm not sure what else we can do here since, as far as I can tell,
> no one else has managed to reproduce the problem.
> 
> Gabor, would you mind doing the try build?

We might want to test DOM-intensive sites with LastPass and non-e10s to see if we see Callek's behaviour. Google Drive documents and spreadsheets come to mind. Treeherder and CNN as well.

Comment 24

[Gabor Krizsanits (INACTIVE)]

(In reply to Bill McCloskey (:billm) from comment #22)
> Gabor, would you mind doing the try build?

Will do it tomorrow.

(In reply to Mike Conley (:mconley) - Needinfo me! from comment #23)
> We might want to test DOM-intensive sites with LastPass and non-e10s

Yeah I agree, I will play with it a bit.

Comment 25

[Gabor Krizsanits (INACTIVE)]

(In reply to Gabor Krizsanits [:krizsa :gabor] from comment #24)
> (In reply to Bill McCloskey (:billm) from comment #22)
> > Gabor, would you mind doing the try build?

Sigh... it's not helping that every tree is closed the whole day :(

Comment 26

[Gabor Krizsanits (INACTIVE)]

(In reply to Gabor Krizsanits [:krizsa :gabor] from comment #25)
> (In reply to Gabor Krizsanits [:krizsa :gabor] from comment #24)
> > (In reply to Bill McCloskey (:billm) from comment #22)
> > > Gabor, would you mind doing the try build?
> 
> Sigh... it's not helping that every tree is closed the whole day :(

It looks good on try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=72ed11375486

anyone feels like experimenting with it, binaries here: https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-72ed11375486

Comment 27

[Mike Conley (:mconley) (:⚙️)]

Any chance you can try the above build, after turning the shims pref back on?

Comment 28

[Justin Wood (:Callek)]

(In reply to Mike Conley (:mconley) - Needinfo me! from comment #27)
> Any chance you can try the above build, after turning the shims pref back on?

I have both bad and great news to go with this test (win32):

Bad news first:
 * The startup time (from click start to UI launched/loaded) has not seemingly improved [over that of the shims disabled case]

Great news:
 * My slowdown in that build (for tab switching) with shims enabled is perceptibly the same as the performance in official Aurora with shims disabled.

I'm open to doing some more perf measurements for my startup path in a different bug, since its a different issue. But I don't anticipate getting that done any sooner than whistler. But imo this test says that yes, we should strive to uplift those shim perf wins.

Comment 29

[Sylvestre Ledru [:Sylvestre]]

OK, we will stop tracking it then.

Comment 30

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

The optimizations landed and it sounds like they helped to some extent. Please open a new bug if you see problems still.