1215970 - "Skype Click to Call" add-on is breaking Firefox

Status: RESOLVED DUPLICATE of bug 1248049

(Core :: Networking: Cache, defect, P1)

Description

[Caspy7]

Within a short period of time I've seen multiple reports that Skype's Click-to-Call addon is breaking Firefox.

https://redd.it/3p0a2f -> resolution: https://redd.it/3p52u7
https://www.reddit.com/r/firefox/comments/3p3rs8/the_latest_firefox_update_4102_is_causing_youtube/cw3axky
https://www.reddit.com/r/firefox/comments/3p7up8/massive_input_lag_since_new_update_after_couple/cw47qcv

Reports include Firefox freezing, crashing and jQuery errors.
Removing the addon fixes the issue but I'm told the option to remove it normally from within Firefox has been removed.  

(I'm upping the severity on this as it has the potential to affect many user and on a high use website.)

Comment 1

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

I want to jump on this ASAP. Who do I need to bother about it?

Comment 2

[Caspy7]

I had an additional reply from someone in a thread about Firefox being "laggy and slow".  They indicated that after removing the skype addon that Firefox returned to a speedy state.

https://www.reddit.com/r/firefox/comments/3p7up8/massive_input_lag_since_new_update_after_couple/cw4fyew?context=10000

This indicates that the addon may be having a larger effect on the browser than just youtube (perhaps it's particularly bad on youtube for some users).

Comment 3

[Jorge Villalobos [:jorgev] (he/him)]

This add-on is notably laggy (has been for years) because it parses all the text in web pages to find and highlight phone numbers. If there are more recent reports than normal, then it's likely something changed in Firefox that makes this problem worse. I'll contact the people in charge of the add-on to see what they have to say about this.

Comment 4

[Chris Peterson [:cpeterson]]

Jorge, can we please block this add-on until we hear from Microsoft/Skype? Given the recent flood of problem reports from people running different Firefox channels, I suspect this is a regression in the add-on, not Firefox.

Comment 5

[Jorge Villalobos [:jorgev] (he/him)]

Not until there's clear evidence that this is a big problem and we've exhausted all other options. This add-on has millions of active users.

Comment 6

[Kris Maglione [:kmag]]

(In reply to Jorge Villalobos [:jorgev] from comment #3)
> This add-on is notably laggy (has been for years) because it parses all the
> text in web pages to find and highlight phone numbers. If there are more
> recent reports than normal, then it's likely something changed in Firefox
> that makes this problem worse. I'll contact the people in charge of the
> add-on to see what they have to say about this.

If that's the case, then it should be fixed or blocked. The add-on has millions of users who didn't install it intentionally. They definitely didn't sign up for that kind of performance hit.

Comment 7

[Chris Peterson [:cpeterson]]

Anthony, since you can reproduce this problem, can you test Firefox <= 40 to verify whether this is a regression in Firefox 41?

Does the add-on have a recent release date or file timestamp? I can't find any release notes on Microsoft's Skype support website.

Comment 8

[Liz Henry (:lizzard) (relman/hg->git project)]

Tracking this for 43+ since it may be a recent (and nasty) regression. 

Andrei, can your team try to reproduce this issue on Firefox 41-44 and give us more details?
 
If this doesn't seem to be correlated with a particular build/release then that would help us figure out if it's our regression or skype's and whether we can work around it, or if we need to block a particular version of the add-on.

Comment 9

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #8)
> Tracking this for 43+ since it may be a recent (and nasty) regression.

Regression or otherwise it is still nasty. I only tested Firefox 41. It should be blocked until the performance issues have been resolved.

"Add-ons should not slow down the application or system significantly."

https://developer.mozilla.org/en-US/Add-ons/Add-on_guidelines#Be_Stable

It is installed with Skype but includes a confirmation page. Also opt-out is a homepage hijack and a search hijack. The search hijack gets blocked. I don't feel positive about the whole trojanware experience.

Comment 10

[Sylvestre Ledru [:Sylvestre]]

We are in touch with Microsoft/Skype about the issue. Anthony, I cced you to the thread.
Tracking in 42 as it is an important issue.

Comment 11

[Bogdan Maris, Desktop QA]

(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #8)
> Tracking this for 43+ since it may be a recent (and nasty) regression. 
> 
> Andrei, can your team try to reproduce this issue on Firefox 41-44 and give
> us more details?
>  
> If this doesn't seem to be correlated with a particular build/release then
> that would help us figure out if it's our regression or skype's and whether
> we can work around it, or if we need to block a particular version of the
> add-on.

I just installed the add-on on Windows 7 and Mac OS X 10.11 but it does not appear in Extensions using Firefox 41, 42 and 44. 'Programs and Features' says that the extension is installed but it's not displayed in any Firefox. 
I downloaded it from http://www.skype.com/en/download-skype/click-to-call/, also searched the web for an .xpi file but with no luck.
Any ideas why this happens?

Comment 12

[Jorge Villalobos [:jorgev] (he/him)]

Attachment: skype_click_to_call-7.5.0.9082-fx-windows.xpi

I'm attaching the version that was submitted to AMO for signing recently. It should be the latest one.

Comment 13

[Jorge Villalobos [:jorgev] (he/him)]

Also, note that the extension only works on Windows (also says so on their website).

Comment 15

[alex_mayorga]

¡Hola!

From https://bugzilla.mozilla.org/show_bug.cgi?id=1216860 where I requested the blocking of {82AF8DCA-6DE9-405D-BD5E-43525BDAD38A} not knowing it was "Skype Click to Call":

"Reasons: {82AF8DCA-6DE9-405D-BD5E-43525BDAD38A} correlates with Crash Reports for UserCallWinProcCheckWow per https://crash-stats.mozilla.com/report/list?range_unit=days&range_value=28&signature=UserCallWinProcCheckWow#tab-correlations and UserCallWinProcCheckWow is #2 top crash for Beta per https://crash-stats.mozilla.com/topcrasher/products/Firefox/versions/42.0b"

Reasons why it should be blacklisted IMHO:

 - It installs behind user's back with Skype (should be opt-in not opt-out
 - Causes performance degradation and seemingly even crashes
 - It is not uninstallable from within Firefox's Add-on management UI

¡Gracias!
Alex

Comment 16

[John Jensen]

> it should be blacklisted IMHO

+1. FWIW I discovered this on my Windows machine last week, when I noticed that my Firefox performance (on a smoking fast machine) had become unusable. I had to find and manually delete the directory to restore performance. I cannot imagine how bad the impact is for less knowledgeable users.

Comment 17

[Caspy7]

Changing the summary to reflect that these negative effects are not limited to Youtube.

Comment 18

[Chris Schanck]

This hit me hard; I switched to Chrome for 2 days out of frustration. Literally everything got slow and laggy. Super annoying. The Skype upgrade was within a day of me updating Firefox, so I did not initially connect Skype with the issue. 

I kept wanting to go back to Firefox, so I was looking at addons and googling around. I saw the reddit thread, saw that Click-To-Call was installed. I uninstalled it (from the Control panel) and restarted Firefox. Night and day. Scrolling, tabs, everything was fast again. 

Nasty, awful plugin.

Comment 19

[Chris Peterson [:cpeterson]]

Jorge, do we need any other information before (temporarily?) blocking this add-on?

This add-on has been shown to make Firefox unusable, break YouTube, is side-loaded by installing Skype, is not removable from Firefox's Add-ons Manager, and is correlated with the #2 top crash for Beta 42 (the "moment in time" marketing campaign release that will get a lot of press and user attention).

Comment 20

[Caspy7]

As I understand there is still the question of whether this issue sprung up because of the 41.0.2 update or the Skype update. Is that right?

If so we should probably test with the the addon (guessing the attached addon is the most recent) in 41.0.1.

Comment 21

[Mike Taylor [:miketaylr]]

(In reply to Jorge Villalobos [:jorgev] from comment #12)
> Created attachment 8676855 [details]
> skype_click_to_call-7.5.0.9082-fx-windows.xpi
> 
> I'm attaching the version that was submitted to AMO for signing recently. It
> should be the latest one.

After installing this XPI my CPU jumps to like 170% when trying to search for anything on Youtube--the UI locks up constantly. It's pretty terrible. 

I can't imagine the people switching to other browsers thinking Firefox is slow are ever going to switch back if and when there is a version that has acceptable performance.

Comment 22

[Chris Peterson [:cpeterson]]

(In reply to Caspy7 from comment #20)
> As I understand there is still the question of whether this issue sprung up
> because of the 41.0.2 update or the Skype update. Is that right?
> 
> If so we should probably test with the the addon (guessing the attached
> addon is the most recent) in 41.0.1.

A Skype contact says they "haven’t changed addon for a couple of years." That suggests this is a Firefox regression. It would be helpful is someone could test Firefox 40 to see if this is indeed a new problem in 41.0.1 or 41.0.2. If it is, then we can bisect the regression with mozregression.

Comment 23

[Jorge Villalobos [:jorgev] (he/him)]

Given the information we currently have, the Click to Call add-on has been marked as incompatible with Firefox 41 and above. This is effectively the same as blocklisting, since the add-on will be disabled for users.

This doesn't resolve this bug yet, since we still need to figure out if there's a regression in Firefox that can be fixed.

Comment 24

[Liz Henry (:lizzard) (relman/hg->git project)]

Sounds like many more people are seeing the problem.  

It's hard to know who should investigate the bug before we have much clue as to when it broke.  Maybe looking at what got uplifted between 41 and 41.0.2 would tell us something. 

Could this also be an issue in Windows itself (or some interference from an external program or another addon?) 

Bogdan, can you test again on Firefox 40 (on Windows) and look for a regression range?   I am not set up right now to test or find regressions on Windows. 

Or, if anyone else who's seen this issue can have a look, here are instructions on finding regression ranges:  https://mozilla.github.io/mozregression/   This can be a little bit time consuming, but the tool works very well!

Comment 25

[Bogdan Maris, Desktop QA]

Unfortunately using this version of the extension (comment 12) I can see that all releases are affected from Firefox 41 until Firefox 32.0.3 (that's as far as I went, did not check any further). 
I tested on two different machines with Windows 7 64-bit and Windows 10 64-bit.

The steps are quite easy:
1. Open Firefox
2. Open a youtube video and scroll the page
3. Open another video and scroll

The page will hang and freeze every time I visit a new video and scroll or jump to another tab and come back and scroll the page.

For now, mozregression will not work, not until bug 1216883 is fixed.

Please needinfo me if there is anything I can help with further.

Comment 26

[Serhiy]

Hi Bogdan Maris.
Could you please describe what other extensions were installed together with Click To Call ?

Comment 27

[Bogdan Maris, Desktop QA]

(In reply to Serhiy from comment #26)
> Hi Bogdan Maris.
> Could you please describe what other extensions were installed together with
> Click To Call ?

None, all my testing was done with fresh new profiles for each Firefox version.

Comment 28

[Caspy7]

Not saying the addon hasn't always caused a general performance problem, but at least two of the original reporters I linked point to the 41.0.2 update as the beginning of the problem for them.
Is there a way we can quantitatively compare, say, 41.0.1 & 41.0.2?  Maybe just look at CPU usage.  Might also include 41.0 on the possibility that they skipped 41.0.1 (I don't remember the time gap).

Comment 29

[Caspy7]

This also has me asking what we might do to better protect our users from addons which unduly hurt performance.  We've likely lost many users who have said that Firefox's performance sucks as a result of this.

And why the heck can't this addon be removed within the addons manager!?
If there's no rule against that, there should be. If there is, it should be grounds for blacklisting.

Comment 30

[Kev Needham [:kev]]

Please see comment #23. It won't be removed, because it's likely it's something that changed in Firefox, but the add-ons team is on it.

(In reply to Caspy7 from comment #29)
> And why the heck can't this addon be removed within the addons manager!?
> If there's no rule against that, there should be. If there is, it should be
> grounds for blacklisting.

Comment 31

[Jorge Villalobos [:jorgev] (he/him)]

(In reply to Caspy7 from comment #29)
> This also has me asking what we might do to better protect our users from
> addons which unduly hurt performance.  We've likely lost many users who have
> said that Firefox's performance sucks as a result of this.

Blocklisting is what we would do in these cases, if they are serious enough ("enough" being the key and highly subjective word in that sentence).

> And why the heck can't this addon be removed within the addons manager!?
> If there's no rule against that, there should be. If there is, it should be
> grounds for blacklisting.

Add-ons that are globally installed can't be removed from individual profiles, though disabling them is essentially the same. See bug 640775.

Comment 32

[Serhiy]

I can see some lags, Win 10 Home, FF 41.0.2, Core i7 4500 1.8GHz, 8GB Ram.
Turning off the plugin caused increasing of productivity while scrolling. But not dramatically.

Comment 33

[John Jensen]

(In reply to Caspy7 from comment #28)

> Is there a way we can quantitatively compare

Not exactly what you are looking for, but I asked the Performance team whether they had any data on the impact of this addon on performance. When looking at contributions to longer Firefox startups & shutdowns using Release-channel opt-in Telemetry data, they found that Skype Click-to-Call prolongs Firefox startup by an average of 771 milliseconds and prolongs shutdown by 12%. They also ran a correlation analysis between add-ons and crashes a while ago and found Skype Click-to-Call was the one of the worst offenders.

Comment 34

[Chris Peterson [:cpeterson]]

(In reply to Caspy7 from comment #28)
> Not saying the addon hasn't always caused a general performance problem, but
> at least two of the original reporters I linked point to the 41.0.2 update
> as the beginning of the problem for them.
> Is there a way we can quantitatively compare, say, 41.0.1 & 41.0.2?  Maybe
> just look at CPU usage.  Might also include 41.0 on the possibility that
> they skipped 41.0.1 (I don't remember the time gap).

Here are the release notes for Firefox 41.0.1 and 41.0.2:

https://www.mozilla.org/en-US/firefox/41.0.1/releasenotes/
https://www.mozilla.org/en-US/firefox/41.0.2/releasenotes/

41.0.1 included a fix for some Flash plugin hangs (bug 1185639), which has caused some problems with other NPAPI plugins. Does Skype Click-to-Call include an NPAPI plugin?

41.0.2 was a CORS security fix that looks unrelated:

https://www.mozilla.org/en-US/security/known-vulnerabilities/firefox/#firefox41.0.2

Comment 35

[Chris Peterson [:cpeterson]]

Actually, 41.0.2 also included:

bug 1209464 - fix some NPAPI plugin hangs
bug 1205987 - block Freecorder DLL for startup crashes

Comment 36

[[:philipp]]

bug 1205987 has never gone into release. comment #25 also concluded that testing didn't show that this would have been caused by a recent update. maybe a recent website change at youtube caused the increase of reports about this now...

Comment 37

[Bogdan Maris, Desktop QA]

Machine (1)
-Windows 7 64-bit
-AMD Radeon HD 6450
-AMD FX-8320 3.50 GHz 

Results:
I can't see any difference in CPU usage between this three builds (40.0, 41.0.1 and 41.0.2). They all vary from 0 to 20% with the add-on enabled or disabled. 

Machine (2)
-Windows 10 64-bit
-Intel HD Graphics 4400 1745 MB
-Intel Core i5 4200U 1.60 GHz

Results:
I can't see any difference in CPU usage between this three builds (40.0, 41.0.1 and 41.0.2) as well. They all vary from 0 to 50% with the add-on enabled or disabled.

Comment 38

[Chris Peterson [:cpeterson]]

Firefox users without Skype Click to Call also report Firefox freezing on YouTube:

https://www.reddit.com/r/youtube/comments/3lcxvj/firefox_started_freezing_on_youtube_a_lot_in/

Two users fixed their freezes by using uBlock to block YouTube XHR requests to fetch "||youtube.com/watch_fragments_ajax*":

https://www.reddit.com/r/youtube/comments/3lcxvj/firefox_started_freezing_on_youtube_a_lot_in/cvhihfr

Two other users fixed their freezes by uninstalling the Steam game client:

https://www.reddit.com/r/youtube/comments/3lcxvj/firefox_started_freezing_on_youtube_a_lot_in/cw14ycb

Comment 39

[Jorge Villalobos [:jorgev] (he/him)]

Since we haven't seen any clear evidence pointing to recent versions of the Skype add-on, I made a modification to the compatibility override. Now it flags older versions of the add-on, which are known to have serious performance problems. New versions, which are based on the SDK, are no longer flagged. We can flag them again if this change causes a new flood of complaints, so we should revisit this in a week or two.

Comment 40

[Caspy7]

(In reply to Jorge Villalobos [:jorgev] from comment #39)
> Since we haven't seen any clear evidence pointing to recent versions of the
> Skype add-on, I made a modification to the compatibility override. Now it
> flags older versions of the add-on, 

Please have a look at the links I posted again. Each one of them indicate that this is a recent occurrence. 

From one of them ( https://redd.it/3p52u7 )
> In the most recent update of Skype (I had no choice but to update it), Skype ended up installing an addon on Firefox called "Skype Click to Call"...

Yesterday a new post was made complaining of Firefox running slowly while Youtube videos are playing. https://redd.it/3r8xia After troubleshooting, the user found and removed the Skype Click to Call addon and the problem was resolved.

Comment 41

[Chris Peterson [:cpeterson]]

(In reply to Caspy7 from comment #40)
> (In reply to Jorge Villalobos [:jorgev] from comment #39)
> > Since we haven't seen any clear evidence pointing to recent versions of the
> > Skype add-on, I made a modification to the compatibility override. Now it
> > flags older versions of the add-on, 
> 
> Please have a look at the links I posted again. Each one of them indicate
> that this is a recent occurrence. 

Jorge, wouldn't we expect users to have the most recent version of the Skype Click to Call add-on? Shouldn't users have been receiving Skype updates, even if the add-on is not distributed through AMO?

Today is a particularly bad time to unblock Skype Click to Call because today is the kick-off of the big Firefox 42 marketing campaign.

Comment 42

[Jorge Villalobos [:jorgev] (he/him)]

(In reply to Chris Peterson [:cpeterson] from comment #41)
> Jorge, wouldn't we expect users to have the most recent version of the Skype
> Click to Call add-on? Shouldn't users have been receiving Skype updates,
> even if the add-on is not distributed through AMO?

Not necessarily. It depends on how they handle automatic updates, if at all. If the add-on is only installed or updated through their application installer, it's likely that many users would be stuck with older versions. I don't have any stats on their version distribution to tell for sure.

> Today is a particularly bad time to unblock Skype Click to Call because
> today is the kick-off of the big Firefox 42 marketing campaign.

If there was clear evidence that this add-on (the latest version in particular) is the culprit, rather than YouTube or other factors, I would keep the block up. We don't have clear steps to reproduce and others have reported the problem without the Skype add-on installed, so I don't think it's fair to continue blocking it.

Comment 43

[Robert Kaiser]

It looks now that in Firefox 43 Beta 1 we have a shutdownhang that is almost 3% of our crash volume on that beta version right now, for which practically every crash report has Skype Click-to-Call installed (add-on ID {82AF8DCA-6DE9-405D-BD5E-43525BDAD38A} seems to be that), a search for those shutdownhang reports is https://crash-stats.mozilla.com/signature/?product=Firefox&process_type=browser&process_type=content&version=43.0b1&signature=shutdownhang+|+WaitForSingleObjectEx+|+WaitForSingleObject+|+PR_WaitCondVar+|+mozilla%3A%3ACondVar%3A%3AWait&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&page=1#reports

Comment 44

[Jorge Villalobos [:jorgev] (he/him)]

Do we know what kind of code would cause these hangs? Holding on to content references from browser code, maybe?

Comment 45

[Fabio Rios [:frios]]

Attachment: whatisthis.JPG

FYI - I'm getting this. I didn't install anything, just started showing up.

I'm on Windows 10. This is Firefox Beta 43, but also showed up last week when 42 was running.

Comment 46

[Jorge Villalobos [:jorgev] (he/him)]

It should only show up after you install or update Skype. The application will drop the extension and Firefox will try to load it next time it starts. That window in the screenshot gives you the option to enable the extension if you want to use it.

Comment 47

[bull500]

(In reply to Jorge Villalobos [:jorgev] from comment #46)
> It should only show up after you install or update Skype. The application
> will drop the extension and Firefox will try to load it next time it starts.
> That window in the screenshot gives you the option to enable the extension
> if you want to use it.

I think that message keeps appearing as long as you deny it.  
I've encountered it in the past. Don't remember the current situation though.

Comment 48

[Robert Kaiser]

OK, and I guess after you allow this installation, Firefox hangs on shutdown and then is force-killed by us and brings up a crash reporter.

Comment 49

[Liz Henry (:lizzard) (relman/hg->git project)]

This is the #2 topcrash for 43.0b1 right now with 2008/22510 crashes in the last 7 days. 
Tracking for 45 as well since it seems to affect Nightly too.

Comment 50

[Chris Peterson [:cpeterson]]

Jorge, can we mark this add-on as incompatible with 41+ again? Or at least 41–44 if we want to watch the crash rate on 45. This is the #17 topcrash on 42.

Comment 51

[alex_mayorga]

(In reply to Fabio Rios [:frios] from comment #45)
> Created attachment 8683779 [details]
> whatisthis.JPG
> 
> FYI - I'm getting this. I didn't install anything, just started showing up.
> 
> I'm on Windows 10. This is Firefox Beta 43, but also showed up last week
> when 42 was running.

¡Hola Fabio!

This is a "Third Party Add-on Warnings"[1].

Given the negative impact of this add-on, you might want to uninstall it, see "How to Disable or Remove the Click and Call"[2]

¡Gracias!
Alex

1 https://wiki.mozilla.org/Extension_Manager:Projects:Third_Party_Add-on_Warnings
2 http://community.skype.com/t5/Other-features/How-to-Disable-or-Remove-the-Click-and-Call-Function/m-p/69602

Comment 52

[Jorge Villalobos [:jorgev] (he/him)]

I'm marking all current versions as incompatible with Firefox 43 and above, to see what impact it has on the shutdown crashes. If there's no impact, I'll revert this change. If there's noticeable impact, I'll extend the flag to 42.

Comment 53

[Arthur K. (he/him)]

Has this been replicated with the 7.14.0.104 just released yesterday? Sometimes Skype's auto-update feature lags behind what's current.

Comment 54

[Jorge Villalobos [:jorgev] (he/him)]

Has there been any change in the crash stats for Firefox 43 and above, Kairo, or is it too early to tell?

Comment 55

[Liz Henry (:lizzard) (relman/hg->git project)]

We're still seeing the crash today, in builds from today and yesterday. example: https://crash-stats.mozilla.com/report/index/3aab7c1a-8fa7-4d9b-b793-3e4e62151112

Comment 56

[Liz Henry (:lizzard) (relman/hg->git project)]

Still a huge top crash for 43.0b3. Is this from the beta e10s experiment or from the Skype addon or something else? Emailing crash-debug list to ask someone to investigate.

Comment 57

[Jorge Villalobos [:jorgev] (he/him)]

I'd like to know (if possible) which versions of the Skype add-on are causing the crashes.

Comment 58

[Robert Kaiser]

(In reply to Jorge Villalobos [:jorgev] from comment #57)
> I'd like to know (if possible) which versions of the Skype add-on are
> causing the crashes.

  shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | PR_WaitCondVar | mozilla::CondVar::Wait|EXCEPTION_BREAKPOINT (440 crashes)
     93% (407/440) vs.  14% (1878/13722) {82AF8DCA-6DE9-405D-BD5E-43525BDAD38A}
          0% (0/440) vs.   0% (2/13722) 7.4.0.9058
         93% (407/440) vs.  14% (1876/13722) 7.5.0.9082

So this is version 7.5.0.9082 according to correlations.

Comment 59

[Arthur K. (he/him)]

(In reply to Robert Kaiser (:[e-mail]) from comment #58)
> (In reply to Jorge Villalobos [:jorgev] from comment #57)
> > I'd like to know (if possible) which versions of the Skype add-on are
> > causing the crashes.
> 
>   shutdownhang | WaitForSingleObjectEx | WaitForSingleObject |
> PR_WaitCondVar | mozilla::CondVar::Wait|EXCEPTION_BREAKPOINT (440 crashes)
>      93% (407/440) vs.  14% (1878/13722)
> {82AF8DCA-6DE9-405D-BD5E-43525BDAD38A}
>           0% (0/440) vs.   0% (2/13722) 7.4.0.9058
>          93% (407/440) vs.  14% (1876/13722) 7.5.0.9082
> 
> So this is version 7.5.0.9082 according to correlations.

So likely fixable by installing 7.14.0.104?

Comment 60

[Jorge Villalobos [:jorgev] (he/him)]

7.5.0.9082 is the latest version available on AMO. It's probably the one most users have installed.

It looks like the compatibility override isn't working as expected, so I'll switch to a block, still limited to 43 and above, and 7.5.0.9082 and lower.

Comment 61

[alex_mayorga]

¡Hola!

Just dropping another breadcrumb to highlight that even newer releases might be causing crashes.

Please see the SUMO question at https://support.mozilla.org/en-US/questions/1096077#answer-811101

The user blames on "Skype agent 7.7.0.103" FWIW

¡Gracias!

Comment 62

[Liz Henry (:lizzard) (relman/hg->git project)]

This signature is still showing up in crash-stats at a fairly high volume for 42 and 43. Are these crashes still mainly caused by the Skype addon?  Jorge, can we block all versions of it?

Comment 63

[Jorge Villalobos [:jorgev] (he/him)]

Done. It's still blocked only on 43 and above, so we need to monitor these crashes in the next week or so to see if the block should remain before the release. If the block isn't being effective, then I don't think there's much we can do.

Comment 64

[Chris Peterson [:cpeterson]]

Liz, has this signature's crash volume changed since the Skype add-on was blocked last week?

Comment 65

[Liz Henry (:lizzard) (relman/hg->git project)]

It was showing up in the RC build and later betas even more often than in earlier betas. I don't see any reports from 43.0 yet. I'll ask on crash-debug for someone to look into this again.

Comment 66

[(No longer employed by Mozilla) Aaron Klotz]

I've looked at 3 or 4 of these dumps and they all show us waiting on cache2 I/O. Honza, could you please comment on why the CLOSE event is a lower priority than other cache2 I/O commands? It seems to me like, given the nature of cache data, we should accept some data loss upon shutdown instead of waiting for all I/O events to happen. I think we're seeing queue starvation where the CLOSE command is never high enough priority to execute on the CacheFileIOManager thread.

Comment 67

[Robert Kaiser]

(In reply to Aaron Klotz [:aklotz] (please use needinfo) from comment #66)
> I've looked at 3 or 4 of these dumps and they all show us waiting on cache2
> I/O. Honza, could you please comment on why the CLOSE event is a lower
> priority than other cache2 I/O commands?

For one thing, we also see bug 1226941 with decent crash volume being new in 43, which is cache IO as well - for the other, closing HTTP connection has been timing out for a long time, see bug 1158189 and bug 1152046, just in case that might be related as well.

Comment 68

[Jorge Villalobos [:jorgev] (he/him)]

Note that bug 1225639 was reverted from "all versions" back to "up to 7.5.0.9082", on December 15th. If we want to change this again, I'll need some clear data of the crash rates and how they are being affected by the blocks.

Comment 69

[Robert Kaiser]

On 43.0.1 this is still pretty much correlated to Skype Click-to-Play:
  shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | PR_WaitCondVar | mozilla::CondVar::Wait|EXCEPTION_BREAKPOINT (2850 crashes)
     86% (2448/2850) vs.  13% (8494/65323) {82AF8DCA-6DE9-405D-BD5E-43525BDAD38A}

That said, it might not be causes by the add-on, maybe Skype running on the system creates an issue with HTTP connections running long?

Comment 70

[(No longer employed by Mozilla) Aaron Klotz]

My guess is that Skype is responsible for the requests that are starving the cache shutdown event.

Comment 71

[(No longer employed by Mozilla) Aaron Klotz]

Really it should not be possible for addons or content to starve cache shutdown in the first place. This is a cache bug as far as I'm concerned.

Comment 72

[Sylvestre Ledru [:Sylvestre]]

42 is gone.

Comment 73

[Liz Henry (:lizzard) (relman/hg->git project)]

Still the #2 topcrash for 43.0.2,  4.7% of crashes reported. Mostly for Win7 and 10.

Comment 74

[Liz Henry (:lizzard) (relman/hg->git project)]

Vladan, can you help find someone to have another look? This is a pretty bad crash for 43, also affects 44+.   
This may not be a "tech evangelism" bug at all as it turns out. I am guessing maybe Aaron means Core::Networking:Cache.

Comment 75

[Liz Henry (:lizzard) (relman/hg->git project)]

Nicolas is this something you could look into?

Comment 76

[Nicholas Nethercote [inactive]]

(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #75)
> Nicolas is this something you could look into?

I'm on PTO until January 4th so I wouldn't be able to look into it until then. Even then, not much about this bug (add-ons, network cache) falls into my areas of expertise, so I don't know if I'll be able to help much. Sorry.

Comment 77

[Vladan Djeric (:vladan)]

(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #74)
> Vladan, can you help find someone to have another look? This is a pretty bad
> crash for 43, also affects 44+.   
> This may not be a "tech evangelism" bug at all as it turns out. I am
> guessing maybe Aaron means Core::Networking:Cache.

I spoke to Aaron Klotz about this crash before he went on PTO. He said it's a network cache issue and that since Honza is on PTO, we should contact Jason Duell. Aaron said he had already briefed jduell on this issue.

Jason, can you take a look?

Comment 78

[Jason Duell]

> could you please comment on why the CLOSE event is a lower priority than other 
> cache2 I/O commands? 

In the general case of browsing we get significantly faster page load times by prioritizing cache hits (reads) over storing new cache entries (writes), which then get stored as things quiet down.  I assume the same is true for lowering Close event priority.

> Really it should not be possible for addons or content to starve cache shutdown in the
> first place. This is a cache bug as far as I'm concerned.

Well, in the general case with a priority scheme where lower priorities never get run if higher ones gobble all the time, you could make the case that the gobbler is the bug.  But I agree we shouldn't let this particular bug block shutdown.

> It seems to me like, given the nature of cache data, we should accept some data loss 
> upon shutdown instead of waiting for all I/O events to happen

Yes, I'd be fine with letting cache entries get corrupted so we could exit earlier (we detect corrupt entries and toss them next time they're used).  But I'm not sure how easy that will be to implement.  I'm hoping :michal or :mayhemer can give us an answer (Can we have the cache service signal it's done with shutdown--maybe using some sort of main thread timer?-- even if all pending cache writes aren't done yet?)

Comment 79

[Honza Bambas (:mayhemer)]

It can well be bug 1152046, which happens for close() on either sockets or files.  Then, it can also be a justified file close delay.  close() call on a file may take some time since it means to flush the buffers, that is the reason it has a lower priority, OTOH see bug 996836 to close sooner.  I don't know from top of my head a way to let close() be just fire-and-forget.  True is we don't much care about the success, since this is just a cache.  One of my ideas for shutdown optimization was to simply let cache files leak (never close them) and let the system handle it when it takes too long to close them all, already having bug 913822 for that.

Comment 80

[Liz Henry (:lizzard) (relman/hg->git project)]

I am still hoping we can fix this for 44, but at this point think it's unlikely this will be fixed on 43. If that changes or you disagree strongly, please let me know with needinfo.

Comment 81

[Michal Novotny [:michal]]

(In reply to Honza Bambas (:mayhemer) from comment #79)
> One of my ideas for shutdown optimization was to simply let cache files leak
> (never close them) and let the system handle it when it takes too long to
> close them all, already having bug 913822 for that.

I don't think the problem is in closing the file on the system level. Don't forget that we cannot have more than 64 opened files at the same time, so how long it could take to close 64 files even on a slow device? The problem is IMO in flushing the unwritten data. Instead of leaking the data I would prefer to doom the unfinished entries. It should be pretty fast and we'll likely end up with an up-to-date cache index on the disk. If we leave corrupted entries on the disk intentionally we would need to update the index almost on every start.

Comment 82

[Honza Bambas (:mayhemer)]

(In reply to Michal Novotny (:michal) from comment #81)
> (In reply to Honza Bambas (:mayhemer) from comment #79)
> > One of my ideas for shutdown optimization was to simply let cache files leak
> > (never close them) and let the system handle it when it takes too long to
> > close them all, already having bug 913822 for that.
> 
> I don't think the problem is in closing the file on the system level. Don't
> forget that we cannot have more than 64 opened files at the same time, so
> how long it could take to close 64 files even on a slow device? 

On a busy and slow device easily one or more minutes?  I've seen this.

> The problem
> is IMO in flushing the unwritten data. Instead of leaking the data I would
> prefer to doom the unfinished entries. 

But that doesn't solve anything.  Dooming means to do I/O.

> It should be pretty fast 

Are you sure about that?  It means to: close the file, rename the file.  Each of actively open.  It's actually worse than to just close them in inconsistent state.

> and we'll
> likely end up with an up-to-date cache index on the disk. If we leave
> corrupted entries on the disk intentionally we would need to update the
> index almost on every start.

Again, are you sure about this?  If we find an entry (a file) that is corrupted, do we really start updating the index?  Maybe yes, just want you to be 100% sure of that.  Also, if yes, should we do the index update for that case?  Index should just not lie about "not having something", but it's probably OK to claim we do what is later found false (found corrupted).

Sometimes being inaccurate is better than being slow.

Comment 83

[Michal Novotny [:michal]]

(In reply to Honza Bambas (:mayhemer) from comment #82)
> (In reply to Michal Novotny (:michal) from comment #81)
> > (In reply to Honza Bambas (:mayhemer) from comment #79)
> > > One of my ideas for shutdown optimization was to simply let cache files leak
> > > (never close them) and let the system handle it when it takes too long to
> > > close them all, already having bug 913822 for that.
> > 
> > I don't think the problem is in closing the file on the system level. Don't
> > forget that we cannot have more than 64 opened files at the same time, so
> > how long it could take to close 64 files even on a slow device? 
> 
> On a busy and slow device easily one or more minutes?  I've seen this.

What exactly have you seen? I was talking about closing the file handle, not about cleanly closing the entry file (i.e. writing all unwritten data and metadata). Calling close() in Linux doesn't flush the buffers and I guess that the same applies to Windows.


> > The problem
> > is IMO in flushing the unwritten data. Instead of leaking the data I would
> > prefer to doom the unfinished entries. 
> 
> But that doesn't solve anything.  Dooming means to do I/O.

Sure, but far less I/O than writing data and metadata.


> > It should be pretty fast 
> 
> Are you sure about that?  It means to: close the file, rename the file. 
> Each of actively open.  It's actually worse than to just close them in
> inconsistent state.

Again, I'm comparing it with the current code, not with "leak everything" solution. 


> > and we'll
> > likely end up with an up-to-date cache index on the disk. If we leave
> > corrupted entries on the disk intentionally we would need to update the
> > index almost on every start.
> 
> Again, are you sure about this?  If we find an entry (a file) that is
> corrupted, do we really start updating the index?  Maybe yes, just want you
> to be 100% sure of that.  Also, if yes, should we do the index update for
> that case?  Index should just not lie about "not having something", but it's
> probably OK to claim we do what is later found false (found corrupted).

Hmm, from a quick look at the code, it seems this wouldn't start updating process.

Comment 84

[Chris Peterson [:cpeterson]]

It doesn't sound like a cache fix will be a simple fix or ready to ship in 44. If Skype Click-to-Play is still the #2 top crash for 43, can we block the add-on again (like bug 1225639)?

Comment 85

[Robert Kaiser]

(In reply to Chris Peterson [:cpeterson] from comment #84)
> It doesn't sound like a cache fix will be a simple fix or ready to ship in
> 44. If Skype Click-to-Play is still the #2 top crash for 43, can we block
> the add-on again (like bug 1225639)?

No, we already tried that and it didn't help The add-on itself seems to not be the cause, even though it's correlated.

Comment 86

[Honza Bambas (:mayhemer)]

(In reply to Michal Novotny (:michal) from comment #83)
> (In reply to Honza Bambas (:mayhemer) from comment #82)
> > (In reply to Michal Novotny (:michal) from comment #81)
> > > (In reply to Honza Bambas (:mayhemer) from comment #79)
> > > > One of my ideas for shutdown optimization was to simply let cache files leak
> > > > (never close them) and let the system handle it when it takes too long to
> > > > close them all, already having bug 913822 for that.
> > > 
> > > I don't think the problem is in closing the file on the system level. Don't
> > > forget that we cannot have more than 64 opened files at the same time, so
> > > how long it could take to close 64 files even on a slow device? 
> > 
> > On a busy and slow device easily one or more minutes?  I've seen this.
> 
> What exactly have you seen? I was talking about closing the file handle, not
> about cleanly closing the entry file (i.e. writing all unwritten data and
> metadata). Calling close() in Linux doesn't flush the buffers and I guess
> that the same applies to Windows.

I remember my tests with an sd card.  Every operation took a large amount of time, write(), close(), open(), each 100-150 ms.  There were just the profile stored, on a busy media it can IMO be even slower.  HDD on a swappy system can similarly slow down as well.

> 
> 
> > > The problem
> > > is IMO in flushing the unwritten data. Instead of leaking the data I would
> > > prefer to doom the unfinished entries. 
> > 
> > But that doesn't solve anything.  Dooming means to do I/O.
> 
> Sure, but far less I/O than writing data and metadata.

But still I/O.  Depends, we would have to try if that changes anything or if we have to leak completely.

> 
> 
> > > It should be pretty fast 
> > 
> > Are you sure about that?  It means to: close the file, rename the file. 
> > Each of actively open.  It's actually worse than to just close them in
> > inconsistent state.
> 
> Again, I'm comparing it with the current code, not with "leak everything"
> solution. 
> 
> 
> > > and we'll
> > > likely end up with an up-to-date cache index on the disk. If we leave
> > > corrupted entries on the disk intentionally we would need to update the
> > > index almost on every start.
> > 
> > Again, are you sure about this?  If we find an entry (a file) that is
> > corrupted, do we really start updating the index?  Maybe yes, just want you
> > to be 100% sure of that.  Also, if yes, should we do the index update for
> > that case?  Index should just not lie about "not having something", but it's
> > probably OK to claim we do what is later found false (found corrupted).
> 
> Hmm, from a quick look at the code, it seems this wouldn't start updating
> process.

So, leaving corrupted items doesn't trigger an index update.  That speaks in favor of leaking completely.  Good to know, thanks.

Comment 87

[Honza Bambas (:mayhemer)]

Marking as dep on bug 913822, let's move forward here.

Comment 88

[Paul Pasca[Away. Please needinfo? Paul Oiegas [:pauloiegasSV]]]

Hi reporter,

I haven't managed to reproduce this issue on the latest release(44.0) nor latest Nightly(47.0a1). Installing the latest Skype click to click version does not reproduce the issue. The attachment from comment 12 can't be installed on the latest Firefox versions, the add-on is blocked for security issues.

User Agent: Mozilla/5.0 (Windows NT 6.1; rv:44.0) Gecko/20100101 Firefox/44.0
Build ID: 20160123151951

User Agent: Mozilla/5.0 (Windows NT 6.1; rv:47.0) Gecko/20100101 Firefox/47.0
Build ID: 20160128030208

I've also tested on Firefox 40.0, 41.0, 41.01, 41.0.2 using the latest click to call version and the issue did not occurred. The attachment from comment 12 does not reproduce the issue on this versions as well. I need to find a bad build in order to perform a regression on this bug, but I'm unable to reproduce this on any Firefox version.

Can you please try to reproduce this on the latest release(44.0) and latest Nightly(47.0a1) and provide the results? When doing this, please try to reproduce with a new clean Firefox profile, maybe even in safe mode, as some of this issues may be caused by third party installed add-ons or custom settings(https://support.mozilla.org/en-US/kb/troubleshoot-and-diagnose-firefox-problems).

Thanks,
Paul.

Comment 89

[Jonathan Howard]

The current experiment is showing a high link with skype click to call.
Skype was blocked to remove an unrelated bug 1210099 in the e10s side.

0.74% [1] The control group; no e10s, mostly excludes skype and also excluded a11y.
2.54% [2] Beta group; excluding e10s, includes a11y and all of the above.

To me the sample size seems large enough to see difference but not to infer any specific fraction.
A11y does not appear to be making any impact.
I can't rule out (with such a simple search) anything coincidental such as perhaps the skype install base may have significantly lower memory.
Patch from bug 1210099 hasn't yet landed, I don't have knowledge of what the fix is doing to rule it out as solution.

[1] https://crash-stats.mozilla.com/search/?product=Firefox&release_channel=beta&ActiveExperiment=%3De10s-beta45-withaddons%40experiments.mozilla.org&ActiveExperimentBranch=%3Dcontrol&date=%3E%3D2016-02-05&date=%3C2016-02-10&build_id=%3E%3D20160204142810&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature
[2] https://crash-stats.mozilla.com/search/?product=Firefox&release_channel=beta&date=%3E%3D2016-02-05&date=%3C2016-02-10&build_id=%3E%3D20160204142810&dom_ipc_enabled=__null__&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature

Comment 90

[Robert Kaiser]

(In reply to Jonathan Howard from comment #89)
> The current experiment is showing a high link with skype click to call.
> Skype was blocked to remove an unrelated bug 1210099 in the e10s side.
> 
> 0.74% [1] The control group; no e10s, mostly excludes skype and also
> excluded a11y.
> 2.54% [2] Beta group; excluding e10s, includes a11y and all of the above.

The shutdownhangs in networking are high on both e10s and non-e10s data and even on 44.0 release, so this is not related to the e10s experiment by itself. Your data just confirms that this is largely (but not completely) correlated to having the Skype CtP add-on, which we established already. We also know already that blocking the add-on doesn't change the frequency of those shutdownhangs, so it may actually be correlated with Skype itself being active on the computer but not caused by the add-on itself.

Comment 91

[Michal Novotny [:michal]]

There is a lot of crash reports with Cache I/O thread removing a file, like this:
https://crash-stats.mozilla.com/report/index/78ee6f2d-66c0-4dfb-a1e8-aceef2160207#allthreads

I assume that Cache IO lag detection is not disabled either by DEBUG define or by preference. So if we spent more that 2 seconds shutting down the cache, then the file was doomed because mInvalid was set to false in ReleaseNSPRHandleInternal(). It's strange that we would have so much doomed files that would cause such delay during shutdown. Anyway we don't have to remove these files because they are already in doomed directory and will be removed on next startup.


There are also few reports with Cache I/O thread reading the data, e.g.
https://crash-stats.mozilla.com/report/index/26a4e1bc-69b4-4259-8f8b-160ef2160208#allthreads
https://crash-stats.mozilla.com/report/index/040ede72-496f-435b-8f14-ede2d2160208#allthreads

I wouldn't expect many read requests during shutdown so I have no theory about this.


And there are also some reports where we're not in CacheFileIOManager::Shutdown() on the main thread and the Cache I/O thread exists, so the process was killed before we started shutting down the cache, e.g.
https://crash-stats.mozilla.com/report/index/8375d922-23fd-4877-a275-2d60b2160208#allthreads
https://crash-stats.mozilla.com/report/index/49740c41-5e47-4611-bb3f-cb3f32160209#allthreads
https://crash-stats.mozilla.com/report/index/5bbbc06e-1d60-425d-a443-b2bef2160207#allthreads
https://crash-stats.mozilla.com/report/index/2df1ec4b-6b97-4aa5-a8d1-7fae92160209#allthreads

So the question here is whether cache causes the majority of the shutdown delay in other cases where it's on the stack. OTOH we would probably have more reports without CacheFileIOManager::Shutdown() if there was another code blocking the shutdown.

Comment 92

[bull500]

Not sure if you people know, if you do forgive me for the spam.  

2nd post that popped up recently where Skype click-to-call was causing connectivity problems for the user  
https://www.reddit.com/r/firefox/comments/453nn2/recent_issue_with_firefox/  
https://www.reddit.com/r/firefox/comments/445z94/at_a_loss_with_this_firefox_suddenly_stops/

Comment 93

[Jason Duell]

Just re-read all 90 comments.  My team is currently short on ideas here.  But I wanted to summarize, to see if I understand correctly:

1) In general, skype addon often causes horrible performance (parsing way too much text).  Yet somehow this wasn't so bad before FF 41?
2) Disabling the addons seems to fix that, but not the top 2 shutdown crash involving cache2 I/O.
3)  Specifically makes youtube really slow (did disabling the addon fix that?)

Given #2, I'm wondering if the addon (or skype in general) is installing an LSP that messes with our networking.  I'm having trouble thinking of how else a disabled addon could cause this.

Comment 94

[Jason Duell]

Honza is installing the addon on a windows 10 VM and will see if he can make any traction here.

Comment 95

[Jason Duell]

I suspect we should split this bug into 2 bugs:  one for the slowness-during-browsing when the addon is enabled (for which the fix is either to block it, or possibly figure out why it started happening more recently), and the other for the shutdown crashes.  Objections?

Comment 96

[Jason Duell]

Do we have any data on whether there's a click-to-call addon for other browsers, and if so, if they're seeing the same slowdowns?

Comment 97

[Robert Kaiser]

(In reply to Jason Duell [:jduell] (needinfo? me) from comment #95)
> I suspect we should split this bug into 2 bugs:  one for the
> slowness-during-browsing when the addon is enabled (for which the fix is
> either to block it, or possibly figure out why it started happening more
> recently), and the other for the shutdown crashes.  Objections?

I'm all for that - and I actually prefer both to be new bugs, with clear descriptions, and this being closed afterwards, so that confusion will be avoided.

Comment 98

[Honza Bambas (:mayhemer)]

(In reply to Jason Duell [:jduell] (needinfo? me) from comment #95)
> I suspect we should split this bug into 2 bugs:  one for the
> slowness-during-browsing when the addon is enabled (for which the fix is
> either to block it, or possibly figure out why it started happening more
> recently), and the other for the shutdown crashes.  Objections?

I don't think it makes any sense.


Looking at the reports here: https://crash-stats.mozilla.com/report/list?signature=shutdownhang%20|%20WaitForSingleObjectEx%20|%20WaitForSingleObject%20|%20PR_WaitCondVar%20|%20mozilla%3A%3ACondVar%3A%3AWait#tab-comments

(I hope it's the right link).

Looking at the comments, many of them talk about slowness (pages don't load etc).  I've checked 9 of reports like that.  


*All* of them had installed *Skype Click to Call* AND also some *antivirus software*.

I found: 
bitdefender 3.12.13582.6399
AVG: 16.12.0.7294, 16.41.0.7442, 16.41.0.7441
BullGuard Spamfilter Helper: 16.0.0.8
Avira: unknown version
Trusteer Rapport: 3.5.1507.109, 3.5.1507.104
Avast: 11.1.2253.1653

Going to check if installing one of them makes me reproduce the problem.

Comment 99

[Arthur K. (he/him)]

(In reply to Honza Bambas (:mayhemer) from comment #98)
> (In reply to Jason Duell [:jduell] (needinfo? me) from comment #95)
> > I suspect we should split this bug into 2 bugs:  one for the
> > slowness-during-browsing when the addon is enabled (for which the fix is
> > either to block it, or possibly figure out why it started happening more
> > recently), and the other for the shutdown crashes.  Objections?
> 
> I don't think it makes any sense.
> 
> 
> Looking at the reports here:
> https://crash-stats.mozilla.com/report/
> list?signature=shutdownhang%20|%20WaitForSingleObjectEx%20|%20WaitForSingleOb
> ject%20|%20PR_WaitCondVar%20|%20mozilla%3A%3ACondVar%3A%3AWait#tab-comments
> 
> (I hope it's the right link).
> 
> Looking at the comments, many of them talk about slowness (pages don't load
> etc).  I've checked 9 of reports like that.  
> 
> 
> *All* of them had installed *Skype Click to Call* AND also some *antivirus
> software*.
> 
> I found: 
> bitdefender 3.12.13582.6399
> AVG: 16.12.0.7294, 16.41.0.7442, 16.41.0.7441
> BullGuard Spamfilter Helper: 16.0.0.8
> Avira: unknown version
> Trusteer Rapport: 3.5.1507.109, 3.5.1507.104
> Avast: 11.1.2253.1653
> 
> Going to check if installing one of them makes me reproduce the problem.

As a user of both Skype (sans Click to Call) and Avira, one thing did pop into my head about interfering component of Antivirus when I read this post. On a standard Avira install, they will install "Web Protection" which I have seen make somewhat intrusive changes to browser traffic and affect network performance.

On many customer PCs I deal with, I disable this component via (Windows 7) Control Panel > Uninstall a Program > right-click Avira Antivirus and choose Change > Modify > uncheck Web Protection.

It seems to dramatically improve the browser experience and stop any wonky network slowness or undesired behavior. I can't speak to the other A/V or if they install some similar component but it could be possible. Just wanted to mention this in case it's of use to you devs. Of course, best way to test is install Skype + Click to Call and Avira with default settings and see if unwanted behavior can be replicated.

Comment 100

[Honza Bambas (:mayhemer)]

I know the cause, at least roughly.

I did the following:
- installed Win10 Pro in virtualbox (currently unactivated, version 10240)
- installed Firefox
- installed Skype + the extension (which is the culprit, probably!)
- installed Trusteer Rupport (probably doesn't have a bit effect on its own)
- installed Bitdefender Total Security 2016 (everything slows down... a lot!)

I cannot reproduce the shutdown crash, but I can see Fx is taking very long time to shutdown.  Cache logging reveals we are closing 2500 files! (after browsing two facebook pages and my blog).

In 5 minutes the Skype extension makes almost 4000 POST requests to https://localhost:26143/skypectoc/v1/pnr/parse.  Those mostly leak (are held in memory as open and active entries) during most of the session.  This has following effects:

- purging on overmemory limit takes enormous amount of time (up to 200ms) because there is huge number of cache entries active that cannot be released (are still used - we leak the XHR requests probably) and is done very very often during shutdown (400 times in my case)
- closing files takes also some time, caused by the AV software probably


So, I can see following bugs to fix here:
- bypass unneeded cache IO after shutdown (already reported as bug 1247432)
- do cache entry memory purge less often
- find out why we leak the XHR requests

Comment 101

[Honza Bambas (:mayhemer)]

(In reply to Honza Bambas (:mayhemer) from comment #100)
> - purging on overmemory limit takes enormous amount of time (up to 200ms)
> because there is huge number of cache entries active that cannot be released
> (are still used - we leak the XHR requests probably) and is done very very
> often during shutdown (400 times in my case)

Err.. not that often.  Only ~25 times, took around a 1000ms total.  This is not the bottle neck.

Comment 102

[Honza Bambas (:mayhemer)]

I can see up to 250 requests during one second, most of them are the skype POSTs.

Comment 103

[Patrick McManus [:mcmanus]]

awesome info honza.

Comment 104

[Honza Bambas (:mayhemer)]

Locally reproduced as f2424747-ba2a-4332-841c-9cab82160211

Comment 105

[Honza Bambas (:mayhemer)]

I think as a quick fix we should block the add-on for a while.  

I can start working on the fixes tomorrow, which may or may not help (!), it will take few days to push them up tho.

Comment 106

[Honza Bambas (:mayhemer)]

The SCtC add-on does every 250ms (a JS interval) a burst of these POSTs for every DOM element that has been mutated (roughly speaking).  Facebook for instance, IIRC, does a lot of DOM modifications during load.  Each burst loop may take only up to 75ms.  Depends on how many requests we are able to make during those 75ms and how many times this scheduling interval is being set up.  I'm not going to investigate further on this add-on evil.  This is just to support the previous findings.

Comment 107

[Honza Bambas (:mayhemer)]

Attachment: example log of one of the POSTs handling in context of shutdown

Captured with release build.  This is only one of the huge number of SCtC XHR requests filtered out from the log.  It's scheduled long ago before shutdown.  Muting cache after shutdown won't have any effect here.  We were able to server 325 requests between shutdown and the final crash (around a minute and half - I've raised the crash timeout limit).

Note: the loopback server is http/2.  Transactions are then probably serialized on a single connection.

Comment 108

[Honza Bambas (:mayhemer)]

Caused by bug 1176988.

Comment 109

[Jorge Villalobos [:jorgev] (he/him)]

I'm pinging Skype about the issues brought up in comment #100 and later.

Comment 110

[Honza Bambas (:mayhemer)]

Confirmed fix by all the open blocking bugs.  Feel free to try a build with all of them at: http://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-667bc07823888fd66de37d40da4ff1993b98737d/

IMPORTANT: Don't forget to copy C:\Program Files (x86)\Mozilla Firefox\browser\extensions\{82AF8DCA-6DE9-405D-BD5E-43525BDAD38A}.xpi to C:\Program Files (x86)\Nightly\browser\extensions directory and allow it!

Comment 111

[Jorge Villalobos [:jorgev] (he/him)]

Honza, which version of the Skype extension are you using for your tests?

Comment 112

[Honza Bambas (:mayhemer)]

(In reply to Jorge Villalobos [:jorgev] from comment #111)
> Honza, which version of the Skype extension are you using for your tests?

8.0.0.9103

Comment 113

[Honza Bambas (:mayhemer)]

The major offender bug 1248049 has landed.