1205987 - startup crash in PR_Read | GetStatusFileContents<T> with freecorder app

Status: RESOLVED WORKSFORME

(Core :: General, defect)

Description

[[:philipp]]

This bug was filed from the Socorro interface and is 
report bp-5117bd90-9b8f-4d27-8547-5e3b52150917.
=============================================================

Crashing Thread
Frame 	Module 	Signature 	Source
0 	nss3.dll 	PR_Read 	nsprpub/pr/src/io/priometh.c
1 	xul.dll 	GetStatusFileContents<32> 	toolkit/xre/nsUpdateDriver.cpp
2 	xul.dll 	GetUpdateStatus 	toolkit/xre/nsUpdateDriver.cpp
3 	xul.dll 	ProcessUpdates(nsIFile*, nsIFile*, nsIFile*, int, char**, char const*, bool, bool, nsIFile*, void**) 	toolkit/xre/nsUpdateDriver.cpp
4 	xul.dll 	XREMain::XRE_mainStartup(bool*) 	toolkit/xre/nsAppRunner.cpp
5 	xul.dll 	XREMain::XRE_main(int, char** const, nsXREAppData const*) 	toolkit/xre/nsAppRunner.cpp
6 	xul.dll 	XRE_main 	toolkit/xre/nsAppRunner.cpp
7 	firefox.exe 	do_main 	browser/app/nsBrowserApp.cpp
8 	firefox.exe 	NS_internal_main(int, char**) 	browser/app/nsBrowserApp.cpp

this low level startup crash has been around for a while, but it's increasing during recent releases.
correlations point to the FLVSrvLib.dll module, which seems part of "Freecorder" by "Applian Technologies".

100% (40/40) vs.   0% (73/84474) FLVSrvLib.dll

Comment 1

[[:philipp]]

cc'ing the "usual suspects" on this bug report... 

although the overall crash numbers for this signature are quite low, this is triggering an exorbitant amount of questions at sumo recently which is a bit worrying: https://support.mozilla.org/en-US/questions/firefox?tagged=bug1205987&show=all

Comment 2

[(Away)]

It's strange that this is only now spiking, when v40 has been out for a while and this freecorder library dates back to 2010.

Comment 3

[[:philipp]]

maybe a two-way interference by third-party programs. other correlations that stick out are:

     45% (5/11) vs.   1% (695/89857) product_info.dll (part of kaspersky)
          0% (0/11) vs.   0% (1/89857) 
          0% (0/11) vs.   0% (28/89857) 15.0.0.463
          0% (0/11) vs.   0% (16/89857) 15.0.1.415
          0% (0/11) vs.   0% (1/89857) 15.0.2.337
          0% (0/11) vs.   0% (79/89857) 15.0.2.361
          0% (0/11) vs.   0% (3/89857) 16.0.0.207
          0% (0/11) vs.   0% (3/89857) 16.0.0.360
          0% (0/11) vs.   0% (6/89857) 16.0.0.424
          0% (0/11) vs.   0% (2/89857) 16.0.0.540
         45% (5/11) vs.   1% (554/89857) 16.0.0.614
          0% (0/11) vs.   0% (2/89857) 16.0.0.667

     45% (5/11) vs.   6% (5658/89857) IPSEng32.dll (part of norton)
          0% (0/11) vs.   0% (2/89857) 
          0% (0/11) vs.   0% (5/89857) 14.0.0.431
          0% (0/11) vs.   0% (21/89857) 14.1.0.4
          0% (0/11) vs.   0% (19/89857) 14.1.1.2
          0% (0/11) vs.   0% (34/89857) 14.1.2.8
          0% (0/11) vs.   0% (9/89857) 14.2.0.84
          0% (0/11) vs.   0% (35/89857) 14.2.1.9
          0% (0/11) vs.   2% (1591/89857) 14.3.0.8
          0% (0/11) vs.   0% (287/89857) 14.4.0.24
         45% (5/11) vs.   4% (3655/89857) 15.0.0.480

the norton IPSEng32.dll 15.0.0.480 in particular seems to have been shipped with a signature update on 2015-09-14 which fits well into the picture/timeframe of the spike.

Comment 4

[Jefferson]

The Freecorder DLL FLVSrvLib.dll also is showing up in recent startup crashes with the signature CreateFileW on Windows Vista. It's hard to generalize but a couple use Norton; in non-startup crashes with that signature, there's often an unfamiliar looking LSP. https://crash-stats.mozilla.com/report/list?product=Firefox&signature=CreateFileW

Comment 5

[[:philipp]]

at the moment & because many of the other top crashers have been fixed in the dot-release, this is now the #1 crash in the score board for firefox 41.0.1 (https://crash-analysis.mozilla.com/rkaiser/crash-report-tools/score/?version=41.0.1&limit=30) and still generating quite an unusually high amount of feedback on sumo and many submitted user comments. 
due of its nature as a persistent startup crash its volume might decrease again as affected users simply move away to other alternatives...

needinfoing dmajor, if something could be done about it on our part?

Comment 6

[[:philipp]]

Attachment: bug1205987v1.patch

adding the FLVSrvLib.dll 1.0.0.0 file to the windows dll blocklist

Comment 7

[Benjamin Smedberg]

Comment on attachment 8670913 [details] [diff] [review]
bug1205987v1.patch

I'm skeptical that we want this kind of injection at all, but yes let's block this version for now. Kairo have you reached out to applian about this, per our blocklisting guidelines? You should do that before this is released.

Comment 8

[Robert Kaiser]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #7)
> Kairo have you reached out to applian about
> this, per our blocklisting guidelines? You should do that before this is
> released.

I'm not usually doing any outreach, AFAIK Release Managers are doing that usually. Or is Jorge doing that when it come to add-on blocks? I hope Ritu (41 release manager) or Sylvestre know.

Comment 9

[Jorge Villalobos [:jorgev] (he/him)]

Is this from an add-on? If so, what's the ID?

Comment 10

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Based on the patch it has to do with flvsrvlib.dll by Applian Technologies. It's described as "FLV Service Library for Ask and Record Toolbar". I don't think it's an addon but I could be wrong.

Comment 11

[Sylvestre Ledru [:Sylvestre]]

afaik, we don't have a contact to reach Applian tech :/

Comment 12

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Philipp, is there a reason this should not land in Nightly now?

Comment 13

[[:philipp]]

hi ritu - since this is my first patch & dmajor helped me get going there, i need the help of somebody else to get that checked in...

in addition since blocking this .dll is rather a speculative fix and the crash volume for this is quite low on pre-release channels, it would probably need to go into beta builds before we can judge by crash stats that this has the desired effect.

Comment 14

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Thanks Philipp. Given that BSmedsberg has r+ this, let's get it checked into Nighlty and see whether it is reproducible or not. You can make Aurora/Beta uplift requests after that.

Comment 15

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/a67ea3f24678

Comment 16

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/a67ea3f24678

Comment 17

[[:philipp]]

i'm able to reproduce the crash in a vm with freecorder 4 & norton trialware.

the patch from comment #6 couldn't fix the crash with the inbound try build. this is a crash triggered: https://crash-stats.mozilla.com/report/index/265a05e8-35aa-434c-8ce4-cf50f2151013 - the FLVSrvLib.dll 1.0.0.0 is still showing up in the list of modules despite being on the blocklist.

Comment 18

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

(In reply to philipp from comment #17)
> i'm able to reproduce the crash in a vm with freecorder 4 & norton trialware.
> 
> the patch from comment #6 couldn't fix the crash with the inbound try build.
> this is a crash triggered:
> https://crash-stats.mozilla.com/report/index/265a05e8-35aa-434c-8ce4-
> cf50f2151013 - the FLVSrvLib.dll 1.0.0.0 is still showing up in the list of
> modules despite being on the blocklist.

Thanks for the follow up Philipp. Given that the fix did not work as expected, I will skip including this in the next 41 dot release.

Comment 19

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/a67ea3f24678

Comment 20

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Backed out in https://hg.mozilla.org/integration/mozilla-inbound/rev/8c36f287c6ce at philipp's request.

Comment 21

[[:philipp]]

since this isn't solvable with a simple blocklist entry, i'm probably the wrong assignee to get this fixed as well :-)

Comment 22

[Loic]

A French user reported that uninstalling "Freecorder" fixed the start-up crashes.
https://forums.mozfr.org/viewtopic.php?f=5&t=126683

Comment 24

[BugBot [:suhaib / :marco/ :calixte]]

Closing because no crashes reported for 12 weeks.