1339083 - Crash in nsObserverService::AddObserver with K7TotalSecurity

Status: VERIFIED FIXED

(External Software Affecting Firefox :: Other, defect)

Description

[Marco Castelluccio [:marco]]

This bug was filed from the Socorro interface and is 
report bp-5bb41c2f-2fe3-4eb5-9909-f01232170213.
=============================================================

(95.88% in signature vs 00.95% overall) Module "k7pswsen.dll" = true
(80.63% in signature vs 00.83% overall) Module "K7OEPlgn.dll" = true
(63.42% in signature vs 00.65% overall) Module "K7Crvr.dll" = true

These modules are from K7TotalSecurity.

Comment 1

[Marco Castelluccio [:marco]]

This is happening most of the time with e10s enabled:
(98.19% in signature vs 56.56% overall) e10s_enabled = 1

Comment 2

[Marco Castelluccio [:marco]]

Dees, do you have contacts within this company?

Should we try to block their DLL? They're #2 top content crash on 52.0b and #4 top content crash on 51.

Comment 3

[Desigan Chinniah [:cyberdees] [:dees] [London - GMT]]

(In reply to Marco Castelluccio [:marco] from comment #2)
> Dees, do you have contacts within this company?
> 
> Should we try to block their DLL? They're #2 top content crash on 52.0b and
> #4 top content crash on 51.

I don't. And I hadn't even heard of them previous to this ping. I'll try — but you may want to take other steps in advance...

Comment 4

[Desigan Chinniah [:cyberdees] [:dees] [London - GMT]]

(In reply to Desigan Chinniah [:cyberdees] [:dees] [London - GMT] from comment #3)
> (In reply to Marco Castelluccio [:marco] from comment #2)
> > Dees, do you have contacts within this company?
> > 
> > Should we try to block their DLL? They're #2 top content crash on 52.0b and
> > #4 top content crash on 51.
> 
> I don't. And I hadn't even heard of them previous to this ping. I'll try —
> but you may want to take other steps in advance...

You have mail. Just connected you to one of their senior engineers, Abhineet.

Comment 5

[Liz Henry (:lizzard) (relman/hg->git project)]

Andrei, can your team try to reproduce the crash? 
Let's prepare a patch to add K7TotalSecurity to the blocklist. It might be best to block the specific version if we can do that. If it works, we could still uplift a block to 52 beta.  Aaron, can you help with that or suggest another owner?

Comment 6

[Liz Henry (:lizzard) (relman/hg->git project)]

I should also mention this is a startup crash at huge volume. Marking as a blocker for 52 release.

Comment 7

[Benjamin Smedberg]

NI?kev for a blocklisting decision.

Comment 8

[(No longer employed by Mozilla) Aaron Klotz]

Attachment: Blocklist patch

Note that this blocklist patch uses ALL_VERSIONS; we don't have good enough correlations ATM to be able to aggregate by module version numbers (and I don't really have time to mess around with that right now).

I have filed bug 1339159 to request that ability in crash-stats.

Comment 9

[Kev Needham [:kev]]

No objections, but we should make sure their eng team is aware of the implications. Suspect most of those affected are in IN given the company, and it'd be great if we can replicate or confirm with K7.

Jorge, concur?

Comment 10

[[:philipp]]

these are the module correlations on 52.0b5 today:

  nsObserverService::AddObserver|EXCEPTION_BREAKPOINT (651 crashes)
    100% (650/651) vs.   7% (753/11324) k7pswsen.dll
          0% (0/651) vs.   0% (20/11324) 15.2.2.89
        100% (650/651) vs.   6% (733/11324) 15.2.2.95
     88% (571/651) vs.   6% (698/11324) K7OEPlgn.dll
         88% (571/651) vs.   6% (697/11324) 12.0.1.3
          0% (0/651) vs.   0% (1/11324) 14.0.1.7
     70% (454/651) vs.   5% (553/11324) K7Crvr.dll
          0% (0/651) vs.   0% (1/11324) 12.0.1.43
          0% (0/651) vs.   0% (1/11324) 12.0.1.51
         60% (389/651) vs.   4% (481/11324) 15.0.1.67
         10% (65/651) vs.   1% (70/11324) 15.0.1.69
     67% (433/651) vs.   5% (523/11324) K7TSHelp.dll
          0% (0/651) vs.   0% (1/11324) 12.0.1.2
         67% (433/651) vs.   5% (522/11324) 15.0.1.4

the file description of "k7pswsen.dll" say it would be a LSP extension (which caused issues with blocklisting in the past) - it doesn't show up in the winsock_lsp field in crash reports though.

i installed a trial version of k7 total security locally but wasn't able to reproduce the crashes with their default setup or when playing around a bit in their software. if you could run a try-build with that blocklisting patch, i could check if it succeeds in getting those dlls out of our process and if it has an immediate impact on connectability at least.

Comment 11

[(No longer employed by Mozilla) Aaron Klotz]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=70b7765f1423

Comment 12

[(No longer employed by Mozilla) Aaron Klotz]

^ today's m-c + the blocklisting patch attached to this bug.

Comment 13

[Gregory Panakkal]

I'm one of the developers at K7. We received independent reports of inconsistent crashes in Firefox 51.0.1 as well. We have scheduled to release a workaround to address this crash today.
It will be addressed in K7PSWSen.dll v15.2.2.96

Comment 14

[Desigan Chinniah [:cyberdees] [:dees] [London - GMT]]

Can we confirm... 

From Gregory (in Comment 13) at K7 via email overnight:

===

Hi,

We have released the update (K7PSWSEn.dll v15.2.2.96) with the workaround.

Please let us know ASAP if the problems persist

===

Comment 15

[Jorge Villalobos [:jorgev] (he/him)]

Agree, but, is there an add-on involved? This seems to be strictly a DLL block.

Comment 16

[Marco Castelluccio [:marco]]

For now, we're still only seeing crash reports exclusively with 15.2.2.95 and 15.2.2.89. In the past two days 6122 with 15.2.2.95 and 35 with 15.2.2.89.

(In reply to Jorge Villalobos [:jorgev] from comment #15)
> Agree, but, is there an add-on involved? This seems to be strictly a DLL
> block.

There is an addon, [e-mail], but only in a small subset of reports.

Comment 17

[[:philipp]]

(In reply to Aaron Klotz [:aklotz] from comment #12)
> ^ today's m-c + the blocklisting patch attached to this bug.
thanks, that blocklist entry seems to have worked fine. the blocked dll modules are no longer hooking into the process, and i didn't notice any functional impact in terms of connectivity.

since the developers of k7 are reacting quite quickly though, we might not need to use it after all...

Comment 18

[[:philipp]]

Attachment: bug1339083.patch

this would be a blocklist patch in case we only want to target the unfixed version of their dll...

Comment 19

[Marco Castelluccio [:marco]]

Attachment: Blocklist old K7TotalSecurity versions

This is the blocklist entry to block up to 15.2.2.95, which should be the last affected version.

Comment 20

[Liz Henry (:lizzard) (relman/hg->git project)]

Thanks, sounds like K7 has fixed the issue in new updates today. Let's land the patch to blocklist the older versions. Best of both worlds!

Comment 21

[[:philipp]]

hi aaron, it looks like relman wants to blocklist the unfixed version of the k7 dll anyway to be on the safe side. could you review either the attachment in comment #18 or comment #19, so that we may get that into 52.0b7?

Comment 22

[[:philipp]]

for attachment #8837174 [details] [diff] [review]

Comment 23

[[:philipp]]

Comment on attachment 8837174 [details] [diff] [review]
Blocklist old K7TotalSecurity versions

Approval Request Comment
[Feature/Bug causing the regression]: third-party update in k7 total security software on feb 10
[User impact if declined]: crashes in the content process for users with an unfixed version of k7 total security
[Is this code covered by automated tests?]: n/a
[Has the fix been verified in Nightly?]: wasn't able to reproduce the crashes in the first place, but manually checked that our blocklist is successful in getting out the offending dll of the firefox process and has no other obvious negative side-effects 
[Needs manual test from QE? If yes, steps to reproduce]: n/a
[List of other uplifts needed for the feature/fix]: n/a
[Is the change risky?]: low risk
[Why is the change risky/not risky?]: this is making use of our purpose-built blocklisting mechanism
[String changes made/needed]: n/a

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/62adae0f5df2
Blocklist old K7TotalSecurity versions. r=aklotz

Comment 25

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Comment on attachment 8837174 [details] [diff] [review]
Blocklist old K7TotalSecurity versions

We need to try this fix because this crash sign is currently ranked #2 on Beta52 top content crasher. (FYI JCristau)

Comment 26

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/62adae0f5df2

Comment 27

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/b3af8fd8abe8

Comment 28

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/24c5c108f6d6

Comment 29

[Bogdan Maris, Desktop Test Engineering]

Hi, I am trying to reproduce the crash and verify the fix but I can't find the versions 15.2.2.95 and 15.2.2.96 to try to reproduce and verify the fix. Can those versions be made available for us to test? I could not find them online.

Comment 30

[Gregory Panakkal]

Attachment: K7PSWSEn_v15.2.2.95.zip

K7 DLL that caused the crash

Comment 31

[Gregory Panakkal]

Attachment: K7PSWSEn_v15.2.2.96.zip

K7 DLL with workaround to address crash

Comment 32

[Gregory Panakkal]

High Level Instructions:
1. Install K7TotalSecurity ( Installer: http://www.k7computing.com/products/eng/K7ts/setup-eng-ts.exe )
2. Register/Activate as trial user
3. Perform product update (reboot the system if prompted)
     MainProductScreen->Update->Run update now
4. Turn off automatic product updates (this is to ensure that K7PSWSEn.dll isn't replaced with the latest version when verifying crash with older file)
     MainProductScreen->Update->Configure update->Uncheck 'Automatically check for updates'
5. Turn off product self protection
     MainProductScreen->Settings->General Settings->Uncheck 'Enable Product Self-Protection'->Close->Choose time period 'permanent'->Yes
6. Rename K7PSWSEn.dll in product installation directory (usually in %Program Files%\K7 Computing\K7TSecurity) to K7PSWSEn.dll_ 
7. Place desired version of K7PSWSEn.dll in product instalation directory
8. Launch Firefox and observe expected behaviour

(I've added the requested K7PSWSEn.dll as attachments )

Comment 33

[Paul Silaghi, QA [:pauly]]

Reproduced the crash on FX 51.0.1 with the K7PSWSEn_v15.2.2.95.zip, Win 10 x64.
No longer crashes with K7PSWSEn_v15.2.2.96.zip, FX 51.0.1.

Comment 34

[Marco Castelluccio [:marco]]

(In reply to Paul Silaghi, QA [:pauly] from comment #33)
> Reproduced the crash on FX 51.0.1 with the K7PSWSEn_v15.2.2.95.zip, Win 10
> x64.
> No longer crashes with K7PSWSEn_v15.2.2.96.zip, FX 51.0.1.

Thanks, Paul. Could you verify that the crash can be reproduced on Nightly 20170215110151 (http://ftp.mozilla.org/pub/firefox/nightly/2017/02/2017-02-15-11-01-51-mozilla-central/), but not on Nightly 20170216030210 (http://ftp.mozilla.org/pub/firefox/nightly/2017/02/2017-02-16-03-02-10-mozilla-central/).

Comment 35

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr52/rev/24c5c108f6d6

Comment 36

[Paul Silaghi, QA [:pauly]]

(In reply to Marco Castelluccio [:marco] from comment #34)
> Thanks, Paul. Could you verify that the crash can be reproduced on Nightly
> 20170215110151
> (http://ftp.mozilla.org/pub/firefox/nightly/2017/02/2017-02-15-11-01-51-
> mozilla-central/)
There are no Windows builds here. I used instead http://ftp.mozilla.org/pub/firefox/nightly/2017/02/2017-02-15-03-03-42-mozilla-central/.
Anyway, I could NOT reproduce the crash on both builds (2017-02-15 and 2017-02-16) using the affected DLL from K7PSWSEn_v15.2.2.95.zip.

Comment 37

[Marco Castelluccio [:marco]]

OK, thanks. We might have to wait for the next beta build to verify then.

Comment 38

[Andrei Vaida [:avaida]]

(In reply to Marco Castelluccio [:marco] from comment #37)
> OK, thanks. We might have to wait for the next beta build to verify then.

Flagging this for verification as a reminder to do this on the first eligible Beta 52 build.

Comment 39

[Gerry Chang [:gchang]]

No plan to have dot release for 51. Mark 51 won't fix.

Comment 40

[Paul Silaghi, QA [:pauly]]

Still reproducible on Fx 52b9 with the affected DLL, so the blocklist doesn't seem to work properly:
https://crash-stats.mozilla.com/report/index/273aef61-4fa7-4bd4-80a1-916482170227

Comment 41

[Marco Castelluccio [:marco]]

(In reply to Paul Silaghi, QA [:pauly] from comment #40)
> Still reproducible on Fx 52b9 with the affected DLL, so the blocklist
> doesn't seem to work properly:
> https://crash-stats.mozilla.com/report/index/273aef61-4fa7-4bd4-80a1-
> 916482170227

Luckily they pushed an update and the volume of the crash has been 0 for a while now, so we don't have to worry at the moment.

Comment 42

[Paul Silaghi, QA [:pauly]]

Marking as verified based on comment 41.