Closed Bug 1384327 Opened 7 years ago Closed 7 years ago

Nightly & kaspersky antivirus cause a big issue at start-up

Categories

(External Software Affecting Firefox :: Other, defect, P2)

Product:
External Software Affecting Firefox
Component:
Other
Platform:
All
Windows
Type:
defect

Tracking

(firefox55 unaffected, firefox56+ fixed, firefox57 fixed)

Status:
RESOLVED WORKSFORME
Tracking Flags:
Tracking Status
firefox55 --- unaffected
firefox56 + fixed
firefox57 --- fixed

People

(Reporter: ratm6, Unassigned)

References

(Blocks 1 open bug)

Details

(Keywords: regression, Whiteboard: [mozfr-community][sb+]) 

User Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:54.0) Gecko/20100101 Firefox/54.0
Build ID: 20170628075643

Steps to reproduce:

I have Kaspersky Anti-virus 17.0.0.611(f) (same problem with KIS)

I've installed 2017-07-18 nightly build then I've updated it to 2017-07-24 nightly build.



Actual results:

A lot of ffcert.exe windows with "missing profile" errors appeared with this latest build.
Then it's impossible to run Nightly.
With 2017-07-18 build there is NO problem.


Expected results:

The latest Nightly build should load correctly. 

I think Kaspersky is not in fault because I've just tried again to install and run older (2017-07-18) nightly build and all is fine.
Component: Untriaged → Other
Product: Firefox → External Software Affecting Firefox
Version: 56 Branch → unspecified
The bug is in the 2017-07-22 build (I made some tests). 
This build (win64) : https://archive.mozilla.org/pub/firefox/nightly/2017/07/2017-07-22-07-26-31-mozilla-central/
As far as I know, ffcert.exe seems to auto-install a Kaspersky certificate for Firefox at start-up, if it is not already done.
OS: Unspecified → Windows
Hardware: Unspecified → All
Whiteboard: [mozfr-community]
(In reply to Julien L. from comment #1)
> The bug is in the 2017-07-22 build (I made some tests). 
> This build (win64) :
> https://archive.mozilla.org/pub/firefox/nightly/2017/07/2017-07-22-07-26-31-
> mozilla-central/

Did you check all the other builds between 07-18 and 07-22 as well? (i.e. is the 07-22 build the first one with the problem?)
Yes, here is my notes about that :
2017-07-23 bad
2017-07-22 bad
2017-07-21 good
2017-07-20 good
2017-07-19 good
2017-07-18 good

So yes, 07-22 build is the first one with the problem.
I'm going to guess this was caused by bug 1381577, it sounds like the sort of change that might move folders around and cause Kapersky's ffcert.exe to not find the profile. Could be something else though, not sure.
Flags: needinfo?(bugspam.Callek)
I don't forsee how our code could have caused that.

It shouldn't be impacting to nightlies until today, the new nightly code signing cert itself was rotated on the 17th.
Flags: needinfo?(bugspam.Callek)
[Tracking Requested - why for this release]: Regression which will prevent any Kaspersky user to start firefox. This will also make their computer unusable, unless they force stop it.

Thank you for filing this bug, Julien! 

I narrowed the regression to bug 1366694. On inbound, bed655e34ed939af69706116bbdb1d97b1f87c77 is clean, and 60ef4d9f30239938dc0769cab69c54f3ccbb7d3f is the first to fail[1]

STEPS TO REPRODUCE
1. Download the most basic edition of Kaspersky from [2]. You can use the free trial (30 days)
2. Download a windows installer from bed655e34ed939af69706116bbdb1d97b1f87c77[3]. This is important to use the installer and make sure that Nightly is the default browser. Otherwise, Kaspersky won't try to mess up with certificates. This particular point made mozregression useless.
3. Install it. See that firefox boots up correctly.
4. Download the installer of the next push[4] and install it.

RESULTS
About 20 error windows pop up. They read:
> Profile missing
> Your Firefox profile cannot be loaded. It may be missing or inaccessible

In the process manager, you may see more than 50 firefox.exe running. The whole system is clogged. I had to force shutdown my machine, each time I hit a bad build.


[1] https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-job_type_symbol=B&filter-tier=1&filter-platform=win&fromchange=bed655e34ed939af69706116bbdb1d97b1f87c77&tochange=60ef4d9f30239938dc0769cab69c54f3ccbb7d3f
[2] https://www.kaspersky.fr/antivirus
[3] https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-win32/1500529701/firefox-56.0a1.en-US.win32.installer.exe
[4] https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-win32/1500533480/firefox-56.0a1.en-US.win32.installer.exe
Blocks: 1366694
Status: UNCONFIRMED → NEW
status-firefox55: --- → unaffected
status-firefox56: --- → affected
tracking-firefox56: --- → ?
Ever confirmed: true
Keywords: regression
Should we tell to Kaspersky about a new way to communicate with the Firefox processes, Bob?
Flags: needinfo?(bobowencode)
Track 56+ as new regression and it impacts Kaspersky users a lot.
tracking-firefox56: ? → +
(In reply to Johan Lorenzo [:jlorenzo] from comment #9)
> Should we tell to Kaspersky about a new way to communicate with the Firefox
> processes, Bob?

I don't think we provide any ways for AVs to communicate with us.

From comment 2, my guess is that they inject some code into all the firefox.exe processes to do this check and the sandbox is preventing it in the child processes now and Kaspersky's code isn't handling that properly.
I'm not really sure why that would cause lots of firefox processes to start.

It's not clear to me why they have to do this through injected code, but if they do, couldn't they just do it for the main process and not any child processes.
Flags: needinfo?(bobowencode)
Do we know how they're injecting to stop it in at least ocntent processes?
Whiteboard: [mozfr-community] → [mozfr-community][sb?]
Dee's do we know anyone from Kaspersky?
Flags: needinfo?(dchinniah)
I just sent an email to two of our contact there. Waiting to hear back and will update the bug when there is news.
Flags: needinfo?(dchinniah)
> A lot of ffcert.exe windows with "missing profile" errors appeared with this
> latest build.

FYI, this exe is part of the Kaspersky install, it's not part of Firefox. Any failure in it is entirely the vendor's responsibility.
Tracy, please test to see if we can repo this in a vm. We might be able to reverse engineer what they're up to.
Flags: needinfo?(twalker)
We investigate the problem in Kaspersky Lab.
Kaspersky products monitor the Fifefox.exe start, and launch ffcert.exe for installing certificate. In new FF build, sandbox process was implemented. As mentioned in comment 12, Kaspersky attempt to install certificate on sandbox process start, but due limited privileges, it is fail.

We already fix it in dev version of Kaspersky software, we will install certificate only on main process start. Now we a planning to patch already released Kaspersky products.

When you a going to release Firefox with sandbox processes?

By the way, Kaspersky have a special email address for any issues, connected on Kaspersky products and browsers compatibility - BrowserIssues@kaspersky.com. 

Alexey Totmakov
Head of Network Technologies Development
Kaspersky Lab
Colleagues,

will it be the robust method to determinate sandbox process, if we check parent on equality with firefox.exe?
Or it will be better to analyze command line parameters?
The firefox.exe binary is used for both content and parent process. The XRE_IsContentProcess function returns the information you need.
Flags: needinfo?(twalker)
(In reply to Alexey Totmakov from comment #18)
> Colleagues,
> 
> will it be the robust method to determinate sandbox process, if we check
> parent on equality with firefox.exe?
> Or it will be better to analyze command line parameters?

Command line params would be good, currently avoiding any firefox.exe process with -contentproc on the cammand line would work.
(In reply to Alexey Totmakov from comment #17)
> We investigate the problem in Kaspersky Lab.
> Kaspersky products monitor the Fifefox.exe start, and launch ffcert.exe for
> installing certificate. In new FF build, sandbox process was implemented. As
> mentioned in comment 12, Kaspersky attempt to install certificate on sandbox
> process start, but due limited privileges, it is fail.
> 
> We already fix it in dev version of Kaspersky software, we will install
> certificate only on main process start. Now we a planning to patch already
> released Kaspersky products.
> 
> When you a going to release Firefox with sandbox processes?

Planning to roll the current changes out in Firefox 56.

https://wiki.mozilla.org/RapidRelease/Calendar


> By the way, Kaspersky have a special email address for any issues, connected
> on Kaspersky products and browsers compatibility -
> BrowserIssues@kaspersky.com. 
> 
> Alexey Totmakov
> Head of Network Technologies Development
> Kaspersky Lab

Thanks Alexey. ni to Dees for the email info.
Flags: needinfo?(dchinniah)
Thanks for the info, Alexey and Jim. Adam will reach out so we can setup a better channel for further engineering communication including the catch-all shared BrowserIssues@ email address.

Dees
Flags: needinfo?(dchinniah) → needinfo?(astevenson)
(In reply to Jim Mathies [:jimm] from comment #21)

> > When you a going to release Firefox with sandbox processes?
> 
> Planning to roll the current changes out in Firefox 56.
> 
> https://wiki.mozilla.org/RapidRelease/Calendar

We are going to release patches for Kaspersky Internet Security 2017 and 2018 till the end of August.
(In reply to Alexey Totmakov from comment #23)
> 
> We are going to release patches for Kaspersky Internet Security 2017 and
> 2018 till the end of August.

Please do not forget to patch KAV too :)
Thanks.
Alexey, will you release a fix for that bug in Patch “C” for KAV/KIS/KTS/KFA 2018 ? That's not obvious to know in your forum web page. Thank you again.
Flags: needinfo?(alexey.totmakov)
(In reply to Julien L. from comment #24)
> Please do not forget to patch KAV too :)
> Thanks.

Of course, the patch will apply to product line.
Flags: needinfo?(alexey.totmakov)
(In reply to Julien L. from comment #25)
> Alexey, will you release a fix for that bug in Patch “C” for KAV/KIS/KTS/KFA
> 2018 ? That's not obvious to know in your forum web page. Thank you again.

Correct, patch C.
Flags: needinfo?(astevenson)
Flags: needinfo?(astevenson)
..waiting on confirmation of the fix.
I'm having this problem with the versions FirefoxDev x64 windows 56.0b1 and 55.0b9, I'm having KIS 17.0.0.611.

But with normal Firefox, not problem.
Problem solved with Beta 56.0, Nightly 57.0, KAV 17.0.0.611 and patch G (on my computer of course).
Flags: needinfo?(jmathies)
(In reply to Julien L. from comment #30)
> Problem solved with Beta 56.0, Nightly 57.0, KAV 17.0.0.611 and patch G (on
> my computer of course).

any idea when this will be released?
Flags: needinfo?(jmathies)
We published patch G for Kaspersky Internet Security 2017 and patch C for Kaspersky Internet Security 2018. The patches should fix the problem. Please check is it true.

Patch for Kaspersky Internet Security 2016 will be available in September.
Could you provide a more specific date for when the fix will be available?
Flags: needinfo?(alexey.totmakov)
n-i on myself to follow up later today.
Flags: needinfo?(lhenry)
(In reply to Panos Astithas [:past] (56 Regression Engineering Owner) (please ni?) from comment #33)
> Could you provide a more specific date for when the fix will be available?

Patch for KIS2016MR1 will be available to the end of September.
Flags: needinfo?(alexey.totmakov)
Firefox 56 release date will be Sept. 26. I think that is close enough that we don't need to block the 2016 version.
(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #36)
> Firefox 56 release date will be Sept. 26. I think that is close enough that
> we don't need to block the 2016 version.

Agree, we are going to fully publish to September 20.
Flags: needinfo?(lhenry)
Flags: needinfo?(astevenson)
We published patch I for Kaspersky Internet Security 2016MR1. From now, all Kaspersky supported products compatible with Firefox 56 sandbox process.
Andrei can you verify that this is now fixed?
Flags: needinfo?(andrei.vaida)
We haven't seen any duplicate reports here, and some possibly related crashes correlated with Kaspersky were fixed in bug 1268470. 

I'm not sure this is worth a release note yet, though we can revisit that if it becomes a problem after the 64-bit migration (after 56 release). So, keeping this bug tracked for 56 till we have more information.
I can confirm this issue is no longer reproducible. I verified using Fx 56.0-build 6 (build ID: 20170926190823), Fx 57.0b3 (build ID: 20170925150345) and Fx 58.0a1 ( build ID: 20170927100120) on Windows 10 x64.
Flags: needinfo?(andrei.vaida)
Priority: -- → P2
Whiteboard: [mozfr-community][sb?] → [mozfr-community][sb+]
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → WORKSFORME
Blocks: injecteject
I am assuming relnote'ing this is not needed anymore. Liz, please correct me if I am wrong.
relnote-firefox: ? → ---
Flags: needinfo?(lhenry)
I believe the fix shipped before most folks updated, all good here and no relnote needed.
Flags: needinfo?(lhenry)
You need to log in before you can comment on or make changes to this bug.