Closed Bug 1591691 Opened 4 years ago Closed 4 years ago

importing enterprise roots can cause network I/O

Categories

(Core :: Security: PSM, defect, P1)

Product:
Core
Component:
Security: PSM
Version:
70 Branch
Type:
defect

Tracking

()

Status:
VERIFIED FIXED
Milestone:
mozilla72
Tracking Flags:
Tracking Status
firefox-esr68 71+ verified
firefox70 --- wontfix
firefox71 --- verified
firefox72 --- verified

People

(Reporter: fred.steiny, Assigned: keeler)

References

Details

(Whiteboard: [psm-assigned])

Attachments

(6 files)

FirefoxLog.zip
2.45 MB, application/x-zip-compressed
Details
Firefox_first_start_delay.PNG
16.34 KB, image/png
Details
Firefox-regression.PNG
86.47 KB, image/png
Details
Firefox-regression2.PNG
76.89 KB, image/png
Details
bug 1591691 - avoid network I/O when importing enterprise roots on Windows r?mhowell
47 bytes, text/x-phabricator-request
pascalc
: approval-mozilla-beta+
lizzard
: approval-mozilla-esr68+
Details | Review
bug 1591691 - avoid network I/O when importing enterprise roots on MacOS r?spohl
47 bytes, text/x-phabricator-request
pascalc
: approval-mozilla-beta+
lizzard
: approval-mozilla-esr68+
Details | Review

User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0

Steps to reproduce:

  1. Start Firefox 70 on Windows 7
  2. Open a website
    => 10 sec delay

Actual results:

On any website it takes 10 sec before page loads for the first website since firefox startup. All followup websites load fast.

Does not happen with Google Chrome.

Happens also in firefox safemode with new profile.

No Proxy or DNS settings set.

Happens also with disabled Kapersky security software

Expected results:

No 10 sec delay on first page load after firefox startup.

Attached file FirefoxLog.zip —

firefox logfile

i'm confirming this bug on the basis of seeing a few similar reports from other users after the firefox 70 update:
https://support.mozilla.org/en-US/questions/1271170
https://old.reddit.com/r/firefox/comments/dm0yxx/firefox_taking_forever_to_load_first_page_when/

also setting some ni's to get it on the radar of the necko and performance teams...

Status: UNCONFIRMED → NEW
status-firefox70: --- → unaffected
status-firefox71: --- → ?
status-firefox72: --- → ?
status-firefox-esr68: --- → unaffected
Ever confirmed: true
Flags: needinfo?(vchin)
Flags: needinfo?(nhnguyen)
Keywords: regression

hello foxy, would you be able to narrow the problem down with the help of the mozregression tool from https://mozilla.github.io/mozregression/ pointing to your profile folder?

Yes, it happens even with empty profile. I attached a screenshot of result, but my last choice was wrong it was not green good but red bad 3d798a3c. (And I could not rewise my wrong choice, so feature request for mozregression tool)

Attached image Firefox-regression.PNG —

is security.enterprise_roots.enabled set to "true" in about:config in your case?

Attached image Firefox-regression2.PNG —

Yes security.enterprise_roots.enabled set to "true" in about:config with locked value by Kaspersky Labs AntiVirus.

After deleting content of file C:\Program Files\Mozilla Firefox\defaults\pref\kl_prefs_62fbb8f7_c917_4cf7_957a_aad2b8fa768c.js and write protect file to not get reverted, Firefox 70 works again without delay on first page after firefox startup.

thank you, so the problem is probably related to bug 1571548 which landed on nightly in the range you came up with (not where it ended up on the autoland branch though).

status-firefox-esr68: unaffected → ?
Flags: needinfo?(dkeeler)
Summary: After firefox startup first page load is delayed for 10 sec → After firefox startup first page load is delayed for some seconds

I can Confim this bug:
Win64 1903 (18362.449)
FF 70 (release) - No matter if full installtion (updated from 69 or new), portable or with new profile or in safe mode.
Kaspersky 20.0.14.1085 - No matter if deactivated or on pause.

I confirm as well that using following empty file (with write protection) resolve this issue. Thanks @foxy!
C:\Program Files\Mozilla Firefox\defaults\pref\kl_prefs_62fbb8f7_c917_4cf7_957a_aad2b8fa768c.js

(In reply to djmaniac44 from comment #12)

I can Confim this bug:
Win64 1903 (18362.449)
FF 70 (release) - No matter if full installtion (updated from 69 or new), portable or with new profile or in safe mode.
Kaspersky 20.0.14.1085 - No matter if deactivated or on pause.

I confirm as well that using following empty file (with write protection) resolve this issue. Thanks @foxy!
C:\Program Files\Mozilla Firefox\defaults\pref\kl_prefs_62fbb8f7_c917_4cf7_957a_aad2b8fa768c.js

As an option we can set
C:\Program Files\Mozilla Firefox\kl_config_62fbb8f7_c917_4cf7_957a_aad2b8fa768c.cfg
lockPref("security.enterprise_roots.enabled", ture);
TO
lockPref("security.enterprise_roots.enabled", false);

Can anyone who can reproduce this bug please attempt to get a profile by following the directions at: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Reporting_a_Performance_Problem
In particular, please include the StreamTrans threads by checking the corresponding box and add in ,ssl manually to the list of threads.
Thanks!

Flags: needinfo?(dkeeler) → needinfo?(fred.steiny)

I have the same Issue. With a new default profile, the homepage upon startup takes over 20 seconds to load (“Performing a TLS handshake").
However, if I change my home page to a NON-https site, the issue does not occur.
==>> Important Hint to this bug: If I delete the CONTENTS of this file:
C:\Program Files\Mozilla Firefox\defaults\pref\kl_prefs_62fbb8f7_c917_4cf7_957a_aad2b8fa768c.js
and then set the file to READ ONLY, the problem is resolved.

Thank You.

I can confirm that this also fixes the issue:

In this File: C:\Program Files\Mozilla Firefox\kl_config_62fbb8f7_c917_4cf7_957a_aad2b8fa768c.cfg
Change "security.enterprise_roots.enabled" to false as below:

// kl_config_62fbb8f7_c917_4cf7_957a_aad2b8fa768c.cfg
lockPref("security.enterprise_roots.enabled", false);

By the way, I use Eset NOD32 v.13.0.22.0

Related:
How to disable the Enterprise Roots preference
https://support.mozilla.org/en-US/kb/how-disable-enterprise-roots-preference
Note: The option security.enterprise_roots.enabled can not be changed in about:config because it's locked.
The C:\Program Files\Mozilla Firefox\kl_config_62fbb8f7_c917_4cf7_957a_aad2b8fa768c.cfg file must be edited manually.

"Starting with Firefox version 68, when a TLS connection error occurs Firefox will automatically enable the Enterprise Roots preference and attempts to connect again. If the issue is resolved, then the Enterprise Roots preference remains enabled. However, you may want to disable this behavior, so this article explains how to do just that without compromising security. "

(In reply to Dana Keeler (she/her) (use needinfo) (:keeler for reviews) from comment #14)

Can anyone who can reproduce this bug please attempt to get a profile by following the directions at: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Reporting_a_Performance_Problem
In particular, please include the StreamTrans threads by checking the corresponding box and add in ,ssl manually to the list of threads.
Thanks!

https://perfht.ml/2Ps6pFJ
https://perfht.ml/36fUgtq

Flags: needinfo?(fred.steiny)

Thank you!

Assignee: nobody → dkeeler
status-firefox71: ? → affected
status-firefox72: ? → affected
status-firefox-esr68: ? → affected
Keywords: regression
Priority: -- → P1
Summary: After firefox startup first page load is delayed for some seconds → importing enterprise roots can cause network I/O
Whiteboard: [psm-assigned]
Flags: needinfo?(vchin)
Flags: needinfo?(nhnguyen)
Component: Untriaged → Security: PSM
Product: Firefox → Core

Yes, fixed.

Flags: needinfo?(fred.steiny)

(In reply to Dana Keeler (she/her) (use needinfo) (:keeler for reviews) from comment #21)

Does this build fix the issue for you?
64-bit: https://queue.taskcluster.net/v1/task/FUMsRpvyTRKvOyqrcEQEeA/runs/0/artifacts/public/build/install/sea/target.installer.exe
32-bit: https://queue.taskcluster.net/v1/task/JIrnaCPeRYyVhX-aJSUFWQ/runs/0/artifacts/public/build/install/sea/target.installer.exe

Yes - way faster now but still a bit slower compared to FF 69.0.3

FF 69.0.3 - https://www.google.de/ (as Homepage) ~ 2 sec after start
FF 72.0a1 (2019-10-29) - https://www.google.de/ (as Homepage) ~ 3 sec after start

But I guess this is OK and may be related to new functions of FF 70+ ?!

Can you get a performance profile with the new build as described in comment 14? Thanks!

Flags: needinfo?(djmaniac44)

(In reply to Dana Keeler (she/her) (use needinfo) (:keeler for reviews) from comment #27)

Can you get a performance profile with the new build as described in comment 14? Thanks!

Don't know what happend but now FF 72.0a1 (2019-10-29) is as fast as FF 69.0.3.
Maybe FF 72.0a1 (2019-10-29) did some stuff after first installation.
Please disregard https://bugzilla.mozilla.org/show_bug.cgi?id=1591691#c26
Thanks for your work here.
I call it fixed as well.

Flags: needinfo?(djmaniac44)
Pushed by dkeeler@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/8a690dff4180
avoid network I/O when importing enterprise roots on Windows r=mhowell

Razvan - next time ping me in slack, please (particularly for a tier-2 breakage).
Apparently mingw's version of that header doesn't include the flag CERT_CHAIN_DISABLE_AIA.

Flags: needinfo?(dkeeler)
Pushed by dkeeler@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/7645d149e3f4
avoid network I/O when importing enterprise roots on Windows r=mhowell
Pushed by dkeeler@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/f7efa3e6089e
avoid network I/O when importing enterprise roots on MacOS r=spohl

https://hg.mozilla.org/mozilla-central/rev/f7efa3e6089e

Status: NEW → RESOLVED
Closed: 4 years ago
status-firefox72: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla72

Will the fix also be released for firefox 70 as hotfix? Because else stable release users will need to wait till next year...

Comment on attachment 9105052 [details]
bug 1591691 - avoid network I/O when importing enterprise roots on Windows r?mhowell

Beta/Release Uplift Approval Request

  • User impact if declined: Slow first start (on the order of many seconds) for users that have enterprise certificates with remote revocation information.
  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Nightly?: Yes
  • Needs manual test from QE?: No
  • If yes, steps to reproduce:
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): This is a small change - it adds a flag to tell the OS to not do network fetches when verifying the candidate enterprise certificates to import. For roots and intermediate DV certificates we don't do active revocation checking (and imported certificates can't be EV anyway), so this behavior is the same as if the user had manually imported the certificates, so it shouldn't be a security risk.
  • String changes made/needed: none

ESR Uplift Approval Request

  • If this is not a sec:{high,crit} bug, please state case for ESR consideration: Enterprise-facing feature.
  • User impact if declined: Slow first start (on the order of many seconds) for users that have enterprise certificates with remote revocation information.
  • Fix Landed on Version: 72
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): This is a small change - it adds a flag to tell the OS to not do network fetches when verifying the candidate enterprise certificates to import. For roots and intermediate DV certificates we don't do active revocation checking (and imported certificates can't be EV anyway), so this behavior is the same as if the user had manually imported the certificates, so it shouldn't be a security risk.
  • String or UUID changes made by this patch: none
Attachment #9105052 - Flags: approval-mozilla-esr68?
Attachment #9105052 - Flags: approval-mozilla-beta?
Attachment #9105056 - Flags: approval-mozilla-esr68?

Comment on attachment 9105052 [details]
bug 1591691 - avoid network I/O when importing enterprise roots on Windows r?mhowell

Low risk fix for a significant Windows perf regression, verified by the reporter in Nightly, uplift approved for 71 beta 8, thanks.

Attachment #9105052 - Flags: approval-mozilla-beta? → approval-mozilla-beta+

Dana, should Attachment #9105056 [details] also have an uplift request to beta? Thanks

Flags: needinfo?(dkeeler)

Comment on attachment 9105056 [details]
bug 1591691 - avoid network I/O when importing enterprise roots on MacOS r?spohl

Beta/Release Uplift Approval Request

  • User impact if declined: (see comment 39)
  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Nightly?: Yes
  • Needs manual test from QE?: Yes
  • If yes, steps to reproduce:
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky):
  • String changes made/needed:
Flags: needinfo?(dkeeler)
Attachment #9105056 - Flags: approval-mozilla-beta?
Attachment #9105056 - Flags: approval-mozilla-beta? → approval-mozilla-beta+

(In reply to foxy from comment #37)

Will the fix also be released for firefox 70 as hotfix? Because else stable release users will need to wait till next year...

It seems this fix won't be released until v.72. In the meantime, here are the steps to fix the issue:

First Close Firefox.

Step 1. Open this File in NOTEPAD: C:\Program Files\Mozilla Firefox\kl_config_62fbb8f7_c917_4cf7_957a_aad2b8fa768c.cfg
Change "security.enterprise_roots.enabled" from true to false as shown below. Then close notepad saving changes.

// kl_config_62fbb8f7_c917_4cf7_957a_aad2b8fa768c.cfg
lockPref("security.enterprise_roots.enabled", false);

Step 2. Open Firefox and type this in the address bar: About:Config
Copy and Paste this into the search box: security.certerrors.mitm.auto_enable_enterprise_roots
Change the value from True to False by simply double-clicking on it.

Note: Keep these instructions. When Firefox is fixed with a new version (probably v.72), reverse these changes!

Sorry, I forget one important Step: You must run Notepad as Administrator by right clicking on Notepad and selecting "Run As Administrator".
You won't be able to save changes otherwise.

[Tracking Requested - why for this release]: performance issue for enterprise-facing feature (see comment 39)

tracking-firefox-esr68: --- → ?

Comment on attachment 9105052 [details]
bug 1591691 - avoid network I/O when importing enterprise roots on Windows r?mhowell

Fix verified in nightly, given this is an enterprise related feature let's uplift to ESR for 68.3.0.

Attachment #9105052 - Flags: approval-mozilla-esr68? → approval-mozilla-esr68+
Attachment #9105056 - Flags: approval-mozilla-esr68? → approval-mozilla-esr68+

This issue occurs on home non-enterprise as well. I've seen the bug in both environments.

Flags: qe-verify+
QA Whiteboard: [qa-triaged]

I am attempting to verify this fix in all relevant channels, but I can't properly reproduce it. I have Kaspersky antivirus installed on my machine, but if I open the Release v70.0.1 (64-bit) and search for pref "security.enterprise_roots.enabled" in about:config, it will be "false" by default and unlocked.
How do I influence the antivirus to flip the pref's value? I need to know what happened exactly so I can properly verify it on a totally new machine.

@philipp: Can you help? Thanks!

Flags: needinfo?(madperson)

when i installed the kaspersky 30-day trial with all the default options (except their KSN and marketing tracking stuff) and performed a signature update afterwards, i did see them creating some autoconfig in C:\Program Files\Mozilla Firefox\defaults\pref turning on the enterprise root pref, but that wasn't creating any observable slowness on startup.
i also tried recreating the scenario of the underlying regressing change in bug 1571548, but that apparently isn't enough to trigger the slowness either. so i assume that you need to have some particular custom certs installed in the OS trust store to suffer from this bug, but can't provide a direct way to reproduce...

Flags: needinfo?(madperson)

I managed to get the "security.enterprise_roots.enabled" pref flipped to true and locked by the "kl_config_62fbb8f7_c917_4cf7_957a_aad2b8fa768c.cfg" file, but I still can't see a delay so I still can't actually verify it correctly.

What I did:

  1. Uninstalled all Firefoxes and Antivirus applications on a Windows7 x32 (and on a W7 x64) system.
  2. Installed Firefox Release v70.0 x32 (and x64).
  3. Installed Kaspersky Total Security (Trial version) from https://www.kaspersky.com/home-security#pc
  4. Performed the Kaspersky Updates and restarted OS.
  5. Opened browser and attempted to open a webpage quickly (www.9gag.com/www.reddit.com/www.yahoo.com)
    Observation:
    The kl_config_62fbb8f7_c917_4cf7_957a_aad2b8fa768c.cfg" file was created, the "security.enterprise_roots.enabled" pref was flipped to true and locked, but when attempting to open a first page of the session (from a bookmark or otherwise), no delay is observed. The site started loading almost instantly.

Some information is still missing from the steps to reproduce.
@foxy: Is there any information you could help me with? Am I doing something wrong? What did you do differently?

Flags: needinfo?(fred.steiny)

@Intrepid: Considering comment 35, it appears there's more to it than we understand. Will you help up validate the 3 fixed channels, please?
Nightly build: http://archive.mozilla.org/pub/firefox/nightly/2019/11/2019-11-29-09-42-47-mozilla-central/
Beta build: http://archive.mozilla.org/pub/firefox/releases/71.0b12/
ESR build: http://archive.mozilla.org/pub/firefox/candidates/68.3.0esr-candidates/build2/
Thank you!

Flags: needinfo?(Marc)

(In reply to Bodea Daniel [:danibodea] from comment #53)

Some information is still missing from the steps to reproduce.
@foxy: Is there any information you could help me with? Am I doing something wrong? What did you do differently?
Mabybe your firefox has a default startup website configured. Then only the start of the default website is delayed. And followup websites load without delay. Try to configure an empty page as startup website. Then website loaded by you is really the first website loaded - and should get a delay.

Flags: needinfo?(fred.steiny)

Yes, it seems to be fixed in all three builds below.

(In reply to Bodea Daniel [:danibodea] from comment #54)

@Intrepid: Considering comment 35, it appears there's more to it than we understand. Will you help up validate the 3 fixed channels, please?
Nightly build: http://archive.mozilla.org/pub/firefox/nightly/2019/11/2019-11-29-09-42-47-mozilla-central/
Beta build: http://archive.mozilla.org/pub/firefox/releases/71.0b12/
ESR build: http://archive.mozilla.org/pub/firefox/candidates/68.3.0esr-candidates/build2/
Thank you!

Flags: needinfo?(Marc)

Thank you for your verification!

Status: RESOLVED → VERIFIED
status-firefox71: fixed → verified
status-firefox72: fixed → verified
status-firefox-esr68: fixed → verified
Flags: qe-verify+

71.0 was just officially released. I tested it and the issue has been fixed. Thank You!

Remember to reverse these temporary changes IF you made them previously as a workaround to the issue:

Resetting back to Default:
STEP 1. Close Firefox. Run Notepad AS ADMINISTRATOR and open this file: C:\Program Files\Mozilla Firefox\kl_config_62fbb8f7_c917_4cf7_957a_aad2b8fa768c.cfg
Change "security.enterprise_roots.enabled" back to true as shown below. Then close notepad saving changes.

// kl_config_62fbb8f7_c917_4cf7_957a_aad2b8fa768c.cfg
lockPref("security.enterprise_roots.enabled", true);

STEP 2. Open Firefox and type this in the address bar: About:Config
Copy and Paste this into the search box: security.certerrors.mitm.auto_enable_enterprise_roots
Change the value back to: True.

Hello, I have still this bug to this day - every time I start mozilla firefox browser, I have few seconds delay before my start page loads.

My system specs: Windows 10 Pro 64 bits, Firefox Browser 73.0.1 (64 Bits), Kaspersky Internet Security (full, bought) 20.0.14.1085 (i)

Please file a new bug at https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=Security%3A%20PSM and attach the profile you get from following the instructions in comment 14.

Flags: needinfo?(specopsbarton)

I can create new topic it's not problem, but sorry, I don't want to start messing with firefox and break something on my pc.

Flags: needinfo?(specopsbarton)

Well, it's up to you, of course, but if you're not able to work with us to narrow down the issue, it's unlikely we'll be able to fix your issue. If it helps, the profiling add-on makes no permanent changes as far as I'm aware and you can remove it when you're done.

Hi specopsbarton,

Have you had any luck producing a profile?

Flags: needinfo?(specopsbarton)

Sorry, but like I said earlier, I dont want mess my firefox or anything else, also is there any other way to make log etc?

Flags: needinfo?(specopsbarton)
You need to log in before you can comment on or make changes to this bug.