Closed Bug 1591691 Opened 22 days ago Closed 15 days ago

importing enterprise roots can cause network I/O

Categories

(Core :: Security: PSM, defect, P1)

70 Branch
defect

Tracking

()

RESOLVED FIXED
mozilla72
Tracking Status
firefox-esr68 ? affected
firefox70 --- affected
firefox71 --- fixed
firefox72 --- fixed

People

(Reporter: fred.steiny, Assigned: keeler)

Details

(Whiteboard: [psm-assigned])

Attachments

(6 files)

User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0

Steps to reproduce:

  1. Start Firefox 70 on Windows 7
  2. Open a website
    => 10 sec delay

Actual results:

On any website it takes 10 sec before page loads for the first website since firefox startup. All followup websites load fast.

Does not happen with Google Chrome.

Happens also in firefox safemode with new profile.

No Proxy or DNS settings set.

Happens also with disabled Kapersky security software

Expected results:

No 10 sec delay on first page load after firefox startup.

Attached file FirefoxLog.zip

firefox logfile

i'm confirming this bug on the basis of seeing a few similar reports from other users after the firefox 70 update:
https://support.mozilla.org/en-US/questions/1271170
https://old.reddit.com/r/firefox/comments/dm0yxx/firefox_taking_forever_to_load_first_page_when/

also setting some ni's to get it on the radar of the necko and performance teams...

Status: UNCONFIRMED → NEW
Ever confirmed: true
Flags: needinfo?(vchin)
Flags: needinfo?(nhnguyen)
Keywords: regression

hello foxy, would you be able to narrow the problem down with the help of the mozregression tool from https://mozilla.github.io/mozregression/ pointing to your profile folder?

Yes, it happens even with empty profile. I attached a screenshot of result, but my last choice was wrong it was not green good but red bad 3d798a3c. (And I could not rewise my wrong choice, so feature request for mozregression tool)

Attached image Firefox-regression.PNG

is security.enterprise_roots.enabled set to "true" in about:config in your case?

Attached image Firefox-regression2.PNG

Yes security.enterprise_roots.enabled set to "true" in about:config with locked value by Kaspersky Labs AntiVirus.

After deleting content of file C:\Program Files\Mozilla Firefox\defaults\pref\kl_prefs_62fbb8f7_c917_4cf7_957a_aad2b8fa768c.js and write protect file to not get reverted, Firefox 70 works again without delay on first page after firefox startup.

thank you, so the problem is probably related to bug 1571548 which landed on nightly in the range you came up with (not where it ended up on the autoland branch though).

Flags: needinfo?(dkeeler)
Summary: After firefox startup first page load is delayed for 10 sec → After firefox startup first page load is delayed for some seconds

I can Confim this bug:
Win64 1903 (18362.449)
FF 70 (release) - No matter if full installtion (updated from 69 or new), portable or with new profile or in safe mode.
Kaspersky 20.0.14.1085 - No matter if deactivated or on pause.

I confirm as well that using following empty file (with write protection) resolve this issue. Thanks @foxy!
C:\Program Files\Mozilla Firefox\defaults\pref\kl_prefs_62fbb8f7_c917_4cf7_957a_aad2b8fa768c.js

(In reply to djmaniac44 from comment #12)

I can Confim this bug:
Win64 1903 (18362.449)
FF 70 (release) - No matter if full installtion (updated from 69 or new), portable or with new profile or in safe mode.
Kaspersky 20.0.14.1085 - No matter if deactivated or on pause.

I confirm as well that using following empty file (with write protection) resolve this issue. Thanks @foxy!
C:\Program Files\Mozilla Firefox\defaults\pref\kl_prefs_62fbb8f7_c917_4cf7_957a_aad2b8fa768c.js

As an option we can set
C:\Program Files\Mozilla Firefox\kl_config_62fbb8f7_c917_4cf7_957a_aad2b8fa768c.cfg
lockPref("security.enterprise_roots.enabled", ture);
TO
lockPref("security.enterprise_roots.enabled", false);

Can anyone who can reproduce this bug please attempt to get a profile by following the directions at: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Reporting_a_Performance_Problem
In particular, please include the StreamTrans threads by checking the corresponding box and add in ,ssl manually to the list of threads.
Thanks!

Flags: needinfo?(dkeeler) → needinfo?(fred.steiny)

I have the same Issue. With a new default profile, the homepage upon startup takes over 20 seconds to load (“Performing a TLS handshake").
However, if I change my home page to a NON-https site, the issue does not occur.
==>> Important Hint to this bug: If I delete the CONTENTS of this file:
C:\Program Files\Mozilla Firefox\defaults\pref\kl_prefs_62fbb8f7_c917_4cf7_957a_aad2b8fa768c.js
and then set the file to READ ONLY, the problem is resolved.

Thank You.

I can confirm that this also fixes the issue:

In this File: C:\Program Files\Mozilla Firefox\kl_config_62fbb8f7_c917_4cf7_957a_aad2b8fa768c.cfg
Change "security.enterprise_roots.enabled" to false as below:

// kl_config_62fbb8f7_c917_4cf7_957a_aad2b8fa768c.cfg
lockPref("security.enterprise_roots.enabled", false);

By the way, I use Eset NOD32 v.13.0.22.0

Related:
How to disable the Enterprise Roots preference
https://support.mozilla.org/en-US/kb/how-disable-enterprise-roots-preference
Note: The option security.enterprise_roots.enabled can not be changed in about:config because it's locked.
The C:\Program Files\Mozilla Firefox\kl_config_62fbb8f7_c917_4cf7_957a_aad2b8fa768c.cfg file must be edited manually.

"Starting with Firefox version 68, when a TLS connection error occurs Firefox will automatically enable the Enterprise Roots preference and attempts to connect again. If the issue is resolved, then the Enterprise Roots preference remains enabled. However, you may want to disable this behavior, so this article explains how to do just that without compromising security. "

(In reply to Dana Keeler (she/her) (use needinfo) (:keeler for reviews) from comment #14)

Can anyone who can reproduce this bug please attempt to get a profile by following the directions at: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Reporting_a_Performance_Problem
In particular, please include the StreamTrans threads by checking the corresponding box and add in ,ssl manually to the list of threads.
Thanks!

https://perfht.ml/2Ps6pFJ
https://perfht.ml/36fUgtq

Flags: needinfo?(fred.steiny)

Thank you!

Assignee: nobody → dkeeler
Keywords: regression
Priority: -- → P1
Summary: After firefox startup first page load is delayed for some seconds → importing enterprise roots can cause network I/O
Whiteboard: [psm-assigned]
Flags: needinfo?(vchin)
Flags: needinfo?(nhnguyen)
Component: Untriaged → Security: PSM
Product: Firefox → Core

Yes, fixed.

Flags: needinfo?(fred.steiny)

(In reply to Dana Keeler (she/her) (use needinfo) (:keeler for reviews) from comment #21)

Does this build fix the issue for you?
64-bit: https://queue.taskcluster.net/v1/task/FUMsRpvyTRKvOyqrcEQEeA/runs/0/artifacts/public/build/install/sea/target.installer.exe
32-bit: https://queue.taskcluster.net/v1/task/JIrnaCPeRYyVhX-aJSUFWQ/runs/0/artifacts/public/build/install/sea/target.installer.exe

Yes - way faster now but still a bit slower compared to FF 69.0.3

FF 69.0.3 - https://www.google.de/ (as Homepage) ~ 2 sec after start
FF 72.0a1 (2019-10-29) - https://www.google.de/ (as Homepage) ~ 3 sec after start

But I guess this is OK and may be related to new functions of FF 70+ ?!

Can you get a performance profile with the new build as described in comment 14? Thanks!

Flags: needinfo?(djmaniac44)

(In reply to Dana Keeler (she/her) (use needinfo) (:keeler for reviews) from comment #27)

Can you get a performance profile with the new build as described in comment 14? Thanks!

Don't know what happend but now FF 72.0a1 (2019-10-29) is as fast as FF 69.0.3.
Maybe FF 72.0a1 (2019-10-29) did some stuff after first installation.
Please disregard https://bugzilla.mozilla.org/show_bug.cgi?id=1591691#c26
Thanks for your work here.
I call it fixed as well.

Flags: needinfo?(djmaniac44)
Pushed by dkeeler@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/8a690dff4180
avoid network I/O when importing enterprise roots on Windows r=mhowell

Razvan - next time ping me in slack, please (particularly for a tier-2 breakage).
Apparently mingw's version of that header doesn't include the flag CERT_CHAIN_DISABLE_AIA.

Flags: needinfo?(dkeeler)
Pushed by dkeeler@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/7645d149e3f4
avoid network I/O when importing enterprise roots on Windows r=mhowell
Pushed by dkeeler@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/f7efa3e6089e
avoid network I/O when importing enterprise roots on MacOS r=spohl
Status: NEW → RESOLVED
Closed: 15 days ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla72

Will the fix also be released for firefox 70 as hotfix? Because else stable release users will need to wait till next year...

Comment on attachment 9105052 [details]
bug 1591691 - avoid network I/O when importing enterprise roots on Windows r?mhowell

Beta/Release Uplift Approval Request

  • User impact if declined: Slow first start (on the order of many seconds) for users that have enterprise certificates with remote revocation information.
  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Nightly?: Yes
  • Needs manual test from QE?: No
  • If yes, steps to reproduce:
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): This is a small change - it adds a flag to tell the OS to not do network fetches when verifying the candidate enterprise certificates to import. For roots and intermediate DV certificates we don't do active revocation checking (and imported certificates can't be EV anyway), so this behavior is the same as if the user had manually imported the certificates, so it shouldn't be a security risk.
  • String changes made/needed: none

ESR Uplift Approval Request

  • If this is not a sec:{high,crit} bug, please state case for ESR consideration: Enterprise-facing feature.
  • User impact if declined: Slow first start (on the order of many seconds) for users that have enterprise certificates with remote revocation information.
  • Fix Landed on Version: 72
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): This is a small change - it adds a flag to tell the OS to not do network fetches when verifying the candidate enterprise certificates to import. For roots and intermediate DV certificates we don't do active revocation checking (and imported certificates can't be EV anyway), so this behavior is the same as if the user had manually imported the certificates, so it shouldn't be a security risk.
  • String or UUID changes made by this patch: none
Attachment #9105052 - Flags: approval-mozilla-esr68?
Attachment #9105052 - Flags: approval-mozilla-beta?
Attachment #9105056 - Flags: approval-mozilla-esr68?

Comment on attachment 9105052 [details]
bug 1591691 - avoid network I/O when importing enterprise roots on Windows r?mhowell

Low risk fix for a significant Windows perf regression, verified by the reporter in Nightly, uplift approved for 71 beta 8, thanks.

Attachment #9105052 - Flags: approval-mozilla-beta? → approval-mozilla-beta+

Dana, should Attachment #9105056 [details] also have an uplift request to beta? Thanks

Flags: needinfo?(dkeeler)

Comment on attachment 9105056 [details]
bug 1591691 - avoid network I/O when importing enterprise roots on MacOS r?spohl

Beta/Release Uplift Approval Request

  • User impact if declined: (see comment 39)
  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Nightly?: Yes
  • Needs manual test from QE?: Yes
  • If yes, steps to reproduce:
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky):
  • String changes made/needed:
Flags: needinfo?(dkeeler)
Attachment #9105056 - Flags: approval-mozilla-beta?
Attachment #9105056 - Flags: approval-mozilla-beta? → approval-mozilla-beta+

(In reply to foxy from comment #37)

Will the fix also be released for firefox 70 as hotfix? Because else stable release users will need to wait till next year...

It seems this fix won't be released until v.72. In the meantime, here are the steps to fix the issue:

First Close Firefox.

Step 1. Open this File in NOTEPAD: C:\Program Files\Mozilla Firefox\kl_config_62fbb8f7_c917_4cf7_957a_aad2b8fa768c.cfg
Change "security.enterprise_roots.enabled" from true to false as shown below. Then close notepad saving changes.

// kl_config_62fbb8f7_c917_4cf7_957a_aad2b8fa768c.cfg
lockPref("security.enterprise_roots.enabled", false);

Step 2. Open Firefox and type this in the address bar: About:Config
Copy and Paste this into the search box: security.certerrors.mitm.auto_enable_enterprise_roots
Change the value from True to False by simply double-clicking on it.

Note: Keep these instructions. When Firefox is fixed with a new version (probably v.72), reverse these changes!

Sorry, I forget one important Step: You must run Notepad as Administrator by right clicking on Notepad and selecting "Run As Administrator".
You won't be able to save changes otherwise.

[Tracking Requested - why for this release]: performance issue for enterprise-facing feature (see comment 39)

You need to log in before you can comment on or make changes to this bug.