Initializing NSS requires too much IO

NEW
Unassigned

Status

()

Core
Security: PSM
P3
normal
a year ago
a year ago

People

(Reporter: florian, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [psm-tracking])

(Reporter)

Description

a year ago
See this cold startup profile where initializing NSS takes more than one second: https://perfht.ml/2sf6j95

This is mostly due to disk IO.

Possible improvements:
- We have some facilities such as mozilla::ReadAhead() for reading a file sequentially before loading it, could this help here?
- Can we look into perhaps folding the NSS code into fewer libraries?
- It seems some DLL are loaded without passing the fully qualified path name to the DLL to LoadLibrary, causing Windows to waste some time searching for the file on disk even though we know exactly where it is.
- once the IO costs are out of the picture (ie. if we fix the above points, or for warm startup) there's also likely more code that runs than we really need. Could someone who knows this code review the above profile to see if there are low hanging fruit? Apparently ehsan already found bug 1370667.

David, could you please look at this to decide what the next steps are, or find someone who can?

Note: If I had to choose, I would give a higher priority to bug 1370516 as I expect it to have a bigger impact on our startup times, but reducing IO (even off main thread) is still something worth doing.
Flags: needinfo?(dkeeler)
(In reply to Florian Quèze [:florian] [:flo] from comment #0)
> See this cold startup profile where initializing NSS takes more than one
> second: https://perfht.ml/2sf6j95
> 
> This is mostly due to disk IO.

Looks like the majority (~60%) of this is due to loading the PKCS#11 module that implements the default trust anchor database (i.e. the built-in CA list). Luckily, this would be fairly easy to make independent of initializing PSM/NSS (and even do it on a background thread a-la bug 1370516). (The only tricky part would be identifying every place where we depend on the built-in trust database and ensuring that that code waits for the initialization to finish.)

> Possible improvements:
> - We have some facilities such as mozilla::ReadAhead() for reading a file
> sequentially before loading it, could this help here?

Potentially could, if we can identify where the file is (see https://dxr.mozilla.org/mozilla-central/rev/5801aa478de12a62b2b2982659e787fcc4268d67/security/manager/ssl/nsNSSComponent.cpp#1100 as it suggests we don't always know exactly where the library is - but maybe this is an outdated assumption).

> - Can we look into perhaps folding the NSS code into fewer libraries?

That might help NSS initialization in general, but it wouldn't help in this case - the built-in CA module must be a separate library due to architecture constraints.

> - It seems some DLL are loaded without passing the fully qualified path name
> to the DLL to LoadLibrary, causing Windows to waste some time searching for
> the file on disk even though we know exactly where it is.

Yeah, that seems strange - we should try to avoid that.

> - once the IO costs are out of the picture (ie. if we fix the above points,
> or for warm startup) there's also likely more code that runs than we really
> need. Could someone who knows this code review the above profile to see if
> there are low hanging fruit? Apparently ehsan already found bug 1370667.

We can probably save 24ms by not initializing CT information since it's disabled by default anyway (bug 1353216). I don't really see any other low-hanging fruit in that profile.

> David, could you please look at this to decide what the next steps are, or
> find someone who can?
> 
> Note: If I had to choose, I would give a higher priority to bug 1370516 as I
> expect it to have a bigger impact on our startup times, but reducing IO
> (even off main thread) is still something worth doing.

Maybe a good approach would be to see about moving the root loading to a background thread first. At that point, it may be more clear how to move the rest of NSS initialization off the main thread.
Flags: needinfo?(dkeeler)

Comment 2

a year ago
(In reply to David Keeler [:keeler] (use needinfo?) from comment #1)
> > Possible improvements:
> > - We have some facilities such as mozilla::ReadAhead() for reading a file
> > sequentially before loading it, could this help here?
> 
> Potentially could, if we can identify where the file is (see
> https://dxr.mozilla.org/mozilla-central/rev/
> 5801aa478de12a62b2b2982659e787fcc4268d67/security/manager/ssl/nsNSSComponent.
> cpp#1100 as it suggests we don't always know exactly where the library is -
> but maybe this is an outdated assumption).

It seems like that's only a by-product of how bug 712579 was fixed, which should be possible to change.

> > David, could you please look at this to decide what the next steps are, or
> > find someone who can?
> > 
> > Note: If I had to choose, I would give a higher priority to bug 1370516 as I
> > expect it to have a bigger impact on our startup times, but reducing IO
> > (even off main thread) is still something worth doing.
> 
> Maybe a good approach would be to see about moving the root loading to a
> background thread first. At that point, it may be more clear how to move the
> rest of NSS initialization off the main thread.

That seems like a great plan!
Depends on: 1291886
Priority: -- → P3
Whiteboard: [psm-tracking]
Depends on: 1372656
No longer depends on: 1291886
You need to log in before you can comment on or make changes to this bug.