1559114 - Performance of CRLite intermediates preload update can be improved

Status: RESOLVED FIXED

(Core :: Security: PSM, enhancement, P2)

Description

[Mathieu Leplatre [:leplatrem]]

A number of operations take a lot a time on my local machine.

For example, the JSON contains 2164 records that are imported in the local DB.

During the first call to .updatePreloadedIntermediates(), 2164 parallel operations will be executed to set cert_import_complete to false. IndexedDB hates that and it takes ages (several minutes). Plus, why set it to false? Why not just rely on falsy values when filtering?
See https://searchfox.org/mozilla-central/rev/c606cdd6d014fee4034da1702d484c0d41b604c9/security/manager/ssl/RemoteSecuritySettings.jsm#168-176

Maybe that's related and we can investigate how to do it, but it would be recommended to use the filtering feature of IndexedDB instead of doing it in memory on the 2100 records. See https://searchfox.org/mozilla-central/rev/c606cdd6d014fee4034da1702d484c0d41b604c9/security/manager/ssl/RemoteSecuritySettings.jsm#177-178

const waiting = await col.list({ filters: { cert_import_complete: false } });

Also, later, by default we take a batch of 100 records.
100 HTTP requests are issued in parallel. I believe it would be better to download those 100 chains in chunks of 4-8 parallel requests.

When we set cert_import_complete to true, we do 100 parallel updates, each in its own transaction. See https://searchfox.org/mozilla-central/rev/c606cdd6d014fee4034da1702d484c0d41b604c9/security/manager/ssl/RemoteSecuritySettings.jsm#204-207
It would be better to use a single IndexedDB transaction:

await kintoCollection.db.execute(transaction => {
    recordsToUpdate.forEach((record) => {
      transaction.update({...record, cert_import_complete: true});
    });
});

Comment 1

[Mathieu Leplatre [:leplatrem]]

Attachment: Bug 1559114 - Improve debug logging of Remote Settings

Comment 2

[Mathieu Leplatre [:leplatrem]]

Attachment: Bug 1559114 - Optimize signature verification of Remote Settings dumps

Depends on D34878

Comment 3

[Mathieu Leplatre [:leplatrem]]

Attachment: Bug 1559114 - Optimize CRLite intermediates download

Depends on D34879

Comment 4

[Mathieu Leplatre [:leplatrem]]

With the current patch the CRLite downloads is a lot more performant, it uses to take minutes and is now almost instantaneous.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=4ff21032a7606e0323c21c6e1a274c39ed1eb24c

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/da24e015b541
Improve debug logging of Remote Settings r=glasserc
https://hg.mozilla.org/integration/autoland/rev/7bcfdec398c1
Optimize signature verification of Remote Settings dumps r=glasserc
https://hg.mozilla.org/integration/autoland/rev/82a474df494f
Optimize CRLite intermediates download r=glasserc,jcj,keeler

Comment 6

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/da24e015b541
https://hg.mozilla.org/mozilla-central/rev/7bcfdec398c1
https://hg.mozilla.org/mozilla-central/rev/82a474df494f