Performance of CRLite intermediates preload update can be improved

RESOLVED FIXED in Firefox 69

Status

()

enhancement
P2
normal
RESOLVED FIXED
Last month
27 days ago

People

(Reporter: leplatrem, Assigned: leplatrem)

Tracking

unspecified
mozilla69
Points:
---

Firefox Tracking Flags

(firefox69 fixed)

Details

Attachments

(3 attachments)

A number of operations take a lot a time on my local machine.

For example, the JSON contains 2164 records that are imported in the local DB.

During the first call to .updatePreloadedIntermediates(), 2164 parallel operations will be executed to set cert_import_complete to false. IndexedDB hates that and it takes ages (several minutes). Plus, why set it to false? Why not just rely on falsy values when filtering?
See https://searchfox.org/mozilla-central/rev/c606cdd6d014fee4034da1702d484c0d41b604c9/security/manager/ssl/RemoteSecuritySettings.jsm#168-176

Maybe that's related and we can investigate how to do it, but it would be recommended to use the filtering feature of IndexedDB instead of doing it in memory on the 2100 records. See https://searchfox.org/mozilla-central/rev/c606cdd6d014fee4034da1702d484c0d41b604c9/security/manager/ssl/RemoteSecuritySettings.jsm#177-178

const waiting = await col.list({ filters: { cert_import_complete: false } });

Also, later, by default we take a batch of 100 records.
100 HTTP requests are issued in parallel. I believe it would be better to download those 100 chains in chunks of 4-8 parallel requests.

When we set cert_import_complete to true, we do 100 parallel updates, each in its own transaction. See https://searchfox.org/mozilla-central/rev/c606cdd6d014fee4034da1702d484c0d41b604c9/security/manager/ssl/RemoteSecuritySettings.jsm#204-207
It would be better to use a single IndexedDB transaction:

await kintoCollection.db.execute(transaction => {
    recordsToUpdate.forEach((record) => {
      transaction.update({...record, cert_import_complete: true});
    });
});
Assignee: nobody → mathieu

With the current patch the CRLite downloads is a lot more performant, it uses to take minutes and is now almost instantaneous.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=4ff21032a7606e0323c21c6e1a274c39ed1eb24c

Pushed by mleplatre@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/da24e015b541
Improve debug logging of Remote Settings r=glasserc
https://hg.mozilla.org/integration/autoland/rev/7bcfdec398c1
Optimize signature verification of Remote Settings dumps r=glasserc
https://hg.mozilla.org/integration/autoland/rev/82a474df494f
Optimize CRLite intermediates download r=glasserc,jcj,keeler
You need to log in before you can comment on or make changes to this bug.