1538372 - Heavy disk writes after an update to (profile)/security_state/data.mdb

Status: RESOLVED FIXED

(Core :: Security: PSM, defect)

Description

[Jim Mathies [:jimm]]

Myk pointed me to this component. This file is associated with a rust module 'rkv' which we've started using for certificate storage.

STR:

1. open resource monitor on Windows
2. open an instance after a nightly update
3. note access to security_state\data.mdb

I see, on average, about 500mb of writes to this file right at startup. This feels like it's dragging browser startup down quite a bit.

Windows 7, 64-bit, multiple profiles. Each profile exhibits the behavior after an update.

Image PID File Read (B/sec) Write (B/sec) Total (B/sec) I/O Priority Response Time (ms)
firefox.exe 7020 C:\Users\jim\AppData\Roaming\Mozilla\Firefox\Profiles\ravw10ms.mozilla-1546466584255\security_state\data.mdb 0 454,174 454,174 Normal 12

Image PID File Read (B/sec) Write (B/sec) Total (B/sec) I/O Priority Response Time (ms)
firefox.exe 7196 C:\Users\jim\AppData\Roaming\Mozilla\Firefox\Profiles\2ykirmgk.stuff\security_state\data.mdb 0 371,371 371,371 Normal 5

Comment 1

[Myk Melez [:myk] [@mykmelez]]

Jim, do you see the same behavior after you restart the browser with a profile that exhibited this on initial startup after Firefox update? I wonder if this is a one-time cost of migrating cert records into the new security_state store. (Even if so, we might be able to improve the efficiency of the migration. I'm only asking in order to understand the scope of the issue better.)

Comment 2

[Jim Mathies [:jimm]]

(In reply to Myk Melez [:myk] [@mykmelez] from comment #1)

Jim, do you see the same behavior after you restart the browser with a
profile that exhibited this on initial startup after Firefox update? I
wonder if this is a one-time cost of migrating cert records into the new
security_state store. (Even if so, we might be able to improve the
efficiency of the migration. I'm only asking in order to understand the
scope of the issue better.)

You're guess seem good, it's only after an update, and only on first run after the update. Also profile specific too since each profile has its own copy of this file.

Comment 3

[Myk Melez [:myk] [@mykmelez]]

One potential issue is that SecurityState.migrate() calls set_revocation_by_subject_and_pub_key() or set_revocation_by_issuer_and_serial() for each (relevant) line in revocations.txt, and those functions call write_entry(), which creates a transaction, writes the entry, and then commits the transaction.

Depending on the number of lines in the source file, it could be much more efficient for SecurityState.migrate() to create and commit a single transaction containing all the entries to be migrated.

Comment 4

[Myk Melez [:myk] [@mykmelez]]

Attachment: migrate revocations in single transaction

cert_storage migrates revocations.txt via one transaction per entry, which can be expensive. This change uses a single transaction to migrate all entries.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ca2216061827
migrate revocations in single transaction r=keeler

Comment 6

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/ca2216061827