Automatically encrypt SQLite databases on disk
Categories
(Core :: SQLite and Embedded Database Bindings, enhancement)
Tracking
()
People
(Reporter: nwipper, Assigned: nwipper)
References
(Blocks 4 open bugs)
Details
(Whiteboard: [size=3.5])
Attachments
(17 files, 5 obsolete files)
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review |
Encrypt all SQLite databases based on a key derived from the primary password, if available, or a default key.
| Assignee | ||
Comment 1•7 months ago
|
||
Updated•7 months ago
|
Updated•4 months ago
|
Updated•4 months ago
|
Updated•3 months ago
|
Updated•2 months ago
|
Updated•2 months ago
|
Updated•2 months ago
|
Comment 2•1 month ago
|
||
Follow-up patch on top of Nikolas's WIP that tightens the architecture
of the keystore + mozStorage integration without changing the
cryptographic primitives.
mozStorageService / Connection
- Centralize NSS init: drop scattered EnsureNSSInitializedChromeOrContent
release-asserts from CookieService, PermissionManager, LSObject, and
XPCOMInit. mozStorageService::Init now registers the keystore profile
observer; Connection::initialize calls EnsureNSSInitializedChromeOrContent
lazily on the encryption path and returns NS_ERROR_FAILURE on failure
instead of crashing the parent process. - Unify Connection::initialize paths: fold initializeSecure into
initialize(nsIFile*), have initialize(nsIFileURL*) also set
mDatabaseEncrypted=true when it injects ?key=, so GetDefaultPageSize
and initializeClone agree on whether the connection is encrypted. - Strip a leading "file:" scheme and any ?key= query from the path
returned by PRAGMA database_list so encrypted clones never produce a
double ?key=.
security/keystore module
- Replace the push-model SetCurrentProfilePath (8 sites in
nsToolkitProfileService.cpp) with a small KeyStorageObserver that
listens on profile-do-change / profile-after-change /
profile-before-change / xpcom-shutdown, resolves the profile dir on
the main thread via NS_GetSpecialDirectory, and caches it under
sKeyMutex. The observer also unregisters itself and clears its
StaticRefPtr at xpcom-shutdown so we don't leak. - Bump the keystore magic to "# mozilla secure key storage v1"; reject
unknown versions cleanly. - Base64-encode per-DB identifiers in the on-disk format so a Windows
path that legitimately contains ':' cannot break the parser. - Replace the appending writer with a full-file rewrite via
NS_NewSafeLocalFileOutputStream, so we get write-to-temp + rename
atomicity. Keep the wrapped system-key bytes in memory
(sSystemKeyWrapped) so each rewrite preserves the system-key line. - Fix GetKeyByPath to use InitWithPath(NS_ConvertUTF8toUTF16(...)) -
callers (e.g. PRAGMA database_list) supply UTF-8, not native bytes. - Reject DBs outside the profile directory with NS_ERROR_NOT_AVAILABLE
so mozStorage callers can fall back to plaintext for temp DBs.
Pref
- Switch security.storage.encryption.sqlite.enabled from mirror: once
to mirror: always (RelaxedAtomicBool) so browser-chrome tests that
set the pref via their manifest see the updated value. All consumers
drop the _AtStartup suffix from the accessor.
Tests
- Document the storage/marionette tests that legitimately cannot run
under encryption (page_size-locked tests, PBM-encryption tests, the
backup compatibility fixtures) and explain the override in each
manifest. - Make browser_connect.js robust under test-verify: use a unique DB
name per task run and remove -wal/-shm/-journal sidecars defensively
so iteration N+1 cannot inherit a stale WAL that obfsvfs cannot
decrypt.
Updated•1 month ago
|
Comment 3•1 month ago
|
||
Lock the pref in modules/libpref/init/all.js so end users cannot toggle
it from about:config or user.js, and add it to the Preferences
enterprise-policy allowlist so admins can still drive it from
policies.json.
The pref stays declared as a StaticPref in StaticPrefList.yaml; this
patch only adds the locking and policy plumbing.
Comment 4•1 month ago
|
||
Snapshot keystore.db / cookies.sqlite presence in key::Init (before any
storage consumer can touch them), so we can still reason about the
profile's pre-storage state even after the patched encryption code has
acted in this session.
Two policy paths:
- pref off + keystore.db existed at startup (encrypted profile, pref
pinned off): live-flip the pref via the unlock/setDefault/relock
dance from key::Init -- before the first DB open -- and continue
running. Idempotent across restarts; no _Exit needed. - pref on + no keystore.db at startup + cookies.sqlite existed at
startup (plaintext profile, pref pinned on by enterprise policy):
refuse to start with a clear FATAL message and std::_Exit(1) from
profile-after-change.
ObfuscatingVFS preserves the SQLite header verbatim, so distinguishing
encrypted from plaintext from on-disk content is unreliable; the
existence-snapshot is the only robust signal.
Comment 5•1 month ago
|
||
When a connection is opened with the encryption pref on and the file
already exists in plaintext form (SQLite header byte 20 == 0), copy
its schema, rows, indexes, views, and triggers into a sibling
.migrating file via obfsvfs and atomically swap. Idempotent on already-
encrypted files; cleans up -wal / -shm / -journal sidecars of the
plaintext source.
Wired into both Connection::initialize overloads (nsIFile* and
nsIFileURL*) right after the per-file key is fetched and before
sqlite3_open_v2. With migration in place, the case-2 fail-stop in
ApplyEncryptionPolicy is no longer reachable, so drop it (and the
profile-after-change observer wiring that called it).
Also adjust GetKeyByFile to fall back to NS_GetSpecialDirectory when
sProfilePath is empty (e.g. from the main-thread synthesized observer
fire before profile-do-change has actually run), so off-main-thread
callers get NS_ERROR_NOT_AVAILABLE rather than a crash.
Migration test added as a browser-chrome test alongside the existing
keystore tests; xpcshell isn't a viable host for these because NSS
SDR isn't initialized in stock xpcshell. The two existing tests now
also do the unlock/setDefault/relock dance to engage encryption,
since the all.js lock blocks the previous prefs = [...] override.
Updated•1 month ago
|
Comment 6•25 days ago
|
||
Follow-up to D270165 addressing reviewer comments outside security/keystore/:
- Drop duplicate hasKey/mDatabaseEncrypted assignment and clarify the
"outside profile" fallback comments in Connection::initialize. - Restore explicit
return rvat the three sqlite3_open failure sites
that the original patch had switched to NS_ENSURE_SUCCESS. - Strengthen the security.storage.encryption.sqlite.enabled pref
description to flag it INTERNAL / DO NOT ENABLE pending the
enterprise-policy gate in the rest of the stack. - Replace the mozilla_net_percent_encode Rust addition in
netwerk/base/idna_glue with a call to NS_EscapeURLSpan(esc_FilePath |
esc_Forced) inside a new shared helper, storage::PreparePathForURI,
exported via storage/StoragePathUtil.h. NS_EscapeURLSpan covers '?',
'#', '&', space, etc. -- not just '%'. - Expose obfsvfs::kObfsPageSize from ObfuscatingVFS.h so that
Connection::GetDefaultPageSize and ObfuscatingVFS.cpp share the same
8192 constant instead of duplicating literals. - Use storage::PreparePathForURI in toolkit/components/places/
Database.cpp::AttachDatabase before building the file: URI, so paths
containing URI-significant bytes don't produce malformed URIs. - Make ExtractURIPathAndQuery tolerant of bare filesystem paths. PRAGMA
database_list returns normalized filenames (no file: prefix), so the
encrypted-clone branch was previously returning NS_ERROR_FAILURE and
breaking Connection::initializeClone for encrypted DBs with attached
databases. - Track the pending mozIStoragePendingStatement returned by
attachDatabase in Sqlite.sys.mjs's _pendingStatements map, matching
the _executeStatement pattern, so shutdown can cancel in-flight
ATTACH operations instead of leaking them. - Parameterize test_page_size_is_32k.js on the encryption pref (8 KiB
when on, 32 KiB when off) and drop its pref override in xpcshell.toml. - Strengthen the comment in dom/indexedDB/test/marionette/
manifest.toml about the PBM x obfsvfs interaction gap and reference a
pending follow-up bug.
Updated•25 days ago
|
Comment 7•25 days ago
|
||
Delete the bespoke security/keystore module introduced by the parent
revision and route per-database obfsvfs keys through security/lockstore
using KekType::LocalKey. The lockstore collection name for each SQLite
database is the SHA-256 of the database's path relative to the active
profile directory; the underlying DEK is created extractable so that
obfsvfs can consume the raw 32-byte key via the ?key=... URI.
- New storage/SQLiteEncryption.{h,cpp} exposes
mozilla::storage::GetEncryptionKey and ShutdownEncryptionKeystore
on top of lockstore_ffi. - Three consumers rewired: mozStorageService::Init drops the explicit
key::Init() call; mozStorageService::Observe(xpcom-shutdown-threads)
calls ShutdownEncryptionKeystore(); mozStorageConnection and
toolkit/components/places/Database.cpp call GetEncryptionKey. - security/keystore/ deleted and dropped from security/moz.build.
- Browser-chrome tests moved to storage/test/browser/encryption/ with
the keystore.db existence assertion removed (lockstore manages its
own DB).
Updated•25 days ago
|
Updated•24 days ago
|
Updated•24 days ago
|
Updated•24 days ago
|
Updated•24 days ago
|
Updated•24 days ago
|
Updated•24 days ago
|
Updated•24 days ago
|
Updated•24 days ago
|
Updated•24 days ago
|
Updated•24 days ago
|
Updated•23 days ago
|
Comment 8•20 days ago
|
||
Flip the YAML default so encrypted-SQLite is on out of the box for any
profile that doesn't already have the pref locked elsewhere. NOTE: the
companion pref(..., false, locked) in modules/libpref/init/all.js from
D297713 still overrides this at runtime; remove that line or flip its
value to actually observe the new default end-to-end.
Comment 9•20 days ago
|
||
- Add browser/branding/enterprise/policies.json shipping
security.storage.encryption.sqlite.enabled=locked-true as the
Enterprise default. Wire it into branding-common.mozbuild so the
packaged distribution picks it up. - Flip the same pref default to true,locked in all.js so the
StaticPrefList.yaml change and all.js agree. - Fix FeltProcessParent::createProfile call to pass the 3rd "source"
argument required since Bug 2026972 (D290460, 2026-05-15) added it
to nsIToolkitProfileService.idl. Without this, Felt enters a
start/stop loop on every launch because createProfile throws
NS_ERROR_XPC_NOT_ENOUGH_ARGS.
Comment 10•20 days ago
|
||
Add an EncryptedDatabases marker under [Compatibility] in the profile's
compatibility.ini. The marker is read in CheckCompatibility at the very
top of XRE_mainStartup -- after profile lock acquisition but before
mDirProvider.SetProfile and any XPCOM/storage initialization. A new gate
CheckEncryptionCompatibility refuses to launch when the marker and the
launching build's security.storage.encryption.sqlite.enabled pref
disagree, so we never silently corrupt a profile by opening encrypted
databases with the wrong VFS (or vice versa).
To disambiguate "marker absent but profile has DBs", a 21-byte SQLite
header peek classifies the existing DBs as encrypted (page=8192,
reserved=32) or plaintext. Already-encrypted profiles from builds that
pre-date the marker self-heal on first launch with this code.
The marker is written from two hook points:
- storage/SQLiteEncryption.cpp ProfileObserver on profile-after-change:
runs before any DB is opened (per the verified startup ordering:
profile-do-change -> policies-startup -> profile-after-change ->
lazy mozStorageService init). Idempotent. - storage/SQLiteEncryption.cpp GetEncryptionKey on first-DEK-create:
defensive backup write for xpcshell / embedding paths where the
observer doesn't fire. Once per session.
nsIToolkitProfileService gains markCurrentProfileEncrypted(bool); the
writer (MaybeWriteEncryptionMarker in nsAppRunner) preserves existing
[Compatibility] keys, uses the same PR_TRUNCATE atomic-replace pattern
as WriteVersion, and skips the write when the on-disk value already
matches.
The gate is skipped in MOZ_BACKGROUNDTASKS mode -- those tasks operate
on profile files independently of any SQLite consumer.
Suggested by mossop. New strings under toolkit/locales for the two
refusal dialogs. New xpcshell tests cover the writer's happy path,
idempotency, and key-preservation behaviour.
Comment 11•11 days ago
|
||
Several components hand-roll a byte->lowercase-hex loop; gcp flagged (D301076) that the SQLite-encryption keystore was about to add yet another copy. Introduce one shared mozilla::HexEncode in xpcom/io (next to Base64Encode, its binary->text sibling) and move sandboxBroker's local copy onto it. mfbt would be the natural home but cannot depend on nsACString, so the helper lives in the XPCOM io/ module.
Comment 12•11 days ago
|
||
Several components hand-roll a byte->lowercase-hex loop; gcp flagged (D301076) that the SQLite-encryption keystore was about to add yet another copy. Introduce one shared mozilla::HexEncode in xpcom/io (next to Base64Encode, its binary->text sibling) and move sandboxBroker's local copy onto it. mfbt would be the natural home but cannot depend on nsACString, so the helper lives in the XPCOM io/ module.
Comment 13•11 days ago
|
||
Updated•11 days ago
|
Updated•10 days ago
|
Comment 14•8 days ago
|
||
Flips both the all.js runtime default and the StaticPrefList.yaml default
of security.storage.encryption.sqlite.enabled from false to true, and
drops the locked attribute from the all.js pref so per-test
prefs=[...=false] opt-outs (page-size-sensitive tests that need an
unencrypted 4096-byte page size) still work.
The YAML flip is required because the encryption gate in
XRE_mainStartup runs before mDirProvider.InitializeUserPrefs(), so a
runtime override in all.js alone wouldn't be visible to the gate; the
gate reads the StaticPrefs accessor which falls back to the YAML
default until prefs are loaded.
For try-side verification of the Case-3 (encryption enabled by default)
codepath only -- not for landing.
Updated•8 days ago
|
Updated•8 days ago
|
Comment 15•7 days ago
|
||
Until this commit, mozStorage's at-rest SQLite encryption stack
bootstrapped on a LocalKey (lockstore::kek::local:sqlite). That
LocalKey's kek_bytes field stores the raw AES-256 KEK plaintext
in lockstore.keys.sqlite -- a file that is itself plaintext
because kvstore-backed databases don't go through obfsvfs. So
anyone with same-UID filesystem access could lift the wrapping
key and decrypt every per-DB DEK, defeating the encryption
layer's confidentiality story.
Promote the SQLite-encryption bootstrap to a Password KEK whose
password is the console-supplied primarySecret. PasswordKekRecord
stores [salt | iterations | AES-GCM(ciphertext)] rather than
raw bytes (lib.rs:172-186) so a lockstore.keys.sqlite leak no
longer yields the KEK without also acquiring primarySecret.
Felt parent (FeltProcessParent.sys.mjs):
- Pre-fetch primarySecret via ConsoleClient.getPrimarySecret()
BEFORE spawning the Firefox child. Cache via Services.felt
.setPrimarySecret(hex) on the parent side. - On the very first post-spawn
.then()step, push it over the
existing Felt IPC channel via Services.felt.sendPrimarySecret(),
ahead ofsendAccessToken/sendReady(no firefoxReady gate).
Felt IPC (rust/{message.rs,utils.rs,components.rs,client.rs} +
nsIFelt.idl): mirror the SSOPassword plumbing for a new
PrimarySecret variant + PRIMARY_SECRET static + set/send/peek/clear
XPCOM surface.
Firefox child storage/SQLiteEncryption.cpp:
- At profile-do-change, EnsurePrimarySecretCached() polls
Services.felt.peekPrimarySecret() every 50ms up to 5s. The
primarySecret stash mirrors sHandle / sKekRef in lifecycle:
established at profile-do-change, cleared at quit-application. - GetEncryptionKey()'s bootstrap branch becomes an
unlock-or-create againstlockstore::kek::password:sqlite:- keystore_unlock_kek with TTL=u32::MAX ms (49.7d == effectively
session-unlimited; lockstore's 0 means "don't cache" not
"infinity"). - On NS_ERROR_NOT_AVAILABLE -> keystore_create_kek with the
same identifier; idempotent on the kek_ref dedup at
keystore.rs:1182-1185. - After the first-ever create, MigrateLocalToPasswordKek()
rotates every collection still carrying a local:sqlite
wrapping (add_kek + switch_kek + remove_kek), then deletes
the orphaned LocalKey record via delete_kek.
- keystore_unlock_kek with TTL=u32::MAX ms (49.7d == effectively
- Cache-expiry safety: if get_dek returns NS_ERROR_NOT_AVAILABLE
mid-session, transparently re-unlock with the cached
primarySecret and retry once before failing.
Failure modes (validated against the existing encryption gate):
- primarySecret never arrives -> Password KEK can't be
unlocked -> get_dek fails -> mozStorage refuses the open
-> CheckEncryptionCompatibility in nsAppRunner refuses launch
via the existing "Encrypted Profile" dialog. - primarySecret rotated server-side -> AEAD tag failure on
unlock surfaces as NS_ERROR_ABORT (LockstoreError::WrongPassword);
the storage layer fails opens, gate refuses launch. Recovery
requires SSO re-auth so the console re-issues primarySecret. - TTL expiry -> existing mozStorage connections survive (the
obfsvfs key was baked into the xFile state at open time per
mozStorageConnection.cpp); new opens hit the re-unlock path.
Out of scope: Felt's own scratch profile (T/felt-default) stays
on LocalKey -- Felt can't gate its own startup on a secret it
hasn't fetched yet. Its content sensitivity is bounded (SSO
session cookies, formhistory email, IdP URLs in places.sqlite);
high-value material (auth tokens, NSS keys) is gated behind the
SSO+primarySecret-derived NSS slot password, not the LocalKey.
Felt-profile hardening (OS keychain / ephemeral profile) is
filed as a follow-up.
Comment 16•6 days ago
|
||
Backported from enterprise-firefox 16d9ea9e4e79.
Threads an explicit key_size through keystore_create_dek (LockstoreService,
lockstore_ffi, keystore.rs, nsILockstore.idl); SQLiteEncryption passes
kDekBytes and static_asserts kDekBytes == IPCStreamCipherStrategy::KeyType size.
Also updates the lockstore gtest call sites (TestLockstoreKeystore.cpp,
TestLockstoreDatastore.cpp, TestLockstoreService.cpp) to the new 5-arg arity;
the enterprise commit updated only TestLockstoreService.cpp, leaving the other
two on the old signature (a latent build break).
Comment 17•6 days ago
|
||
Backported from enterprise-firefox ed9df1f698fa.
obfsvfs is the SQLite default VFS; obfsOpen owns the at-rest policy so keyless
sqlite3_open_v2 (rusqlite/skv/app-services) gets path-aware encryption. Per-file
key cached for shutdown journal finalization; PeekOnDiskHeader uses
page_size==8192 && reserved==32; lockstore.keys.sqlite bootstrap bypass;
no-CREATE-of-missing forwards to lower VFS.
(Enterprise Password-KEK/primarySecret + vault routing remain enterprise-only,
excluded; .cargo vendor-reorder dropped.)
Comment 18•6 days ago
|
||
Not for landing. Included in the stack so the rusqlite/skv default-VFS work can
be exercised with encryption forced ON.
Comment 19•6 days ago
|
||
Two defects broke mozIStorageAsyncConnection.backupToFileAsync when the
database is encrypted through obfsvfs:
-
The backup writes <dest>.tmp then renames it to <dest>. The per-database
key is keyed by the file's profile-relative path, and obfsvfs only strips
-wal/-journal suffixes (not .tmp), so the renamed backup had no key and
could not be reopened. RekeyEncryptedDatabaseForRename moves the DEK from
the .tmp collection to the final one after the rename (the key bytes are
unchanged, so the already-written ciphertext stays valid). -
A multi-step backup never completed. The pager validates that the source
has not changed by reading the 16-byte change-counter region (page 1,
offset 24) directly from the file on every shared-lock acquisition. obfsRead
only decoded full-page reads, so that read returned ciphertext for bytes
32-39 and never matched the decoded value the pager cached -- the pager
concluded the file changed and restarted the backup on every step. obfsRead
now serves that specific read from a decoded copy of page 1.
Also enables test_connection_online_backup.js (made mode-aware for the encrypted
page size) and removes its xpcshell.toml opt-out.
Comment 20•6 days ago
|
||
Instead of opting these storage/toolkit tests out of SQLite encryption, assert
the correct behavior in BOTH the encrypted and plaintext configurations
(detected via the security.storage.encryption.sqlite.enabled pref):
- test_storage_service: a directory-as-database open surfaces NS_ERROR_FAILURE
through obfsvfs (and records no open telemetry) vs NS_ERROR_FILE_ACCESS_DENIED
/ Glean "access" on the plain VFS. - test_vacuum: obfsvfs forces a fixed page size, so a VACUUM cannot change it
(expect unchanged vs 1024); and an encrypted database keeps full auto_vacuum
(1) rather than switching to incremental (2). - test_sqlite_autoVacuum: a fresh encrypted database reports full auto_vacuum
(1), which reclaims freed pages at commit, so there is no freelist for an idle
VACUUM to reclaim; assert that auto-reclaim behavior instead. - test_cache_size: the requested page size is ignored under encryption (cache
size is KiB-based and unchanged). - test_sqlite_secure_delete: an encrypted database never stores the plaintext on
disk, so the pre-delete "string is present" check only applies unencrypted;
the post-delete absence check holds in both modes.
Verified passing both with and without SQLite encryption.
test_connection_online_backup stays opted out for now with a TODO: it exposes a
real backup+encryption defect (the backup destination's DEK is not found on
reopen), which needs a code fix rather than a test change.
Updated•3 days ago
|
| Assignee | ||
Comment 21•3 days ago
|
||
Updated•1 day ago
|
Updated•1 day ago
|
Updated•1 day ago
|
Updated•1 day ago
|
Comment 22•1 day ago
|
||
The keystore previously closed at quit-application (AppShutdownConfirmed), the earliest shutdown phase, which forced obfsvfs to cache the raw per-file DEK (ObfsFile::aKey plus WAL/journal partner-inheritance) so it could finalize a connection's journals after the keystore was already gone. That early teardown was justified by a comment claiming LateWriteChecks fires at AppShutdownNetTeardown; it actually fires at XPCOMShutdownThreads (toolkit.shutdown.lateWriteChecksStage=2), so there is ample headroom to keep the keystore alive much later.
Keep lockstore available for late profile writes and shut it down last, at XPCOMWillShutdown -- after Places (profile-before-change) and QuotaManager/IndexedDB/DOM-storage (profile-before-change-qm) have closed their connections, and after AppShutdownTelemetry (past which nothing writes the profile). Both the storage-internal keystore handle (SQLiteEncryption) and the LockstoreService XPCOM handle now tear down on xpcom-will-shutdown; InitEncryptionKeystore refuses to register/open once already in or beyond XPCOMWillShutdown.
With lockstore alive that late, obfsvfs no longer needs to cache the DEK: WAL and journal opens fall through to the policy branch and re-derive the same per-database key from lockstore on demand (DeriveMainDbPath strips the -wal/-journal suffix), including the final checkpoint at a connection's close during shutdown. Remove ObfsFile::aKey, the partner-inheritance fast path, and the now-dead keyReady flag; pPartner is retained for checkpoint coordination.
Updated•1 day ago
|
Updated•1 day ago
|
Updated•1 day ago
|
Updated•1 day ago
|
Updated•1 day ago
|
Updated•1 day ago
|
Updated•1 day ago
|
Updated•1 day ago
|
Description
•