Closed Bug 2034086 Opened 7 months ago Closed 16 days ago

Inaccurate Usage in navigator.storage.estimate() from MV3 Extension

Categories

(Core :: Storage: Bucket File System, defect)

Firefox 149
defect

Tracking

()

RESOLVED FIXED
161 Branch
Tracking Status
firefox161 --- fixed

People

(Reporter: cam092, Assigned: jari)

Details

Attachments

(1 file, 3 external files)

Attached file metadata.sqlite —

Steps to reproduce:

Call navigator.storage.estimate() from an MV3 web extension's service worker.

Actual results:

It returns

quota: 10737418240
usage: 9674719280

But running (await navigator.storage.getDirectory()).keys().next() shows no files:

Object { done: true, value: undefined }

Restarting the extension does not change anything, neither does restarting the browser. Creating and deleting other files doesn't change anything.
This occurred on Intel Mac OS X 10.15.
UA: UA: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:149.0) Gecko/20100101 Firefox/149.0

Expected results:

Usage should be more accurate.

It appears that the metadata.sqlite became corrupted. Metadata.sqlite should not become corrupted in this way, or if it does, usage should eventually be corrected.

I don't know how it became corrupted, but a theory is that quota being exceeded somehow caused this. The usage is close to the limit, so maybe there was a race condition triggered when quota was exceeded.

I'm attaching the metadata.sqlite here for reference. There appears to be an orphaned file. The 8FA... handle exists in FileIds and Usages but not in Files or Entries.

sqlite> select hex(handle), name from Directories;
134007117DBE8BE37523916C26CBD89EA65678F10E703C652AD036840E93B8E0|root
sqlite> select hex(handle), parent from Entries;
134007117DBE8BE37523916C26CBD89EA65678F10E703C652AD036840E93B8E0|
sqlite> select hex(fileId), handle from FileIds;
sqlite> select hex(handle), usage, tracked from Usages;
8FA29A5FF357A0EE3EF1D9F481742462AED65C1F841E32E7ED9FB0A79E3E4881|9674162224|0
8FA29A5FF357A0EE3EF1D9F481742462AED65C1F841E32E7ED9FB0A79E3E4881|
sqlite> select count(1) from MainFiles;
0
sqlite> select count(1) from Files;
0
sqlite> select count(1) from RehashedFrom001to002;
0

I have access to the Firefox profile folder in which the issue occurred and can provide more details & non-sensitive data if necessary.

The Bugbug bot thinks this bug should belong to the 'WebExtensions::Untriaged' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Product: Firefox → WebExtensions

The Bugbug bot thinks this bug should belong to the 'WebExtensions::Untriaged' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

This is affecting Video DownloadHelper addon. navigator.storage says 10Gb is used while 0Gb is used, making the addon fail to download files.

Thanks for reporting the issue! Moving this to the correct component.

Severity: -- → S3
Component: Untriaged → Storage: Bucket File System
Product: WebExtensions → Core
Flags: needinfo?(jjalkanen)
Assignee: nobody → jjalkanen
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true

Here's an explanation of what happened: There are two interacting defects.

  1. PRAGMA foreign_keys was not enabled at connection open.

SQLite defaults to foreign_keys = OFF and GetStorageConnection in FileSystemDataManager.cpp opened the connection without setting the pragma, so every ON DELETE CASCADE and ON DELETE SET NULL constraint declared in the schema was a dead letter at runtime.

  1. The removal code preserved FileIds rows for failed disk removals, permanently reserving quota.

When RemoveFile or RemoveDirectoryImpl deleted an OPFS entry, it tried to remove each blob from disk, collected failures into failedRemovals, and then only called RemoveFileId for blobs that were successfully removed, skipping it for the failed ones. The intent was to keep the FileIds row around so TryRemoveDuringIdleMaintenance could retry later.

But after that, DeleteEntry runs and cascades through Entries -> Files, removing the file metadata. Because foreign key enforcement was off, the FileIds foreign key did not fire and the FileIds row ended up with a dangling handle referencing a deleted Files row. The Usages row persists too, permanently reserving the blob's quota.

The same pattern existed in V002's MergeFileId, on the write-new-content path: when blob removal failed, it set FileIds.handle = NULL to orphan the row but left the Usages row and cached quota untouched.

How the problematic state was reached:

The extension wrote ~9 GB to OPFS. At some point the file was deleted or its content replaced. The disk blob removal failed. The entry/file metadata was cleaned up through DeleteEntry cascades, and eventually the FileIds row was removed too. But because foreign key constraints were off, the Usages cascade never fired, leaving a single Usages row claiming 9,674,162,224 bytes for a fileId (8FA2...) that no longer exists in any other table. navigator.storage.estimate() reports ~9 GB used; getDirectory().keys() shows no files.

Flags: needinfo?(jjalkanen)

Usages rows with no FileIds parent from the time before foreign key
enforcement was enabled are removed in RemoveFileIdsWithoutMainFile
which runs once per DataManager open.

So if this issue hits and the fix or fixes attached to this bug are not available, there is a workaround to get things working again. However, following the steps below will deletes all data for that origin.

Firefox must be fully closed first.

  1. Open the profile folder from about:support ("Profile Folder" / "Open Folder").
  2. Go to storage/default/ and delete the folder for the affected origin. For a web site that is https+++<host>. For an extension it is moz-extension+++<extension-uuid>; the UUID is in the extension's moz-extension:// URL in about:debugging.
  3. Start Firefox again. Now, the broken origin won't cause trouble again.

For web sites, "Clear Data" for the site in Settings > Privacy & Security > Cookies and Site Data should also remove the same data without touching the filesystem. Extension origins don't appear there, so they need the manual steps above.

Pushed by smolnar@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/5e16dfe8613d https://hg.mozilla.org/integration/autoland/rev/2e57f0eec64c Revert "Bug 2034086: Remove usage rows whose file id no longer exists. r=dom-storage-reviewers,hsingh" for causing xpc failures @ test_orphanUsageCleanup.js

Backed out for causing xpc failures @ test_orphanUsageCleanup.js

Backout link: https://hg.mozilla.org/integration/autoland/rev/2e57f0eec64cde686e3fee5a4b3d0bcbe962ab6d

Push with failures

Failure log ->

Flags: needinfo?(jjalkanen)

Remove orphan blobs before counting their usage so stale records cannot
trigger quota eviction. Share this cleanup with database open for origins
whose quota initialization used cached usage.

Delete metadata only after successful blob removal, and retain failed
candidates and their usage for retry while continuing through the list.
Update live quota accounting after committing each successful cleanup.

Pushed by jjalkanen@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/5644d1997104 https://hg.mozilla.org/integration/autoland/rev/40f30ae08b46 Make fileIds unknown on failed removals. r=dom-storage-reviewers,hsingh https://github.com/mozilla-firefox/firefox/commit/f1622f45913f https://hg.mozilla.org/integration/autoland/rev/addfa34472cf Remove usage rows whose file id no longer exists. r=dom-storage-reviewers,hsingh https://github.com/mozilla-firefox/firefox/commit/e2fb7e45c72c https://hg.mozilla.org/integration/autoland/rev/6042d2d55e07 Clean up orphan OPFS files during quota rescan and database open. r=dom-storage-reviewers,hsingh
Status: ASSIGNED → RESOLVED
Closed: 16 days ago
Resolution: --- → FIXED
Target Milestone: --- → 161 Branch
Flags: needinfo?(jjalkanen)
QA Whiteboard: [qa-triage-done-c162/b161]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: