Closed Bug 934640 Opened 11 years ago Closed 22 days ago

Implement IDBFactory.databases() to enumerate IndexedDB databases

Categories

(Core :: Storage: IndexedDB, enhancement, P2)

Product:
Core
Component:
Storage: IndexedDB
Type:
enhancement

Tracking

()

Status:
RESOLVED FIXED
Milestone:
126 Branch
Tracking Flags:
Tracking Status
relnote-firefox --- ?
firefox126 --- fixed

People

(Reporter: daleharvey, Assigned: janv)

References

(Blocks 2 open bugs, Regressed 1 open bug)

Details

(Keywords: dev-doc-needed, parity-chrome, parity-safari, Whiteboard: DWS_NEXT, [wptsync upstream])

Attachments

(13 files, 3 obsolete files)

Bug 934640 - Added IDBFactory.databases function to webidl; r=#dom-storage
47 bytes, text/x-phabricator-request
Details | Review
Bug 934640 - Let 'Ensure that databases() returns a promise' test case accept a rejected Promise to match its description. r=#dom-workers-and-storage
47 bytes, text/x-phabricator-request
Details | Review
Bug 934640 - Create a new base for factory operations communicating using PBackgroundIDBFactoryRequest; r=#dom-storage
48 bytes, text/x-phabricator-request
Details | Review
Bug 934640 - Add support for factory operations which are not tied up with a specific database; r=#dom-storage
48 bytes, text/x-phabricator-request
Details | Review
Bug 934640 - Extract background actor creation into a separate function; r=#dom-storage
48 bytes, text/x-phabricator-request
Details | Review
Bug 934640 - Extract persistence type deduction into a separate function; r=#dom-storage
48 bytes, text/x-phabricator-request
Details | Review
Bug 934640 - Add a way to get database file managers by database file path; r=#dom-storage
48 bytes, text/x-phabricator-request
Details | Review
Bug 934640 - Add database version to database file managers; r=#dom-storage
48 bytes, text/x-phabricator-request
Details | Review
Bug 934640 - Implement IDBFactory.databases() to enumerate IndexedDB databases; r=#dom-storage
48 bytes, text/x-phabricator-request
Details | Review
Bug 934640 - Save current database versions for already touched databases before GetDatabaseOp gets unblocked by an existing factory operation; r=#dom-storage
48 bytes, text/x-phabricator-request
Details | Review
Bug 934640 - Add methods for checking origin and client directory existence; r=#dom-storage
48 bytes, text/x-phabricator-request
Details | Review
Bug 934640 - Add a way to get all database file managers for given persistence type and origin; r=#dom-storage
48 bytes, text/x-phabricator-request
Details | Review
Bug 934640 - Fix formatting in IndexedDatabaseManager.cpp; r=#dom-storage
48 bytes, text/x-phabricator-request
Details | Review 
https://www.w3.org/Bugs/Public/show_bug.cgi?id=16137 + http://lists.w3.org/Archives/Public/public-webapps/2011JulSep/1537.html for some history

Webkit have implemented a proprietary webkitGetDatabaseNames, implement a matching API with the same semantics (Jonas mentioned adding the version to the information)
Priority: -- → P5
This is actually part of the spec now. See: http://w3c.github.io/IndexedDB/#dom-idbfactory-databases
Let's set this to P3.
Priority: P5 → P3
Spec discussion issue at https://github.com/w3c/IndexedDB/issues/31 and pull request at https://github.com/w3c/IndexedDB/pull/240
URL: https://github.com/w3c/IndexedDB/issu...
Summary: Implement ability to enumerate databases → Implement IDBFactory.databases() to enumerate IndexedDB databases
Whiteboard: DWS_NEXT

Could we get indexedDB.databases() implemented please?

On Firefox it's not possible to remove indexedDB databases by domain. The reason is the lack of indexedDB.databases() support and chrome.browsingData.remove() doesn't support "hostnames" parameter for indexedDB (it only works for cookies and localStorage). So on Firefox WebExtensions can only remove ALL indexedDB databases for all websites or none.

The implication is that privacy related WebExtensions are seriously handicapped on Firefox compared to Chrome and Firefox is supposed to be the go-to-browser for people who care about privacy.

Chrome supports indexedDB.databases();

Assignee: nobody → sgiesecke
Status: NEW → ASSIGNED
Type: defect → enhancement
Status: ASSIGNED → NEW
Depends on: 1593365
Pushed by sgiesecke@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/81e2f38dbf3d
Let 'Ensure that databases() returns a promise' test case accept a rejected Promise to match its description. r=dom-workers-and-storage-reviewers,ytausky
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/20637 for changes under testing/web-platform/tests
Whiteboard: DWS_NEXT → DWS_NEXT, [wptsync upstream]
Upstream web-platform-tests status checks passed, PR will merge once commit reaches central.
Upstream PR merged by moz-wptsync-bot
Assignee: sgiesecke → nobody

The leave-open keyword is there and there is no activity for 6 months.
:jstutte, maybe it's time to close this bug?

Flags: needinfo?(jstutte)

(In reply to Release mgmt bot [:sylvestre / :calixte / :marco for bugbug] from comment #12)

The leave-open keyword is there and there is no activity for 6 months.
:jstutte, maybe it's time to close this bug?

Simon?

Flags: needinfo?(jstutte) → needinfo?(sgiesecke)

This is still waiting for Bug 1593365.

Flags: needinfo?(sgiesecke)

The attached patches only provide a stub for the IDBFactory.databases method right now, returning a rejected promise. They work as such, but landing them might confuse feature-detection code. Further work was halted as it turned out that we can't currently query the set of IndexedDB database for an origin from the quota manager in the parent process. Once the quota manager provides this capability, we can implement a an operation on PBackgroundIDBFactory adding another variant to FactoryRequestParams, and call that from the child process.

At the moment, the Storage view in DevTools makes up for this missing functionality by reading all the databases in the profile, which could of course break if the storage format changes.

It would be great to replace this once we have a standard way to list databases such as this API.

Quota manager is also used for centralized storage management (for example, the unified directory structure on disk), so it's not only about quotas. However, we might implement a workaround which doesn't depend on new quota manager features.

URL: https://github.com/w3c/IndexedDB/issu...
Severity: normal → --
OS: macOS → Unspecified
Hardware: x86 → Unspecified
See Also: → https://github.com/w3c/IndexedDB/issues/31

I was directed here from https://github.com/facebook/sapling/issues/148 because https://reviewstack.dev does not work in Firefox right now because this API is not supported.

From my perspective, I consider this a security issue because not being able to enumerate the databases means that a website has no API it can use to ensure it has removed all of a user's data when they run a "Logout" action of sorts within a web application.

It also prevents a website from removing all IndexedDB data when a user logs in to ensure they start from a clean state.

Note that the Clear-Site-Data header exists and when used with the "storage" key will comprehensively clear storage API data for the origin and should be preferred if the high level goal is to remove all of a user's data for an origin. This is the most reliable and comprehensive mechanism available. In the future, https://github.com/WICG/storage-buckets should allow more granular but still comprehensive clearing on a per-bucket basis as well.

(In reply to bolinfest from comment #19)

I was directed here from https://github.com/facebook/sapling/issues/148 because https://reviewstack.dev does not work in Firefox right now because this API is not supported.

From my perspective, I consider this a security issue because not being able to enumerate the databases means that a website has no API it can use to ensure it has removed all of a user's data when they run a "Logout" action of sorts within a web application.

It also prevents a website from removing all IndexedDB data when a user logs in to ensure they start from a clean state.

Surely a website knows the set of names it may or may not have created ... so can delete all whether they exist or not.

(Unless someone is naming their DBs in a non-deterministic way, which seems unlikely and unwise)

Surely a website knows the set of names it may or may not have created ... so can delete all whether they exist or not.

For a large complex application, this might be true in theory, but in practice keeping track of all databases and making sure they are all cleaned up on logout might not be very practical.

Came across this now, as I run tests on my company's webtools in various browsers and recently upgraded my code to remove all indexedDB databases to reset state. Why does Firefox lag behind every other browser in this regard?

Assignee: nobody → jvarga
Attachment #9354062 - Attachment is obsolete: true
Attachment #9354061 - Attachment is obsolete: true
Attachment #9110558 - Attachment description: Bug 934640 - Added IDBFactory.databases function to webidl. r=#dom-workers-and-storage → Bug 934640 - Added IDBFactory.databases function to webidl; r=#dom-storage
Attachment #9110561 - Attachment description: Bug 934640 - Enable 'Ensure that databases() returns a promise' test case. r=#dom-workers-and-storage → Bug 934640 - Enable 'Ensure that databases() returns a promise' test case; r=#dom-storage
No longer depends on: 1593365
Status: NEW → RESOLVED
Closed: 7 months ago
Resolution: --- → FIXED
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Status: REOPENED → ASSIGNED
Attachment #9110561 - Attachment is obsolete: true
Depends on: 1855142

EnsureBackgroundActor will be also called from IDBFactory::Databases.

Depends on D188979

GetPersistenceType will be also called from IDBFactory::Databases.

Depends on D190017

(In reply to y_broderick from comment #23)

Came across this now, as I run tests on my company's webtools in various browsers and recently upgraded my code to remove all indexedDB databases to reset state. Why does Firefox lag behind every other browser in this regard?

If everything goes well, this bug will be fixed in FF 120.

(In reply to Jan Varga [:janv] from comment #34)

(In reply to y_broderick from comment #23)

Came across this now, as I run tests on my company's webtools in various browsers and recently upgraded my code to remove all indexedDB databases to reset state. Why does Firefox lag behind every other browser in this regard?

If everything goes well, this bug will be fixed in FF 120.

Thanks, much appreciated :)

Priority: P3 → P2
Pushed by jvarga@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/fb741c8c2cb6
Added IDBFactory.databases function to webidl; r=dom-storage-reviewers,webidl,smaug,aiunusov,jari
https://hg.mozilla.org/integration/autoland/rev/ffdb716cfa3f
Create a new base for factory operations communicating using PBackgroundIDBFactoryRequest; r=dom-storage-reviewers,jari
https://hg.mozilla.org/integration/autoland/rev/2e53225b5d81
Add support for factory operations which are not tied up with a specific database; r=dom-storage-reviewers,jari
https://hg.mozilla.org/integration/autoland/rev/9b5fbc925fe5
Extract background actor creation into a separate function; r=dom-storage-reviewers,jari
https://hg.mozilla.org/integration/autoland/rev/067b6058bf70
Extract persistence type deduction into a separate function; r=dom-storage-reviewers,jari
https://hg.mozilla.org/integration/autoland/rev/189686b6ccfa
Add a way to get database file managers by database file path; r=dom-storage-reviewers,jari
https://hg.mozilla.org/integration/autoland/rev/392886344471
Add database version to database file managers; r=dom-storage-reviewers,jari
https://hg.mozilla.org/integration/autoland/rev/133d11049a61
Add methods for checking origin and client directory existence; r=dom-storage-reviewers,asuth
https://hg.mozilla.org/integration/autoland/rev/b4ee14550d3c
Implement IDBFactory.databases() to enumerate IndexedDB databases; r=dom-storage-reviewers,asuth
https://hg.mozilla.org/integration/autoland/rev/74b616bdd9ca
Fix formatting in IndexedDatabaseManager.cpp; r=dom-storage-reviewers,jari
https://hg.mozilla.org/integration/autoland/rev/f607e117912b
Add a way to get all database file managers for given persistence type and origin; r=dom-storage-reviewers,asuth
https://hg.mozilla.org/integration/autoland/rev/b9130d0d185f
Save current database versions for already touched databases before GetDatabaseOp gets unblocked by an existing factory operation; r=dom-storage-reviewers,asuth

Backed out for causing xpcshell failures on test_Store.js

TEST-UNEXPECTED-FAIL | browser/components/newtab/test/xpcshell/test_Store.js | xpcshell return code: -11

Also there were perma failures on this

Could you please take a look?
Thank you!

Flags: needinfo?(jvarga)
Flags: needinfo?(jvarga)

Please also check this wpt failures

Pushed by jvarga@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/ef90771d141d
Added IDBFactory.databases function to webidl; r=dom-storage-reviewers,webidl,smaug,aiunusov,jari
https://hg.mozilla.org/integration/autoland/rev/30dbc38c3e9a
Create a new base for factory operations communicating using PBackgroundIDBFactoryRequest; r=dom-storage-reviewers,jari
https://hg.mozilla.org/integration/autoland/rev/f315683fd3ba
Add support for factory operations which are not tied up with a specific database; r=dom-storage-reviewers,jari
https://hg.mozilla.org/integration/autoland/rev/ed5bb2e976ef
Extract background actor creation into a separate function; r=dom-storage-reviewers,jari
https://hg.mozilla.org/integration/autoland/rev/f3b863806999
Extract persistence type deduction into a separate function; r=dom-storage-reviewers,jari
https://hg.mozilla.org/integration/autoland/rev/184282e69b59
Add a way to get database file managers by database file path; r=dom-storage-reviewers,jari
https://hg.mozilla.org/integration/autoland/rev/8bbe6f8fb695
Add database version to database file managers; r=dom-storage-reviewers,jari
https://hg.mozilla.org/integration/autoland/rev/3a2af6a310ab
Add methods for checking origin and client directory existence; r=dom-storage-reviewers,asuth
https://hg.mozilla.org/integration/autoland/rev/a2769ba4c0ab
Implement IDBFactory.databases() to enumerate IndexedDB databases; r=dom-storage-reviewers,asuth
https://hg.mozilla.org/integration/autoland/rev/ba85838ef7cd
Fix formatting in IndexedDatabaseManager.cpp; r=dom-storage-reviewers,jari
https://hg.mozilla.org/integration/autoland/rev/3cf1e31d16ba
Add a way to get all database file managers for given persistence type and origin; r=dom-storage-reviewers,asuth
https://hg.mozilla.org/integration/autoland/rev/f842e5594224
Save current database versions for already touched databases before GetDatabaseOp gets unblocked by an existing factory operation; r=dom-storage-reviewers,asuth
Keywords: leave-open

Backed out for causing wpt failures in idb-explicit-commit.any.html

  • Backout link
  • Push with failures
  • Failure Log
  • Failure line: TEST-UNEXPECTED-FAIL | /IndexedDB/idb-explicit-commit.any.html | Transactions that handle all errors properly should behave as expected when an explicit commit is called in an onerror handler. - assert_equals: Expected complete event, but got abort event instead expected "complete" but got "abort"
Flags: needinfo?(jvarga)

I re-triggered the test on autoland just to be sure it's not bug 1799374 or something like that.
How is it possible that re-triggered tests are all green ?
https://treeherder.mozilla.org/jobs?repo=autoland&revision=f842e559422478115a52b4f74444a3be2cacbcbb

Flags: needinfo?(jvarga) → needinfo?(ctuns)

I triggered the test also on the try push I did just before landing on autoland:
https://treeherder.mozilla.org/jobs?repo=try&revision=e30a9c222dc3855d1f8444602cf2ddcfa3df0ede&selectedTaskRun=FeHTaKWiQ4SthlZHcBjoxg.0
It's still running ...

Because you did not retrigger it in a good way, you retriggered the wpt17 jobs, where "/IndexedDB/idb-explicit-commit.any.html" did not run. we have backfills in place: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&searchStr=linux%2C18.04%2Cx64%2Cwebrender%2Ctsan%2Copt%2Cweb%2Cplatform%2Ctests%2Ctest-linux1804-64-tsan-qr%2Fopt-web-platform-tests%2Cwpt17&fromchange=33e78b58e39fe602a8fcc6ab12a2f2bc513b66e3&tochange=adf17fc47c84c0fa237aecff550ba42344a5c629&selectedTaskRun=ZcxsCEcoS2C72O9m-QOjIQ.0 and the backfills stop at your push. If you see a wpt17 green that does not mean it ran "/IndexedDB/idb-explicit-commit.any.html" test, they can switch chunks. You can check the log for the green jobs and search for "/IndexedDB/idb-explicit-commit.any.html" to see if the test ran.

Flags: needinfo?(ctuns)

I triggered all the chunks, let's see ...

I explicitly triggered all wpt tests for all shippable builds on the try push prior to landing, and it was all green.
Let's see if it's really only tsan.

It's not only tsan, it shows on multiple platforms: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception%2Cusercancel&revision=adf17fc47c84c0fa237aecff550ba42344a5c629&searchStr=wpt&selectedTaskRun=Tul9xZPpSz-9mq4AagsZFQ.0 and also you have multiple 12 revisions in autoland and only 3 on try, couldn't this be caused by the revisions that are not in you try push?

(In reply to Cristian Tuns from comment #51)

It's not only tsan, it shows on multiple platforms: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception%2Cusercancel&revision=adf17fc47c84c0fa237aecff550ba42344a5c629&searchStr=wpt&selectedTaskRun=Tul9xZPpSz-9mq4AagsZFQ.0 and also you have multiple 12 revisions in autoland and only 3 on try, couldn't this be caused by the revisions that are not in you try push?

The other change sets/patches are there, they are just not shown because I did two other try pushes prior to that:
https://treeherder.mozilla.org/jobs?repo=try&revision=f0779f9faeba7a2cf12b999038ebe98bbf66a6bd
https://treeherder.mozilla.org/jobs?repo=try&revision=4f69bc848c9219b8decd5b89943da5fcfc3188e7
https://treeherder.mozilla.org/jobs?repo=try&revision=e30a9c222dc3855d1f8444602cf2ddcfa3df0ede

I'll do more testing on try, but it looks like it's a variation of existing intermittent bug 1799374
https://treeherder.mozilla.org/jobs?repo=try&revision=8ee684b669a23bd621f6427adf73d66cc3532a7c&selectedTaskRun=ABRD3oXIQFe076OpYK2q8g.0
https://treeherder.mozilla.org/jobs?repo=try&revision=8ee684b669a23bd621f6427adf73d66cc3532a7c&selectedTaskRun=XZ1utCTiT4a8kGdOrenebw.0

I still wonder why triggering of all wpt chunks didn't run IndexedDB/idb-explicit-commit.any.html

This time I rather used: mach try fuzzy testing/web-platform/tests/IndexedDB

Ok, thanks.
The frequency of the intermittent failure is higher and it's especially bad on tsan and some other builds.
https://treeherder.mozilla.org/jobs?repo=try&revision=8ee684b669a23bd621f6427adf73d66cc3532a7c

Depends on: 1799374

Yeah, as I commented in bug 1799374, the frequency of the intermittent failure is affected by multiple factors. Patches for this bug affect it too.
We could just temporarily disable the test and land the patches, but now when I know what the problem is, I think it's better to fix the long standing intermittent failure first.

Pushed by jvarga@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/2bf7b30a3031
Added IDBFactory.databases function to webidl; r=dom-storage-reviewers,webidl,smaug,aiunusov,jari
https://hg.mozilla.org/integration/autoland/rev/44b6e445fae2
Create a new base for factory operations communicating using PBackgroundIDBFactoryRequest; r=dom-storage-reviewers,jari
https://hg.mozilla.org/integration/autoland/rev/752a77c2aeb7
Add support for factory operations which are not tied up with a specific database; r=dom-storage-reviewers,jari
https://hg.mozilla.org/integration/autoland/rev/24996a9ac5b9
Extract background actor creation into a separate function; r=dom-storage-reviewers,jari
https://hg.mozilla.org/integration/autoland/rev/e343ab147db6
Extract persistence type deduction into a separate function; r=dom-storage-reviewers,jari
https://hg.mozilla.org/integration/autoland/rev/3593380ad495
Add a way to get database file managers by database file path; r=dom-storage-reviewers,jari
https://hg.mozilla.org/integration/autoland/rev/2239c6fa5b86
Add database version to database file managers; r=dom-storage-reviewers,jari
https://hg.mozilla.org/integration/autoland/rev/e2c42f46453a
Add methods for checking origin and client directory existence; r=dom-storage-reviewers,asuth
https://hg.mozilla.org/integration/autoland/rev/dd3f65f53626
Implement IDBFactory.databases() to enumerate IndexedDB databases; r=dom-storage-reviewers,asuth
https://hg.mozilla.org/integration/autoland/rev/91aad7241b01
Fix formatting in IndexedDatabaseManager.cpp; r=dom-storage-reviewers,jari
https://hg.mozilla.org/integration/autoland/rev/bcdddd575816
Add a way to get all database file managers for given persistence type and origin; r=dom-storage-reviewers,asuth
https://hg.mozilla.org/integration/autoland/rev/0ead0c93d2e3
Save current database versions for already touched databases before GetDatabaseOp gets unblocked by an existing factory operation; r=dom-storage-reviewers,asuth
Duplicate of this bug: 1878132
Regressions: 1887825

:janv could you consider nominating this for a release note? (Process info)
We could include it in the nightly release notes.

Flags: needinfo?(jvarga)

Yeah, worth a short note in release notes.

Release Note Request (optional, but appreciated)
[Why is this notable]: IDBFactory.databases() is one of the major API methods and FF was the last browser not supporting it.
[Affects Firefox for Android]: Yes
[Suggested wording]: Web Platform, Firefox now supports IDBFactory.databases (for enumeration of IndexedDB databases).
[Links (documentation, blog post, etc)]:
https://developer.mozilla.org/en-US/docs/Web/API/IDBFactory/databases
https://www.w3.org/TR/IndexedDB/#dom-idbfactory-databases

relnote-firefox: --- → ?
Flags: needinfo?(jvarga)

Thanks, added to the Fx126 nightly release notes, please allow 30 minutes for the site to update.
Keeping the relnote-firefox flag as ? to keep it on the radar for inclusion in the final Fx126 release notes

FF126 MDN docs work for this can be tracked in https://github.com/mdn/content/issues/33084.

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: