1741033 - Enhance getAll() implementation to chunk IPC for situations where the result payload would exceed our IPC limit heuristics but is not expected to OOM the content process. (Was: getAll() fails after specific amount of returned data)

Status: UNCONFIRMED

(Core :: Storage: IndexedDB, enhancement, P3)

Description

[[e-mail]]

Steps to reproduce:

I'm trying to retrieve data from an IndexedDB using "getAll()", which is faster for my purposes than opening a cursor. Running Firefox 94.0.1 on both Winows 7 and Windows 10 64-bit. This has been an issue for a while.

... open the database, etc ...
let index = db.transaction("database").objectStore("myStore").index("myIndex");
let range = IDBKeyRange.lowerBound(100000); // That's just a random number for the example
let results = index.getAll(range);
results.onerror = e => {...};
results.onsuccess = e => {...};

Actual results:

getAll() consistently throws an error if the retrieved data is apparently larger than a certain amount.
Altering the database (adding/removing large amounts of data), changing the index or range, etc., all have no effect. At a specific point, I receive a DOMException: The operation failed for reasons unrelated to the database itself and not covered by any other error code.

See more details at:
https://stackoverflow.com/questions/69950861/using-getall-on-an-indexeddb-keeps-throwing-errors-is-there-a-limit-to-the

Expected results:

An array of data rows should have been returned.

Comment 1

[Jens Stutte [:jstutte]]

Hi Jan, can you please help to triage this?

Comment 2

[[e-mail]]

Apparently, this wasn't an issue for me 4 years ago, as evidenced by my post here:
https://stackoverflow.com/questions/44349168/speeding-up-indexeddb-search-with-multiple-workers

I have tried running a regression to see when the issue started, but my extensions keep failing in older versions of Firefox (prior to ~76). If I have time, I will try to generate new databases in prior versions to help with the regression analysis.

Comment 3

[Jens Stutte [:jstutte]]

There are some limitations here in the way. In general, getAll() potentially loads the entire database into JS memory (depending on the database's content and the query). So in general I think it should be avoided for databases / results known to become big. The workaround to use cursors mentioned in the stackoverflow issue is not a workaround but simply the better solution for these cases, IMHO. But there is also a more Firefox specific limitation here, that is getAll() is handled with a single request traveling over IPC (which explains why it is faster), and such requests have an upper limit of 256 MB. The way we fail here is probably not super transparent and should be improved, but basically we estimate the response size and have an upper limit of how much data can be transferred at once and this limit is hard-coded and not really depending on system resources.

Re-triaging to S3 as I cannot really see here "Major functionality/product severely impaired or a high impact issue and a satisfactory workaround does not exist" in the sense of our severity definitions. But we should definitely at least improve how we fail here. We might even think about splitting the payload to more than one IPC message, although this will just move the bar a bit until we hit other memory limitations, probably.

Comment 4

[Andrew Sutherland [:asuth] (he/him)]

I've re-titled this bug into an enhancement. I'm about to dupe bug 1810035 to this bug but note that the reporter has created a fantastic reproduction there on bug 1810035 with source at https://github.com/joshhunt/firefox-indexeddb-repro and hosted at https://idb-perf-repro.netlify.app/.