1499550 - Iterating store with cursor value + continue() is slow

Status: RESOLVED FIXED

(Core :: Storage: IndexedDB, enhancement, P3)

Description

[Mathieu Leplatre [:leplatrem]]

In IndexedDB docs [0], it is recommended to use a cursor in order to step through all the values in the object store.

In Bug 1486980 we figured out that doing so was extremely slow.

I isolated the usage of the high-level lib [1] that we use in RemoteSettings in order to observe the phenomena: 
https://codepen.io/anon/pen/QZaRBe

Iterating on less than 1000 entries using a cursor takes around 1 second on my Firefox.

It is globally 7 times faster in Chromium (Note: other IDB operations are faster too but out of scope here).

Some explanation was given by :asuth in https://bugzilla.mozilla.org/show_bug.cgi?id=1486980#c32 
tldr: Bug 1168606 could help and switching to getAll() might be the workaround for some use-cases.

Having a bug that tracks this precise performance deficiency can help. Especially now that more and more browser features rely on IndexedDB we could face this core issue indirectly.

It might also be a good idea to be more insisting about `getAll()` in the docs, as well as mentioning its downsides regarding I/O, its usage with the `count` parameter etc.


[0] https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API/Using_IndexedDB#Using_a_cursor

[1] https://github.com/Kinto/kinto.js/blob/master/src/adapters/IDB.js

Comment 1

[Mike Conley (:mconley) (:⚙️)]

As I understand it, bug 1168606 will fix this, right? If so, I'll qf- this, since we're tagging bug 1168606 with a qf priority.

Comment 2

[Mike Conley (:mconley) (:⚙️)]

Just double-checking on my understanding of the relationships here with asuth.

Comment 3

[Andrew Sutherland [:asuth] (he/him)]

Yes, bug 1168606 will fix this.

Comment 4

[Mathieu Leplatre [:leplatrem]]

Now that Bug 1168606 was fixed, I tried to run this snippet again, and performance between Chromium and Firefox is now comparable:

With Chromium:

sync: 224.454833984375 ms iframeConsoleRunner-7f4d47902dc785f30dedcac9c996b9f31d4dfcc33567cc48f0431bc918c2bf05.js:1
list all: 6.830810546875 ms iframeConsoleRunner-7f4d47902dc785f30dedcac9c996b9f31d4dfcc33567cc48f0431bc918c2bf05.js:1
get by id: 3.3232421875 ms iframeConsoleRunner-7f4d47902dc785f30dedcac9c996b9f31d4dfcc33567cc48f0431bc918c2bf05.js:1
get by attribute: 13.05712890625 ms

With Firefox:

sync: 585ms - timer ended iframeConsoleRunner-7f4d47902dc785f30dedcac9c996b9f31d4dfcc33567cc48f0431bc918c2bf05.js:1:4080
list all: 4ms - timer ended iframeConsoleRunner-7f4d47902dc785f30dedcac9c996b9f31d4dfcc33567cc48f0431bc918c2bf05.js:1:4080
get by id: 2ms - timer ended iframeConsoleRunner-7f4d47902dc785f30dedcac9c996b9f31d4dfcc33567cc48f0431bc918c2bf05.js:1:4080
get by attribute: 9ms - timer ended

I'm therefore closing this bug! :tada:!

For the reference, here is the snippet:

      const db = new Kinto.default({remote: "https://settings.prod.mozaws.net/v1/", bucket: "main"});
      const collection = db.collection("tippytop");

      const btn = document.getElementById("run");
      btn.onclick = async function () {
        btn.disabled = true;

        console.time("sync");
        await collection.sync();
        console.timeEnd("sync");

        console.time("list all");
        await collection.list();
        console.timeEnd("list all");

        console.time("get by id");
        await collection.list({filters: {id: "7210b4bc-81f6-4f04-976e-1acf5faeb740"}});
        console.timeEnd("get by id");

        console.time("get by attribute");
        await collection.list({filters: {domain: "amazon.es"}});
        console.timeEnd("get by attribute");

        btn.disabled = false;
      }