1522638 - support bulk insert and drop for kvstore

Status: RESOLVED FIXED

(Core :: SQLite and Embedded Database Bindings, enhancement, P3)

Description

[Nan Jiang [:nanj]]

As Activity Stream is planning on replacing the json cache files with kvstore, we've found these two features could be helpful for some use cases.

In particular, for a big dataset, bulk insert, i.e. put multiple key/value pairs to the store in a single transaction, will be way more efficient than calling kvstore.put separately on each key/value pair, because there is an implicit transaction involved for each call. This also benefits the consumer with the "all-or-nothing" semantics when the batch insert is desired.

As a dual feature, a drop API, which drops everything in the given store, would be better than calling delete repeatedly if the consumer wants to truncate the store in a single transaction. LMDB has mdb_drop for this, we can add it to rkv and expose it via kvstore.

Comment 1

[Myk Melez [:myk] [@mykmelez]]

Agreed, both bulk insert and drop would be useful. Is bulk insert different from the support for transactions requested in bug 1499238?

Comment 2

[Nan Jiang [:nanj]]

(In reply to Myk Melez [:myk] [@mykmelez] from comment #1)

Agreed, both bulk insert and drop would be useful. Is bulk insert different from the support for transactions requested in bug 1499238?

Yes, they are related. I was actually commenting this feature request in bug 1499238 first, but then realized that this might want a separate bug on its own.

To implement bulk insert, presumably we will be using cursor plus a write transaction. We can still hide this implementation detail in the kvstore without exposing the transaction to the user at all. And I believe it's better for now than letting the consumers do so with a transaction. I quite like the current API design in kvstore.jsm, it's neat, also hides the unnecessary complexity for the regular uses.

Still trying to wrap my head into this XPCom, hopefully, I can at least help to review some patches, or even better hack on it directly :-)

Comment 3

[Nan Jiang [:nanj]]

Attachment: Bug 1522638 - Add bulk insert to kvstore

This adds the bulk insert to kvstore as discussed in Bug 1522638

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/49ffc038ef84
Add bulk insert to kvstore r=myk,mossop,nika

Comment 5

[Oana Pop-Rus]

https://hg.mozilla.org/mozilla-central/rev/49ffc038ef84