1543861 - consider auto-resizing map of LMDB environments for kvstore

Status: RESOLVED FIXED

(Toolkit :: Storage, enhancement, P3)

Description

[Myk Melez [:myk] [@mykmelez]]

Over in rkv#132, nanj worked on auto-resizing the map of LMDB environments in rkv when an LMDB transaction reports a MapFull error.

That turned out to be complex to achieve, given rkv's promises of atomic transactions and writers in multiple processes. So nanj instead landed rkv#135, which enables consumers to handle MapFull errors by resizing the map themselves.

For kvstore, however, it may still be straightforward to implement map auto-resizing, since kvstore doesn't expose transactions and rather allows batching of changes via its putMany() and clear() methods.

Thus we should consider auto-resizing the map of LMDB environments accessed via kvstore.

Comment 1

[Nan Jiang [:nanj]]

I am on it.

My proposal is that add a map_resize helper function for PutTask and WriteManyTask with the following resizing strategies:

• Active resizing, which is performed before writing any data to the store. It does so by checking the current load ratio of the store, resize will be done if the load ratio reaches the threshold. The resizing rule goes as:

  • Double the size if the current size is below the predefined threshold (such as 10MB, 100MB, or 1GB??).
  • Otherwise, increment it by a constant size to avoid the over resizing.

• Passive resizing, which is performed whenever the MAP_FULL occurs. This is still needed even with the active resizing, since the resized store may still not fit a large payload or a batch of smaller payloads. The target resizing size can be determined as follows:

  • Calculate the total size of the payload, denoted as "S", let's also denote the current store size as "C".
  • If "S < C", then just double the store size; otherwise, increment it by "S" so that the new store size would be "C + S" rounding to the multiple of page size. This extra check of payload size will ensure the passive resizing always succeeds. Otherwise, imagine that inserting a 10MB payload to a 1MB store if we always double the current store size like active resizing.

In summary, the motivation of resizing here is to free our consumers from having to guess the proper size when they're creating the store. With the passive resizing, the active resizing is actually optional, however, it's much cheaper (as it doesn't need to calculate the payload size, nor to retry the failed write(s)) than the passive one and should cover most of the needs for resizing.

myk, what do you think?

Comment 2

[Myk Melez [:myk] [@mykmelez]]

I think this is a great approach. Active resizing will minimize the number of passive resizes without significant performance degradation, while passive resizing will ensure that kvstore write operations never fail due to MAP_FULL. 👍🏼

Comment 3

[Nan Jiang [:nanj]]

Attachment: Bug 1543861 - Add auto resizing for kvstore

This patch implements the auto resizing to handle the MDB_MAP_FULL error. It adds two resizing strategies: 1). Active resizing to increase the store size when it gets opened; 2). Passive resizing to increase the store size when the write transaction encounters the MDB_MAP_FULL error. There are no changes to kvstore APIs, existing consumers will get this feature working without any changes to their code.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1391b448c7b8
Add auto resizing for kvstore r=myk

Comment 5

[Andrei Ciure[:aciure]]

https://hg.mozilla.org/mozilla-central/rev/1391b448c7b8