533856 - storage should use mmap to read database files

Status: RESOLVED INVALID

(Core :: SQLite and Embedded Database Bindings, defect)

Comment 1

[Dietrich Ayala (:dietrich)]

more here, per joel: http://code.google.com/p/chromium/issues/detail?id=26392

Comment 2

[(dormant account)]

Our main motivation for mmap in sqlite is on Windows Mobile. There reading via conventional read() calls is about 3-10x more expensive than mmap()ed io. Since we do a lot of sqlite io, this is quite a hit.

Systems with expensive syscalls such as OSX may benefit too.

Comment 3

[D. Richard Hipp]

SQLite supports databases that are as big as 70 terabytes.  In a move to
mmap(), we'd have to limit that to 2 gibibytes, I suppose.  Maybe less.

Another thing is that SQLite accumulates changes in its internal memory
buffers and then writes them out during a commit.  This allows other processes
to continue reading the unmodified disk image which SQLite is in the middle
of a long-running write transaction.  But with mmap(), we'd have to lock
out all readers as soon as any change is made to the database file.

Why is read() so slow on windows mobile?  If it is really 10x slower than
mmap, couldn't you just reimplement read() such that it used mmap() in the
background?

Comment 4

[(dormant account)]

(In reply to comment #3)
> SQLite supports databases that are as big as 70 terabytes.  In a move to
> mmap(), we'd have to limit that to 2 gibibytes, I suppose.  Maybe less.

Or you could remap chunks of the database as you need them.

> 
> Another thing is that SQLite accumulates changes in its internal memory
> buffers and then writes them out during a commit.  This allows other processes
> to continue reading the unmodified disk image which SQLite is in the middle
> of a long-running write transaction.  But with mmap(), we'd have to lock
> out all readers as soon as any change is made to the database file.

I believe this would be reasonable.

> 
> Why is read() so slow on windows mobile?  If it is really 10x slower than
> mmap, couldn't you just reimplement read() such that it used mmap() in the
> background?

Yes, that's basically what I did in Mozilla's jar reading code.

Comment 5

[(dormant account)]

Just prototyped unixRead replacement using mmap. mmap() seems barely faster than read() in the cold io case(mac+linux), but surprisingly is slower in warm io cache case. It's not worth the hassle.