457158 - 49 kB new leaks from Bug 449443 - Upgrade to SQLite 3.6.3

Status: RESOLVED INACTIVE

(Toolkit :: Storage, defect)

Description

[Karl Tomlinson (:karlt)]

http://graphs.mozilla.org/#show=1474269&sel=1222378976,1222403507
http://graphs.mozilla.org/#show=1473418,1473670&sel=1222388349,1222402576

Comment 1

[Karl Tomlinson (:karlt)]

Attachment: most significant new leaks

from http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1222400063.1222400560.8066.gz

Looks like it's all from sqlite3_open -> setupLookaside.

Comment 2

[Shawn Wilsher :sdwilsh]

Hmm - I was unaware that we'd be using any lookaside memory without telling sqlite to do so...

Comment 3

[D. Richard Hipp]

The lookaside memory allocator is a separate memory allocator for small transient memory allocations associated with each database connection.  Allocations from the lookaside allocator are always the same size.  If an allocation cannot be satisfied by the lookaside allocator, it flows through to the regular system memory allocator.  The reason for doing this is that the lookaside memory allocator is much, much faster than any normal memory allocator.  The end result is that SQLite is as much as 15% faster depending on how often memory allocation requests can be satisfied by the lookaside versus having to use the regular system malloc().

Memory for the lookaside allocator is obtained by a single large allocation from the system malloc() when sqlite3_open() is called.  This memory is released on sqlite3_close().  The size of the lookaside buffer can be changed using sqlite3_config(SQLITE_CONFIG_LOOKASIDE, sz, cnt) where sz is size of each lookaside allocation and typically in the range of 50 to 150 and cnt is the maximum number of lookaside allocations and is typically between 100 to 10000.  The single large malloc() that occurs at sqlite3_open() is sz*cnt bytes in side.  The default values for sz and cnt (if you do not call sqlite3_config() to change them) are 100 and 500.  Thus you normally have a 50000 byte allocation associated with each sqlite3_open().

You can turn lookaside off by setting either sz or cnt to zero.  But you take a performance hit.

Comment 4

[(not currently active) Ted Mielczarek]

Can we just free this buffer on shutdown to get it out of our leak stats? I don't think anyone will really complain about 50k extra memory usage for 15% perf.

Comment 5

[Shawn Wilsher :sdwilsh]

Well, we should be closing all connections, so we should be freeing it.  That is, unless someone isn't closing a connection (but then we should be leaking a mozStorageConnection object too).

Comment 6

[Karl Tomlinson (:karlt)]

(In reply to comment #5)
> Well, we should be closing all connections, so we should be freeing it.  That
> is, unless someone isn't closing a connection (but then we should be leaking a
> mozStorageConnection object too).

Yes, either we're not closing a connection or the sqlite3_close() is not releasing the memory.  But, if sqlite3_close is being called more than once, and we only leaking one pool of memory, then that suggests the blame might lie higher up.

(In reply to comment #3)
> The lookaside memory allocator is a separate memory allocator for small
> transient memory allocations associated with each database connection. 
> Allocations from the lookaside allocator are always the same size.  If an
> allocation cannot be satisfied by the lookaside allocator, it flows through to
> the regular system memory allocator.  The reason for doing this is that the
> lookaside memory allocator is much, much faster than any normal memory
> allocator.  The end result is that SQLite is as much as 15% faster depending
> on how often memory allocation requests can be satisfied by the lookaside
> versus having to use the regular system malloc().

> You can turn lookaside off by setting either sz or cnt to zero.  But you take
> a performance hit.

jemalloc already uses a zone allocator which keeps equal size objects together, so I wonder whether the performance effect of the lookaside allocator in Mozilla may not be similar to the effect with "normal memory allocator"s.  Other special allocators have been disabled (bug 431221).

If a lookaside allocator per connection saves some locking, there may be gain there.

Comment 7

[Shawn Wilsher :sdwilsh]

(In reply to comment #6)
> If a lookaside allocator per connection saves some locking, there may be gain
> there.
As I understand it, using the lookaside allocator avoids locking in SQLite.

Comment 8

[Shawn Wilsher :sdwilsh]

So, we are not seeing this anymore, are we?

Comment 9

[Karl Tomlinson (:karlt)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1274744065.1274747084.14125.gz
says that only 21875 bytes are leaked, which would mean we're not leaking 49 kB, though its hard to believe that we only leak 24 kB.

Comment 10

[BMO Automation]

Per policy at https://wiki.mozilla.org/Bug_Triage/Projects/Bug_Handling/Bug_Husbandry#Inactive_Bugs. If this bug is not an enhancement request or a bug not present in a supported release of Firefox, then it may be reopened.