768312 - extensions.sqlite/addons.sqlite are very space inefficient on disk

Status: RESOLVED WONTFIX

(Toolkit :: Add-ons Manager, defect)

Description

[Blair McBride [:Unfocused] (UNAVAILABLE)]

On my dev profile, extensions.sqlite takes up 448kb. addons.sqlite takes up 480kb. However, there's no where near that much actual data stored in those files.

Running sqlite3_analyzer shows only 6.2% of extensions.sqlite is actually used to store data (bytes of user payload stored) - the rest is wasted space. It also shows that the payload per entry (41.11) is tiny compares to the unused bytes per entry (475.63).

Running VACUUM doesn't help, which makes sense because that database is periodically regenerated anyway (which is what VACUUM does).

AFAICT, this is due to the database having a large page size, yet very small entries. Reducing the page size down to the minimum (512) and VACUUMing resulted in a file of 53kb on disk (~12% of the original size).

That also seems to have reduced the in-memory size from 450kb to 161kb. See also bug 726556 (though, at the time that bug was filed, I'm sure the in-memory size was much larger, around 2Mb).

Comment 1

[Blair McBride [:Unfocused] (UNAVAILABLE)]

Attachment: sqlite3_analyzer output, default page size

Comment 2

[Blair McBride [:Unfocused] (UNAVAILABLE)]

Attachment: sqlite3_analyzer output, page size of 512

Marco/Taras: Any red flags/things I should be concerned about in doing this? 

Looking at mozStorageConnection.cpp, the cache size is set based on the page size, so the reduction in memory footprint makes sense. Also, the page size is set every time a database is opened, but we don't ever VACUUM, so setting that only when we create the DB should be fine.

Comment 3

[Blair McBride [:Unfocused] (UNAVAILABLE)]

As for how I picked the page size... I tested with the smallest size supported. Then looking at the sqlite3_analyzer output, I see there's still some wasted space. So a larger page size would increase the wasted space.

Comment 4

[Blair McBride [:Unfocused] (UNAVAILABLE)]

WONTFIXed, in favour of moving to a JSON data format in bug 853388 / bug 853389.