Open Bug 1631964 Opened 6 months ago Updated 6 months ago

Improve RemoteSettings Reads

Categories

(Firefox :: Remote Settings Client, enhancement)

enhancement

Tracking

()

People

(Reporter: emalysz, Unassigned)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

Attached file ff_startup.zip

For this particular startup profile, we only read 1.639MB of the remote settings file from a range of 13.353 MB. Since this is only a fraction of the range, we should see how we can improve this read performance. This file is read at various points throughout startup (first read ~9s all the way up until ~17.5s)

With the above attachment in https://procmon-analyze.github.io/, search "Rsegmnoittet-es.sqlite" to find the RemoteSettings reads.

Blocks: 1627071

Gijs, do you have any suggestions on how we could improve this? Doug had suggested experimenting with caching or figuring out a way to improve our indices.

Flags: needinfo?(gijskruitbosch+bugs)

(In reply to Emma Malysz from comment #1)

Gijs, do you have any suggestions on how we could improve this? Doug had suggested experimenting with caching or figuring out a way to improve our indices.

This will be because it's 1 db with data for multiple consumers. This is, per my discussion with Andrew earlier today, the "right" way of using indexeddb. RemoteSettings segments the data that comes out of the stores based on who the consumer is (based on idb keyranges), and indexeddb is able to optimize this, AIUI. I think there are already indices in place for listing all data for a given consumer, which is likely what we're doing here. (Mathieu, can you confirm?)

I don't really have context here. Is this the first run on this profile, or a subsequent one? Do we know which remote settings consumers are causing the IO?

I'm not sure how we'd fix this. RS is supposed to provide an intuitive API surface over a DB that is synced to a remote service. It's used by multiple consumers, and some of them run on startup and others do not. Given the ratio of the file size and read size in comment #0, caching the entire thing is clearly silly. But there's a bunch of abstractions (remote settings, indexeddb, sqlite) inbetween the consumers and the actual IO, so I'm not sure how we could reasonably cache this info. My best guess would be duplicating the data to a cache file if we notice we load it at startup, but that would be copy #3 of the data (we also ship initial dumps of a bunch of these datasets in omni.ja - but those will miss updates distributed via remote settings since then), plus we'd need to invalidate it every time the data changes, which can happen every 6 hours. Plus, 2mb is quite a significant amount of data to just duplicate in order to make the read slightly faster...

If we knew what the consumers were, we could look at whether they really need this data so early, or if they could cache the salient details some other way.

Component: General → Remote Settings Client
Flags: needinfo?(gijskruitbosch+bugs) → needinfo?(emalysz)
Product: Core → Firefox

(In reply to :Gijs (he/him) from comment #2)

This will be because it's 1 db with data for multiple consumers. This is, per my discussion with Andrew earlier today, the "right" way of using indexeddb. RemoteSettings segments the data that comes out of the stores based on who the consumer is (based on idb keyranges), and indexeddb is able to optimize this, AIUI. I think there are already indices in place for listing all data for a given consumer, which is likely what we're doing here. (Mathieu, can you confirm?)

Oops, +ni

Flags: needinfo?(mathieu)

This was taken from a cold startup after restarting the system, and the build had been run a few times before the restart. We're hoping to get stacks and incorporate that into our analyzer tool soon, so hopefully that will give more context once it's ready.

Flags: needinfo?(emalysz)

I think there are already indices in place for listing all data for a given consumer, which is likely what we're doing here. (Mathieu, can you confirm?)

Yes, by collection, for the timestamp and the records:
https://searchfox.org/mozilla-central/rev/41c3ea3ee8eab9ce7b82932257cb80b703cbba67/services/settings/Database.jsm#327-330

Flags: needinfo?(mathieu)
You need to log in before you can comment on or make changes to this bug.