Closed Bug 1717068 Opened 2 years ago Closed 1 year ago

Reload collection from packaged dump if it's newer than the existing collection

Categories

(Firefox :: Remote Settings Client, enhancement, P1)

enhancement

Tracking

()

RESOLVED FIXED
91 Branch
Tracking Status
firefox91 --- fixed

People

(Reporter: robwu, Assigned: robwu)

References

Details

(Keywords: sec-want)

Attachments

(3 files)

Remote settings has a feature to allow clients to initialize its database with a packaged dump of records: https://firefox-source-docs.mozilla.org/services/settings/index.html#initial-data

This feature is designed to allow early bootstrapping of the remote settings collection/database on new installations, with the assumption that further updates will be applied remotely. But if the remote updates cannot be downloaded for some reason, then the collection can become out of date. That's not desired. To make sure that this doesn't happen, we should check the recency of the collection when the binary has updated and erase the current collection if it's older than the packaged dump. Checking the recency is not always straightforward (bug 1640126), but we can at least improve the current situation by relying on the timestamps database that's already in use.

This feature can be opt-in to start with (and become opt-out in the future). The security-state collection should not use this, because it stores additional metadata in the records (=whether attachments were downloaded) and should only use this feature once it migrates to the attachment packaging logic (useCache) that's introduced in https://hg.mozilla.org/mozilla-central/rev/c0a18dbb6c502c21d36affbc8c44f5e26e4eb1c3 (deprecation of the old logic is handled in bug 1634127).

Assignee: nobody → rob
Severity: -- → N/A
Status: NEW → ASSIGNED
Priority: -- → P1
Blocks: 1718083

This option will be used in a following commit and in bug 1718083.

Blocks: 1719560

Package a summary of the RemoteSettings dumps with the application, so
that RemoteSettings clients can look up the last_modified value of a
dump without loading the whole JSON dump file.

Attachment #9230171 - Attachment description: Bug 1717068 - Package summary of last_modified → Bug 1717068 - Package summary of last_modified of RemoteSettings dumps

The first two patches already work as desired, and addons-bloomfilter.json has recently been recycled (made more compact by moving the entries to the bloom filter) in https://hg.mozilla.org/mozilla-central/diff/eb766082f46c0cbfd354ef3e9a69afc65018fd1f/services/settings/dumps/blocklists/addons-bloomfilters.json.

Because addons-bloomfilters.json is currently small, I'll land the functionality now. The optimization (third patch) can land later.

Keywords: leave-open
Pushed by rob@robwu.nl:
https://hg.mozilla.org/integration/autoland/rev/7f269f70c52c
Option to load from JSON dump if it's newer than local data r=leplatrem
https://hg.mozilla.org/integration/autoland/rev/1195f23a6393
Use newer blocklist dump if available r=Gijs

I inadvertently registered the test from the third patch in the second patch.

I'll fix that and reland later (possibly the whole stack if r+).

Flags: needinfo?(rob)
Keywords: leave-open
Pushed by rob@robwu.nl:
https://hg.mozilla.org/integration/autoland/rev/cd7ca21b05e9
Option to load from JSON dump if it's newer than local data r=leplatrem
https://hg.mozilla.org/integration/autoland/rev/29ab80ae2bb2
Use newer blocklist dump if available r=Gijs
https://hg.mozilla.org/integration/autoland/rev/e89a9e3bd3a7
Package summary of last_modified of RemoteSettings dumps r=leplatrem,nalexander
Keywords: sec-want
You need to log in before you can comment on or make changes to this bug.