1717068 - Reload collection from packaged dump if it's newer than the existing collection

Status: RESOLVED FIXED

(Firefox :: Remote Settings Client, enhancement, P1)

Description

[Rob Wu [:robwu]]

Remote settings has a feature to allow clients to initialize its database with a packaged dump of records: https://firefox-source-docs.mozilla.org/services/settings/index.html#initial-data

This feature is designed to allow early bootstrapping of the remote settings collection/database on new installations, with the assumption that further updates will be applied remotely. But if the remote updates cannot be downloaded for some reason, then the collection can become out of date. That's not desired. To make sure that this doesn't happen, we should check the recency of the collection when the binary has updated and erase the current collection if it's older than the packaged dump. Checking the recency is not always straightforward (bug 1640126), but we can at least improve the current situation by relying on the timestamps database that's already in use.

This feature can be opt-in to start with (and become opt-out in the future). The security-state collection should not use this, because it stores additional metadata in the records (=whether attachments were downloaded) and should only use this feature once it migrates to the attachment packaging logic (useCache) that's introduced in https://hg.mozilla.org/mozilla-central/rev/c0a18dbb6c502c21d36affbc8c44f5e26e4eb1c3 (deprecation of the old logic is handled in bug 1634127).

Comment 1

[Rob Wu [:robwu]]

Attachment: Bug 1717068 - Option to load from JSON dump if it's newer than local data

This option will be used in a following commit and in bug 1718083.

Comment 2

[Rob Wu [:robwu]]

Attachment: Bug 1717068 - Use newer blocklist dump if available

Comment 3

[Rob Wu [:robwu]]

Attachment: Bug 1717068 - Package summary of last_modified of RemoteSettings dumps

Package a summary of the RemoteSettings dumps with the application, so
that RemoteSettings clients can look up the last_modified value of a
dump without loading the whole JSON dump file.

Comment 4

[Rob Wu [:robwu]]

The first two patches already work as desired, and addons-bloomfilter.json has recently been recycled (made more compact by moving the entries to the bloom filter) in https://hg.mozilla.org/mozilla-central/diff/eb766082f46c0cbfd354ef3e9a69afc65018fd1f/services/settings/dumps/blocklists/addons-bloomfilters.json.

Because addons-bloomfilters.json is currently small, I'll land the functionality now. The optimization (third patch) can land later.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7f269f70c52c
Option to load from JSON dump if it's newer than local data r=leplatrem
https://hg.mozilla.org/integration/autoland/rev/1195f23a6393
Use newer blocklist dump if available r=Gijs

Comment 6

[Dorel Luca [:dluca]]

Backed out 2 changesets (bug 1717068) for Build bustages. CLOSED TREE

Log:
https://treeherder.mozilla.org/logviewer?job_id=344711488&repo=autoland&lineNumber=1127

Push with failures:
https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&revision=1195f23a63939b31df3c1d372ef711cda7a54113

Backout:
https://hg.mozilla.org/integration/autoland/rev/d9af3b01a5bca102470f845d02d1c3d0a88c4432

Comment 7

[Rob Wu [:robwu]]

I inadvertently registered the test from the third patch in the second patch.

I'll fix that and reland later (possibly the whole stack if r+).

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/cd7ca21b05e9
Option to load from JSON dump if it's newer than local data r=leplatrem
https://hg.mozilla.org/integration/autoland/rev/29ab80ae2bb2
Use newer blocklist dump if available r=Gijs
https://hg.mozilla.org/integration/autoland/rev/e89a9e3bd3a7
Package summary of last_modified of RemoteSettings dumps r=leplatrem,nalexander

Comment 9

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/cd7ca21b05e9
https://hg.mozilla.org/mozilla-central/rev/29ab80ae2bb2
https://hg.mozilla.org/mozilla-central/rev/e89a9e3bd3a7