1620621 - (blocklist-v3) Use cascade bloom filters in add-on blocklist

Status: RESOLVED FIXED

(Toolkit :: Blocklist Implementation, enhancement, P1)

Description

[Rob Wu [:robwu]]

To improve the performance and scalability of the add-on blocklist, we're going to replace the current JSON blocklist (and its predecessor, the XML blocklist) with a blocklist based on cascade bloom filters (inspired by CRLite).

We will drop unused features. The only supported operation is querying whether a given add-on is blocked. In particular, the block reason won't be part of the blocklist data itself, but can be looked up on demand.

PRD: https://docs.google.com/document/d/12wGjFjscu2pixwt795aHAyoswHsQzztzVUzWjiQz4vg/edit

Comment 1

[Rob Wu [:robwu]]

Attachment: Bug 1620621 - Add XPCOM FFI for rust_cascade

Comment 2

[Rob Wu [:robwu]]

Attachment: Bug 1620621 - Add caching and recovery mechanisms to RemoteSettings's download() method

Comment 3

[Rob Wu [:robwu]]

Attachment: Bug 1620621 - Add support for fallback to dumps for attachments

With this piece, it is now possible for RemoteSettings clients to always
have a valid attachment.

Comment 4

[Rob Wu [:robwu]]

Attachment: Bug 1620621 - Add bloomfilter-based blocklist for addons

NOTE: This commit does not yet include a dump of the RemoteSettings
collection and attachment. This will be added in the near future.

Comment 5

[Rob Wu [:robwu]]

Attachment: Bug 1620621 - Unit tests for MLBF-based blocklist

Comment 6

[Rob Wu [:robwu]]

Attachment: Bug 1620621 - Implement blocklist stashing behind pref

Comment 7

[Rob Wu [:robwu]]

Attachment: Bug 1620621 - Add initial dump of addons blocklist

The MLBF (addons-mlbf.bin) itself is 64 KB.
Together with the metadata, this is 65 KB.
In contrast, the current JSON-based dump (addons.json) is 913 KB.

Comment 8

[Rob Wu [:robwu]]

Attachment: Bug 1620621 - Temporarily disable test_remote_settings_poll.js

These two tests use the Kinto client to access the RemoteSettings
database. However, due to the database version bump from D72416,
the kinto client is no longer compatible with the RemoteSettings
database.

This will be fixed in bug 1634203.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/bbc991f09d5d
Add XPCOM FFI for rust_cascade r=emilio,vporof
https://hg.mozilla.org/integration/autoland/rev/723a3b4e7bbf
Add caching and recovery mechanisms to RemoteSettings's download() method r=Gijs,leplatrem
https://hg.mozilla.org/integration/autoland/rev/90c08438be0a
Add support for fallback to dumps for attachments r=Gijs,leplatrem
https://hg.mozilla.org/integration/autoland/rev/1a28d1de8f76
Add bloomfilter-based blocklist for addons r=Gijs,aswan
https://hg.mozilla.org/integration/autoland/rev/7abf836343be
Unit tests for MLBF-based blocklist r=Gijs
https://hg.mozilla.org/integration/autoland/rev/cfde27a748fd
Implement blocklist stashing behind pref r=Gijs
https://hg.mozilla.org/integration/autoland/rev/8b4e286967c0
Add initial dump of addons blocklist r=Gijs
https://hg.mozilla.org/integration/autoland/rev/06fccc75c09e
Temporarily disable 2 RemoteSettings tests r=glasserc

Comment 10

[Dorel Luca [:dluca]]

Backed out 8 changesets (bug 1620621) for XPCshell failures in xpcshell/rs-blocklist/test_blocklist_mlbf_fetch.js

Log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=300124869&repo=autoland&lineNumber=4275

Push with failures:
https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&revision=06fccc75c09e6c8e03f58f373109b18be30e0e6d

Backout:
https://hg.mozilla.org/integration/autoland/rev/769b5f27c2d9eba5a13a42e8dbaa4129ba48f457

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/80dcb089ce8e
Add XPCOM FFI for rust_cascade r=emilio,vporof
https://hg.mozilla.org/integration/autoland/rev/12817823a3c9
Add caching and recovery mechanisms to RemoteSettings's download() method r=Gijs,leplatrem
https://hg.mozilla.org/integration/autoland/rev/e0f3e057b311
Add support for fallback to dumps for attachments r=Gijs,leplatrem
https://hg.mozilla.org/integration/autoland/rev/206ad824e1bc
Add bloomfilter-based blocklist for addons r=Gijs,aswan
https://hg.mozilla.org/integration/autoland/rev/c23703684254
Unit tests for MLBF-based blocklist r=Gijs
https://hg.mozilla.org/integration/autoland/rev/6df42b7528ec
Implement blocklist stashing behind pref r=Gijs
https://hg.mozilla.org/integration/autoland/rev/56b2b19a9bc1
Add initial dump of addons blocklist r=Gijs
https://hg.mozilla.org/integration/autoland/rev/14a57e32c414
Temporarily disable test_remote_settings_poll.js r=glasserc

Comment 12

[Razvan Maries]

Backed out for build bustages.

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedTaskRun=H6tfa38gTrKUsRFyK4Tx7Q-0&resultStatus=testfailed%2Cbusted%2Cexception&revision=14a57e32c414009fd0b6bf123b1751dbbe6b2e5d

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=300175058&repo=autoland&lineNumber=62282

TV Failures:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=300177439&repo=autoland&lineNumber=965
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=300178349&repo=autoland&lineNumber=1434

xpcshell failures:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=300177217&repo=autoland&lineNumber=7116
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=300177069&repo=autoland&lineNumber=6769

Backout: https://hg.mozilla.org/integration/autoland/rev/d1eb84b142af186d4c31effc043b954f68679e4d

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/15049f81c652
Add XPCOM FFI for rust_cascade r=emilio,vporof
https://hg.mozilla.org/integration/autoland/rev/c0a18dbb6c50
Add caching and recovery mechanisms to RemoteSettings's download() method r=Gijs,leplatrem
https://hg.mozilla.org/integration/autoland/rev/c4a4b03a7647
Add support for fallback to dumps for attachments r=Gijs,leplatrem
https://hg.mozilla.org/integration/autoland/rev/2a8f00adf40e
Add bloomfilter-based blocklist for addons r=Gijs,aswan
https://hg.mozilla.org/integration/autoland/rev/061107092afd
Unit tests for MLBF-based blocklist r=Gijs
https://hg.mozilla.org/integration/autoland/rev/55b40420f1b5
Implement blocklist stashing behind pref r=Gijs
https://hg.mozilla.org/integration/autoland/rev/ae10e6fa5a55
Add initial dump of addons blocklist r=Gijs
https://hg.mozilla.org/integration/autoland/rev/d34716786f5b
Temporarily disable test_remote_settings_poll.js r=glasserc

Comment 14

[Rob Wu [:robwu]]

The initial backout was due to bug 1620621, which addressed by a work-around in the test.

The build failures for the second backout is due to Lando zeroing the binary from the "initial dump" commit. I fixed that by pushing my local commits to Phabricator and relanded.

Comment 16

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/15049f81c652
https://hg.mozilla.org/mozilla-central/rev/c0a18dbb6c50
https://hg.mozilla.org/mozilla-central/rev/c4a4b03a7647
https://hg.mozilla.org/mozilla-central/rev/2a8f00adf40e
https://hg.mozilla.org/mozilla-central/rev/061107092afd
https://hg.mozilla.org/mozilla-central/rev/55b40420f1b5
https://hg.mozilla.org/mozilla-central/rev/ae10e6fa5a55
https://hg.mozilla.org/mozilla-central/rev/d34716786f5b

Comment 17

[tbodt]

With this flag enabled it's not possible to re-enable a blocked extension like it was before

Comment 18

[Rob Wu [:robwu]]

(In reply to tbodt from comment #17)

With this flag enabled it's not possible to re-enable a blocked extension like it was before

Previously, there was the concept of "hard block" and "soft block", with the latter being optional. In practice, soft-blocking was not used for new blocks any more, so the new implementation only supports hard blocks. If you have a situation where an add-on was unexpectedly blocked, please file a new bug with an example of the blocked add-on and cc me on it.