Support fetching data from Remote Setting
Categories
(Toolkit :: Safe Browsing, enhancement, P2)
Tracking
()
| Tracking | Status | |
|---|---|---|
| firefox104 | --- | fixed |
People
(Reporter: dimi, Assigned: dimi)
References
(Blocks 3 open bugs)
Details
Attachments
(4 files, 1 obsolete file)
Here is a quick note summarizing what needs to be done to support fetching data from Remote Setting in the url-classifier
Implement a service under toolkit/component/url-classifier to fetch data from RemoteSetting
Make URLClassifier aware that the list should come from RemoteSetting instead of shavar. Possible solutions include:
Add a custom url in browser.safebrowsing.provider.mozilla.updateURL pref , ex: rs://collection-name
Process the custom URL in nsUrlClassifierStreamUpdater (probably here). We should fetch data from the remote setting service instead of creating a http channel.
The service should fetch data from RemoteSetting and feed the data to URLClassifier through nsIStreamListener.idl interface
Still need to confirm:
Will the data from the remote setting contain all the available tables? Is it keyed by the table name?
| Assignee | ||
Comment 1•4 years ago
|
||
| Assignee | ||
Comment 2•4 years ago
|
||
| Assignee | ||
Comment 3•4 years ago
|
||
Depends on D135989
Updated•4 years ago
|
Updated•4 years ago
|
Updated•4 years ago
|
| Assignee | ||
Comment 4•4 years ago
|
||
We can use this telemetry to track the statistics of using
RemoteSettings to serve Safe Browsing data.
The can help us understand if we can roll out this feature to more users.
Depends on D135990
Updated•4 years ago
|
Updated•4 years ago
|
Updated•4 years ago
|
Comment 5•3 years ago
|
||
There are some r+ patches which didn't land and no activity in this bug for 2 weeks.
:dimi, could you have a look please?
If you still have some work to do, you can add an action "Plan Changes" in Phabricator.
For more information, please visit auto_nag documentation.
Updated•3 years ago
|
| Assignee | ||
Comment 6•3 years ago
|
||
We plan to add a version number in the RemoteSettings records, so there will be some changes in this patch.
I'm now waiting for the records to be updated to the server so I can continue the work.
| Assignee | ||
Comment 7•3 years ago
|
||
Depends on D136107
Updated•3 years ago
|
Backed out for causing lint failure on test_rsListService.js
Failure line: TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/toolkit/components/url-classifier/tests/unit/test_rsListService.js:19:1 | Unnecessary call to XPCOMUtils.defineLazyGlobalGetters for EventTarget (webidl names are automatically imported) (mozilla/reject-importGlobalProperties)
[vcs 2022-06-30T08:12:46.697Z] TinderboxPrint:<a href=https://hg.mozilla.org/integration/autoland/rev/408c00af6ca7a3952fae070800b82f55f0465649 title='Built from autoland revision 408c00af6ca7a3952fae070800b82f55f0465649'>408c00af6ca7a3952fae070800b82f55f0465649</a>
[setup 2022-06-30T08:12:46.697Z] GECKO_PATH is /builds/worker/checkouts/gecko
[task 2022-06-30T08:12:46.697Z] executing ['bash', '-cx', 'cp -r /build/node_modules_eslint node_modules && cp -r /build/node_modules_eslint-plugin-mozilla/ tools/lint/eslint/eslint-plugin-mozilla/node_modules && ln -s ../tools/lint/eslint/eslint-plugin-mozilla node_modules && ln -s ../tools/lint/eslint/eslint-plugin-spidermonkey-js node_modules && ./mach lint -v -l eslint -f treeherder -f json:/builds/worker/mozlint.json *\n']in /builds/worker/checkouts/gecko
[task 2022-06-30T08:12:46.708Z] + cp -r /build/node_modules_eslint node_modules
[task 2022-06-30T08:13:00.760Z] + cp -r /build/node_modules_eslint-plugin-mozilla/ tools/lint/eslint/eslint-plugin-mozilla/node_modules
[task 2022-06-30T08:13:02.431Z] + ln -s ../tools/lint/eslint/eslint-plugin-mozilla node_modules
[task 2022-06-30T08:13:02.433Z] + ln -s ../tools/lint/eslint/eslint-plugin-spidermonkey-js node_modules
[task 2022-06-30T08:13:02.434Z] + ./mach lint -v -l eslint -f treeherder -f json:/builds/worker/mozlint.json accessible aclocal.m4 AUTHORS browser build build.gradle caps Cargo.lock Cargo.toml chrome client.mk client.py CLOBBER config configure.in configure.py devtools docs docshell dom editor extensions gfx GNUmakefile gradle gradle.properties gradlew gradlew.bat hal image intl ipc js layout LICENSE mach mach.cmd mach.ps1 Makefile.in media memory mfbt mobile modules moz.build moz.configure mozglue mozilla-config.h.in netwerk node_modules nsprpub old-configure.in other-licenses package.json package-lock.json parser python README.txt remote security services servo settings.gradle startupcache storage substitute-local-geckoview.gradle supply-chain taskcluster testing test.mozbuild third_party toolkit tools uriloader view widget xpcom xpfe
[task 2022-06-30T08:13:05.182Z] created virtual environment CPython3.6.9.final.0-64 in 77ms
[task 2022-06-30T08:13:05.182Z] creator CPython3Posix(dest=/builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/lint, clear=False, no_vcs_ignore=False, global=False)
[task 2022-06-30T08:13:05.182Z] activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator
[task 2022-06-30T08:13:05.866Z] 08:13:05.866 mozlint (16) | setup for eslint finished in 0.61 seconds
[task 2022-06-30T08:13:05.870Z] Creating default state directory: /builds/worker/.mozbuild
[task 2022-06-30T08:13:06.007Z] 08:13:06.7 eslint (66) | Passing the following paths:
[task 2022-06-30T08:13:06.007Z] /builds/worker/checkouts/gecko/startupcache
[task 2022-06-30T08:13:06.008Z] /builds/worker/checkouts/gecko/taskcluster
[task 2022-06-30T08:13:06.008Z] /builds/worker/checkouts/gecko/docs
[task 2022-06-30T08:13:06.008Z] /builds/worker/checkouts/gecko/toolkit
[task 2022-06-30T08:13:06.008Z] /builds/worker/checkouts/gecko/xpfe
[task 2022-06-30T08:13:06.008Z] /builds/worker/checkouts/gecko/config
[task 2022-06-30T08:13:06.008Z] /builds/worker/checkouts/gecko/mfbt
[task 2022-06-30T08:13:06.008Z] /builds/worker/checkouts/gecko/docshell
[task 2022-06-30T08:13:06.008Z] /builds/worker/checkouts/gecko/hal
[task 2022-06-30T08:13:06.018Z] 08:13:06.17 eslint (66) | Command: /usr/local/bin/node /builds/worker/checkouts/gecko/node_modules/eslint/bin/eslint.js --ext [mjs,js,jsm,jsx,html,sjs,xhtml] --format json --no-error-on-unmatched-pattern --ignore-pattern toolkit/components/normandy/vendor --ignore-pattern toolkit/components/viaduct/fetch_msg_types.pb.h --ignore-pattern toolkit/components/protobuf --ignore-pattern mfbt/double-conversion/double-conversion --ignore-pattern toolkit/crashreporter/google-breakpad --ignore-pattern toolkit/components/utils/mozjexl.js --ignore-pattern toolkit/components/jsoncpp --ignore-pattern mfbt/lz4 --ignore-pattern toolkit/components/url-classifier/chromium --ignore-pattern toolkit/components/viaduct/fetch_msg_types.pb.cc --ignore-pattern toolkit/crashreporter/breakpad-client --ignore-pattern toolkit/components/certviewer/content/vendor --ignore-pattern toolkit/components/passwordmgr/PasswordRulesParser.jsm /builds/worker/checkouts/gecko/startupcache /builds/worker/checkouts/gecko/taskcluster /builds/worker/checkouts/gecko/docs /builds/worker/checkouts/gecko/toolkit /builds/worker/checkouts/gecko/xpfe /builds/worker/checkouts/gecko/config /builds/worker/checkouts/gecko/mfbt /builds/worker/checkouts/gecko/docshell /builds/worker/checkouts/gecko/hal
[task 2022-06-30T08:13:06.045Z] 08:13:06.45 eslint (63) | Passing the following paths:
[task 2022-06-30T08:13:06.045Z] /builds/worker/checkouts/gecko/accessible
[task 2022-06-30T08:13:06.045Z] /builds/worker/checkouts/gecko/services
[task 2022-06-30T08:13:06.045Z] /builds/worker/checkouts/gecko/caps
[task 2022-06-30T08:13:06.045Z] /builds/worker/checkouts/gecko/gradle
[task 2022-06-30T08:13:06.045Z] /builds/worker/checkouts/gecko/tools
[task 2022-06-30T08:13:06.045Z] /builds/worker/checkouts/gecko/extensions
[task 2022-06-30T08:13:06.046Z] /builds/worker/checkouts/gecko/memory
[task 2022-06-30T08:13:06.046Z] /builds/worker/checkouts/gecko/remote
[task 2022-06-30T08:13:06.046Z] /builds/worker/checkouts/gecko/uriloader
[task 2022-06-30T08:13:06.046Z] /builds/worker/checkouts/gecko/security
[task 2022-06-30T08:13:06.046Z] /builds/worker/checkouts/gecko/widget
[task 2022-06-30T08:13:06.046Z] /builds/worker/checkouts/gecko/ipc
task 2022-06-30T08:13:06.057Z] 08:13:06.56 eslint (63) | Command: /usr/local/bin/node /builds/worker/checkouts/gecko/node_modules/eslint/bin/eslint.js --ext [mjs,js,jsm,jsx,html,sjs,xhtml] --format json --no-error-on-unmatched-pattern --ignore-pattern tools/profiler/core/vtune --ignore-pattern security/nss --ignore-pattern remote/cdp/test/browser/chrome-remote-interface.js --ignore-pattern extensions/spellcheck/hunspell/src --ignore-pattern tools/lint/test/files --ignore-pattern ipc/chromium/src/third_party --ignore-pattern remote/test/puppeteer --ignore-pattern tools/fuzzing/libfuzzer --ignore-pattern security/sandbox/chromium-shim --ignore-pattern security/sandbox/chromium /builds/worker/checkouts/gecko/accessible /builds/worker/checkouts/gecko/services /builds/worker/checkouts/gecko/caps /builds/worker/checkouts/gecko/gradle /builds/worker/checkouts/gecko/tools /builds/worker/checkouts/gecko/extensions /builds/worker/checkouts/gecko/memory /builds/worker/checkouts/gecko/remote /builds/worker/checkouts/gecko/uriloader /builds/worker/checkouts/gecko/security /builds/worker/checkouts/gecko/widget /builds/worker/checkouts/gecko/ipc
[task 2022-06-30T08:13:06.083Z] 08:13:06.83 eslint (64) | Passing the following paths:
[task 2022-06-30T08:13:06.083Z] /builds/worker/checkouts/gecko/chrome
[task 2022-06-30T08:13:06.083Z] /builds/worker/checkouts/gecko/supply-chain
[task 2022-06-30T08:13:06.083Z] /builds/worker/checkouts/gecko/servo
[task 2022-06-30T08:13:06.083Z] /builds/worker/checkouts/gecko/media
[task 2022-06-30T08:13:06.083Z] /builds/worker/checkouts/gecko/storage
[task 2022-06-30T08:13:06.083Z] /builds/worker/checkouts/gecko/modules
[task 2022-06-30T08:13:06.083Z] /builds/worker/checkouts/gecko/testing
[task 2022-06-30T08:13:06.083Z] /builds/worker/checkouts/gecko/mozglue
[task 2022-06-30T08:13:06.083Z] /builds/worker/checkouts/gecko/intl
[task 2022-06-30T08:13:06.083Z] /builds/worker/checkouts/gecko/gfx
[task 2022-06-30T08:13:06.084Z] /builds/worker/checkouts/gecko/view
[task 2022-06-30T08:13:06.084Z] /builds/worker/checkouts/gecko/js
[task 2022-06-30T08:13:06.092Z] 08:13:06.92 eslint (64) | Command: /usr/local/bin/node /builds/worker/checkouts/gecko/node_modules/eslint/bin/eslint.js --ext [mjs,js,jsm,jsx,html,sjs,xhtml] --format json --no-error-on-unmatched-pattern --ignore-pattern media/libdav1d --ignore-pattern testing/talos/talos/tests/dromaeo --ignore-pattern media/libopus --ignore-pattern js/src/jit/arm64/vixl --ignore-pattern testing/xpcshell/odoh-wasm --ignore-pattern intl/unicharutil/util/nsSpecialCasingData.cpp --ignore-pattern modules/zlib --ignore-pattern gfx/vr/service/openvr --ignore-pattern media/libvpx --ignore-pattern gfx/skia --ignore-pattern js/src/vtune/jitprofiling.c --ignore-pattern media/mp4parse-rust --ignore-pattern intl/unicharutil/util/nsUnicodePropertyData.cpp --ignore-pattern media/libtheora --ignore-pattern media/libcubeb --ignore-pattern mozglue/misc/decimal --ignore-pattern testing/talos/talos/tests/kraken --ignore-pattern media/mozva/va --ignore-pattern testing/web-platform/tests/resources/webidl2 --ignore-pattern media/libogg --ignore-pattern media/libaom --ignore-pattern js/src/vtune/ittnotify.h --ignore-pattern js/src/vtune/ittnotify_config.h --ignore-pattern mozglue/tests/glibc_printf_tests --ignore-pattern js/src/dtoa.c --ignore-pattern media/kiss_fft --ignore-pattern media/libmkv --ignore-pattern gfx/ycbcr --ignore-pattern testing/web-platform/tests/tools/third_party --ignore-pattern media/libpng --ignore-pattern js/src/vtune/jitprofiling.h --ignore-pattern gfx/wr --ignore-pattern testing/xpcshell/dns-packet --ignore-pattern media/libnestegg --ignore-pattern gfx/graphite2 --ignore-pattern testing/xpcshell/node-ip --ignore-pattern gfx/harfbuzz --ignore-pattern gfx/ots --ignore-pattern media/openmax_il --ignore-pattern media/webrtc/signaling/gtest/MockCall.h --ignore-pattern testing/modules/sinon-7.2.7.js --ignore-pattern media/libyuv --ignore-pattern testing/xpcshell/node-http2 --ignore-pattern js/src/ctypes/libffi --ignore-pattern js/src/vtune/ittnotify_static.h --ignore-pattern testing/mochitest/tests/MochiKit-1.4.2 --ignore-pattern media/openmax_dl --ignore-pattern media/libspeex_resampler --ignore-pattern media/libtremor --ignore-pattern testing/mochitest/pywebsocket3 --ignore-pattern js/src/zydis --ignore-pattern media/libwebp --ignore-pattern js/src/irregexp/imported --ignore-pattern modules/freetype2 --ignore-pattern testing/mochitest/MochiKit --ignore-pattern js/src/vtune/ittnotify_static.c --ignore-pattern media/libvorbis --ignore-pattern modules/fdlibm --ignore-pattern js/src/editline --ignore-pattern modules/brotli --ignore-pattern media/libsoundtouch --ignore-pattern modules/xz-embedded --ignore-pattern intl/icu --ignore-pattern js/src/vtune/legacy --ignore-pattern gfx/cairo --ignore-pattern js/src/octane --ignore-pattern intl/components/src/UnicodeScriptCodes.h --ignore-pattern gfx/angle/checkout --ignore-pattern js/src/vtune/disable_warnings.h --ignore-pattern testing/mozbase/mozproxy/mozproxy/backends/mitm/scripts/catapult --ignore-pattern media/ffvpx --ignore-pattern testing/talos/talos/tests/v8_7 --ignore-pattern media/libjpeg --ignore-pattern modules/woff2 --ignore-pattern js/src/vtune/ittnotify_types.h --ignore-pattern gfx/qcms /builds/worker/checkouts/gecko/chrome /builds/worker/checkouts/gecko/supply-chain /builds/worker/checkouts/gecko/servo /builds/worker/checkouts/gecko/media /builds/worker/checkouts/gecko/storage /builds/worker/checkouts/gecko/modules /builds/worker/checkouts/gecko/testing /builds/worker/checkouts/gecko/mozglue /builds/worker/checkouts/gecko/intl /builds/worker/checkouts/gecko/gfx /builds/worker/checkouts/gecko/view /builds/worker/checkouts/gecko/js
[task 2022-06-30T08:13:06.122Z] 08:13:06.122 eslint (65) | Passing the following paths:
[task 2022-06-30T08:13:06.122Z] /builds/worker/checkouts/gecko/editor
[task 2022-06-30T08:13:06.122Z] /builds/worker/checkouts/gecko/devtools
[task 2022-06-30T08:13:06.122Z] /builds/worker/checkouts/gecko/layout
[task 2022-06-30T08:13:06.122Z] /builds/worker/checkouts/gecko/netwerk
[task 2022-06-30T08:13:06.122Z] /builds/worker/checkouts/gecko/browser
[task 2022-06-30T08:13:06.122Z] /builds/worker/checkouts/gecko/parser
[task 2022-06-30T08:13:06.122Z] /builds/worker/checkouts/gecko/build
[task 2022-06-30T08:13:06.123Z] /builds/worker/checkouts/gecko/image
[task 2022-06-30T08:13:06.123Z] /builds/worker/checkouts/gecko/dom
[task 2022-06-30T08:13:06.123Z] /builds/worker/checkouts/gecko/python
[task 2022-06-30T08:13:06.123Z] /builds/worker/checkouts/gecko/mobile
[task 2022-06-30T08:13:06.123Z] /builds/worker/checkouts/gecko/xpcom
[task 2022-06-30T08:13:06.131Z] 08:13:06.131 eslint (65) | Command: /usr/local/bin/node /builds/worker/checkouts/gecko/node_modules/eslint/bin/eslint.js --ext [mjs,js,jsm,jsx,html,sjs,xhtml] --format json --no-error-on-unmatched-pattern --ignore-pattern dom/tests/mochitest/dom-level1-core --ignore-pattern dom/u2f/tests/pkijs --ignore-pattern netwerk/dns/nsIDNKitInterface.h --ignore-pattern dom/media/webrtc/transport/third_party --ignore-pattern dom/media/gmp/widevine-adapter/content_decryption_module.h --ignore-pattern dom/tests/mochitest/dom-level2-html --ignore-pattern browser/components/newtab/vendor --ignore-pattern dom/media/webaudio/test/blink --ignore-pattern dom/media/gmp/rlz --ignore-pattern xpcom/build/mach_override.h --ignore-pattern dom/media/platforms/ffmpeg/ffmpeg57 --ignore-pattern devtools/client/shared/sourceeditor/codemirror --ignore-pattern dom/media/webspeech/recognition/energy_endpointer_params.cc --ignore-pattern browser/components/translation/cld2 --ignore-pattern dom/media/webspeech/recognition/energy_endpointer.h --ignore-pattern dom/media/webspeech/recognition/energy_endpointer.cc --ignore-pattern xpcom/io/crc32c.c --ignore-pattern devtools/shared/heapsnapshot/CoreDump.pb.h --ignore-pattern parser/expat --ignore-pattern devtools/client/shared/source-map --ignore-pattern dom/media/gmp/widevine-adapter/content_decryption_module_ext.h --ignore-pattern dom/media/webspeech/recognition/endpointer.cc --ignore-pattern devtools/shared/sprintfjs --ignore-pattern dom/media/webspeech/recognition/energy_endpointer_params.h --ignore-pattern layout/docs/css-gap-decorations --ignore-pattern xpcom/build/mach_override.c --ignore-pattern devtools/client/shared/sourceeditor/test/codemirror --ignore-pattern browser/extensions/formautofill/test/fixtures/third_party --ignore-pattern dom/webauthn/tests/pkijs --ignore-pattern dom/media/gmp/widevine-adapter/content_decryption_module_proxy.h --ignore-pattern dom/media/platforms/ffmpeg/libav55 --ignore-pattern dom/media/webspeech/recognition/endpointer.h --ignore-pattern dom/canvas/test/webgl-conf/checkout --ignore-pattern dom/imptests --ignore-pattern devtools/client/shared/vendor --ignore-pattern devtools/shared/qrcode/decoder --ignore-pattern editor/libeditor/tests/browserscope/lib --ignore-pattern devtools/client/jsonview/lib/require.js --ignore-pattern dom/tests/mochitest/ajax --ignore-pattern devtools/shared/node-properties --ignore-pattern devtools/shared/storage/vendor --ignore-pattern dom/media/platforms/ffmpeg/libav54 --ignore-pattern devtools/shared/qrcode/encoder --ignore-pattern devtools/shared/heapsnapshot/CoreDump.pb.cc --ignore-pattern devtools/client/shared/sourceeditor/test/cm_mode_ruby.js --ignore-pattern dom/media/webvtt/vtt.jsm --ignore-pattern dom/tests/mochitest/dom-level2-core --ignore-pattern dom/media/platforms/ffmpeg/ffmpeg58 --ignore-pattern browser/components/pocket/content/panels/js/vendor --ignore-pattern dom/webauthn/cbor-cpp --ignore-pattern devtools/client/shared/build/babel.js --ignore-pattern browser/extensions/formautofill/content/third-party --ignore-pattern dom/media/platforms/ffmpeg/ffmpeg59 --ignore-pattern dom/media/platforms/ffmpeg/libav53 --ignore-pattern devtools/shared/jsbeautify --ignore-pattern dom/media/gmp/widevine-adapter/content_decryption_module_export.h --ignore-pattern netwerk/sctp/src /builds/worker/checkouts/gecko/editor /builds/worker/checkouts/gecko/devtools /builds/worker/checkouts/gecko/layout /builds/worker/checkouts/gecko/netwerk /builds/worker/checkouts/gecko/browser /builds/worker/checkouts/gecko/parser /builds/worker/checkouts/gecko/build /builds/worker/checkouts/gecko/image /builds/worker/checkouts/gecko/dom /builds/worker/checkouts/gecko/python /builds/worker/checkouts/gecko/mobile /builds/worker/checkouts/gecko/xpcom
[task 2022-06-30T08:14:17.319Z] 08:14:17.318 eslint (64) | Finished in 71.44 seconds
[task 2022-06-30T08:15:20.142Z] 08:15:20.142 eslint (63) | Finished in 134.27 seconds
[task 2022-06-30T08:17:37.727Z] 08:17:37.727 eslint (66) | Finished in 271.85 seconds
[task 2022-06-30T08:26:14.941Z] 08:26:14.940 eslint (65) | Finished in 789.06 seconds
[task 2022-06-30T08:26:14.953Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/toolkit/components/url-classifier/tests/unit/test_rsListService.js:19:1 | Unnecessary call to XPCOMUtils.defineLazyGlobalGetters for EventTarget (webidl names are automatically imported) (mozilla/reject-importGlobalProperties)
[taskcluster 2022-06-30 08:26:15.437Z] === Task Finished ===
[taskcluster 2022-06-30 08:26:15.971Z] Unsuccessful task run with exit code: 1 completed in 1383.104 seconds
| Assignee | ||
Updated•3 years ago
|
Comment 10•3 years ago
|
||
Comment 11•3 years ago
|
||
| bugherder | ||
https://hg.mozilla.org/mozilla-central/rev/792934b2ee56
https://hg.mozilla.org/mozilla-central/rev/9dd22b2b2196
https://hg.mozilla.org/mozilla-central/rev/1221f3ce857c
https://hg.mozilla.org/mozilla-central/rev/d4a800143281
Comment 12•2 years ago
|
||
Update, browser changes have landed, but STE still needs to make script updates to stabilize the sync mechanism on RS. Tagging Andre for awareness and status of SRE work
Comment 13•1 year ago
|
||
hey @cboozarjomehri, I think I'm missing context here .. can you give me a couple of details via Slack if something's missing on our end? BTW, SRE is not using bugzilla, hence the late response .. sorry about that!
Updated•1 year ago
|
Description
•