1451040 - Update JSON dumps of remote settings regularly

Status: RESOLVED FIXED

(Firefox :: Remote Settings Client, enhancement)

Description

[Mathieu Leplatre [:leplatrem]]

Currently, we have JSON dumps of the blocklist settings in https://searchfox.org/mozilla-central/source/services/blocklists

This provides defaults to use if the server has no data (or hasn't done the fetch yet) and reduces the amount of entries to download on first sync.

We would need to setup a task/bot/? to update those files regularly using the latest content available server. 

Something similar is already done for blocklist.xml: https://searchfox.org/mozilla-central/commit/028cd8342899357d80fafba2d528a0fd5819e316

How could you proceed to setup this?

Thanks!

Comment 1

[Mathieu Leplatre [:leplatrem]]

Attachment: Bug 1451040 - Download remote settings dumps regularly

Review commit: https://reviewboard.mozilla.org/r/244278/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/244278/

Comment 2

[Mathieu Leplatre [:leplatrem]]

Comment on attachment 8976090 [details]
Bug 1451040 - Download remote settings dumps regularly

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/244278/diff/1-2/

Comment 3

[Mathieu Leplatre [:leplatrem]]

Comment on attachment 8976090 [details]
Bug 1451040 - Download remote settings dumps regularly

Hi Simon!
I saw you were the latest editor on this script :) 
Are you in charge of this taskcluster job?

In Bug 1257565 we'll get rid of blocklist XML in favor of JSON dumps. We want to update them regularly.

Would you be willing to review/test this patch please?

Thanks!

(Also, first we need Bug 1451050 to land, for dumps to be present in services/settings/dumps)

Comment 4

[Simon Fraser [:sfraser] ⌚️GMT]

Comment on attachment 8976090 [details]
Bug 1451040 - Download remote settings dumps regularly

https://reviewboard.mozilla.org/r/244278/#review251332

`./mach lint -l shellcheck` has some linting errors that will need fixing. 
You will also need to edit runme.sh if you'd like the parameters to be used, and `taskcluster/ci/repo-update/kind.yml` to activate it. Thanks!

::: taskcluster/docker/periodic-updates/scripts/periodic_file_updates.sh:445
(Diff revision 2)
>  }
>  
> +# Copies new remote settings dump files in place, and commits them.
> +function commit_remote_settings_files {
> +  cd "${BASEDIR}"
> +  cp -a ${REMOTE_SETINGS_OUTPUT}/* ${REPODIR}${REMOTE_SETTINGS_DIR}

Typo here, SETINGS should be SETTINGS.

Comment 5

[Mathieu Leplatre [:leplatrem]]

Comment on attachment 8976090 [details]
Bug 1451040 - Download remote settings dumps regularly

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/244278/diff/2-3/

Comment 6

[Simon Fraser [:sfraser] ⌚️GMT]

Comment on attachment 8976090 [details]
Bug 1451040 - Download remote settings dumps regularly

https://reviewboard.mozilla.org/r/244278/#review251838

Looks good, go for it.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3cdec5d6440d
Download remote settings dumps regularly r=sfraser

Comment 8

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/3cdec5d6440d

Comment 9

[Simon Fraser [:sfraser] ⌚️GMT]

Seeing task failures like https://tools.taskcluster.net/groups/CycLNNp6RfS0GsVlkyVFvQ/tasks/cVi2QecTRuGFBfUVbvqXPg/runs/0/logs/public%2Flogs%2Flive.log 

+ cp -a '/home/worker/data/remote-settings.out/*' mozilla-central/services/settings/dumps
cp: cannot stat '/home/worker/data/remote-settings.out/*': No such file or directory

Comment 10

[Mathieu Leplatre [:leplatrem]]

How would you explain that we have a number of `mkdir -p` instructions of /home/worker/data/remote-settings.out ?
And that the .diff command does not fail ?

I'm lost :(

Comment 11

[Simon Fraser [:sfraser] ⌚️GMT]

The remote-settings.diff artifact has a lot of text in it, so it looks like the diff correctly reported some content, and returned 0.

Looking at this line:
cp -a "${REMOTE_SETTINGS_OUTPUT}/*" "${REPODIR}${REMOTE_SETTINGS_DIR}"

I think it should be:

"${REMOTE_SETTINGS_OUTPUT}"/* 

That way it's not looking for a literal '*' as a filename