1594497 - Increase shavar chunk size limit

Status: VERIFIED FIXED

(Core :: Privacy: Anti-Tracking, enhancement)

Description

[Se Yeon Kim]

While pushing all changes to the entity list to include Google's regional domains, we broke production Shavar because of the max size limit on the client which produced sustained increase in requests for download as it failed to download the latest entity list.

As mentioned in this GitHub issue we need to increase the size of the Shavar chunk size limit to be 3.5MB.

Comment 1

[Se Yeon Kim]

Attachment: Bug 1594497 - Increase shavar chunk size limit r?dimi

Comment 2

[Dimi Lee [:dimi]]

Hi Kim,
I think we decided to use multiple lists to make sure the size of each list shouldn't exceed the maximum chunk size limit? If that is true, do we still need to increase the limit?

Comment 3

[Se Yeon Kim]

Hi Dimi,

When I separated the Google entity lists it was about 1.4MB. So I suggested that we break the larger lists but also increase the size limit. There was a conversation about sharding the Google entity lists but we were unable to predict how Google's domains would expand so for now we decided separating the large organization and their entity lists and increasing the max size.

Comment 4

[BugBot [:suhaib / :marco/ :calixte]]

There's a r+ patch which didn't land and no activity in this bug for 2 weeks.
:sykim, could you have a look please?
For more information, please visit auto_nag documentation.

Comment 5

[Se Yeon Kim]

Attachment: Bug 1594497 - Add Google entity lists r?dimi

Depends on D52913

Comment 6

[Se Yeon Kim]

Adding another patch to have google-trackwhite-digest256 be added to all prefs where mozstd-trackwhite-digest256 was, except cryptomining.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a97415a7ce80
Increase shavar chunk size limit r=dimi
https://hg.mozilla.org/integration/autoland/rev/c817e489fd80
Add Google entity lists r=dimi

Comment 8

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f8403e922e55
Backed out 2 changesets causing firefox-ui test failures on test_initial_download.py. CLOSED TREE

Comment 9

[Se Yeon Kim]

Test kicked out because it was unable to check that the google-trackwhite-digest256 list was installed. This failure should be fixed by having the list be available in Shavar.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fdef77d6d362
Increase shavar chunk size limit r=dimi
https://hg.mozilla.org/integration/autoland/rev/dc907aeafad0
Add Google entity lists r=dimi

Comment 11

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/fdef77d6d362
https://hg.mozilla.org/mozilla-central/rev/dc907aeafad0

Comment 12

[Razvan Maries]

Backed out for breaking Twitter pages with blocking settings set to 'Strict'.

Backout: https://hg.mozilla.org/integration/autoland/rev/83ff10838569924efab569b276398f470fed15a4

Comment 13

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Steps to reproduce:

1. Open Nightly's Options.
2. Go to Privacy & Security.
3. Switch Enhanced Tracking Protection to 'Strict'.
4. Load a tweet page, e.g. https://twitter.com/mozilla/status/1208786816704405506

Actual result: Page only rendered as text because some resources are blocked.
Expected result: 'Nice' page layout.

Comment 14

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Backout got merged: https://hg.mozilla.org/mozilla-central/rev/83ff10838569

Comment 15

[Steven Englehardt [:englehardt]]

(In reply to Sebastian Hengst [:aryx] [limited availability until start of 2020] (needinfo on intermittent or backout) from comment #13)

Steps to reproduce:

1. Open Nightly's Options.
2. Go to Privacy & Security.
3. Switch Enhanced Tracking Protection to 'Strict'.
4. Load a tweet page, e.g. https://twitter.com/mozilla/status/1208786816704405506

Actual result: Page only rendered as text because some resources are blocked.
Expected result: 'Nice' page layout.

From this description it sounds like the entity list failed to load or failed to apply entirely. I can reproduce by blocking all resources fromtwimg.com, which is on the blocklist and should be on the Twitter entitylist.

Comment 16

[Se Yeon Kim]

Attachment: Bug 1594497 - Fix typo on socialtracking whitelistTables r?englehardt

Depends on D57389

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/49a332e535d5
Increase shavar chunk size limit r=dimi
https://hg.mozilla.org/integration/autoland/rev/2f44d2e92d64
Add Google entity lists r=dimi
https://hg.mozilla.org/integration/autoland/rev/eb7267486666
Fix typo on socialtracking whitelistTables r=englehardt

Comment 18

[Se Yeon Kim]

Hi Razvan, thanks for informing us about the breakage. The fix has been applied and pushed here. Once the patch is available in Nightly, in addition to visually checking that the Twitter page does not look broken, here are the steps to check that it has been fixed:

1. Go to about:url-classifier
2. To get the latest changes to tracking protection lists, click "Trigger Update" button in the "Mozilla" row in the table "Provider" and check that "Last update status" column states "success".
3. In the "Search" table, change the "List type" from "Blacklist" to "Whitelist"
4. Enter url https://twitter.com/?resource=twimg.com
5. Check that the url is listed under the "socialtracking-protection" and "socialtracking-annotation" features on the results
6. Check that the "List of tables" for the results row for features "socialtracking-protection" and "socialtracking-annotation" have "mozstd-trackwhite-digest256"

Comment 19

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/49a332e535d5
https://hg.mozilla.org/mozilla-central/rev/2f44d2e92d64
https://hg.mozilla.org/mozilla-central/rev/eb7267486666

Comment 20

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Verified according to the steps in comment 18.

Please set the qe-verify flag if a bug fix needs verification in the future (Razvan is not part of the QA team but of the code sheriffs).