Split the Disconnect list into multiple pieces for DNT compliance

RESOLVED FIXED

Status

Cloud Services
Server: Shavar
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: francois, Assigned: groovecoder)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

2 years ago
Once Disconnect has annotated their list with DNT information, we will need to split their list into the trackers that honor DNT and those that don't.
(Reporter)

Updated

2 years ago
Blocks: 1258043
(Reporter)

Comment 1

2 years ago
Disconnect has submitted a PR with a sample annotated entry: https://github.com/mozilla-services/shavar-prod-lists/pull/14

It follows the format I suggested in https://public.etherpad-mozilla.org/p/dnt-loophole
(Reporter)

Comment 2

2 years ago
Here are the new lists to derive from the Disconnect upstream list:

- base-track-digest256 (DNT="", all categories except content)
- basew3c-track-digest256 (DNT==W3C, all categories except content)
- baseeff-track-digest256 (DNT==EFF, all categories except content)
- content-track-digest256 (DNT=="", content category)
- contentw3c-track-digest256 (DNT==W3C, content category)
- contenteff-track-digest256 (DNT==EFF, content category)

Given the latest PR from Disconnect (https://github.com/mozilla-services/shavar-prod-lists/pull/14), all of the lists should be empty except for:

- base-track-digest256 (roughly equivalent to mozstd-track-digest256)
- baseeff-track-digest256 (should have exactly 1 entry)
- content-track-digest256 (should be the difference between mozfull-track-digest256 and mozstd-track-digest256)
(Reporter)

Updated

2 years ago
Assignee: nobody → lcrouch
Status: NEW → ASSIGNED
(Reporter)

Updated

2 years ago
Blocks: 1258041
(Reporter)

Updated

2 years ago
Blocks: 1274671
(Assignee)

Comment 3

2 years ago
Having read thru the lists2safebrowsing.py script now ...

We only need the new separate lists for the tracking-protection section? I.e., no changes to the mozplugin-block nor mozpub-trackwhite lists?
Flags: needinfo?(francois)
(Reporter)

Comment 4

2 years ago
(In reply to Luke Crouch [:groovecoder] from comment #3)
> We only need the new separate lists for the tracking-protection section?
> I.e., no changes to the mozplugin-block nor mozpub-trackwhite lists?

Correct, we're not touching the plugin blocklist or the whitelist.

Looking at the prod config file (https://github.com/mozilla-services/shavar-list-creation-config/blob/master/prod.ini), we'll be adding these new sections:

- tracking-protection-base
- tracking-protection-baseeff
- tracking-protection-basew3c
- tracking-protection-content
- tracking-protection-contenteff
- tracking-protection-contentw3c

and not removing any of the existing ones because of backwards compatibility with old clients.

We'll add the same new lists to stage (https://github.com/mozilla-services/shavar-list-creation-config/blob/master/stage.ini). Note that we're not changing the end-to-end stage lists (tracking-protection-testing and entity-whitelist-testing) because they exist to exercise the entry addition/removal part of the list creation script only.
Flags: needinfo?(francois)
(Assignee)

Comment 5

2 years ago
:ckolos - Can you verify again here (IRC is ephemeral and I've lost track of it since last Wednesday) that Travis used/is-using the readme-and-comment-updates branch of shavar-list-creation to make the lists for the shavar.stage.mozaws.net environment?
Flags: needinfo?(ckolos)
Travis is not involved in this in the least.

Jenkins, however, is using the following branch:

<snip>
Repo:                               https://github.com/mozilla-services/shavar-list-creation/
Branch Specifier (blank for 'any'): */readme-and-comment-updates

The rules for the branch spec are as follows:

<Wildcards>
The syntax is of the form: REPOSITORYNAME/BRANCH. In addition, BRANCH is recognized as a shorthand of */BRANCH, '*' is recognized as a wildcard, and '**' is recognized as wildcard that includes the separator '/'. Therefore, origin/branches* would match origin/branches-foo but not origin/branches/foo, while origin/branches** would match both origin/branches-foo and origin/branches/foo.
Flags: needinfo?(ckolos)
(Assignee)

Comment 7

2 years ago
I think I found this myself ...

In https://deploy.mozaws.net/view/Shavar%20Stage/job/Shavar%20List%20Creation%20Stage/17522/git/ I see refs/remotes/origin/readme-and-comment-updates listed under "Revision" ... I also see refs/remotes/origin/master listed under "Built Branches" ... Are there docs about what each of these mean?

In any case, the console output shows "git checkout -f f361b44e28755739bf13dac1d1d857ca5192b7ed" as the last command for the repo.
(Assignee)

Comment 8

2 years ago
Oops, meant Jenkins. Thanks for the confirmation.
(Assignee)

Comment 9

2 years ago
The effective change is here, along with the script output as a comment:

https://github.com/mozilla-services/shavar-list-creation/commit/1f1d26cc553914e119e931a8947451fd4b65bc48

Will go thru some more verification & QA before sending the PR.
(Reporter)

Comment 10

2 years ago
(In reply to Luke Crouch [:groovecoder] from comment #9)
> The effective change is here, along with the script output as a comment:
> 
> https://github.com/mozilla-services/shavar-list-creation/commit/
> 1f1d26cc553914e119e931a8947451fd4b65bc48
> 
> Will go thru some more verification & QA before sending the PR.

The first part of that change (honoring the new dnt attribute) looks good.

The other part is to change the content lists so that they only include entries from the content category (i.e. not anything that's in the base list). I assume you'll be tackling that in a later commit?
(Assignee)

Comment 11

2 years ago
Ah, I had misunderstood then.

Is that only for the new DNT-separated lists? Or do the old lists need to be updated as well? It looks like the old tracking-protection-full list included Content AND the other categories together.
Flags: needinfo?(francois)
(Reporter)

Comment 12

2 years ago
(In reply to Luke Crouch [:groovecoder] from comment #11)
> Is that only for the new DNT-separated lists? Or do the old lists need to be
> updated as well?

Only for the new lists.

Lists are essentially frozen in time as soon as they get picked up by a Firefox release. So when we want to change them, we have to create new ones and leave the existing ones untouched. Then once there are no more users pulling them down, we can delete the old lists (e.g. bug 1277446).

> It looks like the old tracking-protection-full list
> included Content AND the other categories together.

Correct. That was unnecessary and we'll be fixing that in the new lists.
Flags: needinfo?(francois)
(Assignee)

Comment 13

2 years ago
Okay, updated to include *only* the content category in the new tracking-protection-content list.
(Assignee)

Comment 15

2 years ago
:francois, :rtilder - so, I think the combination of https://github.com/mozilla-services/shavar-list-creation-config/pull/11 and then https://github.com/mozilla-services/shavar-list-creation/pull/37 should deploy the new DNT lists to stage shavar?
Flags: needinfo?(rtilder)
Flags: needinfo?(francois)
(Reporter)

Updated

2 years ago
Flags: needinfo?(rtilder)
Flags: needinfo?(francois)

Comment 17

2 years ago
Commits pushed to master at https://github.com/mozilla-services/shavar-list-creation

https://github.com/mozilla-services/shavar-list-creation/commit/97704040300725180d1f2285b2644e556a90f0d9
Revert "bug 1258038: cleanups, DNT splitting, and content INCLUDE|SKIP|ONLY logic"

https://github.com/mozilla-services/shavar-list-creation/commit/1cecfb3d9049d474bb7ea50589494701b4f57267
Merge pull request #38 from mozilla-services/revert-37-start-list-split-1258038

Revert "bug 1258038: cleanups, DNT splitting, and content INCLUDE|SKIP|ONLY logic"

Comment 18

2 years ago
Commits pushed to master at https://github.com/mozilla-services/shavar-list-creation

https://github.com/mozilla-services/shavar-list-creation/commit/db50ce50660de5bbec7b4bf927a8989e7f31fa15
Revert "Revert "bug 1258038: cleanups, DNT splitting, and content INCLUDE|SKIP|ONLY logic""

This reverts commit 97704040300725180d1f2285b2644e556a90f0d9.

https://github.com/mozilla-services/shavar-list-creation/commit/db237a99a20aa72da684400b1a716551eb99a1d1
Merge pull request #39 from mozilla-services/start-list-split-1258038

Revert "Revert "bug 1258038: cleanups, DNT splitting, and content INC…
(Reporter)

Updated

2 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.