Closed Bug 1258038 Opened 9 years ago Closed 8 years ago

Split the Disconnect list into multiple pieces for DNT compliance

Categories

(Cloud Services :: Server: Shavar, defect)

Product:
Cloud Services
Component:
Server: Shavar
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: francois, Assigned: groovecoder)

References

Details

Once Disconnect has annotated their list with DNT information, we will need to split their list into the trackers that honor DNT and those that don't.
Blocks: 1258043
Disconnect has submitted a PR with a sample annotated entry: https://github.com/mozilla-services/shavar-prod-lists/pull/14 It follows the format I suggested in https://public.etherpad-mozilla.org/p/dnt-loophole
Here are the new lists to derive from the Disconnect upstream list: - base-track-digest256 (DNT="", all categories except content) - basew3c-track-digest256 (DNT==W3C, all categories except content) - baseeff-track-digest256 (DNT==EFF, all categories except content) - content-track-digest256 (DNT=="", content category) - contentw3c-track-digest256 (DNT==W3C, content category) - contenteff-track-digest256 (DNT==EFF, content category) Given the latest PR from Disconnect (https://github.com/mozilla-services/shavar-prod-lists/pull/14), all of the lists should be empty except for: - base-track-digest256 (roughly equivalent to mozstd-track-digest256) - baseeff-track-digest256 (should have exactly 1 entry) - content-track-digest256 (should be the difference between mozfull-track-digest256 and mozstd-track-digest256)
Assignee: nobody → lcrouch
Status: NEW → ASSIGNED
Blocks: 1258041
Blocks: 1274671
Having read thru the lists2safebrowsing.py script now ... We only need the new separate lists for the tracking-protection section? I.e., no changes to the mozplugin-block nor mozpub-trackwhite lists?
Flags: needinfo?(francois)
(In reply to Luke Crouch [:groovecoder] from comment #3) > We only need the new separate lists for the tracking-protection section? > I.e., no changes to the mozplugin-block nor mozpub-trackwhite lists? Correct, we're not touching the plugin blocklist or the whitelist. Looking at the prod config file (https://github.com/mozilla-services/shavar-list-creation-config/blob/master/prod.ini), we'll be adding these new sections: - tracking-protection-base - tracking-protection-baseeff - tracking-protection-basew3c - tracking-protection-content - tracking-protection-contenteff - tracking-protection-contentw3c and not removing any of the existing ones because of backwards compatibility with old clients. We'll add the same new lists to stage (https://github.com/mozilla-services/shavar-list-creation-config/blob/master/stage.ini). Note that we're not changing the end-to-end stage lists (tracking-protection-testing and entity-whitelist-testing) because they exist to exercise the entry addition/removal part of the list creation script only.
Flags: needinfo?(francois)
:ckolos - Can you verify again here (IRC is ephemeral and I've lost track of it since last Wednesday) that Travis used/is-using the readme-and-comment-updates branch of shavar-list-creation to make the lists for the shavar.stage.mozaws.net environment?
Flags: needinfo?(ckolos)
Travis is not involved in this in the least. Jenkins, however, is using the following branch: <snip> Repo: https://github.com/mozilla-services/shavar-list-creation/ Branch Specifier (blank for 'any'): */readme-and-comment-updates The rules for the branch spec are as follows: <Wildcards> The syntax is of the form: REPOSITORYNAME/BRANCH. In addition, BRANCH is recognized as a shorthand of */BRANCH, '*' is recognized as a wildcard, and '**' is recognized as wildcard that includes the separator '/'. Therefore, origin/branches* would match origin/branches-foo but not origin/branches/foo, while origin/branches** would match both origin/branches-foo and origin/branches/foo.
Flags: needinfo?(ckolos)
I think I found this myself ... In https://deploy.mozaws.net/view/Shavar%20Stage/job/Shavar%20List%20Creation%20Stage/17522/git/ I see refs/remotes/origin/readme-and-comment-updates listed under "Revision" ... I also see refs/remotes/origin/master listed under "Built Branches" ... Are there docs about what each of these mean? In any case, the console output shows "git checkout -f f361b44e28755739bf13dac1d1d857ca5192b7ed" as the last command for the repo.
Oops, meant Jenkins. Thanks for the confirmation.
The effective change is here, along with the script output as a comment: https://github.com/mozilla-services/shavar-list-creation/commit/1f1d26cc553914e119e931a8947451fd4b65bc48 Will go thru some more verification & QA before sending the PR.
(In reply to Luke Crouch [:groovecoder] from comment #9) > The effective change is here, along with the script output as a comment: > > https://github.com/mozilla-services/shavar-list-creation/commit/ > 1f1d26cc553914e119e931a8947451fd4b65bc48 > > Will go thru some more verification & QA before sending the PR. The first part of that change (honoring the new dnt attribute) looks good. The other part is to change the content lists so that they only include entries from the content category (i.e. not anything that's in the base list). I assume you'll be tackling that in a later commit?
Ah, I had misunderstood then. Is that only for the new DNT-separated lists? Or do the old lists need to be updated as well? It looks like the old tracking-protection-full list included Content AND the other categories together.
Flags: needinfo?(francois)
(In reply to Luke Crouch [:groovecoder] from comment #11) > Is that only for the new DNT-separated lists? Or do the old lists need to be > updated as well? Only for the new lists. Lists are essentially frozen in time as soon as they get picked up by a Firefox release. So when we want to change them, we have to create new ones and leave the existing ones untouched. Then once there are no more users pulling them down, we can delete the old lists (e.g. bug 1277446). > It looks like the old tracking-protection-full list > included Content AND the other categories together. Correct. That was unnecessary and we'll be fixing that in the new lists.
Flags: needinfo?(francois)
Okay, updated to include *only* the content category in the new tracking-protection-content list.
:francois, :rtilder - so, I think the combination of https://github.com/mozilla-services/shavar-list-creation-config/pull/11 and then https://github.com/mozilla-services/shavar-list-creation/pull/37 should deploy the new DNT lists to stage shavar?
Flags: needinfo?(rtilder)
Flags: needinfo?(francois)
Flags: needinfo?(rtilder)
Flags: needinfo?(francois)
Commits pushed to master at https://github.com/mozilla-services/shavar-list-creation https://github.com/mozilla-services/shavar-list-creation/commit/97704040300725180d1f2285b2644e556a90f0d9 Revert "bug 1258038: cleanups, DNT splitting, and content INCLUDE|SKIP|ONLY logic" https://github.com/mozilla-services/shavar-list-creation/commit/1cecfb3d9049d474bb7ea50589494701b4f57267 Merge pull request #38 from mozilla-services/revert-37-start-list-split-1258038 Revert "bug 1258038: cleanups, DNT splitting, and content INCLUDE|SKIP|ONLY logic"
Commits pushed to master at https://github.com/mozilla-services/shavar-list-creation https://github.com/mozilla-services/shavar-list-creation/commit/db50ce50660de5bbec7b4bf927a8989e7f31fa15 Revert "Revert "bug 1258038: cleanups, DNT splitting, and content INCLUDE|SKIP|ONLY logic"" This reverts commit 97704040300725180d1f2285b2644e556a90f0d9. https://github.com/mozilla-services/shavar-list-creation/commit/db237a99a20aa72da684400b1a716551eb99a1d1 Merge pull request #39 from mozilla-services/start-list-split-1258038 Revert "Revert "bug 1258038: cleanups, DNT splitting, and content INC…
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.