1202082 - Tracking protection lists should filter out public suffixes

Status: RESOLVED WORKSFORME

(Cloud Services :: Server: Shavar, defect)

Description

[François Marier [:francois]]

Steps:

1. Set the following prefs:

  browser.trackingprotection.gethashURL = https://tracking.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2
  browser.trackingprotection.updateURL = https://tracking.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2
  urlclassifier.disallow_completions = test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,goog-downloadwhite-digest256,mozpub-track-digest256,mozpub-trackwhite-digest256,mozpub2-track-digest256
  urlclassifier.trackingTable = test-track-simple,mozpub2-track-digest256

2. Open https://www.youtube.com/ in a Private Browsing window.
3. Open the the devtools
4. Wait for a minute

Expected:

Google-owned resources are not blocked.

Actual:

The devtools show this error:

  The resource at "https://content.googleapis.com/static/proxy.html?jsh=..." was blocked because tracking protection is enabled.

Comment 1

[François Marier [:francois]]

The whitelist includes an entry that should apply to this load:

  [entity] Google >> (canonicalized) youtube.com/?resource=googleapis.com, hash 4f1399ca321d98196f3a1b4f5948decd8c03b0cf327165f540b1e6f60380583e

however that's not what the client looks for:

  30406400[7f8505b6f700]: Checking fragment www.youtube.com/?resource=content.googleapis.com, hash AAD2325E6BA48CA89E7F7B846EEFABE09F4654E63A9E7FDC0967FD23EB0883EF (5E32D2AA)
  30406400[7f8505b6f700]: Probe in test-trackwhite-simple: 5E32D2AA, found 0
  30406400[7f8505b6f700]: Probe in mozpub-trackwhite-digest256: 5E32D2AA, found 0
  30406400[7f8505b6f700]: Checking fragment www.youtube.com/, hash 6BDE51B796663C9B43793D80909B833D077D82DD8D7146FFAED9E874D158247D (B751DE6B)
  30406400[7f8505b6f700]: Probe in test-trackwhite-simple: B751DE6B, found 0
  30406400[7f8505b6f700]: Probe in mozpub-trackwhite-digest256: B751DE6B, found 0
  30406400[7f8505b6f700]: Checking fragment youtube.com/, hash 2EF399A8CF54DDC2BD575AF4E7FE25AF7F1E703BE0CED29219DCB32F86EF13C7 (A899F32E)
  30406400[7f8505b6f700]: Probe in test-trackwhite-simple: A899F32E, found 0
  30406400[7f8505b6f700]: Probe in mozpub-trackwhite-digest256: A899F32E, found 0
  30406400[7f8505b6f700]: Checking fragment youtube.com/?resource=content.googleapis.com, hash FDC21B7CD705C861D5472EB7919E4A5243ABB2BAE4926B61E36997DB41E20656 (7C1BC2FD)
  30406400[7f8505b6f700]: Probe in test-trackwhite-simple: 7C1BC2FD, found 0
  30406400[7f8505b6f700]: Probe in mozpub-trackwhite-digest256: 7C1BC2FD, found 0
  30406400[7f8505b6f700]: Checking fragment youtube.com/, hash 2EF399A8CF54DDC2BD575AF4E7FE25AF7F1E703BE0CED29219DCB32F86EF13C7 (A899F32E)
  30406400[7f8505b6f700]: Probe in test-trackwhite-simple: A899F32E, found 0
  30406400[7f8505b6f700]: Probe in mozpub-trackwhite-digest256: A899F32E, found 0
  30406400[7f8505b6f700]: Found 0 results.
  793257856[7f852e0538c0]: nsChannelClassifier[7f84fa8a3da0]: http://www.youtube.com/?resource=content.googleapis.com is not in the whitelist

and the reason is that googleapis.com is considered a top-level domain name
as per the publicsuffix.org list:

  https://publicsuffix.org/list/public_suffix_list.dat

Comment 2

[François Marier [:francois]]

One option would be to get Disconnect to be more explicit in their blacklist and whitelist about which googleapis.com subdomain they are blocking.

Comment 3

[michael.oneill]

Google can be tracking on these domains, which deliver JS libraries needed by countless sites. The PREF and NID cookies placed there often have the domain attribute set to ".googleapis.com" so they are sent with requests to any subdomains also. 

The Privacy Badger approach of having a list of domains where cookies are stripped from the request, but the request is otherwise not blocked, might work here. An independent check could be made of other tracking techniques (such as embedded UIDs in the delivered script), and if detected the domain could be removed from the "cookie block" list.

Comment 4

[François Marier [:francois]]

(In reply to michael.oneill from comment #3)
> Google can be tracking on these domains, which deliver JS libraries needed
> by countless sites. The PREF and NID cookies placed there often have the
> domain attribute set to ".googleapis.com" so they are sent with requests to
> any subdomains also. 

Yes and on Google properties, we should be letting Google track you because we've targeted TP at third-party tracking only in this iteration.

> The Privacy Badger approach of having a list of domains where cookies are
> stripped from the request, but the request is otherwise not blocked, might
> work here. An independent check could be made of other tracking techniques
> (such as embedded UIDs in the delivered script), and if detected the domain
> could be removed from the "cookie block" list.

I agree that this kind of approach would be better. Instead of blocking ajax.googleapis.com on non-Google sites, we could just strip its cookies. This is new functionality however, so it's not going to make it in Firefox 42 and we should use a new bug for that.

Comment 5

[François Marier [:francois]]

I filed a bug against the list creation script to ensure that we strip out any bare public suffixes from the blacklists: https://github.com/mozilla-services/shavar-list-creation/issues/26

We need to do this before we ship the "B" list.

Comment 6

[Luke Crouch [:groovecoder]]

I have a commit for this fix here: https://github.com/mozilla-services/shavar-list-creation/commit/20cd07edabdbdad9d4880bdd385a3286ccad2609

Need to integrate it with the other outstanding shavar changes for deployment.

Comment 8

[François Marier [:francois]]

That fix doesn't appear to be deployed, or it's not working.

For example, this should be fixed: https://bugzilla.mozilla.org/show_bug.cgi?id=1279706#c0

I also ran the list creation script manually and it does add public suffixes to the list:

[m] cloudfront.net >> cloudfront.net/
[canonicalized] cloudfront.net/
[hash] eee32dcf38bc38f709128a8ece92aafb2137335d5cf881bef2620b31406456c3
...
[m] googleapis.com >> googleapis.com/
[canonicalized] googleapis.com/
[hash] 2c8deec2103fd44639f03ee692201cebb0ddf05230637413b08fe0a6403d2c01

Comment 9

[Luke Crouch [:groovecoder]]

Bah! I never sent the PR for that commit; sorry to waste your time checking it. :(

PR sent: https://github.com/mozilla-services/shavar-list-creation/pull/41

I ran the creation script manually and cloudfront.net and googleapis.com are now removed from all the list (log) files.

Comment 10

[Luke Crouch [:groovecoder]]

:francois - can you check https://github.com/mozilla-services/shavar-list-creation/pull/41 again and merge?

Comment 11

[[github robot]]

Commits pushed to master at https://github.com/mozilla-services/shavar-list-creation

https://github.com/mozilla-services/shavar-list-creation/commit/1b172a49329972dc4008ecdfe5385fc664c33e72
fix #26 bug 1202082: skip publicsuffix domains

https://github.com/mozilla-services/shavar-list-creation/commit/299844abbb5965d092ddfdfa807ba990ba4ac129
Merge pull request #41 from mozilla-services/filter-public-suffixes-26-1202082

fix #26 bug 1202082: skip publicsuffix domains

Comment 12

[Luke Crouch [:groovecoder]]

This change has been merged, and the Jenkins job creating the lists has been fixed to create the new lists for shavar to serve.

I set my Tracking Protection list to strict and checked http://biorxiv.org/content/early/2016/06/10/058305 (mentioned in bug 1279706) - TP no longer blocks the cloudfront assets and the page renders with styles.

:francois: So I think this is RESOLVED:FIXED ?

Comment 13

[François Marier [:francois]]

If you can confirm that the issue in comment 0 is also fixed, then yes I think we can close this bug.

Thanks!

Comment 14

[Luke Crouch [:groovecoder]]

Are those still the right pref names? I don't have any existing values for them. And the new/updated lists are already deployed in prod, so I wouldn't need to change them to stage anyway, right?

If that's the case, I went to youtube.com in a PBM window using the strict blocking list and I got 0 security console messages about blocked content.

Comment 15

[François Marier [:francois]]

(In reply to Luke Crouch [:groovecoder] from comment #14)
> Are those still the right pref names? I don't have any existing values for
> them. And the new/updated lists are already deployed in prod, so I wouldn't
> need to change them to stage anyway, right?

Yeah, the list names have changed.

> If that's the case, I went to youtube.com in a PBM window using the strict
> blocking list and I got 0 security console messages about blocked content.

Sounds like it's fixed then.

Comment 16

[François Marier [:francois]]

According to https://bugzilla.mozilla.org/show_bug.cgi?id=1101005#c22, the public suffixes are no longer filtered out.

Luke, can you take a look to make sure this isn't a bug in the list creation script?

Comment 17

[Luke Crouch [:groovecoder]]

The script is still using & skipping PSL as before ...

:ckolos - is there any way to reach the list .log files created by the script running on jenkins? 

E.g., https://deploy.mozaws.net/view/Shavar%20Prod/job/ShavarListCreationProd/30030/ should have made these log files:

content-track-digest256.log
mozfull-track-digest256.log
mozfullstaging-track-digest256.log

Which should have lines like:

content-track-digest256.log:1020:[Public Suffix] googleapis.com; Skipping.
mozfull-track-digest256.log:1141:[Public Suffix] googleapis.com; Skipping.
mozfullstaging-track-digest256.log:1141:[Public Suffix] googleapis.com; Skipping.

Comment 18

[Chris Kolosiwsky [:ckolos] (ckolos has left the building)]

I can modify the jenkins job so that these log files are echo'd to the console, but other than that, the log files are only available between job runs. The jenkins task is config'd to clean up the workspace prior to the "next" run.

Comment 19

[Chris Kolosiwsky [:ckolos] (ckolos has left the building)]

I've checked the output of the jenkins task when setup to echo the logs output during list creation. I see 6 matches for "Public Suffix" in the output:

content-track-digest256.log
[Public Suffix] cloudfront.net; Skipping.
[Public Suffix] googleapis.com; Skipping.

mozfullstaging-track-digest256.log
[Public Suffix] cloudfront.net; Skipping.
[Public Suffix] googleapis.com; Skipping.

mozfull-track-digest256.log
[Public Suffix] cloudfront.net; Skipping.
[Public Suffix] googleapis.com; Skipping.

Other than that, there are no other matches.

Comment 20

[Luke Crouch [:groovecoder]]

I'm clearing out my old bugs. I can't reproduce the original WebCompat issue or bug comment. Marking RESOLVED:WORKSFORME.