> Looking at the [code](https://github.com/mozilla-services/shavar-list-creation/blob/8176872ed5b49535a31eb8df52f4a5aba09119b6/lists2safebrowsing.py#L620) generating these scripts, it [seems that](https://github.com/mozilla-services/shavar-list-creation/blob/8176872ed5b49535a31eb8df52f4a5aba09119b6/lists2safebrowsing.py#L37) there is no whitelist table support for social tracking protection support at all in the list generation scripts. So not only the entity list for ETP/TP doesn't work automatically for STP but also as far as I can tell there is no entity list for STP whatsoever. If there were then https://www.facebook.com, https://facebook.com (and also https://www.facebook.de) wouldn't be classified in the first place, so none of what I explained above would have happened in the first place. > > BTW I used [this pernosco recording](https://pernos.co/debug/Ot9Y6IBbhA8NmvOITAbdKA/index.html#f{m[AWIK,A4x2_,t[AQ,Zcg_,f{e[AWIK,A4xs_,s{af4ARhnAA,bAUs,oAi9zUg,uAitDEQ___) to debug this, feel free to use it if you'd like to inspect things further. We are now using tracking protection whitelist table, mozstd-trackwhite-digest256, as social tracking protection's whitelist table (and all the other tracking related features are also implemented this way), if this is not right, or not right for all the features, I'll file a bug. So because of "mozstd-trackwhite-digest256" is being used, facebook.de, fbcdn.net ... etc is not classified as social trackers when top-level is facebook.com. But we still see "www.facebook.com", "facebook.com" are classified as trackers in this bug, this is because: 1. Our whitelist generation script doesn't generate whitelist URL with the same domain name, for example, we don't have "facebook.com/?resource=facebook.com" in the whitelist table. I guess we expected this scenario should be filtered out by our third-party check[2]. 2. we don't call third-party check[2] in social tracking annotation(and also for all the other annotation features) as we did in protection features[3]. This cause facebook.com in facebook.com will still be classified by URL Classifier and because it will not be found in whitelist table because of #1, "facebook.com" is marked as social tracker eventually. And My comment 4 was wrong, after adding [2] in FeatureSocailTrackingAnnotation, this did solve this issue. [1] https://searchfox.org/mozilla-central/rev/325c1a707819602feff736f129cb36055ba6d94f/modules/libpref/init/all.js#5069 [2] https://searchfox.org/mozilla-central/rev/325c1a707819602feff736f129cb36055ba6d94f/netwerk/url-classifier/UrlClassifierFeatureTrackingProtection.cpp#95 [3] https://searchfox.org/mozilla-central/rev/325c1a707819602feff736f129cb36055ba6d94f/netwerk/url-classifier/UrlClassifierFeatureTrackingProtection.cpp
Bug 1575811 Comment 6 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
> Looking at the [code](https://github.com/mozilla-services/shavar-list-creation/blob/8176872ed5b49535a31eb8df52f4a5aba09119b6/lists2safebrowsing.py#L620) generating these scripts, it [seems that](https://github.com/mozilla-services/shavar-list-creation/blob/8176872ed5b49535a31eb8df52f4a5aba09119b6/lists2safebrowsing.py#L37) there is no whitelist table support for social tracking protection support at all in the list generation scripts. So not only the entity list for ETP/TP doesn't work automatically for STP but also as far as I can tell there is no entity list for STP whatsoever. If there were then https://www.facebook.com, https://facebook.com (and also https://www.facebook.de) wouldn't be classified in the first place, so none of what I explained above would have happened in the first place. > > BTW I used [this pernosco recording](https://pernos.co/debug/Ot9Y6IBbhA8NmvOITAbdKA/index.html#f{m[AWIK,A4x2_,t[AQ,Zcg_,f{e[AWIK,A4xs_,s{af4ARhnAA,bAUs,oAi9zUg,uAitDEQ___) to debug this, feel free to use it if you'd like to inspect things further. We are now using tracking protection whitelist table, mozstd-trackwhite-digest256, as social tracking protection's whitelist table[1] (and all the other tracking related features are also implemented this way), if this is not right, or not right for all the features, I'll file a bug. So because of "mozstd-trackwhite-digest256" is being used, facebook.de, fbcdn.net ... etc is not classified as social trackers when top-level is facebook.com. But we still see "www.facebook.com", "facebook.com" are classified as trackers in this bug, this is because: 1. Our whitelist generation script doesn't generate whitelist URL with the same domain name, for example, we don't have "facebook.com/?resource=facebook.com" in the whitelist table. I guess we expected this scenario should be filtered out by our third-party check[2]. 2. we don't call third-party check[2] in social tracking annotation(and also for all the other annotation features) as we did in protection features[3]. This cause facebook.com in facebook.com will still be classified by URL Classifier and because it will not be found in whitelist table because of #1, "facebook.com" is marked as social tracker eventually. And My comment 4 was wrong, after adding [2] in FeatureSocailTrackingAnnotation, this did solve this issue. [1] https://searchfox.org/mozilla-central/rev/325c1a707819602feff736f129cb36055ba6d94f/modules/libpref/init/all.js#5069 [2] https://searchfox.org/mozilla-central/rev/325c1a707819602feff736f129cb36055ba6d94f/netwerk/url-classifier/UrlClassifierFeatureTrackingProtection.cpp#95 [3] https://searchfox.org/mozilla-central/rev/325c1a707819602feff736f129cb36055ba6d94f/netwerk/url-classifier/UrlClassifierFeatureTrackingProtection.cpp
> Looking at the [code](https://github.com/mozilla-services/shavar-list-creation/blob/8176872ed5b49535a31eb8df52f4a5aba09119b6/lists2safebrowsing.py#L620) generating these scripts, it [seems that](https://github.com/mozilla-services/shavar-list-creation/blob/8176872ed5b49535a31eb8df52f4a5aba09119b6/lists2safebrowsing.py#L37) there is no whitelist table support for social tracking protection support at all in the list generation scripts. So not only the entity list for ETP/TP doesn't work automatically for STP but also as far as I can tell there is no entity list for STP whatsoever. If there were then https://www.facebook.com, https://facebook.com (and also https://www.facebook.de) wouldn't be classified in the first place, so none of what I explained above would have happened in the first place. > > BTW I used [this pernosco recording](https://pernos.co/debug/Ot9Y6IBbhA8NmvOITAbdKA/index.html#f{m[AWIK,A4x2_,t[AQ,Zcg_,f{e[AWIK,A4xs_,s{af4ARhnAA,bAUs,oAi9zUg,uAitDEQ___) to debug this, feel free to use it if you'd like to inspect things further. We are now using tracking protection whitelist table, mozstd-trackwhite-digest256, as social tracking protection's whitelist table[1] (and all the other tracking related features are also implemented this way), if this is not right, or not right for all the features, I'll file a bug. So because of "mozstd-trackwhite-digest256" is being used, facebook.de, fbcdn.net ... etc are not classified as social trackers when top-level is facebook.com. But we still see "www.facebook.com", "facebook.com" are classified as trackers in this bug, this is because: 1. Our whitelist generation script doesn't generate whitelist URL with the same domain name, for example, we don't have "facebook.com/?resource=facebook.com" in the whitelist table. I guess we expected this scenario should be filtered out by our third-party check[2]. 2. we don't call third-party check[2] in social tracking annotation(and also for all the other annotation features) as we did in protection features[3]. This cause facebook.com in facebook.com will still be classified by URL Classifier and because it will not be found in whitelist table because of #1, "facebook.com" is marked as social tracker eventually. And My comment 4 was wrong, after adding [2] in FeatureSocailTrackingAnnotation, this did solve this issue. [1] https://searchfox.org/mozilla-central/rev/325c1a707819602feff736f129cb36055ba6d94f/modules/libpref/init/all.js#5069 [2] https://searchfox.org/mozilla-central/rev/325c1a707819602feff736f129cb36055ba6d94f/netwerk/url-classifier/UrlClassifierFeatureTrackingProtection.cpp#95 [3] https://searchfox.org/mozilla-central/rev/325c1a707819602feff736f129cb36055ba6d94f/netwerk/url-classifier/UrlClassifierFeatureTrackingProtection.cpp
> Looking at the [code](https://github.com/mozilla-services/shavar-list-creation/blob/8176872ed5b49535a31eb8df52f4a5aba09119b6/lists2safebrowsing.py#L620) generating these scripts, it [seems that](https://github.com/mozilla-services/shavar-list-creation/blob/8176872ed5b49535a31eb8df52f4a5aba09119b6/lists2safebrowsing.py#L37) there is no whitelist table support for social tracking protection support at all in the list generation scripts. So not only the entity list for ETP/TP doesn't work automatically for STP but also as far as I can tell there is no entity list for STP whatsoever. If there were then https://www.facebook.com, https://facebook.com (and also https://www.facebook.de) wouldn't be classified in the first place, so none of what I explained above would have happened in the first place. > > BTW I used [this pernosco recording](https://pernos.co/debug/Ot9Y6IBbhA8NmvOITAbdKA/index.html#f{m[AWIK,A4x2_,t[AQ,Zcg_,f{e[AWIK,A4xs_,s{af4ARhnAA,bAUs,oAi9zUg,uAitDEQ___) to debug this, feel free to use it if you'd like to inspect things further. We are now using tracking protection whitelist table, mozstd-trackwhite-digest256, as social tracking protection's whitelist table[1] (and all the other tracking related features are also implemented this way), if this is not right, or not right for all the features, I'll file a bug. So because of "mozstd-trackwhite-digest256" is being used, facebook.de, fbcdn.net ... etc are not classified as social trackers when top-level is facebook.com. But we still see "www.facebook.com", "facebook.com" are classified as trackers in this bug, this is because: 1. Our whitelist generation script doesn't generate whitelist URL with the same domain name, for example, we don't have "facebook.com/?resource=facebook.com" in the whitelist table. I guess we expected this scenario should be filtered out by our third-party check[2]. 2. we don't call third-party check[2] in social tracking annotation(and also for all the other annotation features) as we do in protection features[3]. This causes facebook.com in facebook.com will still be classified by URL Classifier and because it will not be found in whitelist table because of #1, "facebook.com" is marked as social tracker eventually. And My comment 4 was wrong, after adding [2] in FeatureSocailTrackingAnnotation, this did solve this issue. [1] https://searchfox.org/mozilla-central/rev/325c1a707819602feff736f129cb36055ba6d94f/modules/libpref/init/all.js#5069 [2] https://searchfox.org/mozilla-central/rev/325c1a707819602feff736f129cb36055ba6d94f/netwerk/url-classifier/UrlClassifierFeatureTrackingProtection.cpp#95 [3] https://searchfox.org/mozilla-central/rev/325c1a707819602feff736f129cb36055ba6d94f/netwerk/url-classifier/UrlClassifierFeatureTrackingProtection.cpp
> Looking at the [code](https://github.com/mozilla-services/shavar-list-creation/blob/8176872ed5b49535a31eb8df52f4a5aba09119b6/lists2safebrowsing.py#L620) generating these scripts, it [seems that](https://github.com/mozilla-services/shavar-list-creation/blob/8176872ed5b49535a31eb8df52f4a5aba09119b6/lists2safebrowsing.py#L37) there is no whitelist table support for social tracking protection support at all in the list generation scripts. So not only the entity list for ETP/TP doesn't work automatically for STP but also as far as I can tell there is no entity list for STP whatsoever. If there were then https://www.facebook.com, https://facebook.com (and also https://www.facebook.de) wouldn't be classified in the first place, so none of what I explained above would have happened in the first place. > > BTW I used [this pernosco recording](https://pernos.co/debug/Ot9Y6IBbhA8NmvOITAbdKA/index.html#f{m[AWIK,A4x2_,t[AQ,Zcg_,f{e[AWIK,A4xs_,s{af4ARhnAA,bAUs,oAi9zUg,uAitDEQ___) to debug this, feel free to use it if you'd like to inspect things further. We are now using tracking protection whitelist table, mozstd-trackwhite-digest256, as social tracking protection's whitelist table[1] (and all the other tracking related features are also implemented this way), if this is not right, or not right for all the features, I'll file a bug. So because of "mozstd-trackwhite-digest256" is being used, facebook.de, fbcdn.net ... etc are not classified as social trackers when top-level is facebook.com. But we still see "www.facebook.com", "facebook.com" are classified as trackers in this bug, this is because: 1. Our whitelist generation script doesn't generate whitelist URL with the same domain name, for example, we don't have "facebook.com/?resource=facebook.com" in the whitelist table. I guess we expected this scenario should be filtered out by our third-party check[2]. 2. we don't call third-party check[2] in social tracking annotation(and also for all the other annotation features) as we do in protection features[3]. This causes facebook.com in facebook.com is still classified by URL Classifier and because it will not be found in whitelist table because of #1, "facebook.com" is marked as social tracker eventually. And My comment 4 was wrong, after adding [2] in FeatureSocailTrackingAnnotation, this did solve this issue. [1] https://searchfox.org/mozilla-central/rev/325c1a707819602feff736f129cb36055ba6d94f/modules/libpref/init/all.js#5069 [2] https://searchfox.org/mozilla-central/rev/325c1a707819602feff736f129cb36055ba6d94f/netwerk/url-classifier/UrlClassifierFeatureTrackingProtection.cpp#95 [3] https://searchfox.org/mozilla-central/rev/325c1a707819602feff736f129cb36055ba6d94f/netwerk/url-classifier/UrlClassifierFeatureTrackingProtection.cpp
> Looking at the [code](https://github.com/mozilla-services/shavar-list-creation/blob/8176872ed5b49535a31eb8df52f4a5aba09119b6/lists2safebrowsing.py#L620) generating these scripts, it [seems that](https://github.com/mozilla-services/shavar-list-creation/blob/8176872ed5b49535a31eb8df52f4a5aba09119b6/lists2safebrowsing.py#L37) there is no whitelist table support for social tracking protection support at all in the list generation scripts. So not only the entity list for ETP/TP doesn't work automatically for STP but also as far as I can tell there is no entity list for STP whatsoever. If there were then https://www.facebook.com, https://facebook.com (and also https://www.facebook.de) wouldn't be classified in the first place, so none of what I explained above would have happened in the first place. > > BTW I used [this pernosco recording](https://pernos.co/debug/Ot9Y6IBbhA8NmvOITAbdKA/index.html#f{m[AWIK,A4x2_,t[AQ,Zcg_,f{e[AWIK,A4xs_,s{af4ARhnAA,bAUs,oAi9zUg,uAitDEQ___) to debug this, feel free to use it if you'd like to inspect things further. We are now using tracking protection whitelist table, mozstd-trackwhite-digest256, as social tracking protection's whitelist table[1] (and all the other tracking related features are also implemented this way), if this is not right, or not right for all the features, I'll file a bug. So because of "mozstd-trackwhite-digest256" is being used, facebook.de, fbcdn.net ... etc are not classified as social trackers when top-level is facebook.com. But we still see "www.facebook.com", "facebook.com" are classified as trackers in this bug, this is because: 1. Our whitelist generation script doesn't generate whitelist URL with the same domain name, for example, we don't have "facebook.com/?resource=facebook.com" in the whitelist table. I guess we expected this scenario should be filtered out by our third-party check[2]. 2. we don't call third-party check[2] in social tracking annotation(and also for all the other annotation features) as we do in protection features[3]. This causes facebook.com in facebook.com is still classified by URL Classifier and because it will not be found in whitelist table because of #1, "facebook.com" is marked as social tracker eventually. And My comment 4 was wrong, after adding [2] in FeatureSocailTrackingAnnotation, this did solve this issue. So my question is, should we remove whitelist table from social tracking and let ContentBlocking do all the work? or we shouldn't report first-party trackers in URL Classifier in the first place. [1] https://searchfox.org/mozilla-central/rev/325c1a707819602feff736f129cb36055ba6d94f/modules/libpref/init/all.js#5069 [2] https://searchfox.org/mozilla-central/rev/325c1a707819602feff736f129cb36055ba6d94f/netwerk/url-classifier/UrlClassifierFeatureTrackingProtection.cpp#95 [3] https://searchfox.org/mozilla-central/rev/325c1a707819602feff736f129cb36055ba6d94f/netwerk/url-classifier/UrlClassifierFeatureTrackingProtection.cpp
> Looking at the [code](https://github.com/mozilla-services/shavar-list-creation/blob/8176872ed5b49535a31eb8df52f4a5aba09119b6/lists2safebrowsing.py#L620) generating these scripts, it [seems that](https://github.com/mozilla-services/shavar-list-creation/blob/8176872ed5b49535a31eb8df52f4a5aba09119b6/lists2safebrowsing.py#L37) there is no whitelist table support for social tracking protection support at all in the list generation scripts. So not only the entity list for ETP/TP doesn't work automatically for STP but also as far as I can tell there is no entity list for STP whatsoever. If there were then https://www.facebook.com, https://facebook.com (and also https://www.facebook.de) wouldn't be classified in the first place, so none of what I explained above would have happened in the first place. > > BTW I used [this pernosco recording](https://pernos.co/debug/Ot9Y6IBbhA8NmvOITAbdKA/index.html#f{m[AWIK,A4x2_,t[AQ,Zcg_,f{e[AWIK,A4xs_,s{af4ARhnAA,bAUs,oAi9zUg,uAitDEQ___) to debug this, feel free to use it if you'd like to inspect things further. We are now using tracking protection whitelist table, mozstd-trackwhite-digest256, as social tracking protection's whitelist table[1] (and all the other tracking related features are also implemented this way), if this is not right, or not right for all the features, I'll file a bug. So because of "mozstd-trackwhite-digest256" is being used, facebook.de, fbcdn.net ... etc are not classified as social trackers when top-level is facebook.com. But we still see "www.facebook.com", "facebook.com" are classified as trackers in this bug, this is because: 1. Our whitelist generation script doesn't generate whitelist URL with the same origin, for example, we don't have "facebook.com/?resource=facebook.com" in the whitelist table. I guess we expected this scenario should be filtered out by our third-party check[4]. 2. we don't call third-party check[2] in social tracking annotation(and also for all the other annotation features) as we do in protection features[3]. This causes facebook.com in facebook.com is still classified by URL Classifier and because it will not be found in whitelist table because of #1, "facebook.com" is marked as social tracker eventually. And My comment 4 was wrong, after adding [2] in FeatureSocailTrackingAnnotation, this did solve this issue. So my question is, should we remove whitelist table from social tracking and let ContentBlocking do all the work? or we shouldn't report first-party trackers in URL Classifier in the first place. [1] https://searchfox.org/mozilla-central/rev/325c1a707819602feff736f129cb36055ba6d94f/modules/libpref/init/all.js#5069 [2] https://searchfox.org/mozilla-central/rev/325c1a707819602feff736f129cb36055ba6d94f/netwerk/url-classifier/UrlClassifierFeatureTrackingProtection.cpp#95 [3] https://searchfox.org/mozilla-central/rev/325c1a707819602feff736f129cb36055ba6d94f/netwerk/url-classifier/UrlClassifierFeatureTrackingProtection.cpp [4] https://bugzilla.mozilla.org/show_bug.cgi?id=1189087#c1