1160596 - Create a general blacklist for negative adjacency

Status: RESOLVED FIXED

(Content Services Graveyard :: Tiles, defect)

Description

[Ed Lee :Mardak]

We need a blacklist to hardcode in bug 1159884. We have various sources of blacklist data that we want to combine. We also want to filter probably based on site ranking/popularity to reduce the total number of entries to lessen impact on disk/memory usage of Firefox.

Comment 1

[Matthew Ruttley [:mruttley]]

Here's the most current blacklist: https://github.com/matthewruttley/contentfilter/blob/master/sites.json

I'm continually updating it. In terms of the bloom filter, we have to decide at a legal level what error rate is acceptable (perhaps none?). Maksik mentioned that he will investigate if there is any existing BF implementation in FF. We also have to decide which hashing function is best. My suggestion is that we use a popular function like murmur: https://gist.github.com/raycmorgan/588423

Comment 2

[maxim zhilyaev]

mxr points to C++ implementation of bloom filter
http://mxr.mozilla.org/mozilla-central/source/mfbt/BloomFilter.h

I am not sure how to surface it to the jsm module

Comment 3

[maxim zhilyaev]

When checking false positive rate, please run BloomFilter on 1m of alexa with adult sites removed and see how many actual false positives we get.

Comment 4

[Ed Lee :Mardak]

There's a bloom filter implemented in bug 1138022 https://bugzilla.mozilla.org/attachment.cgi?id=8571008&action=diff

And I don't think it matters too much which exact bloom filter function is used. I expect it to be more of a calculation that can be confirmed with an actual implementation. E.g., given alexa top 1m domains, how what's the collision/false positive rate for a bloom filter with X bits.

Sure, different functions will generate different hashes and which exact sites collide will be different, but in general, I would think it's more of the size of blacklist requiring how many bits and how many additional bits to lessen false positives.

Comment 5

[Ed Lee :Mardak]

There's this wikipedia article:

http://en.wikipedia.org/wiki/Bloom_filter#Probability_of_false_positives

Comment 6

[Ed Lee :Mardak]

This article has a rule of thumb:

http://corte.si/%2Fposts/code/bloom-filter-rules-of-thumb/index.html
1 - One byte per item in the input set gives about a 2% false positive rate.

If we had 1024 blacklisted items, we would get 2% false positive with 1KB bloom filter. If we want 100k items, we would need roughly 100KB bloom filter with 2% false positive rate. Increasing to 10% false positive is roughly half the size.

Comment 7

[Matthew Ruttley [:mruttley]]

Mardak: Are we doing domain, subdomain or path level matching? 

This is a great list we can integrate: http://dsi.ut-capitole.fr/blacklists/index_en.php but requires a bit more than just domain to get at various google groups and live.com specifics.

Comment 8

[Ed Lee :Mardak]

I believe we'll start with just the domain matching for now.

Comment 9

[Matthew Ruttley [:mruttley]]

I've integrated some more data from a comscore source and we now have over 2900 domains: https://github.com/matthewruttley/contentfilter/blob/master/sites.json

The counts per category are as follows:
 - drugs: 19
 - gambling: 154
 - adult: 2743
 - alcohol: 50

All domains are in the latest daily Alexa top 1m sites.

Comment 10

[Ed Lee :Mardak]

mruttley, can you calculate the maxP of those blacklisted sites as one adgroup?

Comment 11

[Matthew Ruttley [:mruttley]]

The MaxP of blacklist groups is currently:

Drugs  0.114959
Gambling  0.094614
Adult  0.020494
Alcohol  0.074813

All Together  0.019026

Comment 12

[maxim zhilyaev]

Matthew, could you generate json file of the form:

{"blacklist": [ base64(bad_site_1), base64(bad_site_2), .... , base64("example.com") ]}

Note that I included "example.com" in the list, so we can browser test negative adjacency without actually going to pron sites.  I am using btoa() call to generate base64 in the client, so it will be advisable to use that for json file generation.

When the file is generated, do, please, attach to the bug

Comment 13

[Matthew Ruttley [:mruttley]]

maksik: Here is the file: https://github.com/matthewruttley/contentfilter/blob/master/sitesb64.json

Comment 14

[Matthew Ruttley [:mruttley]]

Attachment: sitesb64.json - base 64 encoding of the blacklist

Updated with b64 encoding of example.com as requested by maksik

Comment 15

[Matthew N. [:MattN]]

Why are we reinventing the wheel and not using the Safe Browsing data format and server? It supports more than just domain matches and is already usable by JS in Firefox. I don't understand how one second it's argued that we need this ASAP and then on the other hand extra work is made for ourselves.

Comment 16

[Ed Lee :Mardak]

Do you know who can help with using the existing safe browsing data format and server? If it is indeed simple and low risk to uplift to 39, then we could go that route.

Comment 17

[Matthew N. [:MattN]]

:gcp or :mmc (not sure of availability) knows the client code in toolkit and looking at the server code[1] that I linked to before, it would seem like rtilder does. It looks like the client changes would be low risk for uplift IMO. I didn't realize you were trying to rush this into Beta which doesn't seem like a good idea for any of these solutions as there could be performance issues to be handled (especially with a custom solution).

[1] https://github.com/mozilla-services/shavar

Comment 18

[Matthew Ruttley [:mruttley]]

Attachment: md5.json

md5 encoding of the blacklist

Comment 19

[Matthew Ruttley [:mruttley]]

Attachment: md5 and b64 version of the blacklist

Comment 20

[maxim zhilyaev]

We need to change format of the json file. "blacklist" needs to be replaced with :"domains".  It should be

{
  "domains": [
    .....
  ]
}

could you please make a correction

Comment 21

[Matthew Ruttley [:mruttley]]

Attachment: Updated md5+b64 version of the blacklist

Also available here: https://github.com/matthewruttley/contentfilter/blob/master/md5_b64.json

Comment 22

[Ed Lee :Mardak]

What are the changes?

939a940,941
>         "mNlYGAOPc6KIMW8ITyBzIg==", 
>         "bK045TkBlz+/3+6n6Qwvrg==", 
1065d1066
<         "t1O9jSNjg4DTIv/Za4NbtA==", 
1163d1163
<         "+k5lDb+QdNc9iZ01hL5yBg==", 
1244d1243
<         "2qK2ZEY9LgdKSTaLf6VnLA==", 
1369a1369
>         "hIJA+1QGuKEj+3ijniyBSQ==", 
1528d1527
<         "E2lvMXqHdTw0x+KCKVnblg==", 
1973a1973
>         "8DtgIyYiNFqDc5qVrpFUng==", 
2070d2069
<         "RzX2OfSFEd//LhZwRwzBVw==", 
2211a2211
>         "O7JiE0bbp583G6ZWRGBcfw==", 
2925a2926
>         "gYgCu/qUpXWryubJauuPNw==", 
3010a3012
>         "+YVxSyViJfrme/ENe1zA7A==", 
3089a3092
>         "VZX1FnyC8NS2k3W+RGQm4g==",

Comment 23

[Ed Lee :Mardak]

Ah I see the commit:

https://github.com/matthewruttley/contentfilter/commit/048e1410da2408d4e79ef0ea8001691606fe9af4

The commit message doesn't quite explain the changes in the list though. Any reason for those changes?

Comment 24

[Matthew Ruttley [:mruttley]]

The changes reflect updates in how contentfilter is created. Contentfilter is generated by sites only found in the Alexa top 1m sites, which change every day. Thus each time I regenerate the list, a few sites change.

I used backticks in the commit message which seem to have eliminated most/all of it :/ TIL.