1519197 - Add 1st Party Support for AdBlock Plus-style blocklists in Gecko/GeckoView

Status: NEW

(Toolkit :: Safe Browsing, enhancement, P5)

Description

[Colin Lee]

One of the Fenix requirements revolves around adding more complete support for ad blocking. Our managers have asked that we add first-party support rather than just leaning entirely on a WebExtension.

I am proposing this feature as a Google Summer of Code project due to the relatively low number of dependencies involved and the fact we don't have any resources allocated for it yet.

Comment 1

[Andrew Overholt [:overholt]]

I'm moving this to which I think is a better bugzilla component.

Comment 2

[:Gijs (he/him)]

(In reply to Andrew Overholt [:overholt] from comment #1)

I'm moving this to which I think is a better bugzilla component.

Sadly, different kind of blocking. Looks like this bugzilla component needs renaming but I think this is the right place. Johann?

Comment 3

[Johann Hofmann [:johannh]]

(In reply to Colin Lee from comment #0)

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0

Steps to reproduce:

One of the Fenix requirements revolves around adding more complete support for ad blocking.

What does "more complete support" mean exactly?

Our managers have asked that we add first-party support rather than integrating and leaning on existing third-party WebExtensions like uBlock Origin for this functionality. This means that someone will need to write code to parse AdBlock Plus-style lists and handle all of the features it supports, such as regex pattern matching, exceptions, and blocking only scripts, images, etc. from a given URL.

What's the reasoning for not leaning on existing APIs for this? Can't you write the equivalent of a system-addon for this purpose, if you want to support Ad-Blocking natively?

We have the URL-Classifier which powers things like Tracking Protection, so that's very close to first-party ad-blocking support.

Baku, how much do you think it would take to evolve URL-Classifier to support AdBlock Plus-style lists?

As it is presented here, my personal opinion is that this is a waste of time given that we already have the WebExtension APIs.

I am proposing this feature as a Google Summer of Code project due to the relatively low number of dependencies involved and the fact we don't have any resources allocated for it yet.

I think you're underestimating the complexity of this task. Of course it depends on how it's implemented but if we are talking about an effort similar to our existing Tracking Protection code then that's not a GSoC project. Given that you would have it as a GSoC project I take it that this is not a high priority for your team?

Comment 4

[Andrea Marchesini [:baku]]

Baku, how much do you think it would take to evolve URL-Classifier to support AdBlock Plus-style lists?

This project is definitely complex. Let me say why:

1. Shavar server supports google safebrowsing v2 and v4 list formats. We need to see if we can convert the adblock-plus lists in one of those formats.

2. on the browser side, we support just a little of the features required by adblock plus. For instance, we don't support CSS manipulation and full regular expressions.

To be more precise, reading https://adblockplus.org/filters, I would say:

• basic list: I don't know how much we support of wildcards or what is in section "Using regular expressions". Dimi knows definitely more than me and I let him to answer.
• exception rules: this could be probably converted to whitelist vs blacklist.
• Matching at beginning/end of an address: this should be supported
• Marking separator characters: this as well, partially...
• Special comments: nothing like that is supported.
• Specifying filter options: nothing like that is supported.
• CSS manipulation: not supported

Comment 5

[Dimi Lee [:dimi]]

(In reply to Andrea Marchesini [:baku] from comment #4)

• basic list: I don't know how much we support of wildcards or what is in section "Using regular expressions". Dimi knows definitely more than me and I let him to answer.

SafeBrowsing list doesn't support wildcards, regular expression.
The adblock style list requires that we know the rule to see if a URL matches the rule. While Safebrowsing list doesn't know the real data, it contains either 4-prefix hash or 32-bytes full hash. We fragment the suspect URL and hash each of the fragment to see if we can find an "exact" match to any hash in the list.

The only thing I think we can reuse from the SafeBrowsing is the "mechanism" about the timing we trigger an update, the backoff algorithm ...etc . As for the data format in the list and how to use them, I would say, it should be very different.

In addition to that, one goal in the SafeBrowsing protocol is to support partial update and also to minimize the size of the list. If the adblock style list is not a huge one, unless we have a way to make use of the SafeBrowsing list format easily, I don't think SafeBrowsing protocol is required.

Comment 6

[Colin Lee]

(In reply to Johann Hofmann [:johannh] from comment #3)

What does "more complete support" mean exactly?

There are vast differences between what the community supports for ad blocking and what the SafeBrowsing formats support. For an incomplete list of a few issues I've noticed, we lack:

• Support for ad URLs with Punycode characters
• Support for regular expressions
• Support for CSS hiding
• Support for per domain or per resource type exceptions

Each one of these features would allow us to block more ads than we can presently.

What's the reasoning for not leaning on existing APIs for this? Can't you write the equivalent of a system-addon for this purpose, if you want to support Ad-Blocking natively?

We have the URL-Classifier which powers things like Tracking Protection, so that's very close to first-party ad-blocking support.

Baku, how much do you think it would take to evolve URL-Classifier to support AdBlock Plus-style lists?

As it is presented here, my personal opinion is that this is a waste of time given that we already have the WebExtension APIs.

PMs on the project are interested in a more first-party solution. This could possibly be built on top of a WebExtension, but it seemed they did not simply want to say treat an existing WebExtension as a system extension.

I think you're underestimating the complexity of this task. Of course it depends on how it's implemented but if we are talking about an effort similar to our existing Tracking Protection code then that's not a GSoC project. Given that you would have it as a GSoC project I take it that this is not a high priority for your team?

I understand a rough idea of the complexity, but implementing partial support could be done within a summer, especially if building off of existing open source solutions. The AdBlock Plus format has many features and is quite complicated, but there are various open source implementations already to learn from.

Comment 7

[Pradeep Gangwar [:pradeepgangwar]]

Hi Colin, I am Pradeep Gangwar. I was a GSoC'18 student under FOSSASIA. I actively contribute to open source projects, of which few are part of Mozilla. You can take a look at my GitHub profile. I am actively looking for projects for this year's GSoC and this project seems interesting to me. As discussed in above comments, I also feel that completing it within a summer would be a tough task but we can do a part of it and take it to a level where it will be easy to build it up further.

Do you want me to look into it and do a bit of research on it?

Comment 8

[Colin Lee]

We're not entirely sure on the best direction yet. We want to offer users ad blocking without starving content creators. I believe something like this is possible if we default to keeping the feature off, but make it easy to enable and remember the setting per-domain or per-session to meet user needs.

Also, it's fair that producing something from scratch is not reasonable for a summer project. Originally, I was thinking any progress on a subset of the ABP rules would be a success. Now, I'm thinking we want some MVP providing value sooner than end of summer. Perhaps we'll build from open source WebExtensions.

I'll update this bug as I talk to project managers and develop a decision.

Comment 9

[Asif Youssuff]

In terms of user acceptance, it seems like Brave's shield is well liked by users, so that might be a template to look at vs. something more invisible like uBlock Origin (hidden behind the hamburger menu in Fennec): https://techcrunch.com/2016/10/31/brave-the-browser-with-built-in-ad-blocking-tries-again-on-android/

There will be inevitable comparisons to Brave's browser, as it seems to be the most popular Chromium based browser that aligns with ideals in the privacy/ad blocking crowd. Firefox, I think is still the most popular option, but if uBlock Origin is not going to be available (without WebExtensions support), the Brave model seems worth studying (at least the core needs to allow browsers like Fenix to hook into).

Just my 2 cents.

Comment 10

[Laurențiu Nicola]

Please reconsider this. The AdBlock Plus filters are not expressive enough to support the functionality of uBlock Origin.

With Google trying to limit the usefulness of ad blockers, Mozilla should capitalize on that to make users switch. By offering the same capabilities as Chrome, you're effectively throwing away one of the greatest advantages that Firefox has.

Comment 11

[Marco Castelluccio [:marco]]

(In reply to Laurentiu Nicola from comment #10)

Please reconsider this. The AdBlock Plus filters are not expressive enough to support the functionality of uBlock Origin.

With Google trying to limit the usefulness of ad blockers, Mozilla should capitalize on that to make users switch. By offering the same capabilities as Chrome, you're effectively throwing away one of the greatest advantages that Firefox has.

Don't worry, this bug is unrelated and it is not about limiting the possibilities of WebExtensions.

Comment 12

[Laurențiu Nicola]

(In reply to Marco Castelluccio [:marco] from comment #11)

Don't worry, this bug is unrelated and it is not about limiting the possibilities of WebExtensions.

I disagree. This bug is about putting effort into implementing from scratch something that's much worse than uBlock Origin. Even disregarding that effort, what alternative will the users have? Are there any plans to support WebExtensions in GeckoView/Fenix?

Comment 13

[Colin Lee]

Let's please lose the conspiracy theories. Many comments are flat out wrong. There was one discussion where our team wanted ad blocking support and this bug was created to consider whether to obtain resources for some kind of solution. This has nothing whatsoever to do with anything Google is doing. We haven't made any decisions about the nature of a Fenix solution yet.

Comment 14

[Kevin Ar18]

Is there still an interest in making this happen?
If so, would you be willing to expand this feature to include the tools for making custom filters?

Let me explain...
I use adblock plus. For most people, they just subscribe to a list and never touch the settings again.

However, I create custom filters. In fact, I often add multiple items to a list ever single day. However, the ability to create custom filters in adblock plus was completely lost when Firefox switched to webextensions. (To be clear, you can still create custom filters in adblock plus, but all of the tools that made it easy to do are gone now. It's like Firefox with all of the dev tools removed; it will still work for regular users, but it is useless for developers.)

Personally, I'm not sure what I will have to do without this feature... might have to go with waterfox.

Adblock Plus
As far as I know, adblock plus will never add back the tools for making filters. I told them about the missing features long ago. The original creator of the addon-on is long gone. The current developer, Wladimir Palant has indicated that he doesn't work on the add-on much anymore. Furthermore, it seems that the current webextensions Adblock Plus might just be a modified copy of uBlock origin since they are exactly alike in key areas.

uBlock Origin
ublock has managed to add a few tools for making filters; they are actually half decent, but it uses a more restrictive license than Firefox: GPL 3

So, what tools do I mean?

1. The old adblock plus contained an excellent tool for selecting elements on a page. This tool was not made by the adblock devs. It comes from an open source library available under a permissive MIT or BSD license.
2. A tool where a user can right click on an image and then create a block list from that image's url.
3. A way to list all requests on that page (all images, all external files), so you can pick the right one and create a block list based on that.

If you would be willing to expand this feature to also include the "dev tools" needed to create filter lists then I might be willing to help work out some of the details for whoever will be writing it.