1315558 - tld service for webextensions

Status: NEW

(WebExtensions :: General, enhancement, P3)

Description

[Shane Caraveo (:mixedpuppy)]

We should consider providing tld for webextensions.  The SDK provides it here:

https://dxr.mozilla.org/mozilla-central/source/addon-sdk/source/lib/sdk/url.js#313

A simple Host api might be useful, such as https://github.com/NiklasGollenstede/get-tld

CookieMonster is one use case.

Comment 1

[Marat Tanalin]

The `eTLD` service is also used in the UsableHomeButton [1] extension. Namely, the `Services.eTLD.getPublicSuffixFromHost()` function is used.

A pure-JS library is not a reasonable equivalent of built-in `eTLD` features:

* The huge advantage of the built-in `eTLD` service is that the actual list of top-level domains is automatically always up-to-date thanks to browser's automatic updates. This cannot be achieved in a reasonable way with a library.

* Using multiple different implementations of the same feature in the same application leads to inconsistencies and is considered bad practice.

* Using an extra library while already having similar functionality in the application results in wasting memory and CPU power and generally have a negative performance impact.

[1] https://addons.mozilla.org/firefox/addon/usablehomebutton/

Comment 2

[Caitlin Neiman [:caitmuenster]]

To be discussed at Nov. 29, 2016 WE triage meeting. 

Agenda: https://docs.google.com/document/d/1IMBFXHNpg_A-15VdJM1Hh8DUUXF1xNFy87W1w8ZEOBk/edit?usp=sharing

Comment 3

[Shane Caraveo (:mixedpuppy)]

We've had more than one request for this, I think providing an API to the internal implementation is better than making people use external libraries.

Comment 4

[Kris Maglione [:kmag]]

We discussed this and decided that it should probably be implemented as an extension-specific WebIDL interface, probably along the lines of MozURLUtils.

Comment 5

[Martin Kimmerle]

Good to see this feature will be implemented. Is there a schedule?
The RequestPolicy add-on extensively uses the public suffix list.

According to the WE triage meeting, the priority should be P2, no?
Thanks.

Comment 6

[Andy McKay]

A good catch, the meeting did say P2, changing. There's no schedule for this yet, its in the backlog. If its something you'd like to contribute code to, please let us know.

Comment 7

[Tomislav Jovanovic :zombie]

(In reply to Kris Maglione [:kmag] from comment #4)
> We discussed this and decided that it should probably be implemented as an
> extension-specific WebIDL interface, probably along the lines of MozURLUtils.

To confirm I understand the intent here, the idea is to have a `MozURLUtils` property on the `window` global, available only to web extensions (for now), which would include (for this bug), a method like `getTLD(url)`?

Comment 8

[Kris Maglione [:kmag]]

Window and Sandbox globals, yes. Or something along those lines, anyway.

Comment 9

[Franz Erlendorf]

Getting the list of TLDs can be done by the extension itself, an API provided by the browser is not needed. Extensions can cache the list, thereby having offline access to the list.

Comment 11

[Anthony Lieuallen]

+1 for this.  Greasemonkey uses nsIEffectiveTLDService in our legacy version.

Comment 12

[Mike Hommey [:glandium]]

This would be useful for features I want to add to the tab-stat addon (grouping by domain name). Each addon reimplementing something the browser already has built-in (with automatic updates of the list, at that), would be error prone and suboptimal.

Comment 13

[Marat Tanalin]

I’m forced to temporarily remove the eTLD-powered feature entirely from WebExtensions version of my UsableHomeButton extension since bundling an own copy of 12000+-entries database with each extension that needs it looks unreasonable at least in terms of performance.

Comment 14

[sgl4kn]

This would be great, millions of users use currently sth. like this extensive Adblock implementation (with other addons using different and even slower approaches):

var publicSuffixes = {
    "0.bg": 1,
    "1.bg": 1,
    "1kapp.com": 1
};

function getDomain(hostname) {
	let bits = hostname.split(".");
	let cutoff = bits.length - 2;
	for (let i = 0, bitsLen = bits.length; i < bitsLen; i++){
		let offset = publicSuffixes[bits.slice(i).join(".")];
		if (typeof offset != "undefined"){
			cutoff = i - offset;
			break;
		}
	}
	if (cutoff <= 0) return hostname;
	return bits.slice(cutoff).join(".");
}

Comment 15

[Shane Caraveo (:mixedpuppy)]

ni? Fallen to consider and fit in somewhere.

Comment 16

[Philipp Kewisch [:Fallen][📱📆]]

Thanks for tagging me. I'm fine with having this sort of API available, though I'm not sure about the approach. If we are going to be exposing this on the window I'd prefer if we could work towards standardizing it instead of having a WebExtensions specific window property.

Comment 17

[sgl4kn]

BTW: there are also Rust implementations in an attempt to parse PSL quickly:
https://crates.io/crates/psl

There are hundreds of JS libraries and all big extensions like Adblock Plus parse the PSL multiple times on each link on a every website. However, these are all not very efficient.

I use a Regex in my scripts that is executed on every website link. This Regex checks if a link is a referrer link (e.g. http://www.yahoo.de/redirect.cgi?=www.standard.co.uk). For different conditions, the PSL has to be parsed 4 times on each link on a website.
Adbuster's reference implementation is using an array map which results in an overall performance loss of 20% for each website, but it would be awesome if there would be a faster, FF integrated solution.

Comment 18

[Francis :mckenfra]

Attachment: Bug 1315558 - getPublicSuffix(url) for webextensions, r=rpl,robwu

Exposes a call to Services.eTLD.getBaseDomain(url) as a new
webextensions API method: browser.dns.getPublicSuffix(url)

E.g. calling this method with "https://www.mozilla.org" results in "mozilla.org"

This will save Addon creators from having to reinvent the wheel.
The Mozilla Multi-Account Containers Addon team intends to use this functionality
for a Wildcard Subdomains feature that is currently in development, see PR:
https://github.com/mozilla/multi-account-containers/pull/2352

Comment 19

[Francis :mckenfra]

Attachment: Bug 1315558 - getPublicSuffix(url) changed to parent & async, fixed skip-if. r=rpl,robwu

Depends on D148988

Comment 20

[Rob Wu [:robwu]]

I filed an issue to discuss this feature in the WebExtensions Community group:

https://github.com/w3c/webextensions/issues/231
("Extension API to find the public suffix (eTLD) of a given URL/domain")

Comment 21

[Francis :mckenfra]

Attachment: Bug 1315558 - getPublicSuffix(url) for webextensions. r=rpl,robwu

Exposes a call to Services.eTLD.getPublicSuffix(url) as a new
webextensions API method: browser.dns.getPublicSuffix(url)

E.g. getPublicSuffix("https://www.mozilla.co.uk:80") => "co.uk"

The method also takes an optional "additionalParts" integer parameter:

E.g. getPublicSuffix("https://www.mozilla.co.uk:80", 1) => "mozilla.co.uk"

This will save Addon creators from having to reinvent the wheel.
The Mozilla Multi-Account Containers Addon team intends to use this functionality
for a Wildcard Subdomains feature that is currently in development, see PR:
https://github.com/mozilla/multi-account-containers/pull/2352

Comment 22

[BugBot [:suhaib / :marco/ :calixte]]

The following patch is waiting for review from inactive reviewers:

ID	Title	Author	Reviewer Status
D149692	Bug 1315558 - getPublicSuffix(url) for webextensions. r=rpl,robwu	3ecdbelo	rpl: Inactive
			robwu: Inactive

:3ecdbelo, could you please find another reviewer?

For more information, please visit auto_nag documentation.

Comment 23

[Suhaib Mujahid [:suhaib]]

Sorry for the noise, there was a problem with a Bugzilla API that caused the bot to think more people were inactive than normal (bug 1789646).

Comment 24

[BugBot [:suhaib / :marco/ :calixte]]

The severity field for this bug is relatively low, S3. However, the bug has 277 votes and 5 See Also bugs.
:3ecdbelo, could you consider increasing the bug severity?

For more information, please visit auto_nag documentation.

Comment 25

[BugBot (nomail) [:suhaib / :marco/ :calixte]]

The last needinfo from me was triggered in error by recent activity on the bug. I'm clearing the needinfo since this is a very old bug and I don't know if it's still relevant.

Comment 26

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee is inactive on Bugzilla, so the assignee is being reset.