1779770 - webRequest API: Requests blocked by ETP have no flags in urlClassification

Status: NEW

(WebExtensions :: Request Handling, defect, P3)

Description

[Teal Dulcet [:tdulcet] (TB Council)]

Attachment: image.png

Requests blocked by Firefox's Enhanced Tracking Protection (ETP) feature do not have the corresponding classification flags set in the urlClassification property (bug 1589494), which according to MDN is the whole point of that property. The onErrorOccurred() event has the error property set to NS_ERROR_TRACKING_URI, but the urlClassification property for all webRequest events has no flags set (see screenshot). This makes it impossible for extensions to determine why the requests were blocked specifically.

Here is some code to reproduce the screenshot:

const TAB_ID_NONE = browser.tabs.TAB_ID_NONE;

function errorOccurred(details) {
	if (details.tabId && details.tabId !== TAB_ID_NONE) {
		const error = details.error;
		if (error.includes("TRACKING")) {
			const aurl = new URL(details.url);
			console.log(aurl.origin, error, details.urlClassification);
		}
	}
}

browser.webRequest.onErrorOccurred.addListener(errorOccurred,
	{ urls: ["<all_urls>"] }
);

I tested in the latest Firefox 91 ESR, Firefox 102 and Nightly 104.

Comment 1

[Rob Wu [:robwu]]

Shane will take a look.

Some internal classifications are stripped at https://searchfox.org/mozilla-central/rev/3cb31675aeffd10f1f6ae7c40e24b254da7798e5/toolkit/components/extensions/webrequest/WebRequest.jsm#322-329

Comment 2

[Shane Caraveo (:mixedpuppy)]

Verified. When ETP actually blocks a request, we get the onErrorOccurred event without any details for urlClassification, though the error string does come through. When ETP only flags an item, but does not block it, the other WebRequest events will have urlClassification data.

For example, visit https://senglehardt.com/test/trackingprotection/test_pages/fingerprinting_and_trackers.html with strict ETP enabled.

Three items are blocked, and we'll get three calls to onErrorOccurred with NS_ERROR_TRACKING_URI.
One is not and I see (via an event listener I added)

"onHeadersReceived {\"requestId\":\"1895\",\"url\":\"https://content-track-digest256.dummytracker.org/test_not_blocked.png\",\"urlClassification\":{\"firstParty\":[],\"thirdParty\":[\"tracking_content\",\"any_strict_tracking\"]}}"

I suspect that when we block we are not storing the classification on the channel and thus we cannot get that data. Will find someone to verify.

Comment 3

[Shane Caraveo (:mixedpuppy)]

Hi Tim, you're doing some work in this area, do you have any insight on my suspicion above? When ETP blocks an event, should data be available in this code:

https://searchfox.org/mozilla-central/rev/f6a2ef2f028b8f1eb82fa5dc5cb1e39a3baa8feb/toolkit/components/extensions/webrequest/ChannelWrapper.cpp#1017-1029

Comment 4

[Tim Huang[:timhuang]]

Dimi might be a better person to answer this. But I can try to answer it. We don't set the channel classification flags in the case where ETP blocks the request. We only set these flags when annotating the channel. In this case, the channel still loads, but the ETP might do something according to the classification flags, such as limiting the cookie access for trackers.

So, if you wanted to know why the channel was blocked, you have to check the canceling error code to see the reason why it's blocked. For example, we will cancel the channel with the NS_ERROR_SOCIALTRACKING_URI code if the channel was blocked by the social tracking protection.

Comment 5

[Shane Caraveo (:mixedpuppy)]

Ok, then the next question is whether we could set the classification flags even when we block. I'll ni? Dimi.

The other part of this will be documenting current behavior, since it will remain valid even if we did the above change, but would also document for older versions.

The documentation here https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/webRequest/onErrorOccurred should expand the notes for urlClassification to include:

• they are always empty in onErrorOccurred when ETP blocks a request.
• details.error will contain the error code per these codes: https://searchfox.org/mozilla-central/rev/f6a2ef2f028b8f1eb82fa5dc5cb1e39a3baa8feb/js/xpconnect/src/xpc.msg#238-247

Comment 6

[Shane Caraveo (:mixedpuppy)]

See comment 5

Comment 7

[Teal Dulcet [:tdulcet] (TB Council)]

(In reply to Shane Caraveo (:mixedpuppy) from comment #5)

The documentation here https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/webRequest/onErrorOccurred should expand the notes for urlClassification to include:

• they are always empty in onErrorOccurred when ETP blocks a request.

They are empty for all applicable webRequest events in this case, not just onErrorOccurred.

• details.error will contain the error code per these codes: https://searchfox.org/mozilla-central/rev/f6a2ef2f028b8f1eb82fa5dc5cb1e39a3baa8feb/js/xpconnect/src/xpc.msg#238-247

Thanks, that list of error codes and descriptions is very useful. However, the current documentation on MDN says not to rely on those error code values (source here): "This string is an internal error string, may vary from one browser to another, and is not guaranteed to stay the same between releases".

The Chrome documentation is even stronger (source here): "This string is not guaranteed to remain backwards compatible between releases. You must not parse and act based upon its content". It of course would certainly be helpful if Mozilla and Google could come to an agreement on a standard set of error codes, but that is obviously out of scope for this bug.

There are also many other undocumented aspects about this urlClassification property. For example, why does it have both firstParty and thirdParty properties? The details.thirdParty property of course already indicates if the request is third party. Is it possible for a request to simultaneously have both first party and third party tracking?

---

If this helps, the flags are actually NOT empty for requests blocked by ETP when they are replaced with shims by the SmartBlock feature. This is also completely undocumented, but from testing neither the onCompleted nor onErrorOccurred events fire in this case. Instead, the last event seems to be onBeforeRedirect with the redirectUrl property set to a moz-extension:// URL.

Chrome does have similar documentation (source here): "The web request API guarantees that for each request either onCompleted or onErrorOccurred is fired as the final event with one exception: If a request is redirected to a data:// URL, onBeforeRedirect is the last reported event".

Comment 8

[Dimi Lee [:dimi]]

(In reply to Shane Caraveo (:mixedpuppy) from comment #5)

Ok, then the next question is whether we could set the classification flags even when we block. I'll ni? Dimi.

What :tim said in Comment 4 is correct, and yes, URL classifier can set the classification flags even when we block.

Comment 9

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:mixedpuppy, could you have a look please?

For more information, please visit auto_nag documentation.