1499523 - "main_frame" chrome.webRequest events fail to fire upon returning to a page that uses service workers

Status: UNCONFIRMED

(WebExtensions :: Request Handling, defect, P3)

Description

[Alexei]

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36

Steps to reproduce:

1. Load a WebExtensions extension that logs chrome.webRequest.onBeforeRequest (for example) events.
2. Click a link to a page that uses service workers.
3. Navigate back to the page with the link.
4. Reload the page with the link.
5. Click the link again.

You can use the following trivial page (that links to two different sites that use service workers) to help reproduce:

<a href="https://www.netlify.com/docs/">https://www.netlify.com/docs/</a>
<br>
<a href="https://dev.to/aspittel/a-complete-beginners-guide-to-react-2cl6">https://dev.to/aspittel/a-complete-beginners-guide-to-react-2cl6</a>


Actual results:

No "main_frame" event.


Expected results:

I expect to receive a "main_frame" event (before the rest of the page's resources start loading).

This problem does not happen in Chrome.

This problem does not happen when service workers are disabled (set dom.serviceWorkers.enabled to false and restart).

Comment 1

[Alexei]

This leads to several prominent Firefox-specific bugs in Privacy Badger. You can get an idea by browsing the issues linked in https://github.com/EFForg/privacybadger/issues/1144

The main issue is that since Privacy Badger keeps track of the page the user is on using chrome.webRequest.onBeforeRequest's "main_frame" events, Privacy Badger can end up blocking resources belonging to www.netlify.com (for example) while on www.netlify.com because Privacy Badger thinks the tab is still on whatever the previous page's domain was.

This is another issue brought on by not having a straightforward way of synchronously getting the URL loaded in a given tab (bug 1470537, https://bugs.chromium.org/p/chromium/issues/detail?id=856766).

Comment 2

[Manuel Strehl]

I’m the reporter of the original Privacy Badger bug. If there’s anything I can do to diagnose the bug, I’d be glad to help.

Comment 3

[Shane Caraveo (:mixedpuppy)]

Well, navigating back to the netlify page (step 5 in STR) with a service worker results in the service worker doing bunches of XHR.  The webrequest api merely catches events from the http channel and passes them through to the extension, I'm guessing there is some implementation detail with services workers that causes this issue, best to see if we can get input from someone who may know.

Comment 4

[Andrew Sutherland [:asuth] (he/him)]

## Short Story ##

We are overhauling our ServiceWorkers implementation.  This is tracked on bug 1231208.  Until that is fixed, ServiceWorker interception will be invisible to the webRequest API, *but the requests issued by the ServiceWorker will be visible* and *redirections triggered by the ServiceWorker will be visible*.

## Longer Story ##

Currently under e10s, ServiceWorker interceptions occur exclusively in the content process unless a redirect is involved or the interception is reset (due to not invoking respondWith on the FetchEvent).  This will change as we complete the e10s refactor.

Flipping the preference "dom.serviceWorkers.parent_intercept" and restarting Firefox will result in interceptions going to the parent process to experience what the situation will be in the future.  (But it is *NOT SAFE* to run this way in general.  Then the ServiceWorkers will live in the parent process and they will spectre *all* your data.)  Things should then work because interception in the parent (or non-e10s mode) is handled in the Connect() method, which is after OnBeforeConnect().

Comment 5

[Bennett Cyphers]

This is also an issue for users of the new version of Twitter with Privacy Badger.

To reproduce (with the latest version of Privacy Badger installed):

1. Navigate to https://mobile.twitter.com in a new tab and log in.
2. Navigate to a different website (e.g. https://www.eff.org/) in the same tab.
3. Via the browser bar (in the same tab), navigate directly to a mobile.twitter.com url, such as https://mobile.twitter.com/Twitter/status/1087791357756956680.

If you have Privacy Badger installed, this should throw the tab into an ugly redirect loop involving the login page that does not allow you to access Twitter. This happens because the extension doesn't intercept any main_frame requests; all requests to "mobile.twitter.com" are of type "xmlhttprequest". Therefore, Privacy Badger can't see that the first-party host on the tab should be mobile.twitter.com; therefore, it believes requests to *.twitter.com are third-party tracking requests and tries to block them.

Any updates on this? Is there a timeline?

Comment 6

[Andrew Sutherland [:asuth] (he/him)]

The ServiceWorker overhaul should be landing in Fx68, although it's not guaranteed we will flip the preference to enable it, although we're trying hard. It may be best to temporarily set the dom.serviceWorkers.enabled preference to false to disable ServiceWorkers in your profile, noting that this preference will manually need to be flipped back later on.

Comment 7

[Jamie Lokier]

I'm not absolutely certain, but I suspect this bug is responsible for Firefox showing "Corrupted Content Error" on sites reached from search engine results, yet working fine when the exact same URL is pasted into a new tab.

Here is a short discussion:

https://forum.bytemark.co.uk/t/bytemark-website-fails-to-display-when-i-use-google-to-find-it/3053

I've only rarely seen this "Corrupted Content Error" before, but I have seen it occasionally.

Before I've assumed it was a bug in Firefox's TLS or HTTP implementation, or a bug in the site's implementation, and I didn't think to copy the URL and try it in a new tab!

This time, I thought it was just another site with a bad HTTP server, or an annoying "we only test our HTTP server on Chrome" type of issue, until I noticed the console message about a ServiceWorker and thought to look there. Kudos to whoever added the ServiceWorker message to Firefox :-)

Note the error message indicates more than just Privacy Badger misclassifying a page when following a cross-site link:

Failed to load ‘https://www.bytemark.co.uk/’. A ServiceWorker passed a promise to FetchEvent.respondWith() that rejected with ‘TypeError: NetworkError when attempting to fetch resource.’.

When that error occurs, it would be helpful if Firefox displayed a better error page than "Corrupted Content Error".

Comment 8

[Jamie Lokier]

If any site that uses ServiceWorker could suffer from connection errors due to this bug, is there a way for the site developer to detect the condition and turn off the ServiceWorker, or otherwise workaround the glitch?

ServiceWorkers are sometimes used just to cache and accelerate.

If there are detectable situations where a site will break despite a correctly written ServiceWorker (even if it's due to a browser bug, or an unfortunate interaction with an add-on), it would be preferable for the site to catch the error and skip using ServiceWorker altogether.

Comment 9

[Andrew Sutherland [:asuth] (he/him)]

Bug 1503072 covers handling corrupted content errors better. Competing priorities have delayed implementation.

Comment 10

[Jamie Lokier]

(In reply to Andrew Sutherland [:asuth] (he/him) from comment #9)

Bug 1503072 covers handling corrupted content errors better. Competing priorities have delayed implementation.

Thanks.

That's useful, though it's disappointing that such a catastrophic issue (for any site affected permanently by it - the 24 hour recovery timeout never happens in that bug) is marked as mere "enhancement".