Allow some addon functionality to load prior to any content loading

NEW
Unassigned
(NeedInfo from)

Status

()

Toolkit
WebExtensions: Untriaged
P2
normal
4 months ago
10 hours ago

People

(Reporter: andym, Unassigned, NeedInfo)

Tracking

(Depends on: 2 bugs)

unspecified
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox57 wontfix, firefox58 affected)

Details

(Reporter)

Description

4 months ago
Some extensions might want to block on startup to ensure that they load on the first page load in Firefox. Currently they start on startup, but don't block meaning that the background page may or may not have loaded before the page is rendered. 

In testing locally with Windows and uBlock Origin on Nightly it usually renders the first page with the extension loaded. However other people are saying otherwise.

We should probably do this through a permission so it can prompt users about how this can degrade start up performance.

This should probably depend upon out of process extensions? 

Would also be curious if Chrome provides this guarantee. Assuming Rob Wu might know the answer to that question.
(Reporter)

Updated

4 months ago
Duplicate of this bug: 1376239
(Reporter)

Updated

4 months ago
Duplicate of this bug: 1377835
(Reporter)

Comment 3

4 months ago
We have telemetry now for how long background pages take to start up:

https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2017-07-04&keys=__none__!__none__!__none__&max_channel_version=nightly%252F56&measure=WEBEXT_BACKGROUND_PAGE_LOAD_MS&min_channel_version=null&processType=*&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2017-06-12&table=0&trim=1&use_submission_date=0

Comment 4

4 months ago
Instead of adding a separate permission for this, I think we should just do it for extensions that use webRequestBlocking (and possibly others like non-blocking webRequest)
(In reply to Andrew Swan [:aswan] from comment #4)
> Instead of adding a separate permission for this, I think we should just do
> it for extensions that use webRequestBlocking (and possibly others like
> non-blocking webRequest)

I'd rather not conflate webRequestBlocking with blocking startup to allow the load of the background script.

Comment 6

4 months ago
(In reply to Shane Caraveo (:mixedpuppy) from comment #5)
> (In reply to Andrew Swan [:aswan] from comment #4)
> > Instead of adding a separate permission for this, I think we should just do
> > it for extensions that use webRequestBlocking (and possibly others like
> > non-blocking webRequest)
> 
> I'd rather not conflate webRequestBlocking with blocking startup to allow
> the load of the background script.

"conflate" implies that they aren't related.  but under what circumstances would an extension use webRequestBlocking but not actually care if it has the opportunity to examine every outgoing request?
I don't think they are related.  There may be non-webrequest reasons an addon needs early loading of its background page.  I wont pretend to have specific use cases, it just hits me wrong to use webrequestblocking.
(Reporter)

Comment 8

4 months ago
This bug mentions background pages, but it could also be a content script that is loaded at document_start, which I think bug 1332273 comment 0 also affects. Perhaps its just a generic "whole add-on blocks" as opposed to the background?

Comment 9

4 months ago
> the background page may or may not have loaded before the page is rendered

Regarding blockers, my understanding is that the real issue is being able to load all filter rules before a page is rendered, this is especially true for blockers needing to load/parse/instantiate 10Ks of rules. For uBO, my approach is to pre-compile filter lists in order to remove the cost of parsing at load time. So far this seems to work rather well. If this still fails, there is also an experimental setting[1] which allows uBO to block all network requests before its filter rules are all loaded in memory, and as far as I can tell, it takes care of such issue if ever it occurs on some slower systems.[2]

[1] https://github.com/gorhill/uBlock/wiki/Advanced-settings#suspendtabsuntilready-experimental
[2] https://github.com/gorhill/uBlock/issues/1327

Comment 10

4 months ago
Chrome does not provide a guarantee that extensions are loaded when a browser is started*. Even for extension pages (starting Chrome with a chrome-extension://[extensionid]/page.html), I have seen the behavior change multiple times in the past, from randomly not having an extension-privileged script context available, to having partially broken API bindings.

* actually, that is not completely true. Chrome does (or did?) wait for user scripts to have loaded before loading web pages. There is more context in this bug from 2015 (still not "fixed"): https://crbug.com/523634 ("Resource loading should be deferred until webRequest extensions have finished loading").

In Chrome, background pages don't have to be around in order for an extension to function well (they can be transient, in the form of event pages). Ideally we should also go in this direction, that extensions don't depend on the existence of a background page to work.

Chrome's way of offering a good UX with reasonable performance is to allow extensions to declaratively register rules that may be relevant before an extension has started. This includes:

- The contextMenus API (registered menu items persist across restarts).
- declarativeWebRequest API (webRequest handlers that don't require a roundtrip to an extension script, persists across restarts) (this API is restricted to non-release channels due to the lack of engineering efforts, but it is being picked up again - https://crbug.com/696822).
- Content scripts in manifest files (apparently? Not documented except for a comment in https://crbug.com/523634#c4).

If any, we should put this behind a NEW permission, and require manual approval by reviewers on AMO before allowing an extension to block startup. If devs submit add-ons and often get rejected without suitable alternatives for their use case, then we can consider adding APIs to help these add-ons to provide their functionality without blocking startup.
(In reply to Andy McKay [:andym] from comment #0)
> Some extensions might want to block on startup to ensure that they load on
> the first page load in Firefox.

So by "block on startup" I assume (hope) you mean "block content process startup"; this shouldn't block first paint of the browser chrome.

And if the content process creation stays blocked for a user-noticeable amount of time after we have painted the chrome, maybe we should display a warning in the content area saying which extension is to blame for this delay.

Comment 12

4 months ago
(In reply to Shane Caraveo (:mixedpuppy) from comment #7)
> I don't think they are related.  There may be non-webrequest reasons an
> addon needs early loading of its background page.  I wont pretend to have
> specific use cases, it just hits me wrong to use webrequestblocking.

The longer version of what I was trying to suggest is that we should be able to figure out when an extension needs to be loaded based on the apis it uses, rather than making extension authors declare it explicitly.  Documenting how an extension author should decide whether to request that permission will be tricky and no matter how well we document it, many will still get it wrong.  And punting to AMO reviewers also sounds like a bad idea.

As to Florian's questions, extensions shouldn't block the first chrome paint but extensions that examine network requests obviously need to block any content loads.
(Reporter)

Comment 13

4 months ago
(In reply to Rob Wu [:robwu] from comment #10)
> If any, we should put this behind a NEW permission, and require manual
> approval by reviewers on AMO before allowing an extension to block startup.

Manual review is not a sustainable path forward. We are either comfortable with the API being used, or we don't allow it.
(In reply to Andrew Swan [:aswan] from comment #12)
> (In reply to Shane Caraveo (:mixedpuppy) from comment #7)
> > I don't think they are related.  There may be non-webrequest reasons an
> > addon needs early loading of its background page.  I wont pretend to have
> > specific use cases, it just hits me wrong to use webrequestblocking.
> 
> The longer version of what I was trying to suggest is that we should be able
> to figure out

(In reply to Andrew Swan [:aswan] from comment #12)
> The longer version of what I was trying to suggest is that we should be able
> to figure out when an extension needs to be loaded based on the apis it
> uses

As an exercise, what does that look like?

I could presume that an extension not using webrequest or content scripts but has a background script does not need to block.
A content script matching any tab loading on startup needs to block.
A webRequestBlocking permission with a background script and/or content script may or may not need to block, and it's hard to know since the listeners are runtime.

Are there other cases?


> but extensions that examine network requests obviously need to block any
> content loads.

I don't think it's obvious at all.  For instance, a webRequestBlocking addon may only be listening on certain domains, even if <all_urls> is requested.  Those domains may not be in any tab on startup.  

I guess we could punt on that and block startup if "webRequestBlocking && <all_urls> && background || content script" is in the manifest.  I cant imagine an addon with webRequest having no background or content script, but you never know (perhaps a devtools addon).


Of course, by "block startup" we're talking about loading those addons prior to content loads, not a general blocking of startup.
Summary: Allow some background pages to block start up → Allow some addon functionality to load prior to any content loading
(Reporter)

Updated

3 months ago
Priority: -- → P2
(Reporter)

Updated

3 months ago
No longer blocks: 1190679
(Reporter)

Updated

3 months ago
Duplicate of this bug: 1380312

Comment 16

3 months ago
Andy, why do you think that 1380312 is a duplicate? 
1) That bug was introduced on Firefox 54 and can be reproduced on macOS version only.
2) It's not related to loading of extension's background page - background page is loaded as usual and that part of any extension is working fine.
3) Problem described in 1380312 somehow related to initialising first Firefox window. On second window all extensions work fine again. So it's not related to pages loading process.

As far as I know, Firefox 54 introduced "multiprocess" feature. And it looks like, that the cause of 1380312 is a data race that occurs on initialisation of main process and first gui window.

I think 1380312 should not be in resolved state yet.
Duplicate of this bug: 1383470

Comment 18

2 months ago
Hi all,

Any progress on this? I think we are over analyzing it by trying to decide exactly which extensions should be loaded before the first web request, and which ones can wait. Maybe to start we should just load them all before web requests, and maybe find reasons to delay some of them later on. Doing them all up front will at least get us back to the level of reliability provided by the older extension APIs.

If we expose an API that lets an extension block web requests, we shouldn't make it impossible for that extension to do it reliably, just to possibly save a few milliseconds for the very first page load.
(Reporter)

Updated

a month ago
Duplicate of this bug: 1396270

Comment 20

a month ago
Bug #1396270 got marked as a duplicate of this issue, but I reported the issue because it isn't just that add-ons aren't loading before the first web request (as mentioned here); it's that they don't load -at all-, not before, not after. 

I understand that the discussions are related, but there has to be some mechanism to make sure that they are loaded at some point of the boot-up. 

In terms of this bug, I imagine it would be most pertinent to inform the user of the loading process with some sort of icon or dialog, i.e., either allow the user to wait for the add-ons to load properly, or, alternative, allow him or her to skip the process and start viewing content right away. Educating Android users of the problems of the ecosystem seems to me like the right thing to do.

Since the bug is now closed, I will note here that using only WebExtensions addons in Nightly since a week ago or so, the issues I was having with Firefox properly loading the add-ons have lessened considerably. They seem to boot up more reliably.
> I wont pretend to have specific use cases, it just hits me wrong to use webrequestblocking.

Another use-case: proxyAPI

URLs are loading before an addon's background.js (where it calls proxy.register()) has completed. Those URLs are not loading through a proxy server, which causes privacy risks.
(Reporter)

Updated

23 days ago
Duplicate of this bug: 1403086

Updated

23 days ago
Depends on: 1214733
Hi Andy, is this something that will happen in 57 or 58? If this is planned for 57, we should talk on the timeline of uplifting this to beta57.
status-firefox58: --- → affected
tracking-firefox58: --- → ?
Flags: needinfo?(amckay)
(Reporter)

Comment 24

21 days ago
This is too big and risky to land for 57 along with the problem that it will negatively affect Firefox startup performance. It's on our roadmap, but not sure of the timeline.
Flags: needinfo?(amckay)
Thanks, I'll mark this a wontfix for 57 then.
status-firefox57: --- → wontfix
Dusting this off again and thinking a bit about some comments Kris made earlier, I don't believe that getting a background page fully loaded during startup is a sensible thing.  Although its more of a piecemeal effort, I think we should handle this separately for individual APIs.

webRequest is the big one here but I think it isn't conceptually that difficult to handle: we can stash any event listeners and their corresponding filters in addonStartup.json and then during browser startup, install appropriate observers to block any affected network loads until the extension has started up and initialized its listeners.  This works for what I assume is the common case (though I have nothing to back that up) where an extension always creates the same webRequest listeners.  Perhaps it would make sense to try to distinguish listeners created when the extension is starting up from listeners created later (and only apply this technique to the first type).  Regardless, we still have to deal with the scenario where an extension doesn't re-create a listener but we are blocking some outgoing requests.  I don't have a great idea other than using some heuristics to decide when to give up, I would love to hear suggestions.

That leaves the proxy API and probably a bunch of others unaccounted for as well as things like bug 1388270 and that's kind of a bummer, but overall I think its the only feasible option we have.  Unless somebody has a strong objection (and more importantly a workable counter-proposal) I'll do the necessary bugzilla shuffling so we can have a bug for the webRequest work outlined above (which I'll then take a stab at implementing) and a separate one for proxy.  ni to Kris and Shane since their specific feedback would be especially useful.
Flags: needinfo?(mixedpuppy)
Flags: needinfo?(kmaglione+bmo)
It would be great to capture Kris' comments here so we know the issues brought up.

I don't really see that as a reliable approach, there are way too many variables to make sure this works right.  Having proxy not work isn't a bummer, it's a problem that may cause privacy and security issues for users.

If it's possible to pause all outbound network requests, having a permission in the manifest to mark an extension as blocking startup (or blocking network I guess) could work.  We just block everything until that set of extensions are loaded then unblock them.  It removes any guesswork, we know what extensions actually need this, and we know that nothing slips through because we didn't account for it.
Flags: needinfo?(mixedpuppy)
(In reply to Shane Caraveo (:mixedpuppy) from comment #27)
> it's a problem that may cause privacy and security issues for users.
> 
> If it's possible to pause all outbound network requests, having a permission
> in the manifest to mark an extension as blocking startup (or blocking
> network I guess) could work.

This :)
(In reply to Shane Caraveo (:mixedpuppy) from comment #27)
> I don't really see that as a reliable approach, there are way too many
> variables to make sure this works right.  Having proxy not work isn't a
> bummer, it's a problem that may cause privacy and security issues for users.

I didn't mean that we would never fix it, I just think it should be addressed separately.

> If it's possible to pause all outbound network requests, having a permission
> in the manifest to mark an extension as blocking startup (or blocking
> network I guess) could work.  We just block everything until that set of
> extensions are loaded then unblock them.  It removes any guesswork, we know
> what extensions actually need this, and we know that nothing slips through
> because we didn't account for it.

So the common ground here is that we're both talking about just suspending some (or all) outgoing network requests at startup, the disagreement is just about whether that's caused by an explicit permission or inferred from the extension's use of apis like webrequest.  As I see it the downsides of the explicit permission are that it places an additional demand on extension authors (to understand this new permission and use it when appropriate, and there will certainly be cases where extensions that would want it don't know about it as well as cases where extensions that don't use it unnecessarily).  The downside of making it automatic is additional complexity.  I don't believe there's any functional shortcoming.
Well, preferably we'd be able to promise that the extensions are loaded during startup, but there's so much going on with that, I'm not certain it's easy to do.  It's possible blocking outgoing connections is easier, but see my thoughts further down, it's not completely clear that it is.

I'm not convinced that the existence of webRequestBlocking is a good indicator that an extension needs to block for startup, and there is likely a lot of extensions that will have that permission.  The proxy permission however probably is a good indicator.  Overall I think we require authors to know plenty so I'm not concerned about the additional permission, which can also be documented in the webrequest and proxy sections.  I'm also wary of trying to be too smart about it works, we're much more likely to miss something, whether it is a un/necessary filter, or a use case we don't account for.  An explicit flag to block everything is clear, will have less code paths to consider and modify, and will cover any missed or future use cases.

A tricky part is figuring out when to suspend channels.  HTTP Channel for example, doesn't handle on-opening asynchronously right now and on-modify is too late.  That could be fixed, other channels may also need some work to support this.  I've already been thinking about fixing on-opening since an earlier notification could be handy for some proxy handlers.  At that point, relying on on-opening could become murky at best, at worst unusable for proxies on startup.
Depends on: 1407384
Doesn't sound like we need to track this right now.
tracking-firefox58: ? → ---
You need to log in before you can comment on or make changes to this bug.