286159 - xpcom interfaces: No method of globally observing and affecting (redirecting) URIs before the are committed to a socket or content handler

Status: RESOLVED DUPLICATE of bug 421224

(Core :: Networking, enhancement)

Description

[ElZorro]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0

After search for a solution to this problem:
http://forums.mozillazine.org/viewtopic.php?t=233973 and speaking with some
firefox devs on #developers it would appear that there is no sane method of
globally observing all URIs as they are fired by the browser at a stage that
will allow you to filter/modify them before they are committed to a particular
socket.
The aim would be the ability to audit URIs irrespective of protocol (current or
future, eg. http,ftp,gopher,irc,feed etc) or origin (ie. user actioned, scripted
or embedded) and be able to block, allow or modify them.
This will greatly aid development of security, anonymity based extensions aswell
as allow easy integration with cgi style content rewriter/translator and
webproxy services.

Reproducible: Always

Steps to Reproduce:
There is no API in place to achieve the desired goal.

Comment 1

[Darin Fisher]

how much context do you need about the initiator of the URL load?

-> core: networking

Comment 2

[ElZorro]

(In reply to comment #1)
> how much context do you need about the initiator of the URL load?
> 
> -> core: networking

As much as possible? :)
I've only been developing moz extension for a short while, so I'm afraid I can't
give you a definitive answer

I can get by with little or know parent info...
However, it would be handy to have flags to show whether it's a user triggered
event (mouse click, addressbar), scripted ( js or css ) or embedded link (image,
object)
Also a reference to which content window|browser|page|script (or tab?) has it
spawed from? 
Going a step further for markup URIs then perhaps a reference to the tag/node
that has the link as an attribute (<a href=..> <img src=..> <iframe src..>
<object data=..> etc) - this would make it easy if I wanted to highlight all
cross-domain embedded images with red borders for instance.

Comment 3

[Christian :Biesinger (don't email me, ping me on IRC)]

>However, it would be handy to have flags to show whether it's a user triggered
>event (mouse click, addressbar), scripted ( js or css ) or embedded link (image,
>object)
>Also a reference to which content window|browser|page|script (or tab?) has it
>spawed from? 

try nsIContentPolicy

Comment 4

[ElZorro]

(In reply to comment #3)
> >However, it would be handy to have flags to show whether it's a user triggered
> >event (mouse click, addressbar), scripted ( js or css ) or embedded link (image,
> >object)
> >Also a reference to which content window|browser|page|script (or tab?) has it
> >spawed from? 
> 
> try nsIContentPolicy

Ah yes, thanks. From what I can see nsIContentPolicy is good for completely
blocking unwanted content. Some google searching has yielded little in the way
of documentation, but it does seem to have been effectively implemented in the
Adblock extension. Judging by the somewhat daunting size of the extension's
source code it's not exactly straightforward to implement simple URL filtering
and seemingly it's not possible to be able to manipulate URIs before sending
them on their way (eg. "this request is not allowed, use this content instead").
Without the benefit of thorough testing I'm not sure how easy it is to access
contextual information on the source of a URI request, so you can also perform
manipulation there (eg replacing blocked images with alert logos at the dom level).
Perhaps this is all possible through existing interfaces (I'm still searching
for solutions), however it seems to me that even if this is the case the whole
process requires a disproportionately large and complicated lump of scripting to
achieve even limited functionality.
I hope someone can enlighten me to the contrary. (sorry if this is the case, I'm
still finding me feet with the framework)

Comment 5

[Christian :Biesinger (don't email me, ping me on IRC)]

try http://lxr.mozilla.org/seamonkey/source/content/base/public/nsIContentPolicy.idl

context is available.

> the whole
> process requires a disproportionately large and complicated lump of scripting to
> achieve even limited functionality.

that is not my impression of content policy. as far as I know, all you have to
do is add the contractid of your component to the content-policy category, and
implement nsIContentPolicy on an object registered for that contractid.

Comment 6

[Boris Zbarsky [:bzbarsky]]

> it's not exactly straightforward to implement simple URL filtering

Sure it is.  Most of the adblock code is doing things like messing with the page
itself (instead of just blocking things).  It has a very complicated setup for
this (way more complicated than needed), as well as for the multiple different
types of filtering it offers, its UI, etc.

> and seemingly it's not possible to be able to manipulate URIs before sending

With content policy, that's correct.

> I'm not sure how easy it is to access contextual information on the source of
> a URI request

Pretty easy.

Comment 7

[ElZorro]

Thanks for the pointers everyone!

I've spent most of yesterday learning from adblock and playing with content
policy, and it looks like it will fit the bill nicely. It's going to take me a
while, I suspect, to do my own implementation but at least I can see that it's
an attainable goal.
As I've often seen in the framework, half the battle seems to be finding the
right place to look :)

Anyway, going to resolve this one WORKSFORME, unless someone thinks otherwise.

Thanks again.

Comment 8

[ElZorro]

ok, after a brief discussion in #developers, it would seem like a good idea to
keep this open, since being able to easily modify URIs would be undeniably
useful and could save a lot of unnecessary and not 100% effective work-around code.
The question is though would nsIContentPolicy be a good location for this? or is
it best handled somewhere in necko (where this bug is filed)

Cheers.

Comment 9

[Boris Zbarsky [:bzbarsky]]

nsIContentPolicy doesn't sound like a very good place for this....

Depending on what's desired, overriding the nsIIOService contract may be the way
to go, though that would rewrite URIs as the URI objects are being created, not
when the channel is opened.

Comment 10

[ElZorro]

(In reply to comment #9)
> nsIContentPolicy doesn't sound like a very good place for this....
> 
> Depending on what's desired, overriding the nsIIOService contract may be the way
> to go, though that would rewrite URIs as the URI objects are being created, not
> when the channel is opened

I think it would be necessary to intercept them before any channels are opened,
otherwise it would be difficult change the nature of the request. (also from an
efficiency perspective, if you're wanting to filter large numbers of requests)

Comment 11

[Christian :Biesinger (don't email me, ping me on IRC)]

(In reply to comment #9)
> Depending on what's desired, overriding the nsIIOService contract may be the way
> to go, though that would rewrite URIs as the URI objects are being created, not
> when the channel is opened.

One _could_ make that ioservice override's newChannel method return a special
nsIChannel impl that wraps the real channel. care would need to be taken since
things expect nsIHttpChannel etc.

Comment 12

[Boris Zbarsky [:bzbarsky]]

Or one could just override the newURI method to change the URI as desired...

Comment 13

[Christian :Biesinger (don't email me, ping me on IRC)]

(In reply to comment #12)
> Or one could just override the newURI method to change the URI as desired...

Yes, but that would, I think, change things like the URI in the status bar when
hovering over a link, which may or may not be desired.

Comment 14

[ElZorro]

(In reply to comment #13)
> (In reply to comment #12)
> > Or one could just override the newURI method to change the URI as desired...
> 
> Yes, but that would, I think, change things like the URI in the status bar when
> hovering over a link, which may or may not be desired.

As you mentioned to me in #developers, do we still not have the problem that the
IOService is often referenced directly by CID within the source code, so
substituting functionality via contractid may not work in all cases. (perhaps
this doesn't apply to the relevant references though).

Comment 15

[Christian :Biesinger (don't email me, ping me on IRC)]

(the main reason I'm concerned about that is
http://lxr.mozilla.org/seamonkey/source/netwerk/base/public/nsNetUtil.h#90 which
references the ioservice by CID)

Comment 16

[Boris Zbarsky [:bzbarsky]]

Hmm... Yeah, that would make things not really work... :(

Comment 17

[Darin Fisher]

Plus the IO service is loaded very early on during startup.  Even if it were
only referenced by ContractID, you'd have a hard time intercepting all
consumers.  The IO service is initialized by the security system whenever a JS
component is loaded, so basically you have to assume that the IO service is
already initialized by the time the xpcom-startup notification is dispatched.

We could invent an API for watching channels and imposing redirects on those
channels.  That could be used as a way to redirect requests for various
resources to alternate protocols.  We could leverage
nsIChannelEventSink::REDIRECT_TEMPORARY (or maybe REDIRECT_INTERNAL or some
other new flag) to hide the alternate URIs from being shown in the UI.

We have to be careful from a security point of view as well.  Afterall, such a
redirect-based override mechanism would impact JS same-origin checks.

Comment 18

[ElZorro]

Hi just, giving this old report a nudge. I'm curious if any of the structural changes worked into Deer Park have opened the way to resolving this issue (is there a better way?)

Cheers

Comment 19

[Adam Judson]

I'd prefer to see this implimented in the same way as http-on-modify-request,
using the observer service.  e.g. "channel-on-create-request"

A.

Comment 20

[Dark Box]

Any solution for this yet?  Does using nsIIOService to override newChannel or new URI work?  I have spent almost 2 weeks digging into mozilla source to find where I need to intercept before channel opens, but it looks like I hit the dead end.  Any workaround solutions, advices.

-Dark

Comment 21

[ElZorro]

(In reply to comment #15)
> (the main reason I'm concerned about that is
> http://lxr.mozilla.org/seamonkey/source/netwerk/base/public/nsNetUtil.h#90 which
> references the ioservice by CID)

I've just noticed that this issue seems to be resolved with the release of 1.8. IO Service is now being referenced by contractID - if you observe the difference between the old:

http://lxr.mozilla.org/mozilla1.7/source/netwerk/base/public/nsNetUtil.h#90

and the new:

http://lxr.mozilla.org/seamonkey/source/netwerk/base/public/nsNetUtil.h#96

The issue of timing still remains however. Is there no way we can intercept nsIIOService's contract before it gets initialized?

Comment 22

[Christian :Biesinger (don't email me, ping me on IRC)]

How about another idea... call the redirection handlers in nsBaseChannel::AsyncOpen (and call nsIChannelEventSink::onRedirect(this, newChannel, REDIRECT_INTERNAL) once all handlers were called), and forward the open to the new channel.

That'd have some nice features:
- Observers can see the fully-configured channel
- Compared to doing it in nsIIOService::newChannelFromURI, it has no reentrancy problems
- Compared to doing it in IIOService::newURI it has more flexibility (channel rather than URI), and also no reentrancy problem

Downsides:
- Relevant channels don't use basechannel yet

Comment 23

[Christian :Biesinger (don't email me, ping me on IRC)]

further benefits:
- newChannel callers get what they expect
- newURI callers get what they expect (a URI that represents exactly the passed-in string)

Comment 24

[ElZorro]

(In reply to comment #22)
> How about another idea... call the redirection handlers in
> nsBaseChannel::AsyncOpen (and call nsIChannelEventSink::onRedirect(this,
> newChannel, REDIRECT_INTERNAL) once all handlers were called), and forward the
> open to the new channel.

From my somewhat limited knowledge of the framework, this would appear to me to be the most gratifying solution so far. 

> Downsides:
> - Relevant channels don't use basechannel yet
> 

Would seem to be a major stumbling block to me. Although Mr Biesinger has indicated to me that moving existing channels over to the nsBaseChannel implementation is planned for future releases (gecko1.9?).

Personally I'd like to see an interim solution that provides at least a limited functionality from ioService:newChannelFromURI() but is designed in such a way to allow a smooth transition to nsBaseChannel::AsyncOpen when the underlying support is there.

thanks for everyone's comments thus far.

Comment 25

[Darin Fisher]

I like the suggestion of adding observers to anything derived from nsBaseChannel.  A generalization of HTTP's on-modify-request and on-examine-response events would also be nice.

Another thing we can do I think is add observers inside nsIOService:: NewChannelFromURI.  It can call out to consumers and give them the ability to see the newly created channel and substitute that channel for another channel.  This is similar to the way you can implement nsIProtocolHandler today and have your NewChannel method return the channel of another protocol handler.  I know this means that you don't get to intercept a fully configured channel, but it would still be useful in some cases.

Making HTTP and FTP inherit from nsBaseChannel is on my TODO list for Gecko 1.9.

Comment 26

[Christian :Biesinger (don't email me, ping me on IRC)]

(In reply to comment #25)
> Another thing we can do I think is add observers inside nsIOService::
> NewChannelFromURI.  It can call out to consumers and give them the ability to
> see the newly created channel and substitute that channel for another channel. 

I want to note that this has re-entrancy problems - if that impl wants to use a different channel, it has to call NewChannelFromURI (usually). So, that would need to be taken care of somehow.

> Would seem to be a major stumbling block to me.

Well, the code could be done in a helper function that all asyncOpen impls can call.

Comment 27

[ElZorro]

(In reply to comment #26)
> I want to note that this has re-entrancy problems - if that impl wants to use a
> different channel, it has to call NewChannelFromURI (usually). So, that would
> need to be taken care of somehow.

Aye...all I can see to do there is set a flag on re-entry that's used as a conditional for whether to fire the channel redirect handlers.

Using a helper function as you suggest would probably make a smoother entry into gecko1.9, hopefully without breaking compatibility for any future 1.8 sourced components that have already made use of channel redirects.

Comment 28

[jmdi]

This is important.
For me it is the simplest way to 
permit HTTP filtering by extensions at the network level.

Comment 29

[Arturo 'Buanzo' Busleiman]

It would be quite a useful thing to get an input/output filter framework inside Mozilla. For example, for certain enhancements to the HTTP protocol I'm working on (OpenPGP Extensions for HTTP, see enigform and/or mod_auth_openpgp in freshmeat), I had to write an apache input filter of CONNECTION level. Apache also provides filters that can work on higher layers, like a simple content filter (transform XML into PDF on the fly, etc).

It'd be quite nice for XPCom components to register input (http response) and output (http request) filters in a similar fashion.

Comment 30

[Jan Wrobel]

I'm working for more than a year on Firekeeper extesnion (http://firekeeper.mozdev.org) that does some advance content filtering (examines request URL, response body, headers and cancels suspicious requests). I must agree that doing it now is really painful. I had to create a lot work around code and still there are some stability problems. Input/output filter would be really useful for extensions like Firekeeper.

Comment 31

[ElZorro]

Looks like firekeeper does much of what I was planning to do when I opened this bug 2 1/2 years ago (although my goals were for anonymous browsing).

I'm still very interested in getting this feature added - my project has been squarely on a back-burner until I can find a competent coder with the time and will to help implement url auditing.

I did write a little example code for nsIOService.cpp a couple of years back just to illustrate my idea (I'm not a C++ coder though, so it was never intended as a working example), but I'm guessing much has changed since then and there are probably better ways to implement things.

Never-the-less I'll see if I can find it and add it as an attachment for reference sake.

Cheers

Comment 32

[ElZorro]

Attachment: channel redirect/url rewrite illustrative example implementation

Found my example implementation.

Just want to stress that this was for illustrative purposes only, and is most likely obsolete. There was an idea a couple of years back to use nsBaseChannel as the point redirect-point, but it looks like it's still not implemented across the board for all protocol handlers (such as http), is this correct?

Comment 33

[Alex Vincent [:WeirdAl]]

Please see bug 421224 - I'm wondering if it should be duped to this bug.