Closed Bug 209439 Opened 21 years ago Closed 20 years ago

Implement "extension registry", and make all extensions installable and uninstallable in non-application directories

Categories

(Core :: XPCOM, enhancement)

Product:
Core
Component:
XPCOM
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED FIXED

People

(Reporter: netdragon, Assigned: benjamin)

References

(
URL
)

Details

Since XPCOM is supposed to be modular, there needs to be a way 3rd party people
can create 3rd party extensions for the GRE/toolkit that aren't browser plugins
but toolkit extensions that would use interfaces to link in with the main
GRE/toolkit code. If you look at
http://www.mozilla.org/roadmap/app-architecture-diagram.png from the
http://www.mozilla.org/roadmap.html page, it doesn't talk about a way to easily
extend the GRE/toolkit with actual C++ code. That means if you are going to make
an XUL extension, you do it above the application level, for one application, or
a group of them based off the toolkit. Unfortunately, this means to extend the
GRE/toolkit. For a stupid example, let's say you want to add support for
alpha-transparent windows within the GRE. It means you will have to hack it in,
and keep it up-to-date until its accepted into the tree just like you would have
to with the XPFE-based browser. Why can't you make an extension to the interface
that can be installed via an xpinstall package, without requiring a rebuild of
the browser? I mean, COM is theoretically designed for such things.

In this alpha-transparency scenario, half the applications use this feature, and
half don't. It doesn't come built into GRE/toolkit but is a 3rd party
plugin/extension library that connects into the actual interfaces. An
application will query for the existance of this plugin/extension and if you
first install GRE/toolkit, will find out it doesn't exist, and disable that
feature (or possibly offer the plugin/extension). If the user installs the
feature's plugin/extension through an xpinstall package (without requiring to
rebuild or reinstall or even restart the GRE/toolkit), then when the
applications query for the extension, they now find it. The extension was
written in XPCOM or COM, and connected through portions of the actual
GRE/toolkit, and possibly a base interface of certain modules that allows for this.

So basically, what I'm asking for is the ability to design and allow the
installation through .xpi packages of 3rd party extensions for the GRE/toolkit
with the same ease of doing XUL/chrome extensions without having to add it to
the actual tree. With the present XPFE-based browser, it would be the same as
being able to update or extend interfaces like nsIView simply through an .xpi

It should also be designed so that these extensions could be eventually migrated
from the extension form into the actual tree with as little modification as
possible.
For now, the GRE can not be added to, only applications can be added to.
Right, and I imagine you can also see that as something of a liability to the 
project, since all applications depend on the GRE and toolkit, yet can't 
extend it, meaning that any application based on the GRE will always have to 
modify the base GRE/toolkit if they need a low-level extension, which will not 
only cause us more work, but bloat the GRE/toolkit and cause arguments over 
whether things should go in.

Since COM is designed to be binary compatible, it should be possible to extend 
the GRE through an extension that doesn't require a rebuild of the GRE.
This would also be helpful for implementation of http://mozscroll.mozdev.org/
(an XPCOM implementation of autoscroll that will be usable by embedding) because
people could test the code by just downloading an .xpi and not having to build
my patches or get a binary from me.
Would this feature allow SVG support to be included as an xpi rather than having
to be built in?
Brian, much is possible, and equally, much isn't.  :-)

Some extensions that requirement things to be rebuild (like mathml), aren't
going to work the GRE model nicely.  That means, you can't just bolt on features
like a mathml.  

Other things, true addon's, could be shared by multiple applications if they all
buy into these options.  For example a group of applications could agree to
register all components in a given directory.  This could allow all of the
flexiblity required to support this RFE and all me to sleep better a night
knowing that all application using a GRE won't be subject to what addon's are
dropped in.   :-)

Maybe this is in a directory named GRE/extras?

dougt: personally, I am hoping that no extension (application or GRE) will need
to add itself to an application/GRE directory; instead, it can install itself to
a directory C:\Program Files\mozdev.org\mozscroll (or whatever) and register
itself with the application (or with GRE).

To do this, XPCOM should be able to register components from an arbitrary number
of directories. With the multiple-chrome-directories hack that I'm working on,
extensions can be effectively separated from the application (or GRE) they they
work with.
xpcom already can deal with this.  applications can call
nsIComponentRegister::AutoRegister on n number of directories.
dougt: so we could set up some method of registering extensions in the GRE using
"external" component dirs?

is xpti.dat/compreg.dat kept on a per-directory basis or something more global
(for example, if an application chose to disable a particular GRE extension,
would compreg.dat "remember" that component when it shouldn't).
> so we could set up some method of registering extensions in the GRE using
"external" component dirs?

Yes, that would be possible.

is xpti.dat/compreg.dat application specific.  Each application has its own. 
(however, one would like to just able to "register the GRE" and get the minimum
set.  therefore, we decided against throwing optional components into the
gre/components directory)

David: Theoretically, if it could be done -- yes! :-)
In case anyone cares, the answer to comment 4 is "no, unless maybe you ship a
totally new version of gklayout.dll, in which case you had better make sure you
don't make any mistakes..."
Boris: Why is that? Why can't its interfaces just be extended and the extensions
link in through some form of registration? I don't believe theoretically,
especially with dealing with COM, something like SVG support has to be hardcoded
into the GRE. Applications should be able to call on binary extensions on
demand, without requiring it to be built-in. This would, in the end, really make
our lives easier.
Boris: I replied to your newsgroup post:

"You mean you want drop-in XPCOM components?  This has been available all
along... see the MNG-support XPI, for example."

Unless the drop-in support could let you extend any aspect of the GRE, no. For
instance, allowing autoscroll through XPCOM, or (SVG support?) will require a
more direct extensibility of the interface. I also agree that direct
modification of the GRE would be harmful, but allowing them to install anywhere
on the disk, and then somehow register their code to extend certain interfaces
through some sort of extension mechanism that could exist for every component or
interface and activate it on demand would be key. I am not enough of an XPCOM
guru to know how this would fan out. It seems to me, though, that allowing
something of this nature would decrease time needed to extend the interfaces
(especially for embedders) because they could do it on their own right without
it being added to the GRE.
Brian, core layout does NOT use COM for most things.  For the simple reason that
doing so involves an unacceptable performance penalty.  A lot of SVG support
could be added via COM stuff (eg nsSVGElement/nsSVGDocument/company), but the
necessary changes to the CSS parser, style data storage, and style resolution
require either directly changing the code or modifying the architecture to look
for plugins to these processes -- the latter would make things unacceptably slow.


Boris: Would modifying the architecture (your first method) so that such
extensions could be registered and loaded on demand of the application, but not
on startup, make things unacceptably slow? In that way, I was thinking that
Mozilla wouldn't need to look for the extension, because the application would
tell the GRE to load it.
I'll give an example of where this ability would be helpful. I have my Mozscroll
(mozscroll.mozdev.org) code, which is designed as autoscroll code that will be
built into the toolkit and available for embedders. There is currently an XUL
autoscroll project (autoscroll.mozdev.org) that will fill the void until then.
The biggest issue I had when developing it is that I have to provide debug and
non-debug binaries for people to test it, and they can't simply test it by
installing an XPI and activating it when necessary. The reason is because it is
mostly C++. The source code is located here (out of date since I am taking a
break from maintaining it):

http://www.mozdev.org/source/browse/mozscroll/src/

And it touches these parts of Mozilla (might not exist anymore or changed):
mozilla/content/events/src/nsEventStateManager.h (significant additions +
modifications)
mozilla/content/events/src/nsEventStateManager.cpp (significant additions +
modifications)
layout/base/public/nsStyleConsts.h (added new identifiers)
themes/modern/jar.mn
themes/classic/jar.mn
widget/public/nsIWidget.h
widget/public/nsGUIEvent.h
widget/src/build/widget.rc
widget/src/windows/resource.h
widget/src/windows/nsWindow.cpp


I realize this might take a lot of work to implement, but it seems that it would
be very beneficial to the project. It says in the roadmap that we are going to
do thinks more methodical now with a rearchitecurization (probably not a word
:-), so it seems that this would be the best time to do something like this.
Perhaps allowing extensions to directly extend portions of the GRE through this
manner could mean that we could remove some of the code from modules that would
only be used by Mozilla, but not by other applications based on the GRE in favor
of making them extensions that are loaded on demand by Mozilla (or possibly the
rare application that requires it).
On my autoscroll example...

Someone does MozCalc, but it has no need to scroll, so they don't want mozscroll
loaded.

Someone else does MozNotepad, and of course, they want the mozscroll feature.

The problem is that the way its currently designed, you could disable it on an
application basis through a preference, but there is no way to not load the code
because it had to be hardcoded into the GRE because so much of the GRE is
hardcoded. I might be wrong, but I see no way to make it a simple extension that
can be loaded on demand.
Yes, that would make thing unacceptably slow.
OK, let's list the different "functions" of a possible GRE extension:

1) add-on components (binary or js, possible python/etc)
   this is easy: we already have the ability to load multiple component
   directories, we just need a method to "register" them as part of the GRE
   (This should be the focus of this bug).
2) chrome (content/skin/js) (for example, jslib.mozdev.org)
   I'm working on the ability to register an arbitrary number of chrome
   directories (bug 179834), and we already have the ability to register
   multiple resource:// providers
3) hooks (startup, pageload, whatever)
   these can be installed by the components (step 1 above) as necessary
4) replacing "native" parts of the GRE (gklayout.dll, the event code)
   this just aint gonna happen. If you want something this drastic you're gonna
   need to replace the "default" GRE with another build of the GRE (this is
   not at all out of the question, but you have to be careful)

There are a coupla details that need to be worked out:
a) should the extensions load in a particular order? I don't think so, let them
work out their own dependencies if necessary.
b) how should extensions be enabled/disabled by an application? I think that
extensions should be explicitly enabled after GRE_Startup() has be called (no
extensions would be enabled by default).
c) how should GRE extensions be "registered"? through an interface? what does
that mean for xpinstall?

netdemon: I think I would like to take this bug, but not if you're going to spam
it with discussion... let's keep that on the newsgroups, OK?
Ok, if you take this bug I will agree to do most discussions in the newsgroups
and then we can just summarize them here. By the way, if number 4 is not
possible, then 1-3 would at least satisfy most cases where the extensions are
not doing what's in number 4. Number 4 would probably not be a problem if
multiple versions/builds of the GRE or individual module libraries were allowed
to work simultaneously (I don't know if this is possible yet).
There are NOT going to be any "GRE" extensions.  Suites of applications can
agree to install "addon" components in a common directory and each of these
applications can agree to autoreg these directories.  This isn't going to be the
GRE directory.  I would allow something like a GRE/addon's directory to be
created.  However, this will be an optional directory that applications would
have to manually autoreg (see nsIComponentRegistrar.idl).  

Is this what you are asking for Brian?  
OK, taking. dougt: I am not planning on installing *anything* in the GRE
directories, or having any extensions enabled without a specific call from the
application. All I plan on adding to the GRE is a "registry of extensions" so
that multiple applications can find extensions that they need.
Assignee: dougt → bsmedberg
why not just autoreg this addon directory and be done with it?
It will not be a single directory. Each extension will be installed in a
separate location (for example C:\Program Files\mozdev.org\JSlib). I want to
give applications the option of selectively loading extensions as needed.
I see.  

maybe we can just massage xpinstall to return this information.  its
installation data is already stored in a file called registry.dat, iirc.  Maybe
we can have it so that this file is at the "system" level instead of the app level.
I and other developers would definately like the flexibility as Benjamin
mentioned of being to put the extension anywhere on the disk (with permissions
considerations in mind) and to be able to register the components but only load
them on demand. If application A needs the extension, but application B doesn't,
then application B shouldn't have the component loaded. The application should
demand the component and we just have to be careful we don't get into a
condition where the component is constantly loaded and unloaded because of poor
logic in an application that uses it, decreasing system performance.
adjusting summary.  I can agree that there is a need for some kind of optional
system level registry for addon's.  This would be something that the application
can control.
Summary: Need ability to create 3rd-party extensions/patches for GRE/toolkit through .xpi → Need ability to find 3rd-party extensions through .xpi
Depends on: 179834
Summary: Need ability to find 3rd-party extensions through .xpi → Implement "extension registry", and make all extensions installable and uninstallable in non-application directories
Depends on: 210838
Blocks: uninstaller
Depends on: 212212
I have been fleshing out this proposal at
http://bdsmedberg.no-ip.org/extensions.html as discussion pop up... take a look
there for the latest draft (I apologize if you're behind the NS firewall... use
a web DNS lookup to get the IP).
URL: http://bdsmedberg.no-ip.org/extension...
Depends on: 212222
I just read this proposal.  I made comments in the other bug (the one about .moz
files), but I think this may be the more appropriate bug to comment in.  

You mention extensions being able to be disabled/enabled per profile.  The plan
for Firebird is to no longer show extensions in the install directory.  If they
have been placed into that XRE, then they are part of that XRE installation. 
Since they have become a part of that XRE, and as such, they won't show up in
Firebird or Thunderbird Extension Manager dialogs.  I agree with dougt that
installation of extensions that are shared by all profiles/apps that use an XRE
should be strongly discouraged.  The placement (in the XRE directory or outside
of it) is irrelevant.

I see "shared extensions" that become part of an XRE as being installed by e.g.,
a network administrator who can test the extensions and make sure they're ok.

So to reiterate, the only extensions that will show up in Firebird/Thunderbird
will be ones the user explicitly installed into his/her profile directory, so
the set of XRE components will not be mutable. This means you won't have the
profile "leaking" problem you describe in your proposal, and there's no need to
patch XPCOM.

hyatt: so is this bug WONTFIX, then?  I'm OK with that, if you don't like the
new proposal... I thought the new system would be cleaner than the old system of
installing extensions "inside" the applications (and extensions would be
contained within a single file, making them easy to uninstall).
(discussion continued from bug 206358)
In my (unfortunately not humble enough) opinion the current system of extensions
is a mess, which was why I was proposing an alternate mechanism. If that seems
like overkill then let's lay out a more modest set of goals. Hyatt's "install
into the XRE or profile" solution it certainly is simpler/less pervasive, and I
don't mind that solution if that's what the project wants, I just have concerns
about it's scalability.

My point with XPInstall is not that I'm trying to rewrite it... just that in my
plan it is no longer useful. If an extension/application lives in a single JAR
file, there is no need for the complex XPInstall system... an extension just
"exists" and is enabled or disabled... it doesn't need to be "installed".

If we go with hyatt's proposal to install extensions into the profile, how do we
deal with extensions that are not compatible with multiple releases of an
application? (for instance, extensions that use non-frozen interfaces)... are
there multiple versions of the extension installed in the profile? And how do
such extensions deal with binary components?
The idea we discussed in person here in Mountain View (me and mscott, brendan,
bryner, etc.) was that the chrome registry will actually install its files into
subdirectories (both in the install and the profile dirs).  The directory would
basically be chrome-(XRE/app version).  So, for example, Firebird 0.6 would
install its chrome files (in both the install and profile dirs) into chrome-0.6
instead of just chrome.

This means when the user upgrades a Firebird in place, they will get a brand new
chrome registry, and there's no risk that the new Firebird won't run, since it
will be clean initially.  The new chrome registry will then sniff around and
find the previous chrome registry directory.  It will suck in the old chrome
registry and then for each extension it finds, it will attempt to migrate the
extensions to the new chrome registry.

The migration has yet to be designed, but the idea is that you'd negotiate with
a site like Extension Room (or with a site specified by the extension author if
they wish to override the centralized repository) and it would indicate whether
the extension can still work with the new version of the app.  If it can, then
the chrome is re-registered in the new chrome registry.  If it can't, then a new
version of the extension that is compatible should be downloaded and installed
if possible.

Note that nothing in the chrome registry dictates that the JARs have to live in
the install or profile directories.  You could write an XPI *right now* that
would place the extensions in arbitrary directories.  So this is possible right
now.  All that has to live in the profile/install dirs themselves is the actual
chrome registry and that is entirely built through the registration process.

Also, am I happy with the current complexity of the chrome registry? Of course
not.  But I also know how many bugs would be produced by tearing this system
down and trying to rewrite it completely.  That's why I'm shooting for some
intermediate solution that isn't nearly as high-risk, something that would truly
be achievable by Firebird/Thunderbird 1.0.

I also think that those who advocate ripping out RDF still don't understand the
concept of superimposition or realize what RDF is actually doing.  People think
that this functionality can just be done away with easily, but it can't.  If you
were to try to replace this with something other than RDF, I don't really see
what you'd gain.  You still have to know at a minimum the following information:

(1) What packages are installed and where they are
(2) Version information about packages
(3) What skin is in effect for each package
(4) What locale is in effect for each package
(5) All of the locales available for a given package
(6) All the skins available for a given package
(7) All the extension XUL files that overlay a specific XUL file

And all of this is at least two-tier and thus needs to be superimposed.  I don't
see that any of what I said would change in any design others would come up
with, so dropping RDF would just force you to invent some other text file format
that would have to be parsed and that would have to build up a fairly complex
graph structure.

Anyway, I think bsmedberg and I are on the same page regarding the use of RDF
and the collapsing of the chrome information for a given registry into a single
chrome.rdf file, so why don't we do this first while we keep discussing the
other options?

do we all understand "profile directory" to be application specific?

Like .mozilla/default, .firebird/Default\ User, .foo/scratch_my_back?
Or is it just .mozilla, .firebird, .foo, which is where that registry.dat
lingers right now.
Axel: is that a rhetorical question?

All profiles (hidden details of the new app architecture's current
implementation, by the way -- Firebird and Thunderbird do not intend to expose
profile management to end-users) are specific to their applications.  That's good.

What point were you getting at?  That we store registry.dat in the wrong place
(install dir, not profile dir)?

/be
> Axel: is that a rhetorical question?

Yes, just assuring that we talk about the same thing.
No longer depends on: 212212
*** Bug 229265 has been marked as a duplicate of this bug. ***
This was, of course, fixed by a happy team of coders spending an insane amount
of time, for ffox 0.9.
Status: NEW → RESOLVED
Closed: 20 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.