591338 - No clear solution for loading overlaying modules from specific package

Status: RESOLVED WONTFIX

(Add-on SDK Graveyard :: General, defect, P1)

Description

[Irakli Gozalishvili [:irakli] [:gozala] [@gozala]]

There is no clear way of loading module form a specific package. There was an active discussion on CommonJS to address this issue and there are still several proposals but no consensus, think jetpack may try to take a lead and encourage others by supporting one of the proposals or suggest an alternative.

http://wiki.commonjs.org/wiki/Packages/Mappings/C
http://groups.google.com/group/commonjs/browse_thread/thread/77a5e47b58125376

Comment 1

[Myk Melez [:myk] [@mykmelez]]

Irakli: which of the extant proposals would you recommend?

Brian: might your work to fix bug 591525 inform the design of this bug's fix?

Comment 2

[Irakli Gozalishvili [:irakli] [:gozala] [@gozala]]

Version 1.1 makes it solves above mentioned issues:

http://wiki.commonjs.org/wiki/Packages/1.1

It also makes our modules and third party packages incompatible with each other since according to the specs all the modules id's are namespaced under package name.

Comment 3

[Myk Melez [:myk] [@mykmelez]]

(In reply to comment #2)
> Version 1.1 makes it solves above mentioned issues:
> 
> http://wiki.commonjs.org/wiki/Packages/1.1
> 
> It also makes our modules and third party packages incompatible with each other
> since according to the specs all the modules id's are namespaced under package
> name.

I don't think that's actually the case.

The Packages 1.1 "draft" spec mostly doesn't address require(), although it does say:

    main - module that must be loaded when require(name) is called. Definition must be relative to the package description file.
    directories.lib - directory of modules to be loaded under the packages namespace. require(name/subfilename) must return modules from this directory. Definition must be relative to the package description file. 

... where "name" is the name of the package.

The Modules 1.1.1 spec, on the other hand, does address require().  But it doesn't explicitly state that the "module identifier" argument to require must identify the package name for a module, and it in fact implies otherwise.

It does say that the identifier, when "top-level", is "resolved off the conceptual module name space root."  But it also describes a "paths" "attribute" of the require() function that appears to specify a set of search paths for modules, and it implies that "paths" can be provided by the loader rather than the consumer, since "replacing the 'paths' object with an alternate object may have no effect."

And finally, "paths" "may not be an exhaustive list of search paths, as the loader may internally look in other locations before or after the mentioned paths."

I can't speak to the intent of the specification.  And perhaps the conversation around the drafting of the spec would make it clear that the spec authors did intend to require that module identifiers identify the package name in addition to the name of the module.

But based on my reading of the spec, module identifiers are not required to identify the package, and it is possible for the loader to use a precomputed set of search paths to resolve such identifiers when a module name is not explicitly provided.

Thus we can continue to handle require("foo") while adding in the ability to handle require("bar/foo") and maintaining compliance with the spec.  And I think we should continue to support require("foo"), since it's simpler and doesn't cause problems in the common case, although we should definitely add support for require("bar/foo") in the uncommon case where it is needed.


Brian has expressed a willingness to tackle CommonJS compatibility issues for the SDK 0.10 release, so assigning this bug to him.

Comment 4

[Irakli Gozalishvili [:irakli] [:gozala] [@gozala]]

IMO value of been CommonJS complaint is to allow module sharing across different platforms. Currently only platform with significant amount of libraries is nodejs with npm. The package registry is here:

http://npm.mape.me/

If would like to make some of those packages compatible with jetpack we need to allow require("package/module"). IMO code sharing is far more important then simpler module ID's. Having both will just add confusion and incompatibilities. That being said adding I do thin it's fine to allow require('module') type requires with sdk modules itself.

Comment 5

[Myk Melez [:myk] [@mykmelez]]

The Add-on SDK is no longer a Mozilla Labs experiment and has become a big enough project to warrant its own Bugzilla product, so the "Add-on SDK" product has been created for it, and I am moving its bugs to that product.

To filter bugmail related to this change, filter on the word "looptid".

Comment 6

[Brian Warner [:warner :bwarner]]

Myk, Atul, and I spent a long while at the whiteboard trying to figure out
what we wanted from a package search scheme. Here's what we came up with,
which I think is relevant to this ticket:

* sometimes, a CommonJS package contains one specific piece of functionality,
  such as most of the packages in the NPM repository at http://npm.mape.me/ .
  These packages typically have a "main" key in their package.json, which
  hides the internal directory layout from the outside world. For example,
  the Git-in-JS package (https://github.com/christkv/node-git) has {main:
  ./lib/git} which means require("node-git") is satisfied by loading
  node-git/lib/git.js .
* other times, a package contains a library of functionality, such as the
  "addon-kit" and "jetpack-core" packages we ship in the jetpack SDK. For
  these, require("panel") is satisified by loading addon-kit/lib/panel.js .

* we want both forms to work, with the same short import line:
  require("panel") should find the module from our addon-kit package, and
  require("node-git") should find the third-party package (assuming it's been
  installed somewhere we can find it)
* we also don't want to hard-code the addon-kit package into this process.
  The lookup function should be easy to explain and observe in action.

* add-on developers are likely to have a collection of third-party packages
  which they'll want to use from within all of their addons.

* the CommonJS package.json allows a "main" property which indicates which
  .js file should be used to satisfy a require() that names the package as a
  whole. It may also allow a "modules" property which generalizes this, so
  that require(PACKAGE/MODULE) will use PACKAGE/package.json's .modules to
  figure out how to satisfy "MODULE".

The proposal is:

* define an environment variable, putatively named "$JSPATH", which contains
  a list of directories inside of which packages will be found. Either the
  default value will contain $SDK/packages (i.e. the SDK directory which
  holds addon-kit and jetpack-core), or $JSPATH will be prepended to
  $SDK/packages before lookup.
* each element of $JSPATH will contain zero or more packages, each of which
  has its own package.json file. Within a single such directory, all packages
  must obviously have unique names. (the ability to change the name of a
  package from within its package.json makes this non-obviously false,
  however I think we declare any resulting name-collisions to be errors)

* when a module does require(X), the lookup/search will be done in two
  passes:

** first, we walk through $JSPATH looking for a package named X. If found, we
   use its package.json main/modules properties to figure out how to satisfy
   the require-the-package-as-a-whole request. This form is suitable for the
   NPM-style packages which provide one piece of functionality per package,
   so that require("node-git") can find $JSPATH[n]/node-git/lib/git.js .
** second, if that failed to find anything, we walk through each element of
   $JSPATH, then walk through each package therein in alphabetical order. For
   each resulting package, we ask it if it can provide X as a "library
   module". Packages claim this by having a key in their package.json
   putatively named "library-modules", which maps from "X" to the filename
   that should be imported. The first package that provides X terminates the
   search. This form is suitable for the SDK libraries like addon-kit, so
   that require("panel") can find $SDK/packages/addon-kit/lib/panel.js .

* when a module does require(X/Y), the lookup will only use the first pass,
  not the second. Once a package named X is found, we use its package.json
  main/modules properties to figure out which Y.js should satisfy the
  requirement.

The result is that require(X/Y) is explicit/fully-qualified, but the shorter
require(Y) can still find modules that are explicitly provided by a suitable
library. In addition, a package that is *not* explicitly trying to provide
libraries like this won't be able to accidentally interfere with short
require(Y) imports that happen to use the same name as some internal module.

For example, many addons will use require("request") to access
addon-kit/lib/request.js (the XHR api). Both require("addon-kit/request") and
require("request") will work. Suppose a new package is added to $JSPATH, e.g.
an Amazon Web Services package which is normally used by doing
require("AWS/S3") or require("AWS/EC2"). And suppose that this package
happens to contain a module named request.js, which seems likely because the
package is doing lots of things that involve requests of some sort (which may
not involve XHR at all): it's a likely name for a component of a
well-factored design. But since (unlike addon-kit) the AWS package is not
trying to provide a library of individual modules, it does not include the
"library-modules" property in its package.json, so the require("request")
lookup will not peer inside the AWS package to discover request.js . The only
way for the AWS package to "steal" the require("request") role is for it to
declare itself as a library.

A likely extension is for the second lookup pass to keep going after the
initial hit, and make a list of all matching library modules. If there is
more than one result in this list, it should use the first one, but emit a
warning about the others: this tells the developer that the particular
version of e.g. request.js that they got was dependent upon the exact
ordering of packages on their $JSPATH, such that it might change if the path
was rearranged or if the packages had names which sorted differently.

It should be possible to point $JSPATH at different versions of the SDK to
get newer versions of the built-in APIs. $JSPATH should also point at some
inter-project site-wide or user-account-wide directory of useful third-party
packages (much like /usr/lib or /usr/local/lib or ~/.local/lib) which will be
available for linking into any XPI they make. We can imagine a
package-manager which drops packages in this directory, where they can be
used by subsequent addon development.

Third-party API modules which eventually get incorporated into the SDK can
start life in a library package (like ~/.local/lib/my-library/lib/foo.js),
with a suitable package.json to mark it as a library. By having this
directory on $JSPATH, require("foo") will pick up the module. When the module
eventually makes it into the SDK, the developer can manipulate $JSPATH
(putting $SDK/addon-kit either earlier or later than ~/.local/lib) to prefer
either the SDK copy or the local copy (which might be newer).

To implement this, I think we need to define the syntax of the proposed
"library-modules" property, change the require() function to implement the
search path described above (but first finally move the search behavior out
of the JS runtime and into the python linker), and modify
$SDK/packages/addon-kit/package.json to include the magic library-modules
property.

We should also check with the CommonJS world to see if this makes sense. (I
think the overall goal of letting jetpack code do short require("panel")
instead of require("addon-kit/panel") is a valid one: this proposal is all
about making that both possible and somewhat clean).

thoughts?
 -Brian

Comment 7

[Myk Melez [:myk] [@mykmelez]]

(In reply to comment #6)
> thoughts?
>  -Brian

That sounds exactly right to me!

Comment 8

[Irakli Gozalishvili [:irakli] [:gozala] [@gozala]]

(In reply to comment #6)
> Myk, Atul, and I spent a long while at the whiteboard trying to figure out
> what we wanted from a package search scheme. Here's what we came up with,
> which I think is relevant to this ticket:
> 
> * sometimes, a CommonJS package contains one specific piece of functionality,
>   such as most of the packages in the NPM repository at http://npm.mape.me/ .
>   These packages typically have a "main" key in their package.json, which
>   hides the internal directory layout from the outside world. For example,
>   the Git-in-JS package (https://github.com/christkv/node-git) has {main:
>   ./lib/git} which means require("node-git") is satisfied by loading
>   node-git/lib/git.js .
> * other times, a package contains a library of functionality, such as the
>   "addon-kit" and "jetpack-core" packages we ship in the jetpack SDK. For
>   these, require("panel") is satisfied by loading addon-kit/lib/panel.js .
> 
> * we want both forms to work, with the same short import line:
>   require("panel") should find the module from our addon-kit package, and
>   require("node-git") should find the third-party package (assuming it's been
>   installed somewhere we can find it)
> * we also don't want to hard-code the addon-kit package into this process.
>   The lookup function should be easy to explain and observe in action.
> 
> * add-on developers are likely to have a collection of third-party packages
>   which they'll want to use from within all of their addons.
> 
> * the CommonJS package.json allows a "main" property which indicates which
>   .js file should be used to satisfy a require() that names the package as a
>   whole. It may also allow a "modules" property which generalizes this, so
>   that require(PACKAGE/MODULE) will use PACKAGE/package.json's .modules to
>   figure out how to satisfy "MODULE".
> 
> The proposal is:
> 
> * define an environment variable, putatively named "$JSPATH", which contains
>   a list of directories inside of which packages will be found. Either the
>   default value will contain $SDK/packages (i.e. the SDK directory which
>   holds addon-kit and jetpack-core), or $JSPATH will be prepended to
>   $SDK/packages before lookup.
> * each element of $JSPATH will contain zero or more packages, each of which
>   has its own package.json file. Within a single such directory, all packages
>   must obviously have unique names. (the ability to change the name of a
>   package from within its package.json makes this non-obviously false,
>   however I think we declare any resulting name-collisions to be errors)
> 
> * when a module does require(X), the lookup/search will be done in two
>   passes:
> 
> ** first, we walk through $JSPATH looking for a package named X. If found, we
>    use its package.json main/modules properties to figure out how to satisfy
>    the require-the-package-as-a-whole request. This form is suitable for the
>    NPM-style packages which provide one piece of functionality per package,
>    so that require("node-git") can find $JSPATH[n]/node-git/lib/git.js .
> ** second, if that failed to find anything, we walk through each element of
>    $JSPATH, then walk through each package therein in alphabetical order. For
>    each resulting package, we ask it if it can provide X as a "library
>    module". Packages claim this by having a key in their package.json
>    putatively named "library-modules", which maps from "X" to the filename
>    that should be imported. The first package that provides X terminates the
>    search. This form is suitable for the SDK libraries like addon-kit, so
>    that require("panel") can find $SDK/packages/addon-kit/lib/panel.js .
> 
> * when a module does require(X/Y), the lookup will only use the first pass,
>   not the second. Once a package named X is found, we use its package.json
>   main/modules properties to figure out which Y.js should satisfy the
>   requirement.
> 
> The result is that require(X/Y) is explicit/fully-qualified, but the shorter
> require(Y) can still find modules that are explicitly provided by a suitable
> library. In addition, a package that is *not* explicitly trying to provide
> libraries like this won't be able to accidentally interfere with short
> require(Y) imports that happen to use the same name as some internal module.
> 
> For example, many addons will use require("request") to access
> addon-kit/lib/request.js (the XHR api). Both require("addon-kit/request") and
> require("request") will work. Suppose a new package is added to $JSPATH, e.g.
> an Amazon Web Services package which is normally used by doing
> require("AWS/S3") or require("AWS/EC2"). And suppose that this package
> happens to contain a module named request.js, which seems likely because the
> package is doing lots of things that involve requests of some sort (which may
> not involve XHR at all): it's a likely name for a component of a
> well-factored design. But since (unlike addon-kit) the AWS package is not
> trying to provide a library of individual modules, it does not include the
> "library-modules" property in its package.json, so the require("request")
> lookup will not peer inside the AWS package to discover request.js . The only
> way for the AWS package to "steal" the require("request") role is for it to
> declare itself as a library.
> 
> A likely extension is for the second lookup pass to keep going after the
> initial hit, and make a list of all matching library modules. If there is
> more than one result in this list, it should use the first one, but emit a
> warning about the others: this tells the developer that the particular
> version of e.g. request.js that they got was dependent upon the exact
> ordering of packages on their $JSPATH, such that it might change if the path
> was rearranged or if the packages had names which sorted differently.
> 
> It should be possible to point $JSPATH at different versions of the SDK to
> get newer versions of the built-in APIs. $JSPATH should also point at some
> inter-project site-wide or user-account-wide directory of useful third-party
> packages (much like /usr/lib or /usr/local/lib or ~/.local/lib) which will be
> available for linking into any XPI they make. We can imagine a
> package-manager which drops packages in this directory, where they can be
> used by subsequent addon development.
> 
> Third-party API modules which eventually get incorporated into the SDK can
> start life in a library package (like ~/.local/lib/my-library/lib/foo.js),
> with a suitable package.json to mark it as a library. By having this
> directory on $JSPATH, require("foo") will pick up the module. When the module
> eventually makes it into the SDK, the developer can manipulate $JSPATH
> (putting $SDK/addon-kit either earlier or later than ~/.local/lib) to prefer
> either the SDK copy or the local copy (which might be newer).
> 
> To implement this, I think we need to define the syntax of the proposed
> "library-modules" property, change the require() function to implement the
> search path described above (but first finally move the search behavior out
> of the JS runtime and into the python linker), and modify
> $SDK/packages/addon-kit/package.json to include the magic library-modules
> property.
> 
> We should also check with the CommonJS world to see if this makes sense. (I
> think the overall goal of letting jetpack code do short require("panel")
> instead of require("addon-kit/panel") is a valid one: this proposal is all
> about making that both possible and somewhat clean).
> 
> thoughts?
>  -Brian

Sounds good.
Not sure that complexity is worth of implicit but short require('panel') though.

I would also recommend looking at overlay's in npm. From my experience it's a MUST have feature for developing cross-platform packages. (npm help json)

One non-commonjs feature of npm is `require('jetpack-core@1.0.0/panel')`. This allows you to require modules from a specific version of the library. Having ability to load modules from different versions of library in one module may be very useful for devs that update their extensions from one version of dependency to another.

Comment 9

[Irakli Gozalishvili [:irakli] [:gozala] [@gozala]]

NPM dropped support for overlays and I don't know of any other platform that supports it so I think we can close this as wont-fix.