660629 - Simple XPI file layout

Status: RESOLVED FIXED

(Add-on SDK Graveyard :: General, enhancement, P2)

Description

[arky [:arky]]

User-Agent:       Mozilla/5.0 (Macintosh; Intel Mac OS X 10.5; rv:2.0.1) Gecko/20100101 Firefox/4.0.1
Build Identifier: 

Currently the files/directory layout inside XPI is complex, it would great if there was much simpler layout which shows the javascript libraries, assets in toplevel directory. 

Chromeless uses a much simpler file/directory layout which simpler to understand and make it easier to manage.

Reproducible: Always


Actual Results:  
myaddon.xpi
   install.rdf
     - bootstrap.js
     - components/harness.js
     - resource/
     - harness-options.json

Expected Results:  
myapp.xpi
        - install.rdf
  	- images/
        - myapp.html
        - main.js
        - data/

Comment 1

[Myk Melez [:myk] [@mykmelez]]

I'm pretty sure we have a bug on this (or something similar enough) already.

Brian: do you know if there is such a thing?  It seems useful to simplify the directory layout in this fashion and perhaps even go farther (f.e. get rid of data/).

Comment 2

[Brian Warner [:warner :bwarner]]

I can't think of an existing bug, offhand. I'd love to simplify things.

== Simplifying the Jetpack XPI ==

Given that Jetpack addons are really normal addons with a special loader,
part of the file layout is dictated by how normal addons work, which means we
can't get rid of:

 - install.rdf
 - bootstrap.js

(components/harness.js can go away, since we've dropped FF3.6 compatibility)

We need a manifest of sorts, so "harness-options.json" could be renamed but
not removed.

We need a place to put various modules (unlike Chrome addons, Jetpack's core
APIs are shipped in the addon rather than being built-in to the browser),
including the user's top-level code, and a typical addon will have 10-20
modules, so we need names that are distinct enough to make tracebacks
readable. So we need resources/ or code/ or modules/, and probably
PKG/MODNAME.js paths underneath. (we can fix the ugly current
resources/JID-PKG-SECTION/MODNAME.js paths, though).

Data is supposed to be scoped to the package which references it, and we need
to handle name collisions between e.g. package-A/icon.png and
package-B/icon.png . So I think the current approach of putting it inside
resources/ isn't too bad.

== Add-On Layout Forces ==

Overall, I think there are two forces in play here. One is how
Jetpack-generated addons fit into the larger world of Firefox addons. Firefox
wasn't changed to support Jetpack; instead Jetpack produces XPIs that can be
loaded the same way as non-Jetpack addons. We could imagine changing
Firefox's addon interface (loading something other than XPI files), which
would let us factor out some of the jetpack-specific loader files, and would
probably improve security too. This might actually happen once Jetpack is
stable enough to tie ourselves to the FF release cycle. Chrome had a blank
slate to work from and no backwards compatibility to worry about.

Two is the use of modules, making Jetpack addons (and FF addons in general)
more flexible and powerful. Chrome addons use a single .js file because the
platform provides all the functionality they'll ever get; Jetpack bundles
whichever API modules you need in the XPI, and users can write new ones with
require("chrome"). We're aiming for an ecosystem in which anybody can write
new modules that get used by everyone else, sharing code by copying modules
around. I think code-sharing is harder in Chrome because your best option is
to cut-and-paste snippets into your one main.js, and access to platform APIs
is tightly controlled by the platform itself. We could have the SDK
merge/compile/minify all the source files into a single .js file, to make the
XPI appear simpler, but I think that'd be worse.

Finally, in some ways, we're comparing apples and oranges here, or at least
source code and object code. The Jetpack SDK is like a compiler/linker: it
transforms source packages (with their package.json and main.js) into an
object-code bundle (the XPI zipfile). Chrome's "compiler" happens to do very
little transformation, shifting the responsibility to their runtime code.
Jetpack's is a bit more extensive and does the package-search work up-front,
so the output format is rather different than the input. I think, for the
most part, the internals of the XPI matter about as much as the layout of an
ELF or a.out file: addon developers only come into contact with the source
trees.

== Oops, You Said Chromeless ==

Reading more carefully, I just noticed that you weren't talking about the
Google Chrome browser at all. Oops. You meant
https://github.com/mozilla/chromeless .

Much of what I wrote is still applicable, though. I'm still trying to find
the Chromeless docs that describe its generated-bundle layout (which inspired
your "myapp.xpi" expectations), but it sounds like Chromeless defined a new
interface for its apps, so they didn't have backwards-compatibility concerns
or a need to make XPIs that can be loaded by an existing system. They *do*
use modules, but don't need to include quite so many because the platform
(i.e. the Chromeless binary) provides most of the necessary functionality.
And, from what I can tell, Chromeless *source* packages are simple (with an
appinfo.json, index.html, index.js, and some data files), but the generated
"object" files (created by "chromeless appify" or "chromeless package") are
as weird-looking as Jetpack's XPIs (sometimes more so, since there's a whole
browser packed in there).

Comment 3

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

We need to figure specifically what we'll be doing here before it can be prioritized.

Comment 4

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

(Pushing all open bugs to the --- milestone for the new triage system)

Comment 5

[Alexandre Poirot [:ochameau]]

Attachment: wip

Here is my current patch over the no runtime search pull request:
https://github.com/mozilla/addon-sdk/pull/249

It removes URLs from harness-options.json and let loader decide how files should be hosted/handled.
Then files are organised liked this in the XPI:
resources/
  api-utils/
     lib/
       cuddlefish.js
       main.js
     data/
       content-script.js
     tests/
       text-main.js

I'll try to write a better description here or on the group or my blog.

Comment 6

[Brian Warner [:warner :bwarner]]

hm, so the manifest contains paths that start with "/resource/PACKAGE/lib/" ? And the only occurrence of the add-on ID is in a single harness-options.json property named "uriPrefix" or something? And the loader remembers to prepend uriPrefix to build a real URI just before loading each module? And there'd be a single resource-protocol mapping that points this prefix into the XPI?

Seems good to me!

Comment 7

[Alexandre Poirot [:ochameau]]

Attachment: Pull request 276

Here is an updated patch against today's master!
I wrote a more detailed description of this patch in the pull request.

Pull request is way larger than original patch because of the cleaning.
I removed various stuff and renamed all occurences of `URI` to `Path`.
But I haven't changed anything, it just cleans, renames and adapt unit tests to new behavior.

Comment 8

[Brian Warner [:warner :bwarner]]

Comment on attachment 577629 [details]
Pull request 276

as mentioned on the pull-request, looks good! I'll adapt my bug 685378 to match this new layout.

Comment 9

[[github robot]]

Commit pushed to https://github.com/mozilla/addon-sdk

https://github.com/mozilla/addon-sdk/commit/8c6457f40b35464be4bf0e3c505c02a43ab09b72
Merge pull request #276 from ochameau/simple-dirs

Bug 660629: Simple XPI file layout r=@warner

Comment 10

[[github robot]]

Commit pushed to https://github.com/mozilla/addon-sdk

https://github.com/mozilla/addon-sdk/commit/8c6457f40b35464be4bf0e3c505c02a43ab09b72
Merge pull request #276 from ochameau/simple-dirs

Comment 11

[[github robot]]

Commit pushed to https://github.com/mozilla/addon-sdk

https://github.com/mozilla/addon-sdk/commit/03bf4f872d1065aaa6d954283dca3d77bf27b0bb
Merge pull request #282 from ochameau/cleanup

Various cleanup, followup work from bug 660629/PR-276. r=@warner