45937 - reorg chrome dir structure for packageability

Status: VERIFIED FIXED

(Core Graveyard :: Skinability, defect, P3)

Description

[Warren Harris]

The current chrome directory structure doesn't lend itself well to arbitrary 
packaging for several reasons:

1. It is not possible to combine content, skins and locales into a single 
package (due to the subdirectories under bin/chrome). Users may want to do this 
for simple applications, e.g. chatzilla, that wants one package for an en-US 
locale, modern skin, chatzilla app.

2. It is not possible to combine multiple skins or locales into the same package 
without combining their information into a single manifest.rdf file. This is 
because only one manifest.rdf file lives at the top level of each locale or skin 
directory. This makes it significantly more difficult for users to combine 
several things into one package (e.g. by drag/drop one jar file's contents into 
another's).

Hyatt and I worked out the following new scheme (to be instituted after we get 
the initial jar file packaging on-line):

Right now, we're building the chrome directory's jar stuff like this (using 
chatzilla as an example):

     packages/chatzilla.jar!/manifest.rdf
     packages/chatzilla.jar!/chatzilla/content/<stuff>
     skins/modern.jar!/manifest.rdf
     skins/modern.jar!/chatzilla/skin/<stuff>
     locales/en-US.jar!/manifest.rdf
     locales/en-US.jar!/chatzilla/locale/<stuff>

Note that you can't have a simple chatzilla.jar with this scheme that includes 
locales and skins, nor can you easily add chatzilla's manifest.rdf information 
to the global en-US.jar file. 

What we want to do instead is this:

     chatzilla.jar!/content/chatzilla/chatzilla/manifest.rdf
     chatzilla.jar!/content/chatzilla/chatzilla/<stuff>
     chatzilla.jar!/skin/modern/chatzilla/manifest.rdf
     chatzilla.jar!/skin/modern/chatzilla/<stuff>
     chatzilla.jar!/locale/en-US/chatzilla/manifest.rdf
     chatzilla.jar!/locale/en-US/chatzilla/<stuff>

Here the manifest.rdf files are more fine-grained, and apply to particular 
sets of content/skin/locale and the package name. This allows them to be easily 
combined or distributed among one or more jar files. (E.g. "ns6.jar" might 
contain everything for navigator, messenger, and aim components, each with 
modern and classic skins, and a handful of locales.) 

In addition to changing this directory structure, the chrome resolution process 
will have map to the 3 locations based on the current locale and 
skin settings:

     chrome://chatzilla/content/<stuff>
       => 
jar:resource://chrome/chatzilla.jar!/content/chatzilla/chatzilla/<stuff>

     chrome://chatzilla/skin/<stuff>
       => jar:resource://chrome/chatzilla.jar!/skin/modern/chatzilla/<stuff>

     chrome://chatzilla/locale/<stuff>
       => jar:resource://chrome/chatzilla.jar!/locale/en-US/chatzilla/<stuff>

Currently, we're only prepending a resource url prefix to the chrome url 
independent of whether it's a content, skin or locale chrome url.

Comment 1

[Warren Harris]

Actually, the part Hyatt needs to do here is change the chrome registry mapping 
process. Updating the jar.mn files can be done by me and dprice afterwards. 

Hyatt: Can you do this with ifdefs for us?

Comment 2

[Chris Waterson]

Is the problem that it "doesn't lend itself well", or that it doesn't work at 
all? I hate to be a stick-in-the-mud, but if it's the former, can we live with 
the worse-is-better directory structure?

Point 1 seems like a major problem: is it really impossible to do this with the 
existing directory structure?

Point 2 seems pretty unimportant, frankly. How many "users" (what do you mean 
here?) are going to be trying to combine jar files?

Comment 3

[David Hyatt]

Some comments.

Regarding point 1.  It is possible to combine skins, locales, and content into a 
single JAR or "package".  chatzilla does this now.  It's only bad in the sense 
that chatzilla has skin and locale info underneath the packages subdir, i.e., 
you're forced to pick a subdir that may not be logical.  Anyway, I agree that 
the immediate subdirs should go away.  So on (1) we're in agreement.  Let's 
kill the immediate subdirs.

Regarding point 2, I'm not so sure.  Your first statement is incorrect, which 
makes me thing we're crossing wires.  With the current system, it IS possible to 
have multiple skins and locales in the same package and to use different 
manifest.rdf files (one for each skin/locale).

For example, you could have a jar like so...

bigass.jar, which could have as its internal directory structure...
  skins
    modern
      messenger-modern
        manifest.rdf
        messenger
          skin
            <modern messenger skin files>
  locales
  packages

... or you can do pretty arbitrary glomming in the manifest.rdf file and put it 
at a higher level.  It's very flexible.

You can see how that extends.  The suggested reorg that warren and I discussed 
would be to make manifest.rdf look "up" instead of "down" in the dirtree to 
figure out what it applies to, but after thinking about this, I'm a little 
hesitant.

First of all, the current system can already do what you want, albeit with a 
deeper directory structure.  Secondly, however, the deep directory structure 
doesn't make much sense given how we've already agreed to organize our skins and 
locales.

Namely, we decided to have one modern JAR and one en-US JAR, and they don't need 
all those nested subdirs.  We' be making the skins and locales more complex 
with this reorg, not less complex.  

The current system allows for maximal flexibility, but the proposed revision 
limits the flexibility.

But what really clinches it IMO, is the near-lynching we got from the many 
people already using the chrome dir when we re-org'ed.  It doesn't make sense to 
me to screw people with another directory re-org this late in the game.

We've already given public talks at conferences about how this system works, and 
there have been numerous articles about it.  It works.  It isn't fundamentally 
broken.  I don't think we should change it again without really good reason.

We already know that Netscape is not going to bundle multiple locales or skins 
into a single JAR.  If at some later point they do choose to, the current system 
allows for this anyway.

If it ain't broke...

Comment 4

[David Hyatt]

Actually I'm not even sure (1) should go away.  I remember now why dveditz 
wanted that now.  It avoids naming conflicts between skins, locales, and 
packages that have the same filename.  Remember that we have specialized skin 
and locale installers that are different from package installers.  They have to 
put files somewhere.  This helped reduce the odds of having a naming conflict.

"packages" was a place for JARS that were a mixture of skin,content,locale.
"skins" was a place for JARS that contained only skins, which is what the skin 
installer enforces (by preventing registration of content and locale from 
happening, which would be a security hole).
"locales" is the same as "skins", but for locales.

"safe" skins and "safe" locales must be in a different place.  That's why we did 
it.

So I'm voting "Don't change anything."  

  

Comment 5

[Daniel Veditz [:dveditz]]

Wait, I didn't really want the subdirs but accepted it as a compromise. I 
wanted different extensions to fill the same duty, that is "modern.skin", 
"english.locale", and "chatzilla.pkg".

By putting the .jar (why .jar?) files directly under chrome you defeat the 
purpose of these intermediate directories and we should get rid of them. The 
thought at the time was that it would look something like
  chrome
     skins
       modern.jar
       classic.jar
     locale
       en-US.jar
     packages
       chatzilla.jar

That is, the current directories containing the manifest.rdf files would become 
the .jar files in place.  The worry about file conflicts is that if skins 
really do get popular and have the same extension as other chrome the chances 
are really good that people will develop skins that conflict with other 3rd 
party packages. (obviously people will know to avoid the standard packages we 
ship with.)

One off-the-cuff example: falcon.jar might be a sports team skin (or 
bird-lover skin?), or it might be a flight simulator application.

As to 2) the installers have made no allowance for manifest.rdf to be anywhere 
but at the top of an archive, didn't know this was desired. I do at least 
handle chrome being in an archive, but if that's what I'm asked to register I 
just point to the top.  This would be fairly easy to change (but please not for 
beta2), I'll have to add another parameter to the install registerChrome() 
command. An install script would still have to call registerChrome() for each 
manifest.rdf telling me what chrome types to register but that seems 
reasonable. Better than grovelling through each archive searching for anything 
called manifest.rdf

Comment 6

[Warren Harris]

I forgot to mention that the other thing that was motivating this was that our 
current scheme _is not simple_. If you want to combine locales or skins, you 
have to wack manifest.rdf files. If you want to combine packages, you have to 
introduce subdirectories in the jar. Ugh. I can't imagine trying to document 
this.

I think my suggestion simplifies things substantially, making manifest.rdf 
files always appear in a consistent place in the jar file (at the 3rd level 
down). It also makes it trivial to change or combine packages, e.g. by the 
installer, or to enumerate what sorts of things a jar file contains. 

BTW, I don't care if the extension isn't ".jar". Also, jar packaging has moved 
to beta3.

Comment 7

[Peter Trudelle]

current structure is good enough, according to hyatt.

Comment 8

[leger]

Putting brackets in - from nsbeta3- to [nsbeta3-]

Comment 9

[Warren Harris]

I'm going to take this one.

Comment 10

[Daniel Veditz [:dveditz]]

Why the hell are you "taking" this one? Many smart people have commented 
that the current scheme works well enough, and in fact is far more flexible 
than you gave credit for at first. Is there really nothing more important on 
your plate? If not ask around, there are *TONS* more important bugs getting 
pushed off.

Comment 11

[Simon Fraser [no longer active]]

cc self for mac work

Comment 12

[Warren Harris]

part of jar packaging landing today

Comment 13

[Blake Ross]

Is there an easy way for QA to verify this?

Comment 14

[Blake Ross]

per my builds recently, I'm pretty sure this is fixed...QA assigning to warren 
to verify, though.

Comment 15

[Warren Harris]

verif

Comment 16

[Ari Pollak]

Erm. does this have anything to do with the 9/22 & 9/23 builds being rather
large because it includes both the raw & JARed chromes?

Comment 17

[Daniel Veditz [:dveditz]]

No, the latest builds are large because the build process produces both jars 
and flat chrome for different developers. The tarballs simply package up 
whatever the build produces, including extra cruft like test and debugging 
programs, even old dead samples.

The installer builds are smaller because those ship only what are listed in the 
package lists (Debian and Red Hat packages also use these lists, not just the 
mozilla.org installers).