135533 - support preprocessing / compiling of XUL files

Status: RESOLVED FIXED

(SeaMonkey :: Build Config, enhancement)

Description

[Brian Ryner (not reading)]

I've written a simple XUL preprocessor script that allows you to do:

<!--include src="foo.xul"-->

to insert XUL fragments at build time.  To cause a XUL file to be preprocessed,
you just use a "*" next to the filename in jar.mn (similar to the way a + causes
it to override).

Comment 1

[Brian Ryner (not reading)]

Attachment: patch

Comment 2

[Hixie (not reading bugmail)]

I would recommend using <?include ... ?> instead of <!--include ... -->. At
least it gives us the _impression_ of being XMLish... ;-)

Comment 3

[Brian Ryner (not reading)]

Attachment: patch #2

change to hixie's syntax suggestion

Comment 4

[Christian :Biesinger (don't email me, ping me on IRC)]

+    if (/^<!\?include src="(.*)"\?>$/) {

Are you sure the ! should be there?

Comment 5

[Brian Ryner (not reading)]

er, yeah, it shouldn't be

Comment 6

[David Dabbs]

If this hasn't been implemented yet you might consider using the W3C XML
Inclusions Rec. See http://www.w3.org/TR/xinclude/

For example

<foo xmlns:x="uri1">
  <xi:include href="common.xml#xpointer(a/b)"
              xmlns:xi="http://www.w3.org/2001/XInclude"/>
</foo>

This is as XML-like as one could probably get and could allow reusing 
fragments within existing documents (if the XPointer fragment URI syntax 
is supported).

Comment 7

[(not reading, please use [e-mail] instead)]

bryner and I were talking, and we want to take this a step further.

if we can, at build time, resolve what overlays do at run time we could have a
big win.

basically I want to be able to use overlays in my xul, but then when building my
jar, I want them to be merged so that the result xul, is just a .xul file, with
all the xul (and js and css?) in the file.

it might make the .jar bigger, but it should solve the slowness we see from
having overlays, without making the .xul in the tree hard to maintain.

(this even allows me to compile the ns tree!)

bryner takes this one step further:  can we precompile a fastload file?

if we had both the precompiled xul (we'd need it, if we ever had to rebuild the
fastload file) and the pre built fastload, we might have significant
improvements on the first start up.

Comment 8

[(not reading, please use [e-mail] instead)]

as proof of concept, I'm going to try "compiling" the mail compose window by
hand on my branch, and then comparing compiled vs. non-compiled.

Comment 9

[(not reading, please use [e-mail] instead)]

we should be paying for overlays once, on the build machines, instead of at run
time on all machines, right?

Comment 10

[Brendan Eich [:brendan]]

Who cares about first startup?  FastLoad files need to be recomputed on the fly
anyway, and first startup after install can be ~10-20% slower (or whatever the
reciprocal fastload speedup is -- I recall 16% gain on slow machines).

Another "compiling" idea: generate from a set of master and overlay .xul docs
and their associated style sheets a C++ file that calls into the content library
to create instances and append children.  Compile this file at build time and
link it into your app.  We could even arrange things so that if someone touches
any of the .xul files, the compiled code is bypassed in favor of parsing (and
generating a fastload file, of course).

/be

Comment 11

[selmer (gone)]

> we should be paying for overlays once, on the build machines, instead of at run
> time on all machines, right?

At worst, this stuff should be done at install time.  I believe the original
intent was to allow people to drop packages onto the installation and have the
right things happen.  I don't think we're really taking advantage of that
capability in the commercial releases at all, so build time is fine.

---

If we could compile to source, why would we ship the corresponding XUL files? 
It seems like the only valid end-user method for changing XUL files would be to
do some sort of install.  But, if they've installed they should be getting the
new compiled XUL...

Comment 12

[Brendan Eich [:brendan]]

Someone should measure to see that the scapegoat is really to blame.  There have
also been claims that dumb use of overlays, not the overlay mechanism itself,
have added cookies to Mozilla's performance waistline.

selmer, if we could compile hardcoded UI and be happy with it until the next
release, there'd be no reason to ship .xul files.  Themers might not be happy,
and they might even rather remove the hardcoded-UI DLL/DSO, if possible.  I'm
suggesting that we not compile away the freedom to update app UI via just XUL
(and CSS, etc.), at least not yet.

/be

Comment 13

[selmer (gone)]

Hmm.  What if the themer could install their own DLL in place of ours?  (OK,
actually we'd need to be able to switch between the DLLs dynamically.)  

Maybe it's good enough to have the new theme bring all the XUL files and the
install sets a flag that makes us look for them?  (I'm trying to optimize so
that the majority of users see the best possible performance.  Right now,
theming is not being done by large numbers of non-NSCP people.)  

Anyway, I'm just asking.  If we get nearly the same performance then either way
is fine.

Comment 14

[(not reading, please use [e-mail] instead)]

I'm doing some this overlay collapsing manually on a branch.

stephend's going to get some numbers.

that will help convince us that we really do want XUL compiling.

Comment 15

[John Morrison]

Try to measure the memory use as well. i.e., by 'inlining' overlays, we removed
the opportunity for sharing in the xul cache.

Comment 16

[Brian Ryner (not reading)]

*** This bug has been marked as a duplicate of 170737 ***

Comment 17

[Jon Granrose]

v dup.

Comment 18

[selmer (gone)]

This bug is NOT a dup of 170737 since that bug has become a Phoenix only bug
with no current intent to do anything on the trunk.

Comment 19

[Benjamin Smedberg]

Since we have dropped Mac Classic, is there any reason not to enable Hixie's XUL
preprocessor on the trunk? I think all the other build platforms support it.

Comment 20

[Jan Varga [:janv]]

I think it's worth to enable it even before 1.5.
I'd need it in several places. Now I have to use overlays or navigator.platform
which is such a shame.
I talked to bryner about this, it seems there's only one minor issue.
Anyone building on RedHat 6.2 needs newer File:Spec package.
I also heard that mkaply was concerned that it would cause problems for
localization.
Mike, can you provide more details ?
Thanks

Comment 21

[Mike Kaply [:mkaply]]

Currently for platform specific translation, we provide three files in Mozilla,
en-win.jar, en-mac.jar, en-unix.jar.

These are translated by all translation groups, and so we have translations for
all platforms.

If we preprocess DTD files such that the Linux build has a different DTD in the
final product then the Windows build, then when someone translates the Linux
build, there will be some missing entities in the DTD file so the translation
would not be usable for Windows.

We need to ensure that we either uses the same set of DTD files for Windows and
Linux or do something similar to base Mozilla where we have separate JAR files
that contain the files.

Preprocessing DTD files is not a good thing to do.

Comment 22

[Jan Varga [:janv]]

Well, I never thought we would do that (preprocess DTD files).
We need preprocessing mainly for XUL and XBL files.
FYI, preprocessing in Phoenix is not done for all files, only for ones that are
preceeded by '*' in the jar.mn.

Comment 23

[Mike Kaply [:mkaply]]

We preprocess DTD today.

http://lxr.mozilla.org/mozilla/source/browser/base/locale/browser.dtd#265

http://lxr.mozilla.org/mozilla/source/browser/components/prefwindow/locale/pref-navigator.dtd#22

Comment 24

[Jan Varga [:janv]]

Hmm, I was probably not clear about that.
I mean that it would be really good to support it even for mozilla1.4, that is,
on the new stable branch too.
I don't understand why we care about it, that they preprocess DTD files in Phoenix.
The XUL preprocessor is there, it just needs to be enabled (for files preceeded
by *).

Comment 25

[Mike Kaply [:mkaply]]

My point is that if we allow XUL preprocessing in the build, we should
explicitly disallow it for DTD files.

Preprocessing of DTD files has already created headaches in the Phoenix build.

Comment 26

[Jan Varga [:janv]]

I'm ok with that (to disable it for DTD files)

Comment 27

[Benjamin Smedberg]

Attachment: Enable preprocessing for seamonkey

KaiRo and I were talking about bug 154927, and how the .in solution to
localeVersion was rather fragile. Let's go ahead and turn on preprocessing for
seamonkey, and we can use it rather than .in... and begin to encourage use of
the preprocessor in seamonkey where it is appropriate.

Comment 28

[Brian Ryner (not reading)]

Comment on attachment 133587 [details] [diff] [review]
Enable preprocessing for seamonkey

r=bryner

Comment 29

[Benjamin Smedberg]

fixed-on-trunk. We should really update the build documentation somewhere, but I
don't really know where to do it...

Comment 30

[Christian :Biesinger (don't email me, ping me on IRC)]

bsmedberg: how about http://www.mozilla.org/build/jar-packaging.html