55106 - create regular builds for non-default components

Status: RESOLVED WONTFIX

(SeaMonkey :: Build Config, defect, P3)

Description

[Dan Mosedale (:dmosedale, :dmose)]

Peter Van der Beken wrote:

I've been looking into automating the packaging of optional components for
Mozilla into .xpi files. It doesn't look easy, but it should be doable.

1) We have to make a list similar to the packages-{mac,win,unix} in
mozilla/xpinstall/packager. This list is used by pkgcp.pl to create a
staging area.
2) We have to make make .jst files for every component to be installed. This
is used by the makexpi.pl script in mozilla/xpinstall/packager/{win,unix} to
generate the necesary install.js files needed by the installer.
3) We have to figure out how we want to build the .xpi's. We will need three
build machines, one for every platform that build the tree with the optional
components. A plus would be to use these machines as tinderboxes, so people
see when they break optional components. We will also need one of these
machines to package up the necessary .xpi's and push them to an ftp (http?)
server. The other two would push their staging area to the packaging server
who would build .xpi's for the three platforms.
4) We should have a script that generates an "Update Mozilla" page that
lists the optional components.

I think we might need two pages, one with nightlies and another with
"releases"? We could get some volunteers to provide the build machines?
The build scripts are currently not ready for this, they will need to be
modified and that will be the hard part. They are very biased to build just
the default tree.

Looking forward to your thoughts,

[...]

Actually, this is an idea I discussed with Dan in Frankfurt. I think Mozilla
should be providing .xpi files (and an "Update" page) for the optional
components that we have but are not part of the default build (or Netscape
6). The ones that come to mind are: MathML/SVG, Transformiix, Component
Viewer, the LDAP module and XML Extras. MathML/SVG will be the hardest
because the installer should replace the layout library, and so it should
only be done for milestones and releases (M18/NS 6.0). The others are all
drop-in components and could be done nightly for example.
I know how to build the xpi's for the drop-ins but I still need to automate
the process, I'm not planning on doing this by hand. I haven't figured out
how to do the version checking for the layout engine yet (if any of you has
information on this, please share).

Comment 1

[Dan Mosedale (:dmosedale, :dmose)]

I think leaf may have some machines that can do some of this work, and also has
insight into how this could fit into the regular build scheme, so I've started
off by assigning this to him for his comments.  I'm not sure that he's the right
one to own driving this to completion, though.

Thoughts?

Comment 2

[Daniel (Leaf) Nunes]

If the goal is to have mozilla.org machines producing and staging these xpis,
i'm the right person to set it up (because peter doesn't have access to those
machines).

Unfortunately, i haven't really made the release build process modular enough to
say ``drop your additions here''... so i'm going to have to do work to
incorporate whatever peter comes up with.

Comment 3

[Peter Van der Beken]

Okay, so I'll try to come up with a patch.
Leaf, could you explain to me how the installers get built currently? Does each 
different platform build it's own installers? Or do they push their staging area 
to one machine and does that machine build the xpi's then? The reason I'm asking 
is because I didn't find the scripts to build an installer for Macintosh in the 
tree.
Some of these components don't change any code in the default build, eg. 
component viewer, XML Extras, ... but some do eg. MathML. So would we get 
separate build machines for the non-default components?
Also, is there anything specific I should be looking out for that comes to mind, 
so that I can make it easy for you to integrate this?

Comment 4

[Axel Hecht]

Hi all,
some comments:
1) I would like to have the packages files with the components, eg
mozilla/extensions/transformiix/build/packages-transformiix instead of something
in mozilla/xpinstall/packages.
2) the jst files are not used by makexpi.pl, but makejs.pl. See my patch on
22062
3) AFAI figured, building an xpi should be:
a. pkgcp.pl -f some-package -d stagingpath
b. makejs.pl component.jst SOMEVERSION stagingpath builddir/component.js
c. makexpi.pl component stagingpath builddir

There are a few hickups here, at least if we wanna have this rather automated:
The most prominent one, those perl file don't like relative path names, so
having DIST=$(DEPTH)/dist in autoconf.mk.in is bad. If we move to DIST=@prefix@,
we get absolute pathnames for the dist dir.

added dveditz for installer magic, me, and dependency on 22062.

Axel

Comment 5

[Peter Van der Beken]

I have most of this working on Macintosh. Part of the solution comes from my 
patch for bug 56997 [Patch for building an installer on Macintosh]. I'll attach 
a patch, you'll also need http://bugzilla.mozilla.org/
showattachment.cgi?attach_id=17337 (mozilla/build/mac/MozInstaller.pm), http://
bugzilla.mozilla.org/showattachment.cgi?attach_id=17338 (mozilla/xpinstall/
packager/mac/makejs.pl) and http://bugzilla.mozilla.org/
showattachment.cgi?attach_id=17339 (mozilla/xpinstall/packager/mac/makexpi.pl).

The scripts build XPIs in :mozilla:dist_installer:xpi. I have the necessary 
package and jst files for Component viewer, LDAP, XMLExtras, MathML and 
Transformiix on Macintosh. Shall I attach them too (> 10 files)?
In the patch, I assumed that the package files will be distributed over the 
tree, instead of all being in mozilla/xpinstall/packager (see bug 22062). I put 
them in a package/mac directory, named them 'packages' and put the corresponding 
jst file next to them. (eg. mozilla/extensions/cview/package/mac/packages and 
mozilla/extensions/cview/package/mac/cview.jst). I don't know how the XPInstall 
people were planning to do this, dveditz hasn't replied (yet?) to my e-mail 
asking for more info.
I'll try to get to the other platforms (win/linux) after this, though my 
makefile knowledge is only superficial. Anyone wants to help out?

Comment 6

[Peter Van der Beken]

Attachment: Patch to Mac build system

Comment 7

[Daniel (Leaf) Nunes]

cc'ing mac build system experts for r= of patch

Comment 8

[Dan Mosedale (:dmosedale, :dmose)]

Attaching the packages and jst files here might be a good idea, since it would
allow for easy review before checkin.  I think you're right, though, that these
files want to be spread out across the tree, so we should probably wait for
dveditz' thoughts on that.  I can't help out at this moment, as I'm drowning in
other work at the moment.  But I hope to be able to do the road a bit...

Comment 9

[Jon Granrose]

target=future.

Comment 10

[Jon Granrose]

this is build config.

Comment 11

[Dave Miller [:justdave]]

default owners so they know it's been moved?

Comment 12

[Jon Granrose]

back to leaf.  cls isn't going to work on this.

Comment 13

[Dave Miller [:justdave]]

ok, sorry about that.  I see bugs get moved between components without
reassigning a lot (and usually they meant to and overlooked it) so I have a hair
trigger on that :-)

Comment 14

[Peter Van der Beken [:peterv]]

Comment on attachment 17342 [details] [diff] [review]
Patch to Mac build system

CFM is dead.

Comment 15

[Chase Phillips]

Mass reassign of open bugs for [e-mail] to [e-mail].

Comment 16

[Benjamin Smedberg]

Fixed by bug 286147

Comment 17

[Dan Mosedale (:dmosedale, :dmose)]

Re-opening; while it may now be possible to do this work thanks to the fixing of bug 286147, I don't believe that there are yet machines churning out such builds.  If I'm wrong, feel free to re-close.

Comment 18

[Benjamin Smedberg]

What optional components are we talking about, then?

Comment 19

[J. Paul Reed [:preed]]

Mass re-assign of bugs that aren't on the build team radar, so bugs assigned to [e-mail] reflects reality.

If there is a bug you really think we need to be looking at, please *email* [e-mail] with a bug number and explanation.

Comment 20

[Robert Kaiser]

We have no plans on doing this for SeaMonkey. If you want something for the platform or so, please file a new bug that makes really clear what it is you want, and updated to what this really means nowadays. There's not much in this bug that makes sense with the current state of affairs without completely reformulating the actual goals.