101431 - Get jar.mn concatenation into the builds

Status: RESOLVED FIXED

(SeaMonkey :: Build Config, defect)

Description

[Gervase Markham [:gerv]]

I have a plan to allow people to contribute patches to the UI and other
non-compiled files without having to set up a build environment. There's stuff
in n.p.m.seamonkey about this:
news://news.mozilla.org:119/[e-mail]

As part of this, in order for patches to work, we need to ship a file with
Mozilla which gives the mappings from CVS file locations to jar file locations.
This is basically a concatenation of all instances of jar.mn with a little more
path information added.

There are a couple of quirks to this (platform-specific files) but it shouldn't
be too hard. My current idea (cls had an objection to part of this, but I can't
remember what it was):

Find all the jar.mn files in the tree you are building. (This does away with the
need for a master list of jar.mn files.) If this finds old and crufty jar.mn
files, that's fine.  There won't be a clash.

If the path to a jar.mn file contains the wrong ones for your platform of /win/
, /unix/, /os2/ or /mac/, then you don't add that one in.

You munge and concatenate all those files together and ship the result as
chrome/filelist.txt or similar. This can then be used by automated tools to
translate chrome paths into CVS paths.

There is the slight problem that some jar.mn files don't have the double format
for all or part of them, if the referenced file is in the same directory as the
jar.mn file, but the script should be able to cope with this.

I plan to write the correct perl script, and then coordinate with the build team
to try and get it made part of the build.

Gerv

Comment 1

[Gervase Markham [:gerv]]

OK, so here's a first draft of the script. I have no idea whether it will work
cross-platform - it uses Unix-style paths. Does Perl Do The Right Thing in these
cases? Comments and feedback requested :-)

The file it produces is about 161k, which compresses to 16k using gzip.

Gerv

Comment 2

[Gervase Markham [:gerv]]

Attachment: makelist.pl v.0.1

Comment 3

[Gervase Markham [:gerv]]

cls: a reasonable first step would be to make the changes required to include a
chromelist.txt file in the nightlies, but make it so that it gets that file
straight from CVS. I could then check in a chromelist.txt I generated, and
people could use that until we have a cross-platform generating script. How does
that sound?

New version of makelist.pl coming.

Gerv

Comment 4

[Gervase Markham [:gerv]]

Attachment: makelist.pl v.0.2.

Comment 5

[Gervase Markham [:gerv]]

Attachment: Patch v.1 - incorporate making chromelist.txt into Windows and Unix builds

Comment 6

[Gervase Markham [:gerv]]

This patch adds the making of chromelist.txt into Windows and Unix builds.
Because Patch Maker currently doesn't work on the Mac, there's not much point in
adding it to the Mac build system yet (although I've talked to smfr and I have
an idea as to how to do that.)

Looking for review from cls.

Gerv

Comment 7

[Daniel (Leaf) Nunes]

Attachment: build system change to flush the chromelist.txt file if we build from the top. side-effect is that anyone building in config/ is going to whack their chromelist.txt file.

Comment 8

[Daniel (Leaf) Nunes]

Comment on attachment 52602 [details] [diff] [review]
Patch v.1 - incorporate making chromelist.txt into Windows and Unix builds

gerv's changes look good to me; gerv, you want to verify that my change does what you want it to (removal of the list when building from the top, and, therefore, building in mozilla/config)?

Comment 9

[Gervase Markham [:gerv]]

Looks good to me. So... can I check this in?

Gerv

Comment 10

[Daniel (Leaf) Nunes]

i'd rather get at least the once-over from cls before committing.

Comment 11

[Stuart Ballard]

The chromelist.txt as distributed from mozilla.org/hacking/patch-maker doesn't
contain tabbrowser.xml. Does that indicate a problem with makelist.pl, or is
there some genuine reason why tabbrowser.xml (and presumably some other files
but I'm not sure which) can't be edited with patchmaker?

I don't think the distributed chromelist file can be simply outdated, because it
contains linkToolbar* which was added *after* the tab browser landed.

Comment 12

[Gervase Markham [:gerv]]

Yeah, but I had linktoolbar in my tree before then :-) I've updated the
chromelist.txt files on the webserver; just grab a new one.

Gerv

Comment 13

[Stuart Ballard]

Thanks, that worked :)

Comment 14

[hacker formerly known as [e-mail]]

make-list.pl is too generic.  It should be make-chromelist.pl or
make-chrome-list.pl.  When I run it on my RH6.2 system, I get this error:

Can't locate File/Spec/Functions.pm in @INC (@INC contains:
../../../mozilla/config /usr/lib/perl5/5.00503/i386-linux /usr/lib/perl5/5.00503
/usr/lib/perl5/site_perl/5.005/i386-linux /usr/lib/perl5/site_perl/5.005 .) at
../../../mozilla/config/make-list.pl line 49.
BEGIN failed--compilation aborted at ../../../mozilla/config/make-list.pl line 49.

I'm guessing that File::Spec::Functions isn't a standard perl5.004+ module.

From the File::Spec man page:

       Since File::Spec is object oriented, subroutines should
       not called directly, as in:

               File::Spec::catfile('a','b');

       but rather as class methods:

               File::Spec->catfile('a','b');


Removing the 'use File::Spec::Functions;' and making the above change seems to
work for me.

Comment 15

[Gervase Markham [:gerv]]

Attachment: Patch v.2

Comment 16

[Gervase Markham [:gerv]]

Attachment: make-chromelist.pl v.0.5

Comment 17

[Gervase Markham [:gerv]]

Chris - thanks for the feedback. Are these new patches acceptable?

Gerv

Comment 18

[hacker formerly known as [e-mail]]

The perl script still contains a File::Spec::Unix call.  I'm not sure if that's
going to cause problems on win32 or not.  r=cls if you can verify that the
script works as it should on win32 (I can't test that right now).

Comment 19

[Gervase Markham [:gerv]]

Tests on IRC indicate that, on Windows, "use File::Spec::Unix" works, whereas
"use File::Spec::foo" does not. Therefore, it is available.

I'll check this in when the checkin frequency is lower, just in case.

Gerv

Comment 20

[Gervase Markham [:gerv]]

OK, so I broke IRIX. This is somewhat surprising, as http://www.perldoc.com
indicates that File::Spec is available on all Perl version from 5.004 through
5.004_05 upwards.

So, what happens next? Is it my responsibility to work around the quirks of
IRIX's copy of Perl? Do we have a minimum Perl version we require, and is IRIX
meeting it? Can we check this back in, #ifdefed out for IRIX?

And why were the Windows build changes backed out anyway? :-)

Gerv

Comment 21

[hacker formerly known as [e-mail]]

Sorry, it was late, I was tired and I couldn't find you after things went red.
5.004 is the minimum version that we require and is the version installed on
cement.  If there's a problem with that particular installation, then we can
pull the tb and fix it.  I thought that the feature wasn't supported in 5.004 at
all (perldoc.com is news to me).  Sorry about that.

Comment 22

[Gervase Markham [:gerv]]

Hey, no problem :-) I admire you for keeping an eye on the tree at 2am. So,
would you like me to check this stuff back in when the tree opens, or do you
want me to wait until the IRIX Perl question gets investigated?

Gerv

Comment 23

[hacker formerly known as [e-mail]]

I'll check it back in tonight.  I'm going to rebuild perl on cement since it's
using a NFS/IS installed version so who knows what's wrong with it. ;)  Sure
enough, it doesn't have File/Spec.pm .

Comment 24

[hacker formerly known as [e-mail]]

I think http://www.perldoc.com is on crack.  dec1 just went red because of the
same issue so I downloaded perl5.004 & 5.004_05.  File/Spec was added in
5.004_05 not 5.004.  So either the script needs to be rewritten to not use
File::Spec or we need to bump our minimum perl rev to 5.005 and upgrade the
tinderboxes.  Backing out the rules.mk change again.

Comment 25

[Gervase Markham [:gerv]]

<sigh>. What File::Spec gives us is a platform-independent way of manipulating
path information (catdir(), catfile() etc.). I would expect it to be already in
use, except I doubt we have much cross-platform build system Perl code. ;-)

Would it be inconvenient to bump the minimum required Perl version? If and when
build systems converge, we may well hit this problem again. I don't know.

Gerv

Comment 26

[Gervase Markham [:gerv]]

Attachment: Re-enable on Unix with test for File::Spec

Comment 27

[Gervase Markham [:gerv]]

Attachment: Add it to Win and Unix packages as well

Comment 28

[Gervase Markham [:gerv]]

I have r=cls from IRC.

Gerv

Comment 29

[Gervase Markham [:gerv]]

Fixed.

Gerv