194045 - empty chrome directories are left when building jar files

Status: RESOLVED FIXED

(SeaMonkey :: Build Config, defect)

Description

[Wenqing Jiang]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.3b) Gecko/20030217
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.3b) Gecko/20030217

Sorry I can't find a suitable component/target to report this problem.  So
I just pick 'browser'.

I found under the bin/ directory there're 538 files and 1013 directories.
(I got the mozilla by downloading a daily mozilla-win32.zip and unzip it.)
There must be a lot of empty directories under the bin directory.  Why are
they needed?  Isn't there any better arrangement of the directories?  For
example, I found that bin\chrome\classic\skin directory (and all its
subdirectoris) mirrored the directory structure in bin\chrome\classic.jar,
but the former are empty.  Are they really needed?

Reproducible: Always

Steps to Reproduce:
1.
2.
3.

Actual Results:  
too many unnecessary empty directories (causing many disk fragments, etc)


Expected Results:  
files/directories should be more efficiently organized

Comment 1

[Boris Zbarsky [:bzbarsky]]

Sounds like build config to me -- when building with jars we don't need to
create all those dirs....

Comment 2

[hacker formerly known as [e-mail]]

I thought we had a bug on this already but I can't find it.   If you look at
make-jars.pl, you'll see that we create the directory layout inside
$(DIST)/bin/chrome .  If we're using flat files, we'll leave them as is. If
we're using jars, we'll jar up the files at that point.  We cannot remove the
empty directories at that point because you could run into a race condition in
parallel builds where 2 jar.mn files are being processed that use the same
directory.

The directories would have to be removed as a post build step or we'd have to
stage the files somewhere other than the chrome directory when using jar files.

Comment 3

[Wenqing Jiang]

Why can't it be fixed in 1.4 alpha?  There should be nothing
blocking.

Comment 4

[hacker formerly known as [e-mail]]

Because it's a non-critical low-priority bug that I don't have time to work on
right now.  Feel free to contribute a patch if you want it in any sooner.

Comment 5

[hacker formerly known as [e-mail]]

*** Bug 199922 has been marked as a duplicate of this bug. ***

Comment 6

[Roland Mainz]

Christopher Seawood wrote:
> The directories would have to be removed as a post build step or we'd have to
> stage the files somewhere other than the chrome directory when using jar 
> files.

Remember my suggestion in bug 199922 to do a _simple_ % find dist/ -type d |
while read i ; do echo "## Trying to remove $i" ; rmdir "$i" ; done # as part of
the tar.gz build process in xpinstall/packager/ ? That will remove all empty
dirs from dist/ (/usr/bin/rmdir does not delete non-empty dirs) ...

Comment 7

[[e-mail]]

*** Bug 219363 has been marked as a duplicate of this bug. ***

Comment 8

[[e-mail]]

Attachment: Proposed patch

Should simply not storing the directories in the zip file resolve the issue?

Comment 9

[Daniel Veditz [:dveditz]]

Comment on attachment 131556 [details] [diff] [review]
Proposed patch

we should probably add -D ro make-jars.pl too, but that's a separate issue.

r+sr=dveditz

Comment 10

[[e-mail]]

Win32 fix checked in; leaving open for other platforms.

Comment 11

[Jeremy M. Dolan]

Just a quick note:

% find `find -type d|grep -v '^\.$'` -type f|sort|uniq
./icons/default/default.xpm
./overlayinfo/browser/content/overlays.rdf
./overlayinfo/communicator/content/overlays.rdf
./overlayinfo/messenger/content/overlays.rdf
./overlayinfo/navigator/content/overlays.rdf

There are five non-jar files under chrome/ on linux firebird builds. If those
are in the jars as well, or not needed in any other way, that would make another
handful of directories that didn't need to be created.

Not entirely sure why this is a win32 only fix, but for what it's worth, InfoZIP
(what Red Hat uses) supports the -D flag as well. I've always been mildly
annoyed by all of those directories, so I'd like to see the patch make it to
Linux as well.

Comment 12

[[e-mail]]

Jeremy,
.xpm files are needed to display the window icon
overlays.rdf are generated dynamically (although due to be phased out :-)
IIRC only win32 uses .zip installers but correct me if I'm wrong.

Comment 13

[Jeremy M. Dolan]

Ah. Well, if UNIX builds are using GNU tar, --exclude might be an option:
  --exclude US --exclude browser --exclude classic --exclude embed-sample
etc.

There's also -X (exclude from a list in a file). But this would require a
seperate file to reside somewhere, instead of just a makefile change.

I don't fully understand the race condition in the build process, so I don't
know if that's any help. tar doesn't have a -D equivilent though.

Does whatever code that generates overlays.rdf know how to make the parent
directories if it needs to? (no reply necessary, just something to watch for)

Comment 14

[Steffen Wilberg]

Comment on attachment 131556 [details] [diff] [review]
Proposed patch

This should be fixed in the 1.5 branch as well: Don't store empty dirs in zip
packages.

Comment 15

[Asa Dotzler [:asa]]

Comment on attachment 131556 [details] [diff] [review]
Proposed patch

too late for 1.5.

Comment 16

[Martin Mokrejs]

I have seen this also on Tru64 Unix. I'm now hunting throug bugzilla for the
patch I've submitted to make-jars.pl. That perl code just sucks. For example, it
does not care if chdir() succeeds or not, it just blindly continues. The bug
efectively prevented people using mozilla on Tru64 unix, as the .jar file
screated contained no files. The major issue was that make-jars.pl ran mkpath on
wrong variable name, so the target directory was not created. But as I said, I'm
looking for the patch I wrote. The bug manifested also during make install,
because the target structure of chrome directories did not exist and the perl
code just broke it that point.

So, give this high prioritry and improve the perl code quality, or ask me for
new patch. ;)

Comment 17

[Martin Mokrejs]

So, I found it finally. Fix the bug
http://bugzilla.mozilla.org/show_bug.cgi?id=221563 and let's see how many bugs
you can close immediately.

Comment 18

[Steffen Wilberg]

*** Bug 210754 has been marked as a duplicate of this bug. ***

Comment 19

[Wenqing Jiang]

I tested Mozilla 1.7 RC1 for windows and found the problem gone.  Should
the bug be closed or the problem is not solved yet on other platforms?

Comment 20

[Daniel (Leaf) Nunes]

the empty dirs are still present for at least linux; windows is fixed with
somewhat of a workaround in the packagers scripts. I will probably just fix the
linux pacakaging the same way, unless there's a somewhat straightforward way to
fix the build rules when building chrome into .jar files rather than into flatfiles.

Comment 21

[Ryan VanderMeulen [:RyanVM]]

I'm seeing this bug in Windows XP SP2 still.

Comment 22

[Martin Mokrejs]

As I already said, it should get fixed by
http://bugzilla.mozilla.org/show_bug.cgi?id=221563 when someone commits the patch.

Comment 23

[Mark Mentovai]

Attachment: Stage jar files off to the side

This patches make-jars.pl to support a new -j option.  -j takes a directory
argument and tells the script where to place the .jar and .manifest files.  In
the absence of -j, -d is used, preserving backward compatibility.

The build system is updated to make intelligent use of this new option.  If
$(MOZ_CHROME_FILE_FORMAT) is jar, it sets up to use a staging directory,
$(DIST)/stage, or in the case of XPIs, $(DIST)/xpi-stage/$(XPI_NAME).stage. 
The staging directory is used as make-jars.pl's -d argument, the place to put
the .jar and .manifest is $(FINAL_TARGET), which I'm leaving unmolested.  If
$(MOZ_CHROME_FILE_FORMAT) is anything else, the existing behavior is fine.

I'm also pointing make-chromelist.pl to the staging directory, so that ONLY
.jar and .manifest files wind up in $(DIST)/bin/chrome.  chromelist.txt and
installed-chrome.txt stay in the staging directory.  They don't belong in the
app and are stripped at packaging time anyway, or so I think.  I don't think
this should be a problem.

Comment 24

[Mark Mentovai]

*** Bug 256616 has been marked as a duplicate of this bug. ***

Comment 25

[[e-mail]]

(In reply to comment #23)
>I'm also pointing make-chromelist.pl to the staging directory, so that ONLY
>.jar and .manifest files wind up in $(DIST)/bin/chrome.  chromelist.txt and
>installed-chrome.txt stay in the staging directory.  They don't belong in the
>app and are stripped at packaging time anyway, or so I think.
At least for SeaMonkey, installed-chrome.txt is required in $(DIST)/bin/chrome
(it doesn't use .manifest fiels) and chromelist.txt ought to be packaged too.

Comment 26

[Benjamin Smedberg]

Comment on attachment 188503 [details] [diff] [review]
Stage jar files off to the side

I don't mind this patch as a short-term solution: in the 1.9 cycle I actually
intend to stage all the chrome files "flat" and only zip them up at the end of
the build process. In any case, there are two problems with this patch:

1) please don't use $(DIST)/stage, which is used by the installer-packaging
process. Use a unique directory like $(DIST)/stage-chrome or somesuch.

2) the toolkit apps really shouldn't need installed-chrome.txt any more, but
seamonkey does. Please keep chromelist.txt and installed-chrome.txt in the
dist/bin/chrome directory.

Comment 27

[Mark Mentovai]

Attachment: Staging patch, v2

OK.  Changes in this patch:
 - Stage in $(DIST)/chrome-stage for non-xpi.
 - Leave installed-chrome.txt and chromelist.txt in $(FINAL_TARGET)/$jarDir,
   packagers will handle as needed.
 - Put chrome.manifest in the right place ($(FINAL_TARGET)/$jarDir) for
   extensions.

Any shot at 1.8b3 if the reviews are in?

Comment 28

[Benjamin Smedberg]

Comment on attachment 188533 [details] [diff] [review]
Staging patch, v2

Although this patch looks good in our current world, I worry about it if people
start overriding FINAL_TARGET directly instead of using XPI_NAME (which we
already do in extensions/spatialnavigation, but that shouldn't affect chrome
packaging). In any case, since I plan to change this early in 1.9 anyway let's
go with what's here. Not for 1.8b3, but I suppose we can do this for 1.8b4.

Comment 29

[Mark Mentovai]

Comment on attachment 188533 [details] [diff] [review]
Staging patch, v2

Instead of this:
+MAKE_JARS_TARGET = $(if
$(XPI_NAME),$(DIST)/xpi-stage/$(XPI_NAME).stage,$(DIST)/chrome-stage)

I'll use this:
+MAKE_JARS_TARGET = $(if
$(XPI_NAME),$(FINAL_TARGET).stage,$(DIST)/chrome-stage)

That covers bsmedberg's concern in the preceding comment.

Comment 30

[Daniel Veditz [:dveditz]]

Comment on attachment 188533 [details] [diff] [review]
Staging patch, v2

sr=dveditz

Comment 31

[Mark Mentovai]

Approval, anyone?  It'd be nice to have this for 1.8.

Comment 32

[Mark Mentovai]

Checked in.

Note that non-clobber builds will still have empty directories left behind from
before this patch landed.

Comment 33

[[e-mail]]

Comment on attachment 188533 [details] [diff] [review]
Staging patch, v2

>+if ($jarDir !~ /^\//) {
>+    $jarDir = getcwd() . '/' . $jarDir;
>+}
All very well, but my $zipprog doesn't understand cygwin paths, buster!

Comment 34

[Mark Mentovai]

(In reply to comment #33)
> (From update of attachment 188533 [details] [diff] [review] [edit])
> >+if ($jarDir !~ /^\//) {
> >+    $jarDir = getcwd() . '/' . $jarDir;
> >+}
> All very well, but my $zipprog doesn't understand cygwin paths, buster!

My impression of your $zipprog aside...

cygpath is obviously no good either.  What do you need for this to work?  Is it
enough for $jarDir to always be relative to $chromeDir/$jarfile?  If so, can I
rely on File::Spec on your platform, specifically, abs2rel and rel2abs?

Comment 35

[Mark Mentovai]

Attachment: Use relative path for $jarchive

Neil, how does this work?

Comment 36

[[e-mail]]

Attachment: Independently written approach

I wrote this version independently of Mark's patch, but it was only by
comparing the two that I was able to work out how Mark's patch worked ;-)

Comment 37

[[e-mail]]

Comment on attachment 190949 [details] [diff] [review]
Use relative path for $jarchive

I suppose this one is slightly neater because it computes rel2abs of $jarPath
only once, also only the implicit form of abs2rel is necessary.

Comment 38

[Mark Mentovai]

Comment on attachment 190949 [details] [diff] [review]
Use relative path for $jarchive

This fix is for bustage on cygwin builds when $zipprog does not understand
absolute cygwin paths.	None of the tinderboxen are affected.

Comment 39

[Benjamin Smedberg]

What are the relative merits of attachment 190949 [details] [diff] [review] and 191000 ? I'm having
trouble understanding either one, but if they work basically the same way I
prefer the "Independently written approach" because it avoids the use of
file::spec which has unfortunate portability problems.

Comment 40

[[e-mail]]

(In reply to comment #39)
>What are the relative merits of attachment 190949 [details] [diff] [review] and 191000 ?
Attachment 190949 [details] [diff] is like attachment 191000 [details] [diff] [review] but inlining _moz_abs2rel;
the first call to _moz_rel2abs is moved out of the loop to the top level;
the second call to _moz_rel2abs is unnecessary because of the chdir().

Comment 41

[[e-mail]]

Attachment: Another go

This version precomputes the relative path from the staging to the chrome dir,
although I had to throw in an extra .. because I couldn't find any other way to
account for the fact that chrome is built in subdirs for the staging dir.

Comment 42

[Mark Mentovai]

bsmedberg, all three approaches so far rely on File::Spec, as do other parts of
the make-jars script.

Comment 43

[[e-mail]]

Comment on attachment 191233 [details] [diff] [review]
Another go

This is no good; I hadn't tested it properly.

Comment 44

[Benjamin Smedberg]

Comment on attachment 190949 [details] [diff] [review]
Use relative path for $jarchive

ok. Please be prepared to back this out if it causes build system problems.

Comment 45

[Mark Mentovai]

Committed, watching the tree.

Comment 46

[Mark Mentovai]

Backed out.  This made btek go red, but everything else seems fine.  btek may
have an out-of-date File::Spec, it's giving this:

Can't locate object method "rel2abs" via package "File::Spec" at
../../config/make-jars.pl line 293.

Even without the patch, make-jars.pl gives:

Unknown option: -

That's somewhat alarming, but not specific enough to be helpful.  Can someone
look at btek?

Comment 47

[Mark Mentovai]

btek is using perl 5.005_03 (1999).  The Mozilla minimum is 5.004 (1997) per
$PERL_VERSION in configure.in.

5.005_03 contains File::Spec 0.6, which does not have abs2rel or rel2abs
methods.  Those were introduced in File::Spec 0.8, which shipped with 5.6.0
(2000) and the late-shipping 5.005_04 (2004).  As it stands, make-jars.pl relies
on these methods when the symlink chrome format is in use, making the build
require either these perl versions or an updated File::Spec module to package
chrome in this manner.  Furthermore, make-jars.pl unconditionally requires
File::Spec even when none of its methods are used.  File::Spec only shipped with
5.005 (1998) and the late-shipping 5.004_05 (1999), so arguably, the minimum
required perl version should be bumped up.  I'm assuming that support for 5.005
is still desired, so that still doesn't solve this problem.

Since it seems to me that everything relies on Unix-style paths, we can roll our
own rel2abs and abs2rel and drop the File::Spec dependency altogether, making
make-jars.pl work even with an out-of-the-box perl 5.004.

The "unknown option" messages are a result of a Getopt::Std module that does not
understand -- to mean end-of-options.  1.01 doesn't understand it, 1.02
(introduced in perl 5.6.0) does.  make-jars.pl contains a workaround for this
problem, but is unable to prevent the warning from being printed.

Comment 48

[Benjamin Smedberg]

There was previously opposition to bumping the minimum required perl version
(from dbaron IIRC)... I don't remember which bug it came up in, I think it had
to do with preprocessor.pl

Comment 49

[Mark Mentovai]

The point is that the dependency on File::Spec makes the minimum version bogus
already.  make-jars.pl won't even run on perl < 5.004_05 since make-jars.pl 3.61
(2003) unless File::Spec is separately installed.

If we roll our own rel2abs and abs2rel, we can leave the minimum perl version
where it is.

Comment 50

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Upgrading btek is simply too complicated to happen, and it's really useful to us
for performance metrics.  I recall spending hours undoing leaf's attempt to
upgrade perl on btek a few years back because it broke something else.  How hard
is it to avoid requiring all these fancy new perl modules?  Remember that
tinderboxes aren't the only machines building Mozilla, and every additional
requirement could make it harder for people to build.

Comment 51

[[e-mail]]

(In reply to comment #47)
>btek is using perl 5.005_03 (1999).  The Mozilla minimum is 5.004 (1997) per
>$PERL_VERSION in configure.in.
> 
>5.005_03 contains File::Spec 0.6, which does not have abs2rel or rel2abs methods.
Do you know which methods it does have, or a link to the docs for version 0.6?

Comment 52

[Mark Mentovai]

(In reply to comment #51)
> Do you know which methods it does have, or a link to the docs for version 0.6?

Teach a man to fish, etc.  http://www.cpan.org/modules/by-module/File/

Comment 53

[Mark Mentovai]

(In reply to comment #50)
> Upgrading btek is simply too complicated to happen, and it's really useful to
> us for performance metrics.  I recall spending hours undoing leaf's attempt
> to upgrade perl on btek a few years back because it broke something else.
> How hard is it to avoid requiring all these fancy new perl modules?

Not hard - the functions in question aren't intractable.  At some point, the
question becomes "do I really need to reinvent this just to support
five-year-old installations?"  I don't think we're past that point with 5.005 yet.

> Remember
> that tinderboxes aren't the only machines building Mozilla, and every
> additional requirement could make it harder for people to build.

dbaron, if nobody noticed that the build was broken with a standard installation
of pre-5.00405 so far, I'd strongly suspect that it's not an issue, and hasn't
been since 2003.  This is the only reason I mentioned bumping $PERL_VERSION. 
Nobody's suggesting obsoleting 5.005.

Mark

Comment 54

[Boris Zbarsky [:bzbarsky]]

Mark, see bug 227078  

Comment 55

[Bastiaan Jacques]

Comment on attachment 188533 [details] [diff] [review]
Staging patch, v2

Please note that, at least on GNU/Linux, this patch causes files in
dist/chrome-stage to be created by the superuser during `make install'. I guess
fixing that issue can wait until 1.9, though.

Comment 56

[Ere Maijala (slow)]

Comment on attachment 190949 [details] [diff] [review]
Use relative path for $jarchive

Could you check this in, please? My builds don't work at all without this
patch.

Comment 57

[[e-mail]]

Attachment: Yet another approach

ere, does this work for you?

Comment 58

[Ere Maijala (slow)]

Comment on attachment 192688 [details] [diff] [review]
Yet another approach

Yes, this would work too.

Comment 59

[Mark Mentovai]

Comment on attachment 192688 [details] [diff] [review]
Yet another approach

+      splice(@dirs, $_, 2) if $dirs[$_] eq "..";

That's not right.  It splices ../child out of @dirs, but it should splice
parent/.. , or not splice at all if no parent.

Comment 60

[[e-mail]]

Attachment: Bug fixed

Some missing reverses in splitpath were causing an infinite loop.

Comment 61

[Mark Mentovai]

Comment on attachment 192937 [details] [diff] [review]
Bug fixed

This _moz_splitpath splices .. out even if there's no parent directory, so if
you pass in ../../../file, you'll get back only file.

Comment 62

[Benjamin Smedberg]

Comment on attachment 192937 [details] [diff] [review]
Bug fixed

I defer to Mark on this.

Comment 63

[[e-mail]]

So should I rel2abs first, or just ignore ..s? (both paths begin with $DIRS)

Comment 64

[Mark Mentovai]

(In reply to comment #63)
> So should I rel2abs first, or just ignore ..s? (both paths begin with $DIRS)

rel2abs will bust btek and maybe others.  That's what I tried before, comment 47.

We're probably safe in this case for now if we ignore .., but it's not a great
general-purpose solution, since there's not really a guarantee that $destPath
and $jarPath will always be as they are.

What if we implemented our own proper rel2abs and abs2rel and eliminate the
dependency on File::Spec altogether?  The "symlink" format would burn on old
perls as it is.

Anyway, I think we can solve this particular problem with the splitter.  Say,
walk the array of "in" path components in unreversed order, push the component
to the "out" array if it's not . or .., and pop the "out" array if it's .. and
there isn't a .. at the top.

Be careful to do something rational with absolute paths that begins with /.. .
Double slashes in paths, like directory//file, are also possible targets for
canonization.

If you don't want to write this, I can do it.

Comment 65

[[e-mail]]

Attachment: Ignore dots

Comment 66

[[e-mail]]

Attachment: Provisional enhanced version

Comment 67

[Mark Mentovai]

Comment on attachment 192943 [details] [diff] [review]
Ignore dots

I'm a little worried about what happens if destPath has dots in it unmatched in
jarPath, or if destPath is absolute and jarPath is relative.  That should never
happen in our builds now, though.

Comment 68

[Ere Maijala (slow)]

Comment on attachment 192688 [details] [diff] [review]
Yet another approach

I was also a bit too quick to declare a winner. This one hung up later (or
started packing the world into a jar).

Comment 69

[Peter van der Woude [:Peter6]]

from Moz forums:
http://forums.mozillazine.org/viewtopic.php?t=306371&postdays=0&postorder=asc&postsperpage=15&start=105

[quote from AmboyGuy]

If you click on the C of the L/C in the pacifica tinderbox (yellow 6hr one) and
look at the file make.jars.pl (ver 3.86 )
Scroll down to the "skip to line 284" area there appears to be an extra closing
bracket that may have borked the build.

When editing the author may have added an extra closing bracket with no matching
open bracket. ( forgot to delete the existing close bracket ? Those type of
things are damn hard to spot.)

[/quote]

Comment 70

[Mark Mentovai]

3.85, not 3.86.  Braces, brackets, and parentheses look OK to me.  The
interpreter would have kicked out if they had been a problem and the builds
would have gone red instead of entering an infinite loop.

Comment 71

[[e-mail]]

Comment on attachment 192948 [details] [diff] [review]
Provisional enhanced version

This one has passed several cycles on my NT tinderbox.

Comment 72

[Mark Mentovai]

Comment on attachment 192948 [details] [diff] [review]
Provisional enhanced version

I'm good with this one.  $objdir is a little confusing.  Either change its name
to $gObjdir, make it an argument to rel2abs, replace it with a getcwd call
right in that sub, or add a comment in the sub.  (getcwd is not expensive, but
any of these options are fine; personally, I'd make it an arg and fall back to
getcwd if the arg is undefined.)

Note that there are a few minor changes between File::Spec 0.6 and 0.8 in the
subs you're using.  Most of them look benign, but a cygwin-only case was added
in canonpath and I don't know enough about cygwin to tell if it's significant
or not.  As long as you've glanced at 0.6 too and think it's OK, and you make
one of the above changes, r=mark.

Comment 73

[[e-mail]]

Attachment: Addressed review comments

I didn't want to rename $objdir because there are other uses of it.
I couldn't use an optional argument because I'm already using one.
So I just made rel2abs use getcwd instead of $objdir.

Comment 74

[Ere Maijala (slow)]

Comment on attachment 193042 [details] [diff] [review]
Addressed review comments

r- from me as this one doesn't solve the problem. "zip error: Could not create
output file (......bin/chrome/comm.jar)".

Comment 75

[[e-mail]]

Attachment: More eyes please

Comment 76

[Ere Maijala (slow)]

Comment on attachment 193145 [details] [diff] [review]
More eyes please

Works for me. r=emaijala

Comment 77

[[e-mail]]

Comment on attachment 193145 [details] [diff] [review]
More eyes please

ere's going to need to be able to build branch too!

Comment 78

[[e-mail]]

Latest patch seems to be working on trunk :-)

Comment 79

[Mark Mentovai]

Busted seamonkey/creature:

zip error: Could not create output file
(..\C:\builds\tinderbox\MozillaTrunk\WINNT_5.0_Clobber\mozilla\netwerk\resources\..\..\dist\bin\chrome\comm.jar)

http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1124919780.17726.gz&fulltext=1

Comment 80

[Mark Mentovai]

Attachment 193145 [details] [diff] backed out

Comment 81

[[e-mail]]

Attachment: Tested with AS Perl

Basically I just added a line to convert C:\ to c:/ style paths.

Comment 82

[[e-mail]]

Third time's the charm ;-)

Comment 83

[Mark Mentovai]

Nice work, Neil!

Comment 84

[[e-mail]]

Comment on attachment 193808 [details] [diff] [review]
Tested with AS Perl

Ere's finally going to be able to build the branch!

Comment 85

[Ere Maijala (slow)]

Neil, your work is highly appreciated :)