119589 - [mach] Allow a build with cw8 commandline tools

Status: RESOLVED WONTFIX

(Firefox Build System :: General, enhancement)

Description

[Zach Lipton [:zach]]

Because the codewarrior 7 compiler creates faster/tighter/etc code, it 
should be possible to use the commandline compiler and linker included 
with cw7 to build mach-o (while still allowing gcc to be used to eliminate 
the requirement for cw7). I'll work on this when I get a chance.

Comment 1

[Jon Granrose]

qa to jj

Comment 2

[James Turner]

An additional request that would make my life much easier: according to some
Metrowerks people I've spoken too, the CW8 command line tools will support CFM
code gen too. So it would be *really* helpful if this could be setup with an
--enable-cfm to produce a CFM binary. I may have to support Mac OS-9, but I
don't have to run it :-) (and yes, I'm going to be needing a CFM carbon build of
Mozilla for at least 2 years we reckon, schools are really slow to upgrade...)

Comment 3

[hacker formerly known as [e-mail]]

Taking. Bumping req to CW8 as I had link issues with CW7 & CW7.2.
 

Comment 4

[hacker formerly known as [e-mail]]

Attachment: cw nspr v2 patch

This patch lets us build NSPR with the CW8 cmdline tools. A big thanks to
bnesse for testing the initial patch and providing the changes for the tests.  


To setup your build env, you need to
1) edit "/Applications/CodeWarrior Pro 8.1/For OS X Development/For Mac OS X
Command Line Tools/INTO ~"/tcshrc and set CWINSTALL to the location of your CW
installation.
2) add the following to your ~/.cshrc
setenv CWCMDTOOLS "/Applications/CodeWarrior Pro 8.1/For OS X Development/For
Ma
c OS X Command Line Tools/INTO ~"
setenv PATH "${PATH}:${CWCMDTOOLS}/bin"
source "${CWCMDTOOLS}/tcshrc"

To build NSPR, you need to pass CC,CXX & LD via the env:
env CC=mwccppc CXX=mwccppc LD=mwccppc ./configure

When running the runtests.ksh script, you must manually add ../dist/lib to
DYLD_LIBRARY_PATH.  Not all of the tests pass.

Comment 5

[hacker formerly known as [e-mail]]

Attachment: runtests.ksh stdout

Comment 6

[hacker formerly known as [e-mail]]

Attachment: NSPR_TEST_LOGFILE

Comment 7

[Brian Nesse (gone)]

If you build using the disk image I posted to the macdev server, the CC, CXX,
and LD env variables have been pre-added to the ${CWCMDTOOLS}/tcshrc file.

Comment 8

[hacker formerly known as [e-mail]]

Yeah, I saw that. :) I had to comment out those lines as I source the file by
default and they would cause me to build every autoconf'd package I download
with mwcc instead of gcc.  

Speaking of which, the output from mwcc & gcc should be binary compatible,
right?  I'm hitting an assertion when running xpidl and I'm wondering if I need
to compile a new version of libIDL & glib using mwcc instead of fink's gcc
compiled version.

Comment 9

[Wan-Teh Chang]

Comment on attachment 91796 [details]
runtests.ksh stdout

>cltsrv\c
>\t\t\tFAILED

We should find out why this fails.  This is an
important test.

>nameshm1\c
>\t\t\tFAILED

This test failure can be ignored for the Mozilla client.

>provider\c
>\t\t\tFAILED

This test failure should be looked at.

>prpollml\c
>\t\t\tFAILED

You may be running out of file descriptors.  The limit
on descriptors should be set to 512.  In csh you can
use 'limit descriptors 512' to do that.

>sema\c
>\t\t\tFAILED
>semaerr\c
>\t\t\tFAILED
>semaping\c
>\t\t\tFAILED

These test failures can be ignored for the Mozilla client.

>timemac\c
>\t\t\tFAILED
>timetest\c
>\t\t\tFAILED

These should be investigated.

>version\c
>\t\t\tFAILED

This (cross-platform) failure can be ignored.  The test
needs to be updated.

Overall it looks good.

Comment 10

[Steve Dagley]

re: mwcc and gcc binary compatability - I think that only applies to CW8's mwcc
and gcc3 (which Apple hasn't released yet) so I expect you would need to compile
new versions of libIDL & glib using mwcc

Comment 11

[hacker formerly known as [e-mail]]

Attachment: cw8 patch for glib-1.2.10

Here's the patch I needed to apply to get glib built with mwccppc.  The command
I used to compile is:

env CC='mwccppc -ext o -gccincludes' LD='mwldppc' CPPFLAGS='-DMACOSX'
CFLAGS=-O4 ./configure --prefix=/mw --disable-shared powerpc-rhapsody 

The same command (minus the unnecessary powerpc-rhapsody bit) works for
building the libIDL subdir of the ORBit 0.5.12 source tree.  Building all of
ORBit isn't necessary.	Make sure that you add /mw/bin to your path before
building libIDL so that it picks up the proper copy of glib.  xpidl runs fine
now.

Thanks to bryner for pointing out the '-ext o' option to get around libtool's
silliness.

Comment 12

[hacker formerly known as [e-mail]]

I'm stumped.  I hit some problems with xpidl returning large values such as
4294967296 when it was supposed to return 1.  Since the value being printed was
stored as a gint64, I suspected that it was an mwcc issue with printing long
long.  After much further testing, it appears to be an alignment issue with the
union inside struct _IDL_tree_node .  If I build glib & libIDL with |-align
arraymembers| , that problem goes away.  However, I hit another xpidl issue in
xpconnect:

../../../../../dist/bin/xpidl -m header -w -I ../../../../../dist/idl
-I../../../../../../mozilla/js/src/xpconnect/tests/idl -o _xpidlgen/xpctest
../../../../../../mozilla/js/src/xpconnect/tests/idl/xpctest.idl
xpctest.idl:161: Error: unsigned const declaration 'big' initialized with
negative constant

The test that's it's failing is at
http://lxr.mozilla.org/seamonkey/source/xpcom/typelib/xpidl/xpidl_util.c#185 &
http://lxr.mozilla.org/seamonkey/source/js/src/xpconnect/tests/idl/xpctest.idl#161
:

        if (!IDL_TYPE_INTEGER(real_type).f_signed &&
            IDL_INTEGER(dcl->const_exp).value < 0)
 
Again, this looks like an issue with unions.

Comment 13

[hacker formerly known as [e-mail]]

Attachment: moz cw8 patch v1

Here's the patch that got me building up to xpconnect. Things to note:

* There's some general clean up in the patch like making the setting of rtti &
exception handling flags less gcc-specific.
* We need to set HOST_CFLAGS so that jscpucfg will compile.
* TK_CFLAGS & nsSpecialSystemDirectory.h:

Using the FlatCarbon headers with mwccppc doesn't work.  Every FlatCarbon
header just includes a catch all header like <CoreServices/CoreServices.h>. 
This header includes the subsequent headers in the wrong order for mwccppc. 
Even though MacTypes.h is first in our files, CoreServices.h, through as series
of all inclusive framework includes, includes CFBase.h, which uses types
defined in MacTypes.h, before MacTypes.h is actually included.	The -F
framework flag is accepted but doesn't appear to change the include path.

To get around this, I used the recursive include path flag -ir to make it find
the right version of a particular header.  The problem is that this method is
*slow*.  Building in dbm/src/, which uses none of the Carbon headers, takes 5.5
secs without TK_CFLAGS.  It takes 260.91 secs with TK_CFLAGS.

Because of that horrendous slowdown, I'm trying to avoid using TK_CFLAGS unless
absolutely necessary.  Unfortunately, this lead to the hack in
nsSpecialSystemDirectory.h.  I defined FOUR_CHAR_CODE, FourCharCode & OSType to
avoid having to pull in TK_CFLAGS whenever building against xpcom.

* xptcall
CW doesn't come with a standalone assembler.  Cursory googling shows that you
are expected to use inline asm.  Using apple's as allowed me to compile but
TestXPTCInvoke failed.	The .s file will need to be converted into inline asm
that mwccppc can grok.

Comment 14

[Brian Ryner (not reading)]

Chris, take a look at the patch which (hopefully) is in
/sw/fink/dists/stable/main/finkinfo/gnome/orbit-0.5.12-1.patch.  In particular,
the patch to lexer.l (the patch to compat.c may be of interest too, but I'm not
sure what it does).

Comment 15

[hacker formerly known as [e-mail]]

Attachment: moz cw8 patch v2

Thanks bryner.	That did the trick.  With fink's orbit patch, the -align flags
are not needed. 

With this new patch, I can compile all of mozilla except ldap & nss.  That
recursive include flag was *really* slowing things down so I dug through the
frameworks headers and found a better solution.  Defining
__CF_USE_FRAMEWORK_INCLUDES__ allows us to include the proper MacTypes.h header
so that the basic types are not undefined.  Defining __NOEXTENSIONS__ allows us
to work around relation() being undefined (somehow related to CW's <cmath>).  I
have no idea what else though defines do but it's probably mentioned in some
obscure bit of documentation.  Anyway, that hack of copying defines from the
system headers is gone.

Bryner reminded me that CW8 is supposed to be binary compatible with gcc3 so I
was able to get TestXPTCInvoke working by compiling the asm file with gcc and
setting the HAVE_GCC3_ABI define.  It's an ugly hack and I'm not sure if it
fully works as TestXPC & xpcshell still fail.

Apparently, mwccppc's -gccincludes is order dependent so it must come before
any include paths are added in order for it to work properly.  The easiest way
to do this is to just append the flag to CC & CXX.

Comment 16

[hacker formerly known as [e-mail]]

Attachment: ldap cw8 patch v1

Comment 17

[hacker formerly known as [e-mail]]

Attachment: nss cw8 patch v1

Here's a patch to get NSS building.  --enable-crypto still breaks though. I'm
hitting some weird link error in mailnews/extensions/smime/build :

mwccppc -gccincludes  -RTTI off -fno-handle-exceptions
-I/Developer/Headers/FlatCarbon -D__CF_USE_FRAMEWORK_INCLUDES__
-D__NOEXTENSIONS__  -DDEBUG -D_DEBUG -DDEBUG_cls -DTRACING -g -fpic -xm
sharedlibrary -o libmsgsmime.dylib  nsMsgSMIMEFactory.o  nsMsgComposeSecure.o
nsSMimeJSHelper.o nsEncryptedSMIMEURIsService.o 	 
-L../../../../dist/bin -L../../../../dist/lib -lmsgbaseutil
-L../../../../dist/bin -lmozjs -L../../../../dist/bin -lxpcom
-L../../../../dist/bin -L/Users/cls/src/moz/main/obj-cw/dist/lib -lplds4 -lplc4
-lnspr4    -lm	  
 Warning : prebinding disabled because dependent library: ./
	   libmsgbaseutil.dylib is not prebound
   Error : linker '/Applications/CodeWarrior Pro 8.1/For OS X Development/For 
	   Mac OS X Command Line Tools/INTO ~/bin/mwldppc' returned with exit 
	   code -11

Comment 18

[hacker formerly known as [e-mail]]

Attachment: ldap cw8 patch v2

I keep hitting these weird problems where mwccppc has issues resolving symlinks
to far away dirs so I have to resort to using NSDISTMODE=copy to force
nsinstall to copy files instead of symlinking them.

Comment 19

[cls]

I looked into the regxpcom failure and there's something screwy going on.  The
failing trace starts out as:

#0  0x00000000 in ?? ()
Cannot access memory at address 0x0
#1  0x0005b510 in PL_HashTableRawLookup (ht=0xa26e0, keyHash=162440,
key=0x9ea20) at plhash.c:197
#2  0x004c54e0 in _Z11WellOrderedPKvS_PP13nsNamedVectorS2_ (addr1=0x27a88,
addr2=0x9ea20, vec1p=0xa5730, vec2p=0xa2850) at nsAutoLock.cpp:208
#3  0x004c5650 in _ZN14nsAutoLockBaseC1EPvNS_14nsAutoLockTypeE (this=0x9ea20,
addr=0xa5730, type=665680) at nsAutoLock.cpp:255

There's actually another call at #1.5 GetVector() which appears to get
forcefully inlined even though I specified |-inline off| when building.  The
problem in PL_HashTableRawLookup is that the comparator functions of "ht" are
0x0.  The table being passed in is the static |OrderTable| from nsAutoLock.cpp.
  OrderTable is set at nsAutoLock.cpp#147 as

    OrderTable = PL_NewHashTable(64, _hash_pointer,
                                 PL_CompareValues, PL_CompareValues,
                                 &_hash_alloc_ops, 0);

A quick check shows that PL_CompareValues have non-zero values at this point:

(gdb) p PL_CompareValues
$21 = {int (void *, void *)} 0x5bcb0 <PL_CompareValues>

However, when PL_NewHashTable is called, 0x0 is being passed in the place of
PL_CompareValues:

(gdb) n    

Breakpoint 4, PL_NewHashTable (n=0, keyHash=0x4c51cc <_Z13_hash_pointerPKv>,
keyCompare=0, valueCompare=0, allocOps=0x55bef0, allocPriv=0x0) at plhash.c:116
116         if (n <= MINBUCKETS) {

It seems like there's a serious compiler bug here but it could just be gdb
flailing to comprehend cw.  Also note that the first argument of "64" got passed
in as "0". 

Comment 20

[hacker formerly known as [e-mail]]

Attachment: moz cw8 patch v3

Sync'd against this morning's trunk build. No functionality changes.

Comment 21

[hacker formerly known as [e-mail]]

Attachment: comparator crash testcase.tar.gz

Here's the testcase.  Something appears to be broken wrt function prototypes &
shared libs.  

Given:
typedef int (*PLHashComparator)(const void *v1, const void *v2);
int PL_CompareValues(const void *v1, const void *v2);
void  mytest(unsigned int n, PLHashComparator keyCompare, PLHashComparator
valueCompare)

If we dynamically link against the library containing PL_CompareValue(), then
keyCompare & valueCompare show up as 0x0 inside of mytest() .  If we statically
link against the library or PL_CompareValue() is defined in the same file as
mytest(), then everything works fine.

I included a Makefile with the tarball so you should be able to just do 'make'
to get the static link test and 'make SHAREDLIB=1' for the dynamic link test.

Comment 22

[hacker formerly known as [e-mail]]

Ok, so no sooner do I shoot off an email to c.s.m.p.codewarrior than I discover
a workaround.  If I use the __declspec(export) declarations as defined in the
XP_MAC section of prtypes.h, then the testcase works fine in the shared library
case.  

Comment 23

[hacker formerly known as [e-mail]]

Attachment: nspr cw8 patch v3

Ok, with the change to prtypes.h to use the XP_MAC section of the PR_EXPORT
defintions all of the necessary tests pass and regxpcom runs w/o the bus error.

nameshm1, sema, semaerr, & semaping still fail.

However, I wasn't able to build correctly when I used the XP_MAC sections of
jstypes.h and nscore.h.  I hit an internal compiler error.  It appears as
though cw doesn't like it if there's a declspec mismatch between the function
declaration and the function definition but I'm probably just missing something
obvious again.

Comment 24

[hacker formerly known as [e-mail]]

Attachment: nspr cw8 patch v4

Oops. Attached the wrong patch. A directory with spaces in your path makes
using AC_PATH_PROGS impractical.

Comment 25

[Wan-Teh Chang]

Comment on attachment 95380 [details] [diff] [review]
nspr cw8 patch v4

1. In nsprpub/pr/tests/mbcs.c, instead of the type casts,
you should declare the TraverseDirectory function to take
a 'const char *' parameter.

2. In nsprpub/pr/tests/testfile.c, we should remove the
'scope' parameter for the create_new_thread function and
declare it as a local variable in that function.

3. In nsprpub/pr/tests/dll/Makefile.in, use NS_USE_GCC
instead of GNU_CC for consistency with the original NSPR
makefile code.	Also, your substitution for the gcc case
should be the other way around.  I suggest that you make
the new code look like this:

ifdef NS_USE_GCC
DSO_LDOPTS = -bundle
else
DSO_LDOPTS = -xm bundle
endif

Good job on figuring out the __declspec(export) problem!

Comment 26

[hacker formerly known as [e-mail]]

Attachment: nspr cw8 patch v5

Updated patch with Wan-Teh's suggested changes.

Comment 27

[hacker formerly known as [e-mail]]

Still working on building JS with the proper declspec declarations.  If I use
the existing XP_MAC ifdefs in jstypes.h, then the build fails with an internal
compiler error.

   Error : internal compiler error (report to <[e-mail]>)
           while executing in file 'ObjGenMachO.c' line: 1480
           (compiling '' in 'jsapi.c')

If I modify JS_FRIEND_DATA(t) to not use declspecs at all, JS compiles.   If I
use declspec(export) with JS_FRIEND_DATA and comment out all internal
JS_FRIEND_DATA uses except jsemit.h, then the JS build works as well.  Based
upon that, it seems as though the cmdline utils do not like using declspec on
types that have been forwarded declared, like JSClass in jspubtd.h .  Since
JSSrcNoteSpec, in the jsemit.h case, has been fully declared, the build works fine.

Comment 28

[Wan-Teh Chang]

Comment on attachment 96132 [details] [diff] [review]
nspr cw8 patch v5

In  nsprpub/pr/tests/dll/Makefile.in

>-DSO_LDOPTS = -bundle
>+ifdef NS_USE_GCC
>+DSO_LDOPTS := $(subst -dynamiclib,-bundle,$(DSO_LDOPTS))
>+else
>+DSO_LDOPTS := $(subst -xm sharedlibrary,-xm bundle,$(DSO_LDOPTS))
>+endif

This should say

+ifdef NS_USE_GCC
 DSO_LDOPTS = -bundle
+else
+DSO_LDOPTS = -xm bundle
+endif

Otherwise I believe you'll break the build on Mac OS X 10.2
(see bug 159976).

Comment 29

[hacker formerly known as [e-mail]]

ARGH! I reinstalled CW8 from cd, applied the 8.2 update and now more of the nspr
tests are failing.  They still fail even if I remove the change from prtypes.h.
cltsrv, nameshm1, provider, sema, semaerr, semaping, timemac, timetest & version
are the failed tests.

[soundwave:obj-nspr/pr/tests] cls% mwccppc -v

Metrowerks C/C++ Compiler for Mach-O/PPC.
Copyright (c)1993-2000 Metrowerks, Inc.
All rights reserved.
Version 3.0.2 build 336
Runtime Built: Aug 21 2002 17:13:42

Comment 30

[TimMaroney]

I am looking at making the same kind of build for Mac Classic PPC and Carbon
builds with the MPW Shell. All the major GNU tools exist as MPW tools except
gmake. My initial investigation indicates gmake is easy to port. If we can build
for Classic without CodeWarrior project files, that should reduce the
maintenance cost of the Classic platform as it moves towards the end of its life.

Comment 31

[Conrad Carlen (not reading bugmail)]

> My initial investigation indicates gmake is easy to port.

That would be so cool! There would be some additional challenges, like the IDL
compiler, though.

Comment 32

[TimMaroney]

xpidl is a command-line tool on other platforms so I don't imagine it would 
be much trouble to make it an MPW tool. But yes, it's not just gmake. The 
Windows build instructions list about a dozen required cygwin packages, 
and the worst case would be that every command in every one of those 
packages would be needed in MPW. In practice, though, it looks like most 
of them aren't  needed. Basic commands like mkdir, cp, rm, mv and touch 
could be hooked up to MPW scripts through aliasing.

My impression is that Mozilla does not use shell scripts, only perl scripts. 
If I'm right about that, the chance of requiring large sections of the UNIX 
command set is mostly limited to make instructions (that is, build rules in 
makefiles). If I'm wrong, though, the task could be much bigger. I'm 
thinking of trying to build with a crippled cygwin to see what UNIX 
commands are required.

Comment 33

[TimMaroney]

config/rules.mk might need some MPW build rules. Not a problem.

From eyeballing rules.mk, the commands I'm seeing are perl, cat, cd, cp, 
echo, make, rm, sed, sleep, test and touch. Not too bad as far as porting 
needs go. I think all those exist except test and make, and I'm well along 
on make.

There is a good deal of shell-ism in rules.mk, such as if, for and &&. 
There are only 5 for's, but 24 &&'s and 18 if's. (There are no ||'s outside 
sed commands.) MPW is compatible with && and ||, and the minor 
idiosyncracies of MPW if and for syntax can be handled by running the 
commands through sed.

Serialization may present an issue on MPW, in that one has to put out 
commands and then execute them, rather than being able to run them 
inside the gmake tool. Creating commands may be dependent on what 
has happened before, and in MPW gmake, the predecessor commands 
would not have executed yet. This could be solved by having make stop 
itself at points and insert a continuation make command at the end of its 
output in certain make rules, or in other ways. It was not really a problem 
for MPW's own Build and may not be one here.

Comment 34

[Wan-Teh Chang]

Please open a new bug for making Mac Classic PPC and Carbon
builds with the MPW Shell.  Each bug should focus on one
issue.  This bug is about using the CodeWarrior commandline
tools to build mach-o.

Comment 35

[Scott Collins]

Building under MPW would be great; this would significantly lower the bar
(certainly the financial bar, possibly the complexity bar) for people who want
or need CFM builds after our current focus has shifted to mach-o.

My understanding is that MPW will remain a Classic application.  This probably
is not a hindrance.  Do we think MPW will remain available? (forget supported)

Comment 36

[TimMaroney]

See bug 174405 for Classic command-line build issues.

Comment 37

[hacker formerly known as [e-mail]]

Mass reassign to new default build assignee

Comment 38

[Simon Fraser [no longer active]]

CodeWarrior is dead.