370766 - nspr: 64 bit MAC OS X support

Status: RESOLVED FIXED

(NSPR :: NSPR, enhancement)

Description

[glen beasley]

cipher.sh consistently fails for RC4 in 64 bit mode

Comment 1

[glen beasley]

Attachment: in 64 bit mode on MAC OS X define "WORD" as unsigned long long

all RC4 cipher.sh tests now pass in 32 bit and 64 bit mode on MAC OS X

Comment 2

[Wan-Teh Chang]

Comment on attachment 255492 [details] [diff] [review]
in 64 bit mode on MAC OS X define "WORD"  as unsigned long long 

Glen, the proper fix is to define IS_64 in
mozilla/nsprpub/pr/include/md/_darwin.cfg for
64-bit Mac OS X.  In fact, you need to have a
section for __x86_64 in _darwin.cfg.  Use
_solaris.cfg and _linux.cfg as examples.

Comment 3

[Wan-Teh Chang]

mozilla/nsprpub/pr/include/gencfg.c is a stand-alone
program that generates a section of _xxxos.cfg for you.
You can try to use that program to generate the section
for 64-bit Mac OS X.  gencfg.c is out of date, so you'll
need to augment its output.

Comment 5

[glen beasley]

Attachment: work in progress

This patch is a work in progress, since there seems to be current interest in this bug I am adding what I have so far meaning I know I will be changing this patch (possibly a great deal).

Currently I can build and test 32 and 64 bit on MAC OS X with this patch. 
64 bit build:
file libnspr4.dylib: Mach-O 64-bit dynamically linked shared library x86_64
32 bit build:
file libnspr4.dylib: Mach-O dynamically linked shared library i386


10 NSPR tests fail for 32 bit:
cltsrv, layer, nameshm1, nblayer, provider, sema, semaerr,			
semaping, socket, testfile 
12 NSPR tests fail for 64 bit:
cltsrv, *dlltest*, layer, *libfilename*, nameshm1, nblayer, provider, sema, semaerr, semaping, socket 

dlltest and libfilename are the extra failures for 64 bit. 

The file prlink.c uses the 32 bit Darwin Code Fragment Manager Functions that 
are not available to 64-bit Darwin applications. I hoped to make use of the existing USE_MACH_DYLD macro but I need to research/test more.

http://developer.apple.com/documentation/Carbon/Reference/Code_Fragment_Manager/DeprecationAppendix/AppendixADeprecatedAPI.html#//apple_ref/c/func/CountSymbols

http://developer.apple.com/documentation/DeveloperTools/Conceptual/MachORuntime/Reference/reference.html#//apple_ref/doc/uid/TP40000895-CH248-95693

I also want to verify _darwin.cfg

Comment 6

[glen beasley]

Looking at the mac os x documentation while -m64 works with gcc, I believe the preferred option on mac os x is -arch x86_64.

For Dynamic Loading:

http://developer.apple.com/documentation/DeveloperTools/Reference/MachOReference/Reference/reference.html

"These functions are for loading Mach-O bundle files. They are declared in /usr/include/mach-o/dyld.h. The use of these functions is discouraged. You should use the more efficient functions described in “Dynamic Loader Compatibility Functions.”

so in prlink.c I am trying not to use the code path "USE_MACH_DYLD" and get "XP_UNIX + USE_DLFCN" code path working for "XP_MACOSX + __x86_64__".

Also for 32 bit XP_MACOSX many of the current CodeFragments functions are deprecated as of 10.5 so at some point they may not work in the future.

Comment 7

[Wan-Teh Chang]

Comment on attachment 330184 [details] [diff] [review]
work in progress

Glen, thanks for the patch.

A trick to add -arch x86_64 in only one place in NSPR
is to add it to the CC variable in the configure script.
For example, this is how we take care of 64-bit Solaris

For _darwin.cfg, we should try to verify that the
PR_XXX_XXX_XXX macros, especially the PR_ALIGN_OF_XXX macros,
all have the same values between __x86_64__ and __ppc64__.
Do you have a 64-bit PowerPC Mac?

In prlink.c, we should use a new macro to indicate that we
don't want to support loading "CodeFragments", rather than
using the __x86_64__ macro directly.

Comment 8

[glen beasley]

Attachment: add x86_64 support

in prlink.c I made use of the existing macro USE_DLFCN to indicate that we
want to use the Mac OS X Dynamic Loader Compatibility Functions.

http://developer.apple.com/documentation/DeveloperTools/Reference/MachOReference/Reference/reference.html

"Dynamic Loader Compatibility Functions

These are the recommended functions to use to interact with the dynamic loader. These functions work in Mac OS X v10.3 and v10.4. However, in Mac OS X v10.4 they are more efficient than other image-loading functions."

this patch only uses USE_DLFCN for x86_64 on MAC OS X, but we may want to change this for 32 bit as well since CodeFragments functions are
deprecated as of 10.5.

I do not have access to 64-bit PowerPC Mac. since the 64 bit support is a new feature maybe we should choose not to support 64-bit PowerPC Mac? 

Note: if you use Xcode "To run Xcode in 64-bit mode on a 64-bit machine, use the Finder’s Get Info command and uncheck the “Open in 32-Bit Mode” checkbox."

http://lists.apple.com/archives/Xcode-users/2007/Oct/msg00400.html

Comment 9

[glen beasley]

Attachment: add x86_64 support 2

new patch due to bug 417044 changes. I had not tried do get the universal build working x86_64.  

note on the behavior of dlopen on MAC OS X  

http://developer.apple.com/documentation/DeveloperTools/Reference/MachOReference/Reference/reference.html#//apple_ref/c/func/dlopen

When path is a filename, the dynamic loader searches for the library in the search paths in the following order:

$LD_LIBRARY_PATH
$DYLD_LIBRARY_PATH
The process’s working directory
$DYLD_FALLBACK_LIBRARY_PATH

on solaris for dlopen

A path name containing an embedded '/' is interpreted as an absolute path or relative to the current directory, otherwise the set of search paths currently in effect by the run-time linker will be used to locate the specified file

Comment 10

[Wan-Teh Chang]

Attachment: add x86_64 support 3

Glen,

I reviewed your patch and made some changes.  I still
need to review the changes to prlink.c.

Here is your patch with my changes.  (Some of the
changes fix compiler warnings.)

Comment 11

[Wan-Teh Chang]

One more thing: I set LD_LIBRARY_PATH to ../../dist/lib when
I run the NSPR tests, but it doesn't work.  Setting
DYLD_LIBRARY_PATH works.

Comment 12

[Wan-Teh Chang]

Attachment: add x86_64 support 3 (except prlink.c; checked in)

I checked in Glen's patch, except the changes to prlink.c,
on the NSPR trunk (NSPR 4.7.2).

Glen, I'd appreciate if you could review my changes to your
patch.

Checking in configure;
/cvsroot/mozilla/nsprpub/configure,v  <--  configure
new revision: 1.236; previous revision: 1.235
done
Checking in configure.in;
/cvsroot/mozilla/nsprpub/configure.in,v  <--  configure.in
new revision: 1.240; previous revision: 1.239
done
Checking in pr/include/md/_darwin.cfg;
/cvsroot/mozilla/nsprpub/pr/include/md/_darwin.cfg,v  <--  _darwin.cfg
new revision: 3.13; previous revision: 3.12
done
Checking in pr/include/md/_darwin.h;
/cvsroot/mozilla/nsprpub/pr/include/md/_darwin.h,v  <--  _darwin.h
new revision: 3.21; previous revision: 3.20
done
Checking in pr/src/md/unix/os_Darwin.s;
/cvsroot/mozilla/nsprpub/pr/src/md/unix/os_Darwin.s,v  <--  os_Darwin.s
new revision: 1.2; previous revision: 1.1
done
RCS file: /cvsroot/mozilla/nsprpub/pr/src/md/unix/os_Darwin_x86_64.s,v
done
Checking in pr/src/md/unix/os_Darwin_x86_64.s;
/cvsroot/mozilla/nsprpub/pr/src/md/unix/os_Darwin_x86_64.s,v  <--  os_Darwin_x86_64.s
initial revision: 1.1
done

Comment 13

[Wan-Teh Chang]

Attachment: Glen's patch for prlink.c

The remaining work is to port prlink.c.

Comment 14

[glen beasley]

Comment on attachment 331488 [details] [diff] [review]
add x86_64 support 3 (except prlink.c; checked in)

patch looks good.

Comment 15

[glen beasley]

Comment on attachment 331491 [details] [diff] [review]
Glen's patch for prlink.c

R+; sorry I took so long to review.

Comment 16

[Nomis101]

If this patch for prlink.c is checked in and if bug 456662 is fixed, is it than possible to build 64-bit mozilla apps on Mac OS X? I ask, because I have applied both patches on my own build (Thunderbird) and tried than to build the application as x86_64. But I get an error.
I've tried to trigger 64bit building by adding "--enable-optimize=-m64" and "--enable-64bit" to my mozconfig. The error I get was:

> /temp/src/mozilla/nsprpub/config/now.c
> gcc-4.2 now.o -isysroot /Developer/SDKs/MacOSX10.5.sdk -o now
> ld warning: in now.o, file is not of required architecture
> Undefined symbols:
> "_main", referenced from:
> start in crt1.10.5.o
> ld: symbol(s) not found
> collect2: ld returned 1 exit status
> make[5]: *** [now] Error 1
> make[4]: *** [export] Error 2
> make[3]: *** [tier_nspr] Error 2
> make[2]: *** [default] Error 2
> make[1]: *** [default] Error 2
> make: *** [build] Error 2

I'am to hasty and there is a lot more to do bevor it is possible to build 64bit apps on OS X? Or do I make something wrong?

Comment 17

[Simon Howes]

The way I have found is to use

export CC="gcc-4.2 -arch x86_64"
export CXX="g++-4.2 -arch x86_64"

prlink.c then builds fine, however it fails when it comes to nanojit, which is for another bug.

Comment 18

[Nomis101]

(In reply to comment #17)
> The way I have found is to use
> 
> export CC="gcc-4.2 -arch x86_64"
> export CXX="g++-4.2 -arch x86_64"
Ooooh, yes. Thanks, this workes much better. :) :)


(In reply to comment #17)
> prlink.c then builds fine, however it fails when it comes to nanojit, which is
> for another bug.
Yes, I see this also:

-I/usr/X11/include -DMOZILLA_CLIENT -include ./mozilla-config.h -Wp,-MD,.deps/avmplus.pp /temp/src/mozilla/js/src/nanojit/avmplus.cpp
{standard input}:24393:`pusha' is not supported in 64-bit mode
{standard input}:24397:`popa' is not supported in 64-bit mode
{standard input}:34582:suffix or operands invalid for `call'
make[5]: *** [jstracer.o] Error 1
make[4]: *** [default] Error 2
make[3]: *** [tier_js] Error 2
make[2]: *** [default] Error 2
make[1]: *** [default] Error 2
make: *** [build] Error 2

Comment 19

[Josh Aas]

lets keep this bug about 64-bit Mac OS X support for nspr, not Mozilla/Gecko in general

Comment 20

[Wan-Teh Chang]

Josh: isn't it superfluous to qualify the summary of an NSPR bug with
"nspr:"?  Perhaps your bug query needs to filter out bugs under the NSPR
product?

Comment 21

[Josh Aas]

Perhaps superfluous, but it makes it easier to distinguish between this bug and the bug that is strictly "64 bit MAC OS X support" in Mozilla. I'm not trying to set a standard or anything, just make this situation a little easier. You can remove that if it really matters to you.

Comment 22

[Wan-Teh Chang]

Comment on attachment 331491 [details] [diff] [review]
Glen's patch for prlink.c

Glen, this patch is the prlink.c portion of your "add x86_64 support 2" patch (attachment 331360 [details] [diff] [review]).
So you don't need to review it :-)  I was planning to work on this right away, but it fell through the
cracks.  Sorry about the confusion.

Comment 23

[Wan-Teh Chang]

Attachment: Proposed patch for prlink.c (checked in)

I edited Glen's patch for prlink.c and checked it in on the NSPR trunk (NSPR 4.7.4).

Josh, you just need to review the fact that the 64-bit Mac OS X NSPR uses the
dlopen functions (declared in <dlfcn.h>) for dynamic library loading, and does
not support CFBundle.

Note that as a result, the 64-bit Mac OS X NSPR doesn't depend on the Core
Foundation, so it doesn't need the -I$(MACOS_SDK_DIR)/Developer/Headers/FlatCarbon
compiler flag and the -framework CoreServices -framework CoreFoundation
linker flags.  But I left these flags unchanged because NSPR now supports
building universal binaries in a single pass (bug 417044), which does not
allow arch-specific compiler or linker flags.

Basically, this patch changes
    defined(XP_MACOSX)
to
    defined(XP_MACOSX) && defined(USE_MACH_DYLD)
so that the original (32-bit) Mac OS X code is not perturbed.  For the new
64-bit Mac OS X code, USE_DLFCN is defined (in _darwin.h), and
USE_MACH_DYLD is not defined.  Perhaps we should define a new
macro for "defined(XP_MACOSX) && defined(USE_MACH_DYLD)"
because that combination is now repeated several times in prlink.c.

Checking in prlink.c;
/cvsroot/mozilla/nsprpub/pr/src/linking/prlink.c,v  <--  prlink.c
new revision: 3.94; previous revision: 3.93

Comment 24

[Wan-Teh Chang]

A few NSPR test programs are failing.  Most of the failures are caused by
the socket function shutdown() failing with the ENOTCONN error.  The
tests that fail because of this are cltsrv, layer, nblayer, provider, and socket.
This could be an OS bug.

testfile fails with this message:
File Test failed: file data corrupted
testfile failed to rmdir /tmp/testfile_dir
File Test failed

The failures of nameshm1, sema, semaerr, and semaping are a known
problem of Mac OS X (PR_OpenSemaphore not implemented).

Comment 25

[Wan-Teh Chang]

Attachment: Use the predefined __LITTLE_ENDIAN__ macro (checked in)

This was suggested in bug 466531 comment 90.
I checked this in on the NSPR trunk (NSPR 4.7.4).

Checking in _darwin.cfg;
/cvsroot/mozilla/nsprpub/pr/include/md/_darwin.cfg,v  <--  _darwin.cfg
new revision: 3.14; previous revision: 3.13
done

Comment 26

[Josh Aas]

Comment on attachment 352853 [details] [diff] [review]
Proposed patch for prlink.c (checked in)

> Josh, you just need to review the fact that the 64-bit Mac OS X NSPR uses the
> dlopen functions (declared in <dlfcn.h>) for dynamic library loading, and does
> not support CFBundle.

Looks fine to me in that respect. I don't think we use the CFBundle loading code in Gecko and I agree that moving to the dlopen functions is a good idea.

Comment 27

[Wan-Teh Chang]

I'm marking this bug FIXED again.  None of the test failures are
specific to 64-bit x86_64 builds.  32-bit i386 builds have the same
test failures.  I opened three bugs to track the test failures:
bug 469744, bug 469745, and bug 469747.