237684 - AIX: negotiateauth extension fails to build

Status: RESOLVED FIXED

(Core :: Networking, defect)

Description

[Philip K. Warren]

AIX ships the following include files and libraries in the krb5.client.rte and
krb5.toolkit.adt filesets:

/usr/include/com_err.h
/usr/include/gssapi
/usr/include/gssapi/gssapi.h
/usr/include/gssapi/gssapi_generic.h
/usr/include/gssapi/gssapi_krb5.h
/usr/include/ibm_svc
/usr/include/ibm_svc/krb5_svc.h
/usr/include/ibm_svc/krb5adm.h
/usr/include/ibm_svc/krb5gss.h
/usr/include/ibm_svc/krb5kdc.h
/usr/include/ibm_svc/krb5krb.h
/usr/include/ibm_svc/krb5utl.h
/usr/include/ibm_svc/krb5vdb.h
/usr/include/k5profile.h
/usr/include/kadm5
/usr/include/kadm5/admin.h
/usr/include/kdb_types.h
/usr/include/krb5.h

/usr/krb5/lib
/usr/krb5/lib/libdyn.a
/usr/krb5/lib/libgssapi_krb5.a
/usr/krb5/lib/libgssrpc.a
/usr/krb5/lib/libk5profile.a
/usr/krb5/lib/libkadm5clnt.a
/usr/krb5/lib/libkdb5.a
/usr/krb5/lib/libkrb5.a
/usr/krb5/lib/libksvc.a
/usr/krb5/lib/libss.a
/usr/lib/libdyn.a -> /usr/krb5/lib/libdyn.a
/usr/lib/libgssapi_krb5.a -> /usr/krb5/lib/libgssapi_krb5.a
/usr/lib/libgssrpc.a -> /usr/krb5/lib/libgssrpc.a
/usr/lib/libk5profile.a -> /usr/krb5/lib/libk5profile.a
/usr/lib/libkadm5clnt.a -> /usr/krb5/lib/libkadm5clnt.a
/usr/lib/libkdb5.a -> /usr/krb5/lib/libkdb5.a
/usr/lib/libkrb5.a -> /usr/krb5/lib/libkrb5.a
/usr/lib/libksvc.a -> /usr/krb5/lib/libksvc.a
/usr/lib/libss.a -> /usr/krb5/lib/libss.a

It does not ship the libcom_err or libk5crypto libraries, so the check for the
kerberos libraries in configure fails on AIX:

configure:12513: checking for gss_init_sec_context in -lgssapi_krb5
configure:12532: xlc_r -o conftest    -I/usr/include -bh:5 -Wl,-brtl -blibpath:/
usr/lib:/lib -L/usr/lib conftest.c -lgssapi_krb5  -lC_r -ldl -lm -lc_r  -lgssapi
_krb5 -lkrb5 -lk5crypto -lcom_err 1>&5
ld: 0706-006 Cannot find or open library file: -l k5crypto
    ld:open(): No such file or directory
ld: 0706-006 Cannot find or open library file: -l com_err 
    ld:open(): No such file or directory

Comment 1

[Philip K. Warren]

Attachment: Patch v1

I have tested a build with the negotiateauth extension with this small change
to configure.in, and everything seems to be working properly.

Comment 2

[Darin Fisher]

Comment on attachment 144071 [details] [diff] [review]
Patch v1

looks good.  we may end up needing to add other platform exceptions in the same
manner.  i'm sure AIX isn't the only platform that doesn't have those
libraries.

Comment 3

[Darin Fisher]

i hear the book is closed on 1.7b ... we'll probably have to wait for 1.7 final
to get this in.

Comment 4

[Christian :Biesinger (don't email me, ping me on IRC)]

Comment on attachment 144071 [details] [diff] [review]
Patch v1

hm, why not always check first if it works without these extra libraries, then
check with these libraries?

Comment 5

[Darin Fisher]

hmm... yeah, that should work too.  pkw?

Comment 6

[Philip K. Warren]

That was what I tried initially, but doing this:

AC_CHECK_LIB(gssapi_krb5, gss_init_sec_context,
            [GSSAPI_LIBS="-L$GSSAPI_DIR/lib -lgssapi_krb5 -lkrb5 -lk5crypto
-lcom_err"],)

followed by this:

AC_CHECK_LIB(gssapi_krb5, gss_init_sec_context,
            [GSSAPI_LIBS="-L$GSSAPI_DIR/lib -lgssapi_krb5 -lkrb5"],)

autoconf caches the first result, and thinks the second test is identical, so
never runs the second test.

Comment 7

[Darin Fisher]

Comment on attachment 144071 [details] [diff] [review]
Patch v1

requesting sr= from bryner in case he has some other suggestions.

Comment 8

[Brian Ryner (not reading)]

(In reply to comment #6)
> That was what I tried initially, but doing this:
> 
> AC_CHECK_LIB(gssapi_krb5, gss_init_sec_context,
>             [GSSAPI_LIBS="-L$GSSAPI_DIR/lib -lgssapi_krb5 -lkrb5 -lk5crypto
> -lcom_err"],)
> 
> followed by this:
> 
> AC_CHECK_LIB(gssapi_krb5, gss_init_sec_context,
>             [GSSAPI_LIBS="-L$GSSAPI_DIR/lib -lgssapi_krb5 -lkrb5"],)
> 
> autoconf caches the first result, and thinks the second test is identical, so
> never runs the second test.
> 

That's because it is identical; you've only changed what it should do if the
test succeeds.  What I think you want to do is make use of the OTHER-LIBRARIES
parameter to AC_CHECK_LIB, i.e. get rid of setting LIBS before the AC_CHECK_LIB
and then do:

_GSS_LIBS=-L$GSSAPI_DIR/lib -lgssapi_krb5
AC_CHECK_LIB(gssapi_krb5, gss_init_sec_context, 
             [GSSAPI_LIBS="$_GSS_LIBS -lk5crypto -lcom_err"],
             [AC_CHECK_LIB(gssapi_krb5, gss_init_sec_context,
                           [GSSAPI_LIBS="$_GSS_LIBS"])],
             "-lk5crypto -lcom_err")

does that work?

Comment 9

[Philip K. Warren]

Attachment: Patch v2

This changes the behavior a bit, so we now check for the existance of the
k5crypto and com_err libraries before testing for the gss_init_sec_context
function. I have verified that this works properly on AIX and Linux.

AIX:
pkw@ut:~/sb/mozilla/trunk/mozilla$ grep '^GSSAPI_LIBS' config/autoconf.mk 
GSSAPI_LIBS	= -L/usr/lib -lgssapi_krb5 -lkrb5

Linux:
[pkw@boost mozilla]$ grep '^GSSAPI_LIBS' config/autoconf.mk 
GSSAPI_LIBS	= -L/usr/lib -lgssapi_krb5 -lkrb5 -lk5crypto -lcom_err

Comment 10

[Philip K. Warren]

Comment on attachment 145046 [details] [diff] [review]
Patch v2

Requesting approval for patch which allows the negotiateauth extension to be
built on AIX and other operating systems which do not have the k5crypto or
com_err libraries.

Comment 11

[Darin Fisher]

fixed-on-trunk for 1.7 final