Closed Bug 724243 Opened 8 years ago Closed 8 years ago

dladdr not detected in certain environments

Categories

(NSPR :: NSPR, defect, P1)

All
Linux
defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: wolfiR, Assigned: wolfiR)

Details

Attachments

(1 file, 1 obsolete file)

The automated tests about FIPS are failing in most (but not all) compilations I've tried for NSS 3.13.2beta2. (On systems with very old toolchains on Linux it doesn't appear to be an issue but on more recent ones they fail on different architectures and toolchain combinations):

cert.sh: Enable FIPS mode on database -----------------------
modutil -dbdir /usr/src/packages/BUILD/nss-3.13.2/mozilla/tests_results/security/localhost.1/fips -fips true 

WARNING: Performing this operation while the browser is running could cause
corruption of your security databases. If the browser is currently running,
you should exit browser before continuing this operation. Type 
'q <enter>' to abort, or <enter> to continue: 
security library: invalid arguments.
ERROR: Unable to switch FIPS modes.
cert.sh: #124: Enable FIPS mode on database for FIPS PUB 140 Test Certificate (11)  - FAILED
cert.sh ERROR: Enable FIPS mode on database for FIPS PUB 140 Test Certificate failed 11

fips.sh: FIPS 140 Compliance Tests ===============================
fips.sh: Verify this module is in FIPS mode  -----------------
modutil -dbdir ../fips -list

Listing of PKCS #11 Modules
-----------------------------------------------------------
  1. NSS Internal PKCS #11 Module
	 slots: 2 slots attached
	status: loaded

	 slot: NSS Internal Cryptographic Services
	token: NSS Generic Crypto Services

	 slot: NSS User Private Key and Certificate Services
	token: NSS Certificate DB

  2. RootCerts
	library name: /usr/src/packages/BUILD/nss-3.13.2/mozilla/dist/Linux2.6_x86_glibc_PTH_OPT.OBJ/lib/libnssckbi.so
	 slots: 1 slot attached
	status: loaded

	 slot: NSS Builtin Objects
	token: Builtin Object Token
-----------------------------------------------------------
FIPS mode disabled.
fips.sh: #410: Verify this module is in FIPS mode (modutil -chkfips true) . - FAILED

fips.sh: Run PK11MODE in FIPSMODE  -----------------
pk11mode -d ../fips -p fips- -f ../tests.fipspw
Loaded FC_GetFunctionList for FIPS MODE; slotID 0 

FIPS MODE PKM_Error: C_Initialize failed with 0x00000005, CKR_GENERAL_ERROR         
Loaded FC_GetFunctionList for FIPS MODE; slotID 0 
**** Total number of TESTS ran in FIPS MODE is 2. ****
fips.sh: #429: Run PK11MODE in FIPS mode (pk11mode) . - FAILED


fips.sh: Run PK11MODE in Non FIPSMODE  -----------------
pk11mode -d ../fips -p nonfips- -f ../tests.fipspw -n
loaded C_GetFunctionList for NON FIPS MODE; slotID 1 
loaded C_GetFunctionList for NON FIPS MODE; slotID 1 
loaded C_GetFunctionList for NON FIPS MODE; slotID 1 
loaded C_GetFunctionList for NON FIPS MODE; slotID 1 
Hybrid MODE PKM_Error: FC_Initialize failed with 0x00000005, CKR_GENERAL_ERROR         
Hybrid MODE PKM_Error: PKM_HybridMode failed with 0x00000005, CKR_GENERAL_ERROR         
loaded C_GetFunctionList for NON FIPS MODE; slotID 1 
**** Total number of TESTS ran in NON FIPS MODE is 99. ****
fips.sh: #430: Run PK11MODE in Non FIPS mode (pk11mode -n) . - FAILED

fips.sh: Detect mangled softoken--------------------------
mangling /usr/src/packages/BUILD/nss-3.13.2/mozilla/tests_results/security/localhost.1/fips/mangle/libsoftokn3.so
mangle -i /usr/src/packages/BUILD/nss-3.13.2/mozilla/tests_results/security/localhost.1/fips/mangle/libsoftokn3.so -o -8 -b 5
Changing byte 0x000d5d0c (875788): from 49 (73) to 69 (105)
LD_LIBRARY_PATH=/usr/src/packages/BUILD/nss-3.13.2/mozilla/tests_results/security/localhost.1/fips/mangle dbtest -r -d ../fips
fips.sh: #431: Init NSS with a corrupted library (dbtest -r) . - FAILED

[some more examples skipped]

The following is the short summary of failed tests:

+ grep FAILED ../../../tests_results/security/localhost.1/output.log
cert.sh: #124: Enable FIPS mode on database for FIPS PUB 140 Test Certificate (11)  - FAILED
fips.sh: #410: Verify this module is in FIPS mode (modutil -chkfips true) . - FAILED
fips.sh: #429: Run PK11MODE in FIPS mode (pk11mode) . - FAILED
fips.sh: #430: Run PK11MODE in Non FIPS mode (pk11mode -n) . - FAILED
fips.sh: #431: Init NSS with a corrupted library (dbtest -r) . - FAILED
ssl.sh: #1105:  (modutil -fips true) produced a returncode of 11, expected is 0 - FAILED
ssl.sh: #1107:  (grep "FIPS PKCS #11") produced a returncode of 1, expected is 0 - FAILED
ssl.sh: #1108:  (modutil -fips true) produced a returncode of 11, expected is 0 - FAILED
ssl.sh: #1110:  (grep "FIPS PKCS #11") produced a returncode of 1, expected is 0 - FAILED
ssl.sh: #1240:  (modutil -fips false) produced a returncode of 13, expected is 0 - FAILED
ssl.sh: #1243:  (modutil -fips false) produced a returncode of 13, expected is 0 - FAILED
ssl.sh: #1246:  (modutil -fips true) produced a returncode of 11, expected is 0 - FAILED
ssl.sh: #1248:  (grep "FIPS PKCS #11") produced a returncode of 1, expected is 0 - FAILED
ssl.sh: #1339:  (modutil -fips false) produced a returncode of 13, expected is 0 - FAILED
cert.sh: #2167: Enable FIPS mode on database for FIPS PUB 140 Test Certificate (11)  - FAILED
fips.sh: #2438: Verify this module is in FIPS mode (modutil -chkfips true) . - FAILED
fips.sh: #2457: Run PK11MODE in FIPS mode (pk11mode) . - FAILED
fips.sh: #2458: Run PK11MODE in Non FIPS mode (pk11mode -n) . - FAILED
fips.sh: #2459: Init NSS with a corrupted library (dbtest -r) . - FAILED
fips.sh: #3845: Verify this module is in FIPS mode (modutil -chkfips true) . - FAILED
fips.sh: #3864: Run PK11MODE in FIPS mode (pk11mode) . - FAILED
fips.sh: #3865: Run PK11MODE in Non FIPS mode (pk11mode -n) . - FAILED
fips.sh: #3866: Init NSS with a corrupted library (dbtest -r) . - FAILED
cert.sh: #4362: Enable FIPS mode on database for FIPS PUB 140 Test Certificate (11)  - FAILED
fips.sh: #4648: Verify this module is in FIPS mode (modutil -chkfips true) . - FAILED
fips.sh: #4667: Run PK11MODE in FIPS mode (pk11mode) . - FAILED
fips.sh: #4668: Run PK11MODE in Non FIPS mode (pk11mode -n) . - FAILED
fips.sh: #4669: Init NSS with a corrupted library (dbtest -r) . - FAILED
A bit more testing showed that the errors apparently only happen when NSS is built against NSPR_4_9_0beta6. If I build against NSPR 4.8.9 the tests are successful.
More information:
Starting from a working Firefox stack with
- nspr 4.8.9
- nss 3.13.1
- firefox 10
where I can enable FIPS without a problem within Firefox

Now I _only_ installed nspr 4.9beta6 into that stack (w/o recompiling anything) and it's immediately impossible to switch Firefox 10's NSS into FIPS mode.
From previous observations I'm pretty sure this has all the same origin and is something in NSPR.
Assignee: nobody → wtc
Component: Libraries → NSPR
Product: NSS → NSPR
QA Contact: libraries → nspr
Target Milestone: --- → Future
Version: 3.13.2 → 4.9
most likely caused by the fix for bug 712281.

On my Linux systems dladdr cannot be found:

configure:5858: checking for dladdr
configure:5886: gcc -o conftest -fmessage-length=0 -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g -Wall  -ldl  conftest.c  1>&5
/tmp/ccTELoB3.o: In function `main':
/usr/src/packages/BUILD/nspr-4.9.0/mozilla/nsprpub/configure:5880: undefined reference to `dladdr'
collect2: ld returned 1 exit status
What kind of linux system doesn't have dladdr in -ldl? Could if be that dladdr is actually a macro to something else?
I still cannot explain what's happening here. From within the rpmbuild I get exactly the error above. When I change after the build into the chroot and call configure again the same way (after removing config.cache) it suddenly finds dladdr(). It's likely a local problem caused by some RPM building effects (while I have absolute no idea yet why).
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → WORKSFORME
In the end it turned out to be an error in configure which only is triggered for certain linker configurations (which are enabled in our RPM build environment)
See http://www.gentoo.org/proj/en/qa/asneeded.xml
(Code Listing 2.4: example of mistake in library checks) which is exactly the issue happening here.
Severity: critical → normal
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
Summary: FIPS tests failing for 3.13.2beta2 → dladdr not detected in certain environments
Attached patch patch (obsolete) — Splinter Review
Assignee: wtc → mozilla
Attachment #594950 - Flags: review?(wtc)
Comment on attachment 594950 [details] [diff] [review]
patch

Review of attachment 594950 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks for the patch.  I have a question.

::: configure.in
@@ -2743,3 +2743,3 @@
> >  AC_PROG_GCC_TRADITIONAL
> > -_SAVE_LDFLAGS="$LDFLAGS"
> > +_SAVE_LIBS="$LIBS"
> > -LDFLAGS="$OS_LIBS"
> > +LIBS="$OS_LIBS"

Should we say
    LIBS="$LIBS $OS_LIBS"
here?
I changed
    LIBS="$OS_LIBS"
to
    LIBS="$LIBS $OS_LIBS"
and checked in the patch on the NSPR trunk (NSPR 4.9).

Checking in configure;
/cvsroot/mozilla/nsprpub/configure,v  <--  configure
new revision: 1.320; previous revision: 1.319
done
Checking in configure.in;
/cvsroot/mozilla/nsprpub/configure.in,v  <--  configure.in
new revision: 1.322; previous revision: 1.321
done
Attachment #594950 - Attachment is obsolete: true
Attachment #594950 - Flags: review?(wtc)
Status: REOPENED → RESOLVED
Closed: 8 years ago8 years ago
Priority: -- → P1
Resolution: --- → FIXED
Target Milestone: Future → 4.9
You need to log in before you can comment on or make changes to this bug.