459626 - Certutil is not building properly on Scratchbox ARM builds

Status: VERIFIED FIXED

(Firefox for Android Graveyard :: General, defect)

Description

[cmtalbert]

When we run a scratchbox build with --enable-tests, the build goes into an infinite loop when running  python ../../_profile/pgo/genpgocert.py --gen-server

With some debugging I found out that the busy wait we end up in is because this in turn calls <objdir>/dist/bin/certutil which errors out.  Python will wait forever because it doesn't get a proper return status from the .process call.

If you call certutil directly using the same set of arguments that genpgocert.py is using: certutil -N -d ../../_profile/pgo/certs -f ../../_profile/pgo/certs/.crtdbpw

then you get the following error: Inconsistency detected by ld.so: ../sysdeps/unix/sysv/linux/dl-origin.c: 48: _dl_get_origin: Assertion `linkval[0] == '/'' failed!

If you run ldd certutil it seg faults.

So, certutil is not being built properly.  I'm not entirely sure why yet.  I'll re-run my build and capture the output to see if I can tell why, but this is pretty bad.  I thought a bug was already on file for this but I didn't see one.

Any ideas or help debugging this issue is greatly appreciated.

This is marked critical because we need to build with --enable-tests in order to get our testing system to run on Fennec on the device itself.  This is blocking that effort.  It needs to be resolved as soon as possible.

== Expected ==
Run a build with --enable-tests
The build completes with a testable product and certutil is functioning properly.

Comment 1

[cmtalbert]

Here is the command that builds certutil.  The build step succeeds.

/scratchbox/tools/bin/make -C /home/mozilla/mobile-src/security/nss/cmd/lib MAKE="/scratchbox/tools/bin/make -j1" -j1 CC="gcc" SOURCE_MD_DIR=/home/mozilla/mobile-debug/xulrunner/dist DIST=/home/mozilla/mobile-debug/xulrunner/dist NSPR_INCLUDE_DIR=/home/mozilla/mobile-debug/xulrunner/dist/include/nspr NSPR_LIB_DIR=/home/mozilla/mobile-debug/xulrunner/dist/lib MOZILLA_CLIENT=1 NO_MDUPDATE=1 NSS_ENABLE_ECC=1 BUILD_TREE=/home/mozilla/mobile-debug/xulrunner NS_USE_GCC=1 NS_USE_NATIVE=

However, the generated utility is useless.  There is a #ifdef XP_UNIX in the certutil.c file here: http://mxr.mozilla.org/mozilla-central/source/security/nss/cmd/certutil/certutil.c#55.

Perhaps that's not getting included (though if that were a problem, then we'd have compilation errors). I'm really not sure why this library isn't linking properly, and less sure as to why no errors are thrown.

Comment 2

[(not currently active) Ted Mielczarek]

I think the problem here is that we're building an ARM binary for certutil, and then trying to execute it in the scratchbox environment, so it fails. We discussed this in bug 428009 comment 72 and on, but never filed a followup bug. My suggestion was to require the presence of certutil/pk12util on the host machine for cross-compiled builds that don't --disable-tests. It's easy enough to get a copy of those (download an NSS package, or build Firefox for your native architecture), and it wouldn't require a large change in the build system.

Comment 3

[(not currently active) Ted Mielczarek]

Might be a dupe of bug 454881, although this seems to explain it better.

Comment 4

[cmtalbert]

(In reply to comment #3)
> Might be a dupe of bug 454881, although this seems to explain it better.

FWIW, I think there are two issues here.  I think that this certutil issue is what's causing the root problem.  But I think that the structure of the python calls as outlined in bug 454881, comment 10 shows that the call chain is a bit broken and if anything goes wrong with the process spawn the calling python process doesn't hear back from it's children, hence the hang.

Comment 5

[(not currently active) Ted Mielczarek]

Per bug 459699 we're no longer running certutil at build time, we have a prebuilt cert db checked in, and it can be updated as necessary by developers outside of the normal build process. This should no longer be an issue.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

this is not an issue