874261 - "B2G Arm opt" and "B2G Arm debug" build bots need to build libnssutil3.so for certutil and pk12util

Status: RESOLVED WORKSFORME

(Firefox Build System :: General, defect, P1)

Description

[Wan-Teh Chang]

[I am not familiar with the way Mozilla builds NSS right now, so parts
of my description of the problem may be inaccurate.]

In NSS_3_15_BETA5, a new function SECITEM_ReallocItemV2 is added to
libnssutil3.so.

When I pushed NSS_3_15_BETA5 to mozilla-inbound, it broke three Mozilla
build bots: "B2G Arm opt", "B2G Arm debug", and "Android 2.2 Armv6 opt".

The builds failed when linking the NSS command-line tools certutil and
pk12util.  Here are the linker command line and the error message:

/builds/slave/m-in-ics_a7_g-0000000000000000/build/gonk-toolchain/prebuilt/linux-x86/toolchain/arm-linux-androideabi-4.4.x/bin/arm-linux-androideabi-gcc -o /builds/slave/m-in-ics_a7_g-0000000000000000/build/obj-b2g/security/nss/cmd/pk12util/pk12util -O2 -gdwarf-2 -std=gnu89 -fPIC -DLINUX2_1 -DANDROID -isystem /builds/slave/m-in-ics_a7_g-0000000000000000/build/gonk-toolchain/bionic/libc/arch-arm/include -isystem /builds/slave/m-in-ics_a7_g-0000000000000000/build/gonk-toolchain/bionic/libc/include/ -isystem /builds/slave/m-in-ics_a7_g-0000000000000000/build/gonk-toolchain/bionic/libc/kernel/common -isystem /builds/slave/m-in-ics_a7_g-0000000000000000/build/gonk-toolchain/bionic/libc/kernel/arch-arm -isystem /builds/slave/m-in-ics_a7_g-0000000000000000/build/gonk-toolchain/bionic/libm/include -I/builds/slave/m-in-ics_a7_g-0000000000000000/build/gonk-toolchain/system -I/builds/slave/m-in-ics_a7_g-0000000000000000/build/gonk-toolchain/system/core/include -isystem /builds/slave/m-in-ics_a7_g-0000000000000000/build/gonk-toolchain/bionic -I/builds/slave/m-in-ics_a7_g-0000000000000000/build/gonk-toolchain/hardware/libhardware/include -I/builds/slave/m-in-ics_a7_g-0000000000000000/build/gonk-toolchain/frameworks/base/opengl/include -I/builds/slave/m-in-ics_a7_g-0000000000000000/build/gonk-toolchain/frameworks/base/native/include -I/builds/slave/m-in-ics_a7_g-0000000000000000/build/gonk-toolchain/frameworks/base/include -I/builds/slave/m-in-ics_a7_g-0000000000000000/build/gonk-toolchain/frameworks/base/services/camera -I/builds/slave/m-in-ics_a7_g-0000000000000000/build/gonk-toolchain/frameworks/base/include/media/stagefright -I/builds/slave/m-in-ics_a7_g-0000000000000000/build/gonk-toolchain/frameworks/base/include/media/stagefright/openmax -I/builds/slave/m-in-ics_a7_g-0000000000000000/build/gonk-toolchain/frameworks/base/media/libstagefright/rtsp -I/builds/slave/m-in-ics_a7_g-0000000000000000/build/gonk-toolchain/frameworks/base/media/libstagefright/include -I/builds/slave/m-in-ics_a7_g-0000000000000000/build/gonk-toolchain/external/dbus -I/builds/slave/m-in-ics_a7_g-0000000000000000/build/gonk-toolchain/external/bluetooth/bluez/lib -I/builds/slave/m-in-ics_a7_g-0000000000000000/build/gonk-toolchain/dalvik/libnativehelper/include/nativehelper  -Wall -Wpointer-arith -Wdeclaration-after-statement -Werror=return-type -Wtype-limits -Wempty-body -Wsign-compare -Wno-unused -mandroid -fno-short-enums -fno-exceptions -march=armv7-a -mthumb -mfpu=vfp -mfloat-abi=softfp -std=gnu99 -fgnu89-inline -fno-strict-aliasing -ffunction-sections -fdata-sections -pipe  -DNDEBUG -DTRIMMED -g -Os -freorder-blocks -fno-reorder-functions -fomit-frame-pointer -DCHECK_FORK_GETPID -DRTLD_NOLOAD=0 -include /builds/slave/m-in-ics_a7_g-0000000000000000/build/security/manager/android_stub.h -Wall -Werror-implicit-function-declaration -Wno-switch -pipe -DLINUX -Dlinux -DHAVE_STRERROR -DXP_UNIX -DNSPR20 -UDEBUG -DNDEBUG -D_REENTRANT -DNSS_ENABLE_ECC -DNSS_DISABLE_DBM -DUSE_UTIL_DIRECTLY -DNO_NSPR_10_SUPPORT -I/builds/slave/m-in-ics_a7_g-0000000000000000/build/obj-b2g/dist/include/nspr -I/builds/slave/m-in-ics_a7_g-0000000000000000/build/obj-b2g/dist/include/nspr -I/builds/slave/m-in-ics_a7_g-0000000000000000/build/obj-b2g/security/build/../../dist/include/nss  -I/builds/slave/m-in-ics_a7_g-0000000000000000/build/obj-b2g/security/build/../../dist/private/nss  -I/builds/slave/m-in-ics_a7_g-0000000000000000/build/obj-b2g/security/build/../../dist/include/dbm -I/builds/slave/m-in-ics_a7_g-0000000000000000/build/obj-b2g/security/build/../../dist/include/seccmd  /builds/slave/m-in-ics_a7_g-0000000000000000/build/obj-b2g/security/nss/cmd/pk12util/pk12util.o -mandroid -L/builds/slave/m-in-ics_a7_g-0000000000000000/build/gonk-toolchain/out/target/product/generic/obj/lib -Wl,-rpath-link=/builds/slave/m-in-ics_a7_g-0000000000000000/build/gonk-toolchain/out/target/product/generic/obj/lib --sysroot=/builds/slave/m-in-ics_a7_g-0000000000000000/build/gonk-toolchain/out/target/product/generic/obj/  -mthumb -Wl,-z,noexecstack -Wl,--build-id -Wl,--icf=safe   -Wl,-rpath-link,/builds/slave/m-in-ics_a7_g-0000000000000000/build/obj-b2g/dist/bin -Wl,-rpath-link,/usr/local/lib  -L/builds/slave/m-in-ics_a7_g-0000000000000000/build/obj-b2g/dist/lib -lmozglue -Wl,--wrap=getaddrinfo,--wrap=freeaddrinfo,--wrap=gai_strerror -Wl,--wrap=PR_GetEnv,--wrap=PR_SetEnv /builds/slave/m-in-ics_a7_g-0000000000000000/build/obj-b2g/security/build/../../dist/lib/libsectool.a  -L/builds/slave/m-in-ics_a7_g-0000000000000000/build/obj-b2g/security/build/../../dist/lib -lssl3 -lsmime3 -lnss3 -L/builds/slave/m-in-ics_a7_g-0000000000000000/build/obj-b2g/security/build/../../dist/lib -lnssutil3 -L/builds/slave/m-in-ics_a7_g-0000000000000000/build/obj-b2g/dist/lib -lplc4 -lplds4 -lnspr4   -ldl -lc

/builds/slave/m-in-ics_a7_g-0000000000000000/build/gonk-toolchain/prebuilt/linux-x86/toolchain/arm-linux-androideabi-4.4.x/bin/../lib/gcc/arm-linux-androideabi/4.4.3/../../../../arm-linux-androideabi/bin/ld: /builds/slave/m-in-ics_a7_g-0000000000000000/build/obj-b2g/security/build/../../dist/lib/libsectool.a(secutil.o): in function SECU_ReadDERFromFile:secutil.c:533: error: undefined reference to 'SECITEM_ReallocItemV2'
collect2: ld returned 1 exit status

Note that we pass -lnssutil3 to the linker. The build log showed the static
library libnssutil.a was rebuilt but the shared library libnssutil3.so was
not rebuilt.  Note that "3" is only in the shared library's name.

Comment 1

[Wan-Teh Chang]

NSS_3_15_BETA5 was built successfully on some Android build bots, such
as "Android 2.2 x86 opt", "Android 2.2 NoIon opt", and "Android 2.2 opt".
On those bots, libnss3.so was rebuilt, and libnssutil3.so was a symbolic
link to libnss3.so.

So the problem seems to be that "B2G Arm opt", "B2G Arm debug", and
"Android 2.2 Armv6 opt" did not rebuild libnss3.so.

Comment 2

[Phil Ringnalda (:philor)]

Because I believe anything about our build system can be fixed by a clobber until proven otherwise, I clobbered those three jobs and retriggered them on your push.

Comment 3

[Wan-Teh Chang]

Mike, Ted: could you take a look at this build system problem?
This is preventing us from pushing an NSS patch to mozilla-inbound.

This problem seems to be a dependency bug in some makefile.
libnss3.so is not relinked after security/nss/lib/util/secitem.c
is recompiled.

Comment 4

[Wan-Teh Chang]

Attachment: "B2G Arm opt" build log

This build log is downloaded from
http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-ics_armv7a_gecko/1369088923/mozilla-inbound-ics_armv7a_gecko-bm61-build1-build164.txt.gz

You can see that secitem.c was recompiled into secitem.o, and libnssutil.a
was rebuilt to contain the new secitem.o, but libnss3.so was not rebuilt.

Comment 5

[Wan-Teh Chang]

As a comparison, please see the "Android 2.2 opt" clobber build log at
http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-inbound-android/1369088924/mozilla-inbound-android-bm65-build1-build132.txt.gz

(This build log is too big to upload to Bugzilla.)

This build was done after a manual clobber by philor (see comment 2).
This build succeeded.

You will see libnss3.so was built with libnssutil.a as one of the
inputs.

Comment 6

[Kai Engert [:KaiE:]]

Hello Mike, it would be great if you could have a look, your help would be much appreciated, thanks in advance!

Comment 7

[Mike Hommey [:glandium]]

There is no obvious reason why this would be happening. Building libnss3.so the first time creates a .deps/libnss3.so.pp file which contains the proper dependencies, used in subsequent (incremental) builds. The only way i can see this failing is if .deps/libnss3.so.pp is not populated correctly, but i don't see why that would happen either.

IOW, without isolating a failing buildbot and looking at the tree state, there's nothing i can do.

Comment 8

[Kai Engert [:KaiE:]]

I've started a try build, for android and b2g platforms, with a quick crashtest, only (should be sufficient, since we're only testing if the build succeeds or not).

I understand we need to revert the reversion patch in mozilla/security/patches for the test. That's what I did.

Try build here: https://tbpl.mozilla.org/?tree=Try&rev=3aafa9e95a35

Comment 9

[Kai Engert [:KaiE:]]

This error didn't happen again.

Closing.