686462 - shlibsign fails on Linux when trying to sign libraries

Status: VERIFIED WORKSFORME

(NSS :: Build, defect)

Description

[(no longer active)]

Today I noticed that my mozilla-central build is failing on Linux like this:

make[6]: Entering directory `/home/ehsan/moz/src/security/nss/cmd/shlibsign'
make[6]: warning: -jN forced in submake: disabling jobserver mode.
cd mangle; make -j1 export
make[7]: Entering directory `/home/ehsan/moz/src/security/nss/cmd/shlibsign/mangle'
make[7]: Leaving directory `/home/ehsan/moz/src/security/nss/cmd/shlibsign/mangle'
cd mangle; make -j1 libs
make[7]: Entering directory `/home/ehsan/moz/src/security/nss/cmd/shlibsign/mangle'
make[7]: Leaving directory `/home/ehsan/moz/src/security/nss/cmd/shlibsign/mangle'
/home/ehsan/moz/src/objdir-ff-dbg/nss/shlibsign -v -i /home/ehsan/moz/src/objdir-ff-dbg/security/manager/../../dist/lib/libsoftokn3.so
moduleSpec configdir='' certPrefix='' keyPrefix='' secmod='' flags=noCertDB, noModDB
Generate a DSA key pair ... 
DSA key pair generation failed: 0x00000005, CKR_GENERAL_ERROR                   
NSPR error code: -8182: Unknown code ___f 10
/home/ehsan/moz/src/objdir-ff-dbg/nss/shlibsign -v -i /home/ehsan/moz/src/objdir-ff-dbg/security/manager/../../dist/lib/libfreebl3.so
moduleSpec configdir='' certPrefix='' keyPrefix='' secmod='' flags=noCertDB, noModDB
Generate a DSA key pair ... 
DSA key pair generation failed: 0x00000005, CKR_GENERAL_ERROR                   
NSPR error code: -8182: Unknown code ___f 10
/home/ehsan/moz/src/objdir-ff-dbg/nss/shlibsign -v -i /home/ehsan/moz/src/objdir-ff-dbg/security/manager/../../dist/lib/libnssdbm3.so
moduleSpec configdir='' certPrefix='' keyPrefix='' secmod='' flags=noCertDB, noModDB
Generate a DSA key pair ... 
DSA key pair generation failed: 0x00000005, CKR_GENERAL_ERROR                   
NSPR error code: -8182: Unknown code ___f 10
make[6]: Leaving directory `/home/ehsan/moz/src/security/nss/cmd/shlibsign'
/home/ehsan/moz/src/objdir-ff-dbg/config/nsinstall: cannot access /home/ehsan/moz/src/objdir-ff-dbg/security/manager/../../dist/lib/libsoftokn3.chk: No such file or directory
make[5]: *** [libs] Error 1
make[5]: Leaving directory `/home/ehsan/moz/src/objdir-ff-dbg/security/manager'
make[4]: *** [libs_tier_platform] Error 2

I ran shlibsign under gdb for libsoftokn3.so, and the failure is stemming from here: <http://mxr.mozilla.org/mozilla-central/source/security/nss/lib/freebl/dsa.c#592> while verifying the DSA key, with this backtrace:

(gdb) bt
#0  DSA_VerifyDigest (key=0x44cda0, signature=0x7fffffffc340, digest=0x7fffffffc328) at dsa.c:592
#1  0x00007ffff7f557a2 in DSA_VerifyDigest (key=0x44cda0, signature=0x7fffffffc340, digest=0x7fffffffc328) at loader.c:277
#2  0x00007ffff7f30830 in nsc_DSA_Verify_Stub (ctx=0x44cd90, sigBuf=0x4120e0, sigLen=40, dataBuf=0x7ffff7f5c592, dataLen=20) at pkcs11c.c:1889
#3  0x00007ffff7f30a3d in NSC_Verify (hSession=1, pData=0x7ffff7f5c592 "Mozilla Rules World!", ulDataLen=20, pSignature=0x4120e0 "QvK\304\315ѕ\237\373b\305\346;\235\363\264\212\377", ulSignatureLen=40) at pkcs11c.c:2694
#4  0x00007ffff7f34ee1 in sftk_PairwiseConsistencyCheck (hSession=1, publicKey=0x4491a0, privateKey=0x44a960, keyType=1) at pkcs11c.c:3699
#5  0x00007ffff7f347a7 in NSC_GenerateKeyPair (hSession=1, pMechanism=0x7fffffffccc8, pPublicKeyTemplate=0x7fffffffcc50, ulPublicKeyAttributeCount=5, pPrivateKeyTemplate=0x7fffffffcbd0, ulPrivateKeyAttributeCount=5, 
    phPublicKey=0x7fffffffcbc8, phPrivateKey=0x7fffffffcbc0) at pkcs11c.c:4259
#6  0x0000000000401d6d in main (argc=4, argv=0x7fffffffe288) at shlibsign.c:849
(gdb) bt full
#0  DSA_VerifyDigest (key=0x44cda0, signature=0x7fffffffc340, digest=0x7fffffffc328) at dsa.c:592
        p = {sign = 0, alloc = 64, used = 16, dp = 0x44d630}
        q = {sign = 0, alloc = 64, used = 3, dp = 0x44d840}
        g = {sign = 0, alloc = 64, used = 16, dp = 0x44f710}
        u1 = {sign = 0, alloc = 64, used = 3, dp = 0x44e8c0}
        w = {sign = 0, alloc = 64, used = 3, dp = 0x44ead0}
        err = 0
        verified = SECFailure
        r_ = {sign = 0, alloc = 64, used = 3, dp = 0x44de70}
        u2 = {sign = 0, alloc = 64, used = 3, dp = 0x44e290}
        s_ = {sign = 0, alloc = 64, used = 3, dp = 0x44e080}
        v = {sign = 0, alloc = 64, used = 3, dp = 0x44e6b0}
        y = {sign = 0, alloc = 64, used = 16, dp = 0x44f500}
#1  0x00007ffff7f557a2 in DSA_VerifyDigest (key=0x44cda0, signature=0x7fffffffc340, digest=0x7fffffffc328) at loader.c:277
No locals.
#2  0x00007ffff7f30830 in nsc_DSA_Verify_Stub (ctx=0x44cd90, sigBuf=0x4120e0, sigLen=40, dataBuf=0x7ffff7f5c592, dataLen=20) at pkcs11c.c:1889
        signature = {type = 4511104, data = 0x4120e0 "QvK\304\315ѕ\237\373b\305\346;\235\363\264\212\377", len = 40}
        digest = {type = 4493392, data = 0x7ffff7f5c592 "Mozilla Rules World!", len = 20}
        key = 0x44cd90
#3  0x00007ffff7f30a3d in NSC_Verify (hSession=1, pData=0x7ffff7f5c592 "Mozilla Rules World!", ulDataLen=20, pSignature=0x4120e0 "QvK\304\315ѕ\237\373b\305\346;\235\363\264\212\377", ulSignatureLen=40) at pkcs11c.c:2694
        session = 0x449050
        context = 0x44d580
        crv = 0
#4  0x00007ffff7f34ee1 in sftk_PairwiseConsistencyCheck (hSession=1, publicKey=0x4491a0, privateKey=0x44a960, keyType=1) at pkcs11c.c:3699
        isEncryptable = 0
        ciphertext = 0x7fffffffc4d0 "\001"
        text_compared = 0x4491a0 ""
        known_digest = 0x7ffff7f5c592 "Mozilla Rules World!"
        mech = {mechanism = 17, pParameter = 0x0, ulParameterLen = 0}
        canSignVerify = 1
        isDerivable = 0
        signature_length = 40
        modulusLen = 140737353339156
        crv = 0
        known_message = 0x7ffff7f5c57d "Known Crypto Message"
        plaintext = "ؚD\000\000\000\000\000\001\000\000\000\000\000\000\000\207\310\377\377"
        bytes_decrypted = 140737353339849
        bytes_encrypted = 4496088
        bytes_compared = 4493728
        signature = 0x4120e0 "QvK\304\315ѕ\237\373b\305\346;\235\363\264\212\377"
#5  0x00007ffff7f347a7 in NSC_GenerateKeyPair (hSession=1, pMechanism=0x7fffffffccc8, pPublicKeyTemplate=0x7fffffffcc50, ulPublicKeyAttributeCount=5, pPrivateKeyTemplate=0x7fffffffcbd0, ulPrivateKeyAttributeCount=5, 
    phPublicKey=0x7fffffffcbc8, phPrivateKey=0x7fffffffcbc0) at pkcs11c.c:4259
        session = 0x449050
        cktrue = 1 '\001'
        slot = 0x412130
        bitSize = 1018
        pubExp = {type = 4294953920, data = 0x7fffffffcbc8 "", len = 4294953936}
        rsaPriv = 0x7fffffffccc8
        ecPriv = 0x0
        privateKey = 0x44a960
        pubClass = 2
        publicKey = 0x4491a0
        key_type = 1
        crv = 0
        rv = SECSuccess
        ecEncodedParams = {type = 4160294688, data = 0x7fffffffcbc8 "", len = 0}
        privClass = 3
        i = 5
        ecParams = 0xff000000000000
        public_modulus_bits = 0
        pqgParam = {arena = 0xff000000000000, prime = {type = 4198592, data = 0x44c000 ";*\376^\"\302\223\233B\364b/\017\214\221duStW,\037\234", len = 129}, subPrime = {type = siBuffer, data = 0x40b5a0 "0\321@", len = 21}, base = {
            type = 4294953072, data = 0x44c090 ";*\376^\"\302\223\233B\364b/\017\214\221duStW,\037\234", len = 128}}
        dhParam = {arena = 0x0, prime = {type = siBuffer, data = 0x0, len = 0}, base = {type = 4159741952, data = 0x7fffffffcbd0 "\001", len = 0}}
        dsaPriv = 0x44c3c0
        private_value_bits = 0
        dhPriv = 0x0
#6  0x0000000000401d6d in main (argc=4, argv=0x7fffffffe288) at shlibsign.c:849
        rv = PR_SUCCESS
        verify = 0
---Type <return> to continue, or q <return> to quit---
        stat_buf = {st_dev = 140737488346928, st_ino = 140737488346952, st_nlink = 140737354052952, st_mode = 0, st_uid = 0, st_gid = 4160158080, __pad0 = 32767, st_rdev = 140737353543680, st_size = 140737349688826, 
          st_blksize = 140737343948576, st_blocks = 140737349681664, st_atim = {tv_sec = 4294967296, tv_nsec = 4294967455}, st_mtim = {tv_sec = 140737351932842, tv_nsec = 140737354127832}, st_ctim = {tv_sec = 140737488347056, 
            tv_nsec = 140737488347264}, __unused = {140737354131232, 29848917, 1}}
        digestmech = {mechanism = 544, pParameter = 0x0, ulParameterLen = 0}
        fd = 0x7fffffffe2b0
        bytesWritten = 0
        count = 0
        hRwSession = 1
        hDSApubKey = 0
        link_file = 0x0
        configDir = 0x0
        pSlotList = 0x40b540
        sign = '\000' <repeats 39 times>
        lib = 0x40a460
        link_buf = '\000' <repeats 888 times>"\375, \272\336\367\377\177\000\000\000\000\000\000\000\000\000\000\242\334d\367\377\177\000\000\000\000\000\000\000\000\000\000\312B\336\367\377\177", '\000' <repeats 70 times>, "\001", '\000' <repeats 147 times>"\375, \272\336\367\377\177\000\000\000\000\000\000\000\000\000\000i\270\233\367\377\177\000\000\000\000\000\000\000\000\000\000\312B\336\367\377\177", '\000' <repeats 58 times>, "~\270\233\367\377\177\000\000\000\000\000\000\001", '\000' <repeats 211 times>"\375, \272\336\367\377\177\000\000\000\000\000\000\000\000\000\000\373/\274\367\377\177\000\000\000\000\000\000\000\000\000\000\312B\336\367\377\177", '\000' <repeats 58 times>, "~\270\233\367\377\177\000\000\000\000\000\000\001", '\000' <repeats 163 times>"\375, \272\336\367\377\177\000\000\000\000\000\000\000\000\000\000\317(\371\367\377\177\000\000\000\000\000\000\000\000\000\000\312B\336\367\377\177", '\000' <repeats 43 times>"\240, \027"...
        digest = '\000' <repeats 19 times>
        input_file = 0x7fffffffe583 "/home/ehsan/moz/src/objdir-ff-dbg/security/manager/../../dist/lib/libsoftokn3.so"
        signLen = 0
        dsaKeyPairGenMech = {mechanism = 16, pParameter = 0x0, ulParameterLen = 0}
        dsaPrivKeyTemplate = {{type = 1, pValue = 0x407170, ulValueLen = 1}, {type = 2, pValue = 0x407158, ulValueLen = 1}, {type = 259, pValue = 0x407158, ulValueLen = 1}, {type = 264, pValue = 0x407158, ulValueLen = 1}, {type = 354, 
            pValue = 0x407170, ulValueLen = 1}}
        file_buf = "\000\000\000\000\000\000\000\000\360\337\377\377\377\177\000\000?\000\000\000\377\177\000\000p\366\366\367\377\177\000\000\210\341\377\377\377\177\000\000\b\000\000\000\000\000\000\000.N=\366\000\000\000\000Df\336\367\377\177\000\000\330\331\377\367\377\177\000\000\070\365\330\003\000\000\000\000.\000\000\000\000\000\000\000L\272c\367\377\177\000\000\000\000\000\000\000\000\000\000\360\337\377\377\377\177\000\000@\274c\367\377\177\000\000h{d\367\377\177\000\000\320\341\377\377\377\177\000\000\370\346\377\367\377\177\000\000\017\347w\367\377\177\000\000\060\262\377\367\377\177\000\000\240\341\377\377\377\177\000\000\000\000\000\000\000\000\000\000h\341\377\367\377\177\000\000\000\000\000\000\000\000\000\000\260\366\366\367\377\177\000\000\000\360\366\367\377\177\000\000\067\a@\000\000\000\000\000 \207d\367\377\177\000\000\310\002@\000\000\000\000\000\000\000\000\000\001\000\000\000\367\a\000\000\001\000\000\000bh\336\367\377\177\000\000\000\000\000\000\000\000\000\000\300\344\377\367\377\177\000\000\300\341\377\377\377\177\000\000\260\366\366\367\377\177\000\000\350\341\377\377\377\177\000\000h\341\377\367\377\177\000\000.N"...
        i = -136422486
        pFunctionList = 0x7ffff7f6a7f0
        crv = 0
        digestLen = 0
        signMech = {mechanism = 17, pParameter = 0x0, ulParameterLen = 0}
        optstate = 0x40a3e0
        libname = 0x40a3e0 ""
        output_file = 0x0
        pC_GetFunctionList = 0x7ffff7f20700 <C_GetFunctionList>
        tokenInfo = {label = "NSS Generic Crypto Services     ", manufacturerID = "Mozilla Foundation", ' ' <repeats 14 times>, model = "NSS 3", ' ' <repeats 11 times>, serialNumber = '0' <repeats 16 times>, flags = 1539, 
          ulMaxSessionCount = 0, ulSessionCount = 1, ulMaxRwSessionCount = 0, ulRwSessionCount = 1, ulMaxPinLen = 0, ulMinPinLen = 0, ulTotalPublicMemory = 0, ulFreePublicMemory = 0, ulTotalPrivateMemory = 0, ulFreePrivateMemory = 0, 
          hardwareVersion = {major = 4 '\004', minor = 0 '\000'}, firmwareVersion = {major = 0 '\000', minor = 0 '\000'}, utcTime = '0' <repeats 16 times>}
        program_name = 0x7fffffffe573 "shlibsign"
        ret = 1910330751
        disableUnload = 0x0
        dsaPubKeyValue = {type = 0, pValue = 0x0, ulValueLen = 0}
        bytesRead = 0
        pwd = 0x0
        dbPrefix = 0x0
        slotIndex = 0
        dsaPubKeyTemplate = {{type = 304, pValue = 0x403d10, ulValueLen = 129}, {type = 305, pValue = 0x403da0, ulValueLen = 21}, {type = 306, pValue = 0x403dc0, ulValueLen = 128}, {type = 1, pValue = 0x407170, ulValueLen = 1}, {
            type = 266, pValue = 0x407158, ulValueLen = 1}}
        hDSAprivKey = 0
        dsaPubKey = '\000' <repeats 127 times>
        FIPSMODE = 0

Comment 1

[(no longer active)]

So, I discovered that this only happens with clang.  It happens for me on clang trunk, as well as the older version that I used to have before it:

ehsan@ehsan-Macmini:~/moz/src$ clang --version
clang version 3.0 (trunk 139648)
Target: x86_64-unknown-linux-gnu
Thread model: posix

I'll attach my full build log in a moment.

Comment 2

[(no longer active)]

Attachment: Full build log

Comment 3

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

I think this got fixed along the way.

I just fixed the last c++ 0x bug I was having and was able to build firefox with clang on linux and libstdc++ 4.6 headers.

Comment 4

[(no longer active)]

Indeed.  A new build was successful!