1419009 - Sigsegv at Hacl_EC_crypto_scalarmult on Solaris

Status: RESOLVED INVALID

(NSS :: Libraries, defect)

Description

[Petr Sumbera]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0
Build ID: 20171112125346

Steps to reproduce:

Firefox (trunk) cores dump when accessing sites over TLS on Solaris intel (I couldn't verify sparc as it doesn't build now). Last version I can really confirm it's working is 52.

t@6 (l@6) terminated by signal SEGV (Segmentation Fault)
0x00007fffbea55d4a: __lwp_sigqueue+0x000a:      jae      __lwp_sigqueue+0x18    [ 0x7fffbea55d58, .+0xe ]
(dbx) where
current thread: t@6
=>[1] __lwp_sigqueue(0x0, 0x6, 0xffffa100275240c0, 0x0, 0xffffffff, 0x0), at 0x7fffbea55d4a
  [2] thr_kill(), at 0x7fffbea4c5c2
  [3] raise(), at 0x7fffbe9874a9
  [4] nsProfileLock::FatalSignalHandler(), at 0x7fffb79dcc91
  [5] js::UnixExceptionHandler(), at 0x7fffb7df31b3
  [6] WasmFaultHandler<(Signal)0>(), at 0x7fffb7fe99c9
  [7] __sighndlr(), at 0x7fffbea4f116
  [8] call_user_handler(), at 0x7fffbea40bc1
  [9] sigacthandler(), at 0x7fffbea40fde
  ---- called from signal handler with signal 11 (SIGSEGV) ------
  [10] Hacl_EC_crypto_scalarmult(), at 0x7fff9e47654c
  [11] Curve25519_crypto_scalarmult(), at 0x7fff9e4768e7
  [12] ec_Curve25519_mul(), at 0x7fff9e43fe9a
  [13] ec_Curve25519_pt_mul(), at 0x7fff9e44340c
  [14] ec_NewKey(), at 0x7fff9e43e4e8
  [15] EC_NewKey(), at 0x7fff9e43e604
  [16] EC_NewKey(), at 0x7fffbd634246
  [17] NSC_GenerateKeyPair(), at 0x7fff9e63b2d4
  [18] PK11_GenerateKeyPairWithOpFlags(), at 0x7fffbdc8be3a
  [19] SECKEY_CreateECPrivateKey(), at 0x7fffbdc74120
  [20] ssl_CreateECDHEphemeralKeyPair(), at 0x7fffbd647ccb
  [21] tls13_CreateKeyShare(), at 0x7fffbd659c46
  [22] tls13_SetupClientHello(), at 0x7fffbd659d48
  [23] ssl3_SendClientHello(), at 0x7fffbd6412db
  [24] ssl_BeginClientHandshake(), at 0x7fffbd64e11d
  [25] ssl_Do1stHandshake(), at 0x7fffbd64fbce
  [26] SSL_ForceHandshake(), at 0x7fffbd650190
  [27] nsNSSSocketInfo::DriveHandshake(), at 0x7fffb7858784
  [28] mozilla::net::nsHttpConnection::EnsureNPNComplete(), at 0x7fffb4e0cfe5
  [29] mozilla::net::nsHttpConnection::OnSocketWritable(), at 0x7fffb4e0d987
  [30] mozilla::net::nsHttpConnection::OnOutputStreamReady(), at 0x7fffb4e0de31
  [31] mozilla::net::nsHttpConnection::Activate(), at 0x7fffb4e0e241
  [32] mozilla::net::nsHttpConnectionMgr::DispatchAbstractTransaction(), at 0x7fffb4e0e44e
  [33] mozilla::net::nsHttpConnectionMgr::DispatchTransaction(), at 0x7fffb4e0e6d4
  [34] mozilla::net::nsHttpConnectionMgr::nsHalfOpenSocket::SetupConn(), at 0x7fffb4e107c6
  [35] mozilla::net::nsHttpConnectionMgr::nsHalfOpenSocket::OnOutputStreamReady(), at 0x7fffb4e11189
  [36] mozilla::net::nsSocketOutputStream::OnSocketReady(), at 0x7fffb4b15202
  [37] mozilla::net::nsSocketTransport::OnSocketReady(), at 0x7fffb4b18ceb
  [38] mozilla::net::nsSocketTransportService::DoPollIteration(), at 0x7fffb4b1b086
  [39] mozilla::net::nsSocketTransportService::Run(), at 0x7fffb4b1f5fe
  [40] nsThread::ProcessNextEvent(), at 0x7fffb4a8ad96
  [41] NS_ProcessNextEvent(), at 0x7fffb4a919bb
  [42] mozilla::ipc::MessagePumpForNonMainThreads::Run(), at 0x7fffb4e6a8e5
  [43] MessageLoop::RunInternal(), at 0x7fffb4e43ccb
  [44] MessageLoop::Run(), at 0x7fffb4e43ef9
  [45] nsThread::ThreadFunc(), at 0x7fffb4a872ce
  [46] _pt_root(), at 0x7fffbe63bd5d
  [47] _thrp_setup(), at 0x7fffbea4ed14
  [48] _lwp_start(), at 0x7fffbea4eff0
  
  
firefox:core> libfreebl3.so`Hacl_EC_crypto_scalarmult+0x39::dis
libfreebl3.so`Hacl_EC_crypto_scalarmult+0x12:   movq   %rdi,%rbx
libfreebl3.so`Hacl_EC_crypto_scalarmult+0x15:   leaq   -0x70(%rbp),%rdi
libfreebl3.so`Hacl_EC_crypto_scalarmult+0x19:   movl   $0x0,%eax
libfreebl3.so`Hacl_EC_crypto_scalarmult+0x1e:   movl   $0xa,%ecx
libfreebl3.so`Hacl_EC_crypto_scalarmult+0x23:   rep stosq %rax,(%rdi)
libfreebl3.so`Hacl_EC_crypto_scalarmult+0x26:   movq   (%rdx),%rdi
libfreebl3.so`Hacl_EC_crypto_scalarmult+0x29:   movq   0x6(%rdx),%rcx
libfreebl3.so`Hacl_EC_crypto_scalarmult+0x2d:   movq   0xc(%rdx),%r11
libfreebl3.so`Hacl_EC_crypto_scalarmult+0x31:   movq   0x13(%rdx),%r8
libfreebl3.so`Hacl_EC_crypto_scalarmult+0x35:   movq   0x18(%rdx),%r10
libfreebl3.so`Hacl_EC_crypto_scalarmult+0x39:   movzbl 0x7(%rcx),%r9d  <========= rcx is zero
libfreebl3.so`Hacl_EC_crypto_scalarmult+0x3e:   shll   $0x8,%r9d
libfreebl3.so`Hacl_EC_crypto_scalarmult+0x42:   movzbl 0x6(%rcx),%edx
libfreebl3.so`Hacl_EC_crypto_scalarmult+0x46:   orl    %edx,%r9d
libfreebl3.so`Hacl_EC_crypto_scalarmult+0x49:   shll   $0x10,%r9d
libfreebl3.so`Hacl_EC_crypto_scalarmult+0x4d:   movzbl 0x5(%rcx),%r14d
libfreebl3.so`Hacl_EC_crypto_scalarmult+0x52:   shll   $0x8,%r14d
libfreebl3.so`Hacl_EC_crypto_scalarmult+0x56:   movzbl 0x4(%rcx),%edx
libfreebl3.so`Hacl_EC_crypto_scalarmult+0x5a:   orl    %edx,%r14d
libfreebl3.so`Hacl_EC_crypto_scalarmult+0x5d:   orl    %r9d,%r14d
libfreebl3.so`Hacl_EC_crypto_scalarmult+0x60:   movq   %r14,%r9

Comment 1

[Franziskus Kiefer [:franziskus]]

Petr, do you have a debug build at hand to get a little more information on the issue?
How's the Firefox/NSS build patched on Solaris to work around bug 1405268?

Comment 2

[Petr Sumbera]

I don't have debug build. I can to produce it (based on https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Building_Firefox_with_Debug_Symbols).

As for workaround for bug 1405268 I used patch from the bug (https://bugzilla.mozilla.org/attachment.cgi?id=8914707).

Comment 3

[Franziskus Kiefer [:franziskus]]

Without more information it's a little hard to tell what's happening. The actual line number for 

>  [10] Hacl_EC_crypto_scalarmult(), at 0x7fff9e47654c

would be great. And infos on the build (compiler, flags, etc.).
Since this code changed completely from 56 to 57 it would also be interesting if 56 works.

Comment 4

[Petr Sumbera]

Following is stack with debug information:

t@6 (l@6) terminated by signal SEGV (Segmentation Fault)
0x00007fffbea55d4a: __lwp_sigqueue+0x000a:      jae      __lwp_sigqueue+0x18    [ 0x7fffbea55d58, .+0xe ]
Current function is Hacl_EC_crypto_scalarmult
 1024   {
(dbx) where
current thread: t@6
  [1] __lwp_sigqueue(0x0, 0x6, 0xffffa10024606000, 0x0, 0xffffffff, 0x0), at 0x7fffbea55d4a
  [2] thr_kill(), at 0x7fffbea4c5c2
  [3] raise(), at 0x7fffbe9874a9
  [4] nsProfileLock::FatalSignalHandler(), at 0x7fffb79e662d
  [5] js::UnixExceptionHandler(), at 0x7fffb7dfcfaf
  [6] WasmFaultHandler<(Signal)0>(), at 0x7fffb7ff3db3
  [7] __sighndlr(), at 0x7fffbea4f116
  [8] call_user_handler(), at 0x7fffbea40bc1
  [9] sigacthandler(), at 0x7fffbea40fde
  ---- called from signal handler with signal 11 (SIGSEGV) ------
=>[10] Hacl_EC_crypto_scalarmult(mypublic = 0x653267690a346631 "<bad address 0x653267690a346631>", secret = 0xa6e3167690a6132 "<bad address 0x0a6e3167690a6132>", basepoint = 0x67690a6965723367 "<bad address 0x67690a6965723367>"), line 1024 in "hacl_curve25519_64.c"
  [11] Curve25519_crypto_scalarmult(mypublic = 0xa70316d3261326c "<bad address 0x0a70316d3261326c>", secret = 0x316c320a75326131 "<bad address 0x316c320a75326131>", basepoint = 0x6c0a7a3172326535 "<bad address 0x6c0a7a3172326535>"), line 1042 in "hacl_curve25519_64.c"
  [12] ec_Curve25519_mul(mypublic = 0x65636f7270006563 "<bad address 0x65636f7270006563>", secret = 0x766974614e495255 "<bad address 0x766974614e495255>", basepoint = 0x6d616e74736f4872 "<bad address 0x6d616e74736f4872>"), line 10 in "curve25519_64.c"
  [13] ec_Curve25519_pt_mul(X = 0x9480020808948006, k = 0x80b98010e9970100, P = 0x894800208089480), line 104 in "ecp_25519.c"
  [14] ec_NewKey(ecParams = 0x445f544e45564500, privKey = 0x5f544e4556450044, privKeyBytes = 0x4e45564500444550 "<bad address 0x4e45564500444550>", privKeyLen = 1415071060), line 192 in "ec.c"
  [15] EC_NewKey(ecParams = 0x40b900001a000000, privKey = 0x51ba000027000000), line 389 in "ec.c"
  [16] EC_NewKey(params = 0xa03140201, privKey = 0x2100000000000000), line 1166 in "loader.c"
  [17] NSC_GenerateKeyPair(hSession = 7883916677068649317U, pMechanism = 0x63656c655365766f, pPublicKeyTemplate = 0x630074657366664f, ulPublicKeyAttributeCount = 7453001440112756480U, pPrivateKeyTemplate = 0x474e494445434552, ulPrivateKeyAttributeCount = 6867513658151486796U, phPublicKey = 0x7fffab5fdb60, phPrivateKey = 0x7fffab5fdb68), line 4712 in "pkcs11c.c"
  [18] PK11_GenerateKeyPairWithOpFlags(slot = 0xa3273723265702e, type = 3317598428068802606U, param = 0x63316d65722e0a6f, pubKey = 0x732e0a6f67316134, attrFlags = 1633891443U, opFlags = 8299582678994806131U, opFlagsMask = 526336U, wincx = 0x6509c88), line 1140 in "pk11akey.c"
  [19] SECKEY_CreateECPrivateKey(param = 0x492800364ee0000, pubk = 0xb0ee00000001002f, cx = 0x747865746e6f43), line 212 in "seckey.c"
  [20] ssl_CreateECDHEphemeralKeyPair(ss = 0x6f6c6c41676e6967, ecGroup = 0x69724f746567006c, keyPair = 0x6d6f437465470064), line 455 in "ssl3ecc.c"
  [21] tls13_CreateKeyShare(ss = 0x73726564616548, groupDef = 0x7265766f00657461), line 345 in "tls13con.c"
  [22] tls13_SetupClientHello(ss = 0x6f69746163696c70), line 397 in "tls13con.c"
  [23] ssl3_SendClientHello(ss = 0x656c646e61486c6f, type = <unknown enum member 1630614585>), line 4945 in "ssl3con.c"
  [24] ssl_BeginClientHandshake(ss = 0x7453657275747061), line 121 in "sslcon.c"
  [25] ssl_Do1stHandshake(ss = 0x680b9a8018c1a01), line 56 in "sslsecur.c"
  [26] SSL_ForceHandshake(fd = 0x680b98001152801), line 370 in "sslsecur.c"
  [27] nsNSSSocketInfo::DriveHandshake(), at 0x7fffb7861f38
  [28] mozilla::net::nsHttpConnection::EnsureNPNComplete(), at 0x7fffb4e1e2c3
  [29] mozilla::net::nsHttpConnection::OnSocketWritable(), at 0x7fffb4e1ec65
  [30] mozilla::net::nsHttpConnection::OnOutputStreamReady(), at 0x7fffb4e1f10f
  [31] mozilla::net::nsHttpConnection::Activate(), at 0x7fffb4e1f51f
  [32] mozilla::net::nsHttpConnectionMgr::DispatchAbstractTransaction(), at 0x7fffb4e1f72c
  [33] mozilla::net::nsHttpConnectionMgr::DispatchTransaction(), at 0x7fffb4e1f9b2
  [34] mozilla::net::nsHttpConnectionMgr::nsHalfOpenSocket::SetupConn(), at 0x7fffb4e21aa4
  [35] mozilla::net::nsHttpConnectionMgr::nsHalfOpenSocket::OnOutputStreamReady(), at 0x7fffb4e22467
  [36] mozilla::net::nsSocketOutputStream::OnSocketReady(), at 0x7fffb4b24fda
  [37] mozilla::net::nsSocketTransport::OnSocketReady(), at 0x7fffb4b28ac3
  [38] mozilla::net::nsSocketTransportService::DoPollIteration(), at 0x7fffb4b2ae5e
  [39] mozilla::net::nsSocketTransportService::Run(), at 0x7fffb4b2f3d6
  [40] nsThread::ProcessNextEvent(), at 0x7fffb4a9a0da
  [41] NS_ProcessNextEvent(), at 0x7fffb4aa0cff
  [42] mozilla::ipc::MessagePumpForNonMainThreads::Run(), at 0x7fffb4e7bced
  [43] MessageLoop::RunInternal(), at 0x7fffb4e550d3
  [44] MessageLoop::Run(), at 0x7fffb4e55301
  [45] nsThread::ThreadFunc(), at 0x7fffb4a96612
  [46] _pt_root(arg = 0x680c30192), line 125 in "ptthread.c"
  [47] _thrp_setup(), at 0x7fffbea4ed14
  [48] _lwp_start(), at 0x7fffbea4eff0
(dbx)

Comment 5

[Petr Sumbera]

Ok, the core file doesn't give enough information. But adding printfs helps...

It crashes in:

Hacl_EC_crypto_scalarmult()
-> Hacl_EC_Format_fexpand()
-> uint64_t i0 = load64_le(input);
-> #define load64_le(b) (le64toh(load64(b)))
-> #define le64toh(x) LE_IN64(x)

The last definition is entirely Solaris specific and my problem. It's exactly what Franziskus asked about above. I need to look again on my fix for bug 1405268. Sorry for the noise!

Comment 6

[Petr Sumbera]

For record LE_IN64() expects pointer not the value. That was the reason for SIGSEGV.

But still HTTPS pages still doesn't work. It says "Performing TLS handshake" and then "Timed Out". And on console there are following erros:

JavaScript error: resource://gre/modules/TelemetrySession.jsm, line 1698: NS_ERROR_NOT_AVAILABLE: Component returned failure code: 0x80040111 (NS_ERROR_NOT_AVAILABLE) [nsIMemoryReporterManager.residentUnique]

Comment 7

[Benjamin Beurdouche (do not use this account)]

(In reply to Petr Sumbera from comment #6)
> For record LE_IN64() expects pointer not the value. That was the reason for
> SIGSEGV.

When you make it work, can you please send us the proper defines for Solaris
so that we can backport that in Kremlin and in NSS ?
https://github.com/FStarLang/kremlin/blob/master/kremlib/kremlib.h#L241

Comment 8

[Petr Sumbera]

I believe that macros are now just fine: https://github.com/FStarLang/kremlin/pull/70