539744 - NSS libraries Seg fault on Intel Westmere chips

Status: RESOLVED DUPLICATE of bug 536485

(NSS :: Libraries, defect)

Description

[Colm Harrington]

User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7
Build Identifier: NSS 3.12.3

NSS libraries are seg faulting on Intels Westmere chips.

[17/Dec/2009:18:53:24] catastrophe ( 2222): CORE3260: Server crash detected (signal SIGSEGV)
[17/Dec/2009:18:53:24] info ( 2222): CORE3262: Crash occurred in function FREEBL_GetVector from module /sun/webs
erver7/lib/amd64/libfreebl3.so

# mdb core.webservd.2222.1261076004 
mdb: core file data for mapping at fffffd7fff600000 not saved: Bad address
Loading modules: [ libumem.so.1 libc.so.1 libuutil.so.1 ld.so.1 ]
> > $C
fffffd7f75761ca0 libfreebl3.so`intel_aes_decrypt_cbc_256+0x226()
fffffd7f75761d40 libssl3.so`ssl3_HandleRecord+0x5f9()
fffffd7f75761db0 libssl3.so`ssl3_GatherCompleteHandshake+0x233()
fffffd7f75761dd0 libssl3.so`ssl_GatherRecord1stHandshake+0x44()
fffffd7f75761e20 libssl3.so`ssl_SecureRecv+0x176()
fffffd7f75761e60 libssl3.so`ssl_Recv+0x7f()
fffffd7f75761f50 libns-httpd40.so`__1cNDaemonSessionDrun6M_v_+0x5c1()
fffffd7f75761f70 libnsprwrap.so`ThreadMain+0x27()
fffffd7f75761fc0 libnspr4.so`_pt_root+0xe5()
fffffd7f75761fe0 libc.so.1`_thrp_setup+0xbc()
fffffd7f75761ff0 libc.so.1`_lwp_start()


Chip info:

oaf578# isainfo -v
64-bit amd64 applications
	pclmulqdq aes sse4.2 sse4.1 ssse3 popcnt tscp cx16 mon sse3 pause sse2 
	sse fxsr mmx cmov amd_sysc cx8 tsc fpu 
32-bit i386 applications
	pclmulqdq aes sse4.2 sse4.1 ssse3 popcnt tscp ahf cx16 mon sse3 pause 
	sse2 sse fxsr mmx cmov sep cx8 tsc fpu 


oaf578# /export/bench/cpu_tools/cpuid 
CPU:
   vendor_id = "GenuineIntel"
   version information (1/eax):
      processor type  = primary processor (0)
      family          = Intel Pentium Pro/II/III/Celeron, AMD Athlon/Duron, Cyrix M2, VIA C3 (6)
      model           = 0xc (12)
      stepping id     = 0x0 (0)
      extended family = 0x0 (0)
      extended model  = 0x2 (2)
      (simple synth)  = Intel Pentium II / Pentium III / Pentium M / Celeron / Mobile Celeron / Celeron M / Core Solo / Core Duo / Core 2 / Core 2 Extreme Processor / Xeon Processor LV / Xeon Processor 5100 (unknown model)
   miscellaneous (1/ebx):
      process local APIC physical ID = 0x2 (2)
      cpu count                      = 0x20 (32)
      CLFLUSH line size              = 0x8 (8)
      brand index                    = 0x0 (0)
   brand id = 0x00 (0): unknown
   feature information (1/edx):
      x87 FPU on chip                        = true
      virtual-8086 mode enhancement          = true
      debugging extensions                   = true
      page size extensions                   = true
      time stamp counter                     = true
      RDMSR and WRMSR support                = true
      physical address extensions            = true
      machine check exception                = true
      CMPXCHG8B inst.                        = true
      APIC on chip                           = true
      SYSENTER and SYSEXIT                   = true
      memory type range registers            = true
      PTE global bit                         = true
      machine check architecture             = true
      conditional move/compare instruction   = true
      page attribute table                   = true
      page size extension                    = true
      processor serial number                = false
      CLFLUSH instruction                    = true
      debug store                            = true
      thermal monitor and clock ctrl         = true
      MMX Technology                         = true
      FXSAVE/FXRSTOR                         = true
      SSE extensions                         = true
      SSE2 extensions                        = true
      self snoop                             = true
      hyper-threading / multi-core supported = true
      therm. monitor                         = true
      IA64                                   = false
      pending break event                    = true
   feature information (1/ecx):
      PNI/SSE3: Prescott New Instructions    = true
      MONITOR/MWAIT                          = true
      CPL-qualified debug store              = true
      VMX: virtual machine extensions        = true
      Secure mode extensions                 = true
      Enhanced Intel SpeedStep Technology    = true
      thermal monitor 2                      = true
      Supplemental SSE3 instructions         = true
      context ID: adaptive or shared L1 data = false
      cmpxchg16b available                   = true
      xTPR disable                           = true
      processor serial number                = true
      direct cache access                    = true
      SSE4.1 instructions                    = true
      SSE4.2 instructions                    = true
      POPCNT instructions                    = true
      AES instructions                       = true
   cache and TLB information (2):
      0x5a: unknown
      0x03: data TLB: 4K pages, 4-way, 64 entries
      0x55: unknown
      0xff: unknown
      0xb2: unknown
      0xf0: 64 byte prefetching
      0xca: unknown
   processor serial number: 0002-06C0-0000-0000-0000-0000
   deterministic cache parameters (4):
      cache type                        = no more caches (0)
      cache level                       = 0x0 (0)
      self-initializing cache level     = false
      fully associative cache           = false
      extra threads sharing this cache  = 0x0 (0)
      extra processor cores on this die = 0x0 (0)
      system coherency line size        = 0x0 (0)
      physical line partitions          = 0x0 (0)
      ways of associativity             = 0x0 (0)
      number of sets - 1 (s)            = 0
   MONITOR/MWAIT (5):
      smallest monitor-line size (bytes)      = 0x40 (64)
      largest monitor-line size (bytes)       = 0x40 (64)
      enum of Monitor-MWAIT exts supported    = true
      supports intrs as break-event for MWAIT = true
      number of C0 sub C-states using MWAIT   = 0x0 (0)
      number of C1 sub C-states using MWAIT   = 0x2 (2)
      number of C2 sub C-states using MWAIT   = 0x1 (1)
      number of C3 sub C-states using MWAIT   = 0x1 (1)
      number of C4 sub C-states using MWAIT   = 0x0 (0)
   Thermal and Power Management Features (6):
      digital thermometer                     = true
      operating point protection              = true
      digital thermometer thresholds          = 0x7 (7)
      ACNT/MCNT supported performance measure = true
   Architecture Performance Monitoring Features (0xa/eax):
      version ID                               = 0x3 (3)
      number of counters per logical processor = 0x4 (4)
      bit width of counter                     = 0x30 (48)
      length of EBX bit vector                 = 0x7 (7)
   Architecture Performance Monitoring Features (0xa/ebx):
      core cycle event not available           = false
      instruction retired event not available  = false
      reference cycles event not available     = true
      last-level cache ref event not available = false
      last-level cache miss event not avail    = false
      branch inst retired event not available  = false
      branch mispred retired event not avail   = false
   0x0000000b: eax=0x00000000 ebx=0x00000000 ecx=0x0000002c edx=0x00000002
   extended feature flags (0x80000001/edx):
      SYSCALL and SYSRET instructions        = false
      execution disable                      = true
      64-bit extensions technology available = true
   Intel feature flags (0x80000001/ecx):
      LAHF/SAHF supported in 64-bit mode = true
   brand = "Genuine Intel(R) CPU             000  @ 3.07GHz"
   L1 TLB/cache information: 2M/4M pages & L1 TLB (0x80000005/eax):
      instruction # entries     = 0x0 (0)
      instruction associativity = 0x0 (0)
      data # entries            = 0x0 (0)
      data associativity        = 0x0 (0)
   L1 TLB/cache information: 4K pages & L1 TLB (0x80000005/ebx):
      instruction # entries     = 0x0 (0)
      instruction associativity = 0x0 (0)
      data # entries            = 0x0 (0)
      data associativity        = 0x0 (0)
   L1 data cache information (0x80000005/ecx):
      line size (bytes) = 0x0 (0)
      lines per tag     = 0x0 (0)
      associativity     = 0x0 (0)
      size (Kb)         = 0x0 (0)
   L1 instruction cache information (0x80000005/ecx):
      line size (bytes) = 0x0 (0)
      lines per tag     = 0x0 (0)
      associativity     = 0x0 (0)
      size (Kb)         = 0x0 (0)
   L2 TLB/cache information: 2M/4M pages & L2 TLB (0x80000006/eax):
      instruction # entries     = 0x0 (0)
      instruction associativity = L2 off (0)
      data # entries            = 0x0 (0)
      data associativity        = L2 off (0)
   L2 TLB/cache information: 4K pages & L2 TLB (0x80000006/ebx):
      instruction # entries     = 0x0 (0)
      instruction associativity = L2 off (0)
      data # entries            = 0x0 (0)
      data associativity        = L2 off (0)
   L2 unified cache information (0x80000006/ecx):
      line size (bytes) = 0x40 (64)
      lines per tag     = 0x0 (0)
      associativity     = 8-way (6)
      size (Kb)         = 0x100 (256)
   Advanced Power Management Features (0x80000007/edx):
      temperature sensing diode      = 0x0 (0)
      frequency ID (FID) control     = 0x0 (0)
      voltage ID (VID) control       = 0x0 (0)
      thermal trip (TTP)             = 0x0 (0)
      thermal monitor (TM)           = 0x0 (0)
      software thermal control (STC) = 0x0 (0)
      TscInvariant                   = 0x1 (1)
   Physical Address and Linear Address Size (0x80000008/eax):
      maximum physical address = 0x28 (40)
      maximum linear address   = 0x30 (48)
   Logical CPU cores (0x80000008/ecx):
      number of logical CPU cores - 1 = 0x0 (0)
      ApicIdCoreIdSize                = 0x0 (0)
   (multi-processing synth): hyper-threaded (t=32)
   (synth) = Intel Pentium II / Pentium III / Pentium M / Celeron / Mobile Celeron / Celeron M / Core Solo / Core Duo / Core 2 / Core 2 Extreme Processor / Xeon Processor LV / Xeon Processor 5100 (unknown model)




--------------------------------------------------------------
t@353 (l@353) terminated by signal SEGV (Segmentation Fault)
0xfffffd7ffa49b006: intel_aes_decrypt_cbc_256+0x0226:   ***ERROR--unknown op code***

Crash seems to happen at 0xfffffd7ffa49b006

0xfffffd7ffa49b001: intel_aes_decrypt_cbc_256+0x0221:   ***ERROR--unknown op code***
0xfffffd7ffa49b004: intel_aes_decrypt_cbc_256+0x0224:   fmulp    %st,%st(2)
0xfffffd7ffa49b006: intel_aes_decrypt_cbc_256+0x0226:   ***ERROR--unknown op code***
0xfffffd7ffa49b009: intel_aes_decrypt_cbc_256+0x0229:   fisttp   (%rdi)
0xfffffd7ffa49b00b: intel_aes_decrypt_cbc_256+0x022b:   pxor     %xmm0,%xmm1
0xfffffd7ffa49b00f: intel_aes_decrypt_cbc_256+0x022f:   movdqu   %xmm1,(%rsi,%rax)
0xfffffd7ffa49b014: intel_aes_decrypt_cbc_256+0x0234:   movdqu   (%r8,%rax),%xmm0
0xfffffd7ffa49b01a: intel_aes_decrypt_cbc_256+0x023a:   addq     $0x0000000000000010,%rax
0xfffffd7ffa49b01e: intel_aes_decrypt_cbc_256+0x023e:   cmpq     %rax,%r9
0xfffffd7ffa49b021: intel_aes_decrypt_cbc_256+0x0241:   jne      intel_aes_decrypt_cbc_256+0x1d3        [ 0xfffffd7ffa49afb3, .-0x6e ]
0xfffffd7ffa49b023: intel_aes_decrypt_cbc_256+0x0243:   movdqu   %xmm0,(%rdx)
0xfffffd7ffa49b027: intel_aes_decrypt_cbc_256+0x0247:   xorl     %eax,%eax
0xfffffd7ffa49b029: intel_aes_decrypt_cbc_256+0x0249:   ret
------------------------------------------------

From source code :


intel_aes_decrypt_ecb_256:
...
	.byte 0x66,0x0f,0x38,0xde,0xcb	/* aesdec	%xmm3, %xmm1 */
	.byte 0x66,0x0f,0x38,0xde,0xca	/* aesdec	%xmm2, %xmm1 */
	.byte 0x66,0x0f,0x38,0xdf,0x0f	/* aesdeclast (%rdi), %xmm1 */
	pxor	%xmm0, %xmm1
	movdqu	%xmm1, (%rsi, %rax)
	movdqu	(%r8, %rax), %xmm0
	addq	$16, %rax
	cmpq	%rax, %r9
	jne	4b

5:	movdqu	%xmm0, (%rdx)

	xor	%eax, %eax
	ret
	.size intel_aes_decrypt_cbc_256, .-intel_aes_decrypt_cbc_256

-----------------------------------


The problem is the first operation to the AESDECLAST instruction is NOT aligned.  That will generate an exception.  It must be aligned on a 0 mod 16 address:



intel_aes_decrypt_ecb_256 + 0x226:
.byte 0x66,0x0f,0x38,0xdf,0x0f	/* aesdeclast (%rdi), %xmm1 */

The value of the %rdi register is 0x0000000018ed9da8. This is not aligned and will generate a SEGV.

To fix, please ensure the imputs are aligned 0 mod 16. 


Reproducible: Always

Steps to Reproduce:
Apply an SSL load to a webserver using NSS libraries on a Westmere class system, eg: sun webserver 7.0 update 6
Actual Results:  
Libs seg fault

Comment 1

[Dan Anderson]

This bug looks similar to bug 536485.