53479 - Deadlock caused by lock ordering error in NSS/SSL

Status: RESOLVED DUPLICATE of bug 588698

(NSS :: Libraries, defect, P3)

Description

[Nelson Bolyard (seldom reads bugmail)]

The file nss/lib/ssl/notes.txt defines the intended lock aquisition
ordering for SSL in NSS.  

ssl3_HandleRecord can, under some circumstances, attempt to aquire
the handshake lock while it already holds the RecvBuf lock.

The deadlock that was seen had one thread in this call stack:

PR_Poll -> ssl_Poll -> SSL_DataPending

holding the 1stHandshake and SSL3Handshake locks, trying to aquire
the RecvBuf lock,  and a second thread in this call stack:

PR_Recv -> ssl_SecureRecv -> DoRecv -> ssl3_GatherAppDataRecord 
 -> ssl3_GatherCompleteHandshake -> ssl3_HandleRecord

holding the RecvBuf lock, trying to aquire the SSL3Handshake lock.

It is unusual to Poll a socket while doing a read on it, and the
application that did this was found to have been polling on the 
wrong socket.  Still, it should not have deadlocked.

Comment 1

[Nelson Bolyard (seldom reads bugmail)]

Target fix == 3.2.

Comment 2

[Nelson Bolyard (seldom reads bugmail)]

Deferred to NSS 3.3

Comment 3

[Wan-Teh Chang]

Changed the QA contact to Bishakha.

Comment 4

[Wan-Teh Chang]

Set target milestone to NSS 3.5.

Comment 5

[Wan-Teh Chang]

Moved to target milestone 3.8 because the original
NSS 3.7 release has been renamed 3.8.

Comment 6

[Nelson Bolyard (seldom reads bugmail)]

Remove target milestone of 3.8, since these bugs didn't get into that release.

Comment 7

[Wan-Teh Chang]

A similar deadlock caused by the same locking order problem
was reported in bug 588698.  Although this bug was filed first,
I'm marking this bug as a duplicate because bug 588698 has
more info.