529450 - receipt of no_renegotiation alert doesn't terminate SSL_ForceHandshake

Status: RESOLVED WONTFIX

(NSS :: Libraries, defect, P5)

Description

[Nelson Bolyard (seldom reads bugmail)]

This is a old bug that has just recently come to light, now that a lot more
systems are refusing to renegotiate than before.  

NSS's libSSL has a function that initiates a renegotiation (SSL_ReHandshake), 
and a function (SSL_ForceHandshake) that allows the application to wait for 
the renegotiation (or current handshake in progress) to end.  The receipt of 
a "fatal" alert always causes SSL_ForceHandshake to end, but the receipt of a
"warning" alert typically does not.  

The "no_renegotiation" alert is, by definition, a "warning" alert.  It leaves
the connection in a valid SSL state where it is able to continue to move data
securely, so it is not fatal to the SSL session in progress.  The trouble is 
that we don't do anything to cause the application to stop waiting for the 
handshake to complete.  So, a failed renegotiation may cause some thread to 
hang until some timeout expires (if one has been set by the application).

We need to mark the handshake as not in progress, (that's easy) and make sure that the SSL connection is back in the state it was before we attempted to do 
a renegotiation.  That latter part may be very tricky.  

It's unclear to me if this must be done before we can release 3.12.5 or not.

Comment 1

[Nelson Bolyard (seldom reads bugmail)]

Oh, we also must determine whether or not to call the handshake completion 
callback in this case.

Comment 2

[Nelson Bolyard (seldom reads bugmail)]

Attachment: untested patch v1 - to play with

This patch returns the handshake state machine to the idle state and calls the 
completion callback.  There's no way to tell the callback that it completed 
unsuccessfully. :(  That's why I'm not sure we should call it at all.  
Still, an app could be waiting for that call, and hang if it doesn't occur.
Comments invited.

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee is inactive on Bugzilla, so the assignee is being reset.