265298 - SSL_ERROR_RX_UNEXPECTED_CHANGE_CIPHER (-12237) when communicating with Apache 2.0.51

Status: RESOLVED INCOMPLETE

(NSS :: Libraries, defect, P3)

Description

[Semyon Reyfman]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.3) Gecko/20040910
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.3) Gecko/20040910

This problem occurs intermittently for several NSS-based browsers (Netscape,
Mozilla) on various platforms (Windows, Linux, Solaris) communicating  over SSL
with our internal Apache 2.0 server running on a Linux box.  It manisfests
itself as a pop-up

"hostname has sent an incorrect or unexpected message. Error Code: -12237 "

I looked at NSS code and found out that the error corresponds to

SSL_ERROR_RX_UNEXPECTED_CHANGE_CIPHER -12237 "SSL received an unexpected Change
Cipher Spec record."

Apache debug tracing of the problem suggests that it receives "client hello"
message with a valid session id and responds with "server hello" followed by
"change cipher", which I believe is a valid response in this case.  But it
appears that for some reason the client doesn't like "server hello" message
(session id?) and because of that rejects the following "change cipher" message
with the result described above.  "unexpected message" is sent to the server and
shows up in the server log.



Reproducible: Sometimes
Steps to Reproduce:
1.   The problem is intermittent and occurs randomly more or less frequently.
To me it occurs once every couple of days with rather intensive use of the
server, but one of our developers complains that she gets it in one in ten
server hits.

3.

Comment 1

[Semyon Reyfman]

Forgot to mention that according to the server log it was sslv3 handshake.

Comment 2

[Wan-Teh Chang]

By the way, you can find a listing of NSS error
codes in our documentation:
http://www.mozilla.org/projects/security/pki/nss/ref/ssl/sslerr.html

Comment 3

[Semyon Reyfman]

(In reply to comment #2)
> By the way, you can find a listing of NSS error
> codes in our documentation:
> http://www.mozilla.org/projects/security/pki/nss/ref/ssl/sslerr.html

That's what I used before downloading the source:-)

Comment 4

[Nelson Bolyard (seldom reads bugmail)]

There's nothing that can be done with this bug until either
a) instructions are supplied that enable me to reproduce this at will, or 
b) a complete trace of the problem is obtained with NSS's ssltap tool.

In the meantime, I'd move this bug back to "unconfirmed" if I could.

Comment 5

[timeless]

reporter: i don't suppose you could provide a test server or an nss log (if you need instructions for getting an nss log, just ask)?

Comment 6

[Joshua Schmidlkofer]

I am getting this problem, and I have been able to reproduce it a lot.   

An error occurred during a connection to trac.myisteam.com.
SSL received an unexpected Change Cipher Spec record.
(Error code: ssl_error_rx_unexpected_change_cipher)


This site "https://trac.myisteam.com", is mine, and I am using NGinx. (http://wiki.codemongers.com).   FF3b2 started with these errors.   B3 and B5 has each made it successively worse.  Safari, IE, FF2 all work fine.  

4/9 Minefeild actually seems to be worse than ever, however it may just be that I hit the error a lot.

I can't be sure that this comment belongs with this ticket.  This however is the closest to my symptoms of any ticket I have read so far.  

Comment 8

[Nelson Bolyard (seldom reads bugmail)]

In bug 430703, Kai Engert reported:

> About 25% of the connection attempts to https://limedaley.com/webmail/
> fail with ssl_error_rx_unexpected_change_cipher

In reply to comment 6, I do not experience any problem with the server at 
https://trac.myisteam.com

Comment 9

[Joshua Schmidlkofer]

(In reply to comment #8)
> In reply to comment 6, I do not experience any problem with the server at 
> https://trac.myisteam.com
> 

I don't know how many pages you were able to view, since it's a locked site.  
It seemed to be predominately with POSTs and GETs.  However, it's hit and miss.  I had fairly consistent bad luck at the login page.  Sometimes I can barely make pages functions, requiring 10 or 15 retries.  Other times, it
won't happen for 10 or 15 pages clicks w/in the site.  Most
often a 'good' browsing session would end on my first POST.

If someone seriously wants to test this, I will create a test case, with TLS
enabled, and we can work with it from there.

The problem, apparently is TLS.   This is a recommendation sent to me by Chip
Parker:

"""
in nginx conf:
ssl_protocols SSLv3;

OR, in ff3b5, disable TLSv1 (tools -> options -> advanced -> encryption)
"""

In any case, after disabling TLS on nginx, the problem has gone away.  I have
not done tons of testing, but it's working.

Also, I have seen this problem on a few websites asking about it WRT FF3b5.  It
does appear that TLS is to blame.  Are there any SSL/TLS geniuses who can tell
me if it's FF3 or nginx?

Comment 10

[Kai Engert (:KaiE:)]

Updating subject to include the error name.

Maybe it's easier to debug this with https://limedaley.com/webmail/ as it happens when simply loading the page (sometimes).

Comment 11

[Nelson Bolyard (seldom reads bugmail)]

Joshua, I was able to reproduce this problem *once* on your server 
https://trac.myisteam.com .  

I found that your server is supporting and using the new Japanese national 
standard cipher, Camellia, a new feature in Firefox 3.  That may be relevant 
to the problem, or may be just a coincidence.  

However, since that experience, your server now seems to be behaving 
differently.  Now, when I visit it, it redirects me to 
http://trac.myisteam.com/is  which is not an https (SSL) web site.  
Consequently, I can no longer reproduce the problem. :(  

In any case, I think you should file a new bug about your server's problem.
I think it is unlikely that your problem has the same cause as the reason
this bug was filed 4 years ago.  

I am resolving this old bug incomplete.  We'll deal with your new bug when
you've filed it.

Comment 12

[Nelson Bolyard (seldom reads bugmail)]

Footnote:  the problem seen in Bug 430703, using host limedaley.com,
has the same user visible symptom as this old bug, but absolutely does not
have the same cause, so I have opened that bug and added comments to it.

So, there are (at least) 3 bug reports with the same basic symptom, with 
separate causes.  
This bug - 4 years old, incomplete
Bug 430703, caused somehow by TLS Session Ticket Extension (new in FF3)
Joshua's bug (not yet filed), which also uses a different brand new feature
of FF3.  

Comment 13

[Joshua Schmidlkofer]

(In reply to comment #11)
> Joshua, I was able to reproduce this problem *once* on your server 
> https://trac.myisteam.com .  


> 
> I found that your server is supporting and using the new Japanese national 
> standard cipher, Camellia, a new feature in Firefox 3.  That may be relevant 
> to the problem, or may be just a coincidence.  

This is a gentoo box, running the latest OpenSSL and up-to-date nginx.  I don't know what to say about this WRT crypto.  

As far as the SSL goes, I need this server for my daily work, so I have forced SSLv3 in the SSL config of nginx, hence the TLS error is no longer possible:

         ssl                     on;
+	ssl_protocols 		SSLv3;


> 
> However, since that experience, your server now seems to be behaving 
> differently.  Now, when I visit it, it redirects me to 
> http://trac.myisteam.com/is  which is not an https (SSL) web site.  
> Consequently, I can no longer reproduce the problem. :(  

I have been entirely unable to reproduce the case in which I get redirected to http:// -  This is news to me.   That server should _exclusively_ redirect to https://

If you want to test, I have another server running ahsay backup behind an nginx proxy.  If you send me a private email I will setup an account for you.  I can afford to leave the standard crypto mode on that.  It would make an effective test case.

Comment 14

[Nelson Bolyard (seldom reads bugmail)]

Joshua, I have filed bug 430769 about the issue you experience.  
While disabling TLS may seem to help, I believe there is another solution
that will not only solve the problem, but also will reduce the workload 
on your server for the same amount of traffic.  Please see that bug.