167764 - OSX-Cannot visit a TLS intolerant site.

Status: VERIFIED FIXED

(Core Graveyard :: Security: UI, defect, P3)

Description

[John Unruh]

1.) With a new profile, visit the above site with TLS enabled.
What happens: I cannot reach the site. Win32 and Linux are OK.

Comment 1

[Kai Engert (:KaiE:)]

Confirming bug on Mac OS X 10.1.5 using 29/9 build.

Comment 2

[Kai Engert (:KaiE:)]

I traced this on the Mac.

The connection does not succeed, because we do not automatically retry (as we
are supposed to do with TLS intolerant servers).


We do not automatically retry, because the error code we encounter is not in the
retry-without-tls code list.

The error code returned from NSPR/SSL is => PR_UNKNOWN_ERROR


The failure happens during the 1st SSL handshake in macsockotpt.c,
SendReceiveStream, the first call to OTRcv returns 0xfffff3ad, which is
kOTOutStateErr.

I don't know what kOTOutStateErr means, do you know that?


But I made a wild guess and guessed "state" could be related to the
blocking/nonblocking state of the socket. I remembered we are doing a blocking
write initially and I thought that could cause the problem.

And indeed, after having worked on bug 163605 and tested the patch, the latest
patch for bug 163605 also fixes the problem in this bug 167764.

Comment 3

[Wan-Teh Chang]

kOTOutStateErr (-3155) means "The endpoint was not in
an appropriate state when you called this function."

I am afraid that the fix for bug 163605 merely masks
this problem.  In any case, as a general-purpose library,
NSPR should be able to handle this case properly.

Based on your description, I think the OTRcv() call in
SendReceiveStream() returns kOTOutStateErr, so we come
to this point:
                    
1543                 case kOTOutStateErr:    // if provider already closed, fall
through to handle error
1544                     if (fd->secret->md.orderlyDisconnect) {
1545                         amount = 0;
1546                         goto NormalExit;
1547                     }
1548                     // else fall through
1549                 default:
1550                     err = result;
1551                     goto ErrorExit;
[...]
1561 ErrorExit:
1562     PR_ASSERT(opCode == kSTREAM_SEND ? fd->secret->md.write.thread == NULL :
1563                                        fd->secret->md.read.thread  == NULL);
1564     macsock_map_error(err);
1565     return -1;

So apparently fd->secret->md.orderlyDisconnect is false and
macsock_map_error does not map kOTOutStateErr to a useful
NSPR error.  Simon, Gordon, do you know what kind of error
condition it is for fd->secret->md.orderlyDisconnect to be
false at that point?  That will tell us how we should map
kOTOutStateErr to an NSPR error code.  I think it probably
should be mapped to PR_CONNECT_RESET_ERROR.  Kai, is
PR_CONNECT_RESET_ERROR in the retry-without-tls error code
list?

Comment 4

[Kai Engert (:KaiE:)]

>  Kai, is PR_CONNECT_RESET_ERROR in the retry-without-tls error code list?

Yes

Comment 5

[Matthias Versen [:Matti]]

is this bug related to bug 167764 (also recent regression)

Comment 6

[Wan-Teh Chang]

Matti, this bug is bug 167764. :-)

Comment 7

[Kai Engert (:KaiE:)]

If you have time today and want to help test the fix for this bug, please grab a
build from http://www.kuix.de/mozilla/163605/ and test whether it works, and
there are no regressions in SSL functionality. Please give feedback today or
early tomorrow. Thanks!

Comment 8

[Kai Engert (:KaiE:)]

This should be fixed starting with tomorrow's build, by the patch for bug
163605. Please reopen if you still can reproduce the problem.

Comment 9

[John Unruh]

Verified.