Firefox 36 fails to connect to Cisco 5505, cannot override sec_error_unknown_issuer, connection stalls

RESOLVED INCOMPLETE

Status

()

Core
Security: PSM
RESOLVED INCOMPLETE
3 years ago
3 years ago

People

(Reporter: kaie, Unassigned)

Tracking

36 Branch
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

(Reporter)

Description

3 years ago
In bug 1138332 comment 10, Dick Riegner reported failure to override bad certificates with Firefox 36.

Based on the screenshots that were provided, the error code reported is sec_error_unknown_issuer.

See attachment 8573578 [details], attachment 8573580 [details], attachment 8573582 [details], and the comments from Dick in bug 1138332
(Reporter)

Comment 1

3 years ago
Dick, your issue seems to be different in several ways. We must analyze what's happening for you.

In bug 1138332, people weren't even able to add an override. The bug in that scenario was: The error page shown for some sites displayed an error message, but didn't over to override.

Based on your screenshots, you were given the option to override.

However, as you say, adding an override doesn't help with Firefox 36, you're still not able to access the host.

The other unusual (different from the bugs I had seen) detail of your scenario is: In the past, you weren't able to add permanent overrides.

Regarding the inability to override in the past: I wonder if your device generates a different certificate every time. The overrides in Firefox are bound to the combination of {hostname, port, certificiate, error produced}. If anything changed on the next execution, the remembered override is ignored. If you had connected with the same browser to the same host again, then the error would usually be the same (unless it had expired in the meantime). Thus, the only way to differ is if the server presents a different certificate.

With Firefox 36, as you explained in the other bug, you are able to override, but then your connections are stuck, and Firefox doesn't show the web page.

This means, Firefox is able to initiate the connection to the server, and is able to complete the initial part of the handshake, in which the certificate is sent by the server (which is deemed untrusted, und thus you are asked to override).

Once you had added the override, Firefox will pass beyond that step, and attempt to establish a connection to your server (which is the Cisco 5505 VPN hardware box you mentioned).

If Firefox "hangs", it seems that Firefox is expecting the server to send more data, but it isn't. If the server were simply disconnecting, you'd a different error page. So it seems, the server is unable to understand the communication with Firefox, and falsely assumes that it needs to wait for additional data, and with both sides waiting, you see that it hangs and never loads.

(That's my guess.)

If this problem, Firefox hangs on that site, was first seen with Firefox 36, then we should investigate which SSL/TLS options have changed between Firefox 35 and Firefox 36.

I really wish I was able to connect to your system myself. If your device can be reached from the public internet, then it would be very helpful if you could enable me to connect to it myself, for purpose of diagnosing the server's beahviour.

Comment 2

3 years ago
Kai,

Thanks for all your help with this problem.

I cannot think of a way to give you access to my VPN box.  Firefox is connecting to it through one of the VPN box's private network ports, not through the public network port.

My top priority is getting Firefox back to its release 35 and previous behavior.  It would then be nice to fix Firefox so that it could permanently store the VPN box's certificate.  If storing the certificate is not feasible that is OK, I have lived with that limitation for years now.

Let me know how you think we should proceed.  I will let you know if I can think of a way to give you access to the private side of the VPN box.

Dick

Comment 3

3 years ago
Has anyone taken a look at my problem?
(Reporter)

Comment 4

3 years ago
Without more details, we can't.

Comment 5

3 years ago
What details do you need?  I can gather whatever documentation you need.
(Reporter)

Comment 6

3 years ago
I need to be able to connect to the service.

If you cannot allow me to do that, someone with knowledge of how SSL/TLS works, and is able to use the tools, must run tests against your service, and tell us which versions work, and which version don't, and what the behaviour of those versions is.

Preferably executed using command line tools like "ssltap -s", that can dump details of the handshake, which can give us a clue which detail is responsible for the failure.

Comment 7

3 years ago
The Cisco 5505 VPN hardware box is directly connected to my iMac over ethernet.  I am running Mac OS X 10.8.5 and can run any tool that is available for Mac OS X.

As an option, can you spin-up a debug version of Firefox that can dump the needed details?

Since this is a regression from Firefox 35, any chance you can find the mod that caused this problem?

Thanks.
(Reporter)

Updated

3 years ago
Summary: Cannot override sec_error_unknown_issuer on some sites with Firefox 36 → Firefox 36 fails to connect to Cisco 5505, cannot override sec_error_unknown_issuer, connection stalls
(Reporter)

Comment 8

3 years ago
A debug version of firefox isn't easy to distribute and wouldn't give us the necessary information in a logfile. That won't help.

It would help if you can compile NSS and run the diagnostic utilities on the command line.

Create a new local directory.
Download https://ftp.mozilla.org/pub/mozilla.org/security/nss/releases/NSS_3_18_RTM/src/nss-3.18-with-nspr-4.10.8.tar.gz and place it into that directory.
Open a terminal and use "cd" to go into that directory.

Use the following commands to build:

tar xzf nss-3.18-with-nspr-4.10.8.tar.gz
cd nss-3.18
cd nss
export USE_64=1
make nss_build_all
cd ..

Now execute the following commands to prepare the environment to run the NSS tools that you just built:

OBJDIR=`cd nss/tests/common/ && make objdir_name`
export PATH=$PWD/dist/$OBJDIR/bin:$PATH
export DYLD_LIBRARY_PATH=$PWD/dist/$OBJDIR/lib:$DYLD_LIBRARY_PATH
export ROOTS=$PWD/dist/$OBJDIR/lib/libnssckbi.dylib

Now, if you type the name of a NSS tool, it should print it's help output, e.g. type
  tstclnt

If that works, you can test connections to your 5505 box. Get the IP address and set it as a variable, so you can copy paste the following commands.

export IP=1.2.3.4 (use the IP of your cicso 5505)

I assume you connect to the regular https port, 443.

Now run each of the following tests.

Open a text editor where you can paste text into.

For each of the tests you have executed, copy/paste the output that you see on the terminal, INCLUDING the respective command in front of the output, into your editor.

tstclnt -D -p 443 -h $IP -R $ROOTS -V tls1.2:tls1.2 -C
tstclnt -D -p 443 -h $IP -R $ROOTS -V tls1.1:tls1.1 -C
tstclnt -D -p 443 -h $IP -R $ROOTS -V tls1.0:tls1.0 -C
tstclnt -D -p 443 -h $IP -R $ROOTS -V ssl3:ssl3 -C
tstclnt -D -p 443 -h $IP -R $ROOTS -V ssl2:ssl2 -C

Please provide output of all 5 commands. At least one of the commands should print some diagnostic information.

Then, run another command that will print much more information about the server's certificate, and after the -V use the respective parameter from above, which didn't result in an error, but printed more information.

tstclnt -D -p 443 -h $IP -R $ROOTS -V replace-me -CCC

Again, copy paste the results, including the command, into the editor.

Send all that information by email to me.

Leave the terminal window open, in case I need you to run additional commands later.
(Reporter)

Comment 9

3 years ago
You should copy/paste the commands to the terminal, don't try to type them (except the very last command when you have to replace), because the commands contain some special characters, which are easy to get wrong, if you're aren't used to them.
(Reporter)

Comment 10

3 years ago
After each time you run the tstclnt utility, after it printed some output, and if the connection worked, it will still run and wait for input.

So, after each tstclnt command is done, you will have to press CTRL-C to terminate tstclnt.

Comment 11

3 years ago
Can I build NSS on a default Mac OS X 10.8.5 install or do I need to install some build tools?  I have no special build tools installed.  Is there an NSS binary already built that I can download?

I won't be able to leave the window open.  Once I am done with my VPN session into the private corporate network for the day, I revert my ethernet connection to the public network.  My company's VPN blocks too many sites it deems unnecessary for business use.

Also, once I have completed my login to the VPN box, I think that session is terminated.  I usually then access our company's home page.  

For what it's worth, there may be ways to access the VPN box's certificate and security options through other pages that I currently don't access to.  I should be able to get complete access to the VPN box administrator pages if needed.
(Reporter)

Comment 12

3 years ago
Type gcc into a terminal. Is a program found? If not, you probably must install "Xcode", which is free.

No, we don't usually provide binaries of NSS.

Keeping the terminal was simply a convenience suggestion, no problem to open it later again, simply cd into the directory you made earlier, and again run these commands:

OBJDIR=`cd nss/tests/common/ && make objdir_name`
export PATH=$PWD/dist/$OBJDIR/bin:$PATH
export DYLD_LIBRARY_PATH=$PWD/dist/$OBJDIR/lib:$DYLD_LIBRARY_PATH
export ROOTS=$PWD/dist/$OBJDIR/lib/libnssckbi.dylib

Then you can execute test tools again.

Comment 13

3 years ago
Any idea where I can get an NSS binary?

I am willing to put some effort into helping debug this Firefox regression, but installing an entire devleopment environment and building NSS is getting a bit much.

Migrating to Chrome to interface with my VPN box looks to be a very practical alternative.

Thanks for your effort.
(Reporter)

Comment 14

3 years ago
You earlier said you're willing to execute any tool on your machine to diagnose, and I've spent at least 30 minutes to write up and test the steps on a Mac for you, it's disappointing that you're no longer willing to analyze.
(Reporter)

Comment 15

3 years ago
Firefox is a community project, and can only work if people contribute a little piece of work to make it better. This is your chance to help.
(Reporter)

Comment 16

3 years ago
I only have access to a 32 bit Mac mini with OSX 10.6

I don't know if the binaries I built will work on your potentially newer 64 bit Mac with OSX 10.8

Comment 17

3 years ago
I am willing to help with the tools and environment I have at hand.  At this time, I am unwilling to install a complete Mac OS X development environment just to build one tool.

If I can find an NSS binary, I will gather the documentation you requested.

Comment 18

3 years ago
I suspect that a 32-bit binary will run fine on my 64-bit Mac OS X 10.8.5 machine.  How can I get the binary?
(Reporter)

Comment 19

3 years ago
I've uploaded binaries to
  https://kuix.de/mozilla/nss/nss-3.18-darwin-32bit.zip

I cannot give any guarantees whatsoever, but I think running them should be safe.
If you're worried these binaries might destroy any of your data, you could create a separate user account on your machine for running them.

Save the file to your disk, e.g. to the "Downloads" folder.
Open a terminal, "cd" into the directory where you stored the zip file, e.g. 
  cd $HOME/Downloads
and run
  unzip nss-3.18-darwin-32bit.zip
  cd nss-3.18-darwin-32bit

The commands to prepare your terminal to run these NSS tools is different from what I said earlier. Execute these commands in the terminal:

export PATH=$PWD/bin:$PATH
export DYLD_LIBRARY_PATH=$PWD/lib:$DYLD_LIBRARY_PATH
export ROOTS=$PWD/lib/libnssckbi.dylib

Now, go to comment 8 above, please follow the instructions from
  "Now, if you type the name of a NSS tool"

Let's cross fingers that the 32 bit binaries will work for you.

Comment 20

3 years ago
Ok, I downloaded and installed nss;  it seems to run fine.

Firefox attempts to connect to this IP and port on the VPN box, not 443:

https://134.15.84.97:1443/netaccess/connstatus.html

How should I set the IP variable?
(Reporter)

Comment 21

3 years ago
export IP=134.15.84.97

and don't use -p 443, use:
  -p 1443

Comment 22

3 years ago
1) I ran these various tstclnt commands before attempting to connect to the VPN box:

Riegner-iMac3:nss-3.18-darwin-32bit dick$ tstclnt -D -p 1443 -h $IP -R $ROOTS -V tls1.2:tls1.2 -C
tstclnt: read from socket failed: SSL_ERROR_UNSUPPORTED_VERSION: Peer using unsupported version of security protocol.

Riegner-iMac3:nss-3.18-darwin-32bit dick$ tstclnt -D -p 1443 -h $IP -R $ROOTS -V tls1.1:tls1.1 -C
tstclnt: read from socket failed: SSL_ERROR_UNSUPPORTED_VERSION: Peer using unsupported version of security protocol.

Riegner-iMac3:nss-3.18-darwin-32bit dick$ tstclnt -D -p 1443 -h $IP -R $ROOTS -V tls1.0:tls1.0 -C
==== certificate(s) sent by server: ====
Certificate:
    Data:
        Serial Number: -316472491 (0xed230355)
        Signature Algorithm: PKCS #1 SHA-1 With RSA Encryption
        Issuer: "CN=134.15.84.97"
        Validity:
            Not Before: Fri Mar 13 17:52:45 2015
            Not After : Mon Mar 10 17:52:45 2025
        Subject: "CN=134.15.84.97"
    Fingerprint (SHA-256): FC:B2:2B:B9:01:1B:B6:E1:65:DC:D0:B0:E7:72:11:6E:40:A8:B9:71:20:6A:37:8C:01:4E:CA:EC:4F:D0:41:D4
    Fingerprint (SHA1): 66:88:7C:DB:C0:0D:B1:2F:BE:25:FB:F3:04:B9:2A:B7:63:C6:CE:DB
==== locally found issuer certificate(s): ====
==== end of certificate chain information ====
tstclnt: authentication of server cert failed: SEC_ERROR_CA_CERT_INVALID: Issuer certificate is invalid.

Riegner-iMac3:nss-3.18-darwin-32bit dick$ tstclnt -D -p 1443 -h $IP -R $ROOTS -V ssl3:ssl3 -C
==== certificate(s) sent by server: ====
Certificate:
    Data:
        Serial Number: -316472491 (0xed230355)
        Signature Algorithm: PKCS #1 SHA-1 With RSA Encryption
        Issuer: "CN=134.15.84.97"
        Validity:
            Not Before: Fri Mar 13 17:52:45 2015
            Not After : Mon Mar 10 17:52:45 2025
        Subject: "CN=134.15.84.97"
    Fingerprint (SHA-256): FC:B2:2B:B9:01:1B:B6:E1:65:DC:D0:B0:E7:72:11:6E:40:A8:B9:71:20:6A:37:8C:01:4E:CA:EC:4F:D0:41:D4
    Fingerprint (SHA1): 66:88:7C:DB:C0:0D:B1:2F:BE:25:FB:F3:04:B9:2A:B7:63:C6:CE:DB
==== locally found issuer certificate(s): ====
==== end of certificate chain information ====
tstclnt: authentication of server cert failed: SEC_ERROR_CA_CERT_INVALID: Issuer certificate is invalid.

Riegner-iMac3:nss-3.18-darwin-32bit dick$ tstclnt -D -p 1443 -h $IP -R $ROOTS -V ssl2:ssl2 -C
tstclnt: error setting SSL/TLS version range : SSL_ERROR_INVALID_VERSION_RANGE: SSL version range is not valid.


2)  I then attempted to connect to the VPN box using Firefox 36 and got this:

Riegner-iMac3:nss-3.18-darwin-32bit dick$ tstclnt -D -p 1443 -h $IP -R $ROOTS -V tls1.0:tls1.0 -C
==== certificate(s) sent by server: ====
Certificate:
    Data:
        Serial Number: -316472491 (0xed230355)
        Signature Algorithm: PKCS #1 SHA-1 With RSA Encryption
        Issuer: "CN=134.15.84.97"
        Validity:
            Not Before: Fri Mar 13 17:52:45 2015
            Not After : Mon Mar 10 17:52:45 2025
        Subject: "CN=134.15.84.97"
    Fingerprint (SHA-256): FC:B2:2B:B9:01:1B:B6:E1:65:DC:D0:B0:E7:72:11:6E:40:A8:B9:71:20:6A:37:8C:01:4E:CA:EC:4F:D0:41:D4
    Fingerprint (SHA1): 66:88:7C:DB:C0:0D:B1:2F:BE:25:FB:F3:04:B9:2A:B7:63:C6:CE:DB
==== locally found issuer certificate(s): ====
==== end of certificate chain information ====
tstclnt: authentication of server cert failed: SEC_ERROR_CA_CERT_INVALID: Issuer certificate is invalid.

Riegner-iMac3:nss-3.18-darwin-32bit dick$ tstclnt -D -p 1443 -h $IP -R $ROOTS -V ssl3:ssl3 -C
==== certificate(s) sent by server: ====
Certificate:
    Data:
        Serial Number: -316472491 (0xed230355)
        Signature Algorithm: PKCS #1 SHA-1 With RSA Encryption
        Issuer: "CN=134.15.84.97"
        Validity:
            Not Before: Fri Mar 13 17:52:45 2015
            Not After : Mon Mar 10 17:52:45 2025
        Subject: "CN=134.15.84.97"
    Fingerprint (SHA-256): FC:B2:2B:B9:01:1B:B6:E1:65:DC:D0:B0:E7:72:11:6E:40:A8:B9:71:20:6A:37:8C:01:4E:CA:EC:4F:D0:41:D4
    Fingerprint (SHA1): 66:88:7C:DB:C0:0D:B1:2F:BE:25:FB:F3:04:B9:2A:B7:63:C6:CE:DB
==== locally found issuer certificate(s): ====
==== end of certificate chain information ====
tstclnt: authentication of server cert failed: SEC_ERROR_CA_CERT_INVALID: Issuer certificate is invalid.


3) I will now run the verbose version of the previous two commands.

Comment 23

3 years ago
4) I switched back to the VPN network, tried to connect to the VPN box using Firefox 36, and ran these commands:

Riegner-iMac3:nss-3.18-darwin-32bit dick$ tstclnt -D -p 1443 -h $IP -R $ROOTS -V tls1.0:tls1.0 -CCC
==== certificate(s) sent by server: ====
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: -316472491 (0xed230355)
        Signature Algorithm: PKCS #1 SHA-1 With RSA Encryption
        Issuer: "CN=134.15.84.97"
        Validity:
            Not Before: Fri Mar 13 17:52:45 2015
            Not After : Mon Mar 10 17:52:45 2025
        Subject: "CN=134.15.84.97"
        Subject Public Key Info:
            Public Key Algorithm: PKCS #1 RSA Encryption
            RSA Public Key:
                Modulus:c9:88:bf:42:de:f8:78:42:b8:e7:b7:d7:4f:1b:77:16:a0:c7:de:2b:29:e5:95:43:ff:df:4c:5a:23:76:ee:43:68:9b:20:4d:62:17:73:c7:63:fb:1c:a3:9d:88:07:29:24:c8:33:9a:f1:36:0e:7f:87:0d:de:b7:a5:b5:ea:61:8a:9f:69:68:14:01:78:56:9c:71:c4:af:5a:e6:44:23:1b:ef:5f:53:88:f9:8f:54:07:ee:42:42:4b:92:88:17:51:df:b5:37:bb:fb:0e:3a:51:ce:a8:b2:7e:c3:e8:7a:78:61:78:e3:29:1c:00:cb:a9:86:38:b4:7a:10:5a:23
                Exponent: 65537 (0x10001)
    Fingerprint (SHA-256): FC:B2:2B:B9:01:1B:B6:E1:65:DC:D0:B0:E7:72:11:6E:40:A8:B9:71:20:6A:37:8C:01:4E:CA:EC:4F:D0:41:D4
    Fingerprint (SHA1): 66:88:7C:DB:C0:0D:B1:2F:BE:25:FB:F3:04:B9:2A:B7:63:C6:CE:DB
-----BEGIN CERTIFICATE-----
MIIBpTCCAQ6gAwIBAgIE7SMDVTANBgkqhkiG9w0BAQUFADAXMRUwEwYDVQQDEwwx
MzQuMTUuODQuOTcwHhcNMTUwMzEzMTc1MjQ1WhcNMjUwMzEwMTc1MjQ1WjAXMRUw
EwYDVQQDEwwxMzQuMTUuODQuOTcwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGB
AMmIv0Le+HhCuOe3108bdxagx94rKeWVQ//fTFojdu5DaJsgTWIXc8dj+xyjnYgH
KSTIM5rxNg5/hw3et6W16mGKn2loFAF4VpxxxK9a5kQjG+9fU4j5j1QH7kJCS5KI
F1HftTe7+w46Uc6osn7D6Hp4YXjjKRwAy6mGOLR6EFojAgMBAAEwDQYJKoZIhvcN
AQEFBQADgYEAourbskHcBlTMFcNgzjS++So8qd4PFt4lidGfIj/HpT1x7IvG18Dp
cE9kLn4zHkEk2cUcx7uhxlL0Vr55OnmxTLEeE8voPxbnYpXz7mcplUe+6Nc20Xnn
4iv6ikrQQtPQM/kyffckMVxtC5qlFFHSesOH+uRmpbn5HPNhsBgwI+Q=
-----END CERTIFICATE-----
==== locally found issuer certificate(s): ====
==== end of certificate chain information ====
tstclnt: authentication of server cert failed: SEC_ERROR_CA_CERT_INVALID: Issuer certificate is invalid.

Riegner-iMac3:nss-3.18-darwin-32bit dick$ tstclnt -D -p 1443 -h $IP -R $ROOTS -V ssl3:ssl3 -CCC
==== certificate(s) sent by server: ====
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: -316472491 (0xed230355)
        Signature Algorithm: PKCS #1 SHA-1 With RSA Encryption
        Issuer: "CN=134.15.84.97"
        Validity:
            Not Before: Fri Mar 13 17:52:45 2015
            Not After : Mon Mar 10 17:52:45 2025
        Subject: "CN=134.15.84.97"
        Subject Public Key Info:
            Public Key Algorithm: PKCS #1 RSA Encryption
            RSA Public Key:
                Modulus:c9:88:bf:42:de:f8:78:42:b8:e7:b7:d7:4f:1b:77:16:a0:c7:de:2b:29:e5:95:43:ff:df:4c:5a:23:76:ee:43:68:9b:20:4d:62:17:73:c7:63:fb:1c:a3:9d:88:07:29:24:c8:33:9a:f1:36:0e:7f:87:0d:de:b7:a5:b5:ea:61:8a:9f:69:68:14:01:78:56:9c:71:c4:af:5a:e6:44:23:1b:ef:5f:53:88:f9:8f:54:07:ee:42:42:4b:92:88:17:51:df:b5:37:bb:fb:0e:3a:51:ce:a8:b2:7e:c3:e8:7a:78:61:78:e3:29:1c:00:cb:a9:86:38:b4:7a:10:5a:23
                Exponent: 65537 (0x10001)
    Fingerprint (SHA-256): FC:B2:2B:B9:01:1B:B6:E1:65:DC:D0:B0:E7:72:11:6E:40:A8:B9:71:20:6A:37:8C:01:4E:CA:EC:4F:D0:41:D4
    Fingerprint (SHA1): 66:88:7C:DB:C0:0D:B1:2F:BE:25:FB:F3:04:B9:2A:B7:63:C6:CE:DB
-----BEGIN CERTIFICATE-----
MIIBpTCCAQ6gAwIBAgIE7SMDVTANBgkqhkiG9w0BAQUFADAXMRUwEwYDVQQDEwwx
MzQuMTUuODQuOTcwHhcNMTUwMzEzMTc1MjQ1WhcNMjUwMzEwMTc1MjQ1WjAXMRUw
EwYDVQQDEwwxMzQuMTUuODQuOTcwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGB
AMmIv0Le+HhCuOe3108bdxagx94rKeWVQ//fTFojdu5DaJsgTWIXc8dj+xyjnYgH
KSTIM5rxNg5/hw3et6W16mGKn2loFAF4VpxxxK9a5kQjG+9fU4j5j1QH7kJCS5KI
F1HftTe7+w46Uc6osn7D6Hp4YXjjKRwAy6mGOLR6EFojAgMBAAEwDQYJKoZIhvcN
AQEFBQADgYEAourbskHcBlTMFcNgzjS++So8qd4PFt4lidGfIj/HpT1x7IvG18Dp
cE9kLn4zHkEk2cUcx7uhxlL0Vr55OnmxTLEeE8voPxbnYpXz7mcplUe+6Nc20Xnn
4iv6ikrQQtPQM/kyffckMVxtC5qlFFHSesOH+uRmpbn5HPNhsBgwI+Q=
-----END CERTIFICATE-----
==== locally found issuer certificate(s): ====
==== end of certificate chain information ====
tstclnt: authentication of server cert failed: SEC_ERROR_CA_CERT_INVALID: Issuer certificate is invalid.
(Reporter)

Comment 24

3 years ago
Dick, thanks for the output.

FYI, for the tools it shouldn't matter whether you have used Firefox previously or not. The output of (1) and (2) is identical.

We have learned that your server doesn't support the most modern TLS 1.2 and TLS 1.1 protocols.

Firefox 36 will by default try to run TLS 1.2, and might attempt to fall back to using older protocols.

I wonder if something is wrong with that mechanism of falling back.

You could try to *temporarily* disable TLS 1.2 and TLS 1.1 in Firefox 36.

Go to about:config
In the search box, type: tls
Locate the line with security.tls.version.max
This is equivalent to the highest version number enable.
3 == TLS 1.2
2 == TLS 1.1
1 == TLS 1.0

Double click the line, enter 1 and confirm.

(I don't recommend to keep that setting, because you really should use those protocols with any site that supports it. Only use it for testing.)

Restart Firefox. Try to connect with Firefox to your box. Does it work?

(If it worked, it could mean bad implementation of TLS in your box, or a problem with the automatic fallback to older protocols in Firefox.)
(Reporter)

Comment 25

3 years ago
Only after you have tried the instructions from comment 24, here is another test, which might produce more helpful information.

- go to about:config again
- enter tls
- set security.tls.version.max back to 3
- restart firefox

Open a terminal where you have prepared the NSS tools.
Run the following command, which will execute a tunneling debug utility:

cd /tmp/
ssltap -l -s 134.15.84.97:1443 2>&1 | tee /tmp/ssltap.log

The tool will now listen on your local host port 1924 for connections.

Go back to Firefox with tls max set to 3, and paste the following address into the location bar:

https://127.0.0.1:1924

After it stops loading in Firefox, close Firefox.

Go back to the terminal, and hit CTRL-C.

Your terminal should have a long log of information. The same information will also have been written to the file /tmp/ssltap.log

Please provide file ssltap.log 

The logfile will contain detailed SSL/TLS protocol information of the interaction between firefox and your server. If Firefox attempts to fall back or not, we'll learn that from the log.
(Reporter)

Comment 26

3 years ago
(In reply to Kai Engert (:kaie) from comment #25)
> Please provide file ssltap.log 

When you do, don't paste it into a bugzilla comment. Please either use the file attach mechanism in this bug, or please send the file to me by email. Thanks.

Comment 27

3 years ago
I set this variable:

security.tls.version.max = 1

and restarted Firefox.  But Firefox 36 still hangs when attempting to connect to the VPN box.  I got the same prompts and warnings as shown in the attachments in bug 1138332.

I will try the other test tonight.

Comment 28

3 years ago
Created attachment 8582724 [details]
Here is the ssltap log file.
(Reporter)

Comment 29

3 years ago
I assume when you produced the ssltap.log file, Firefox showed an error page with a hostname mismatch.

Please add an override. I need a logfile of the scenario where Firefox doesn't show an error page, but hangs.

Thanks

Comment 30

3 years ago
Created attachment 8582879 [details]
Here is the ssltap log file with the Firefox override to accept the untrusted certificate.
(Reporter)

Comment 31

3 years ago
Thanks.

What I see is:
- a first connection attempt, client disconnects after the handshake
- now you've probably added the override
- 8 seconds later, a second connection attempt
- the handshake is completed
- client sends a full request with encrypted data to the server,
  which most likely contains the http request
- 15 seconds with no data exchanged
- client disconnects

I believe this means, client and server complete the handshake, in which they agree with each other on the parameters of the encryption.

The client sends the http request to the server, to which the server is expected to reply.

Instead of replying, the server doesn't send any response within 15 seconds.

After 15 seconds, the client sends an alert message to the server, probably telling it that it gives up.

Both client and server close the connection.

It seems like the server doesn't understand the request that Firefox sends, and misbehaves by not sending any response.

This is all I can guess without being able to connect to the server myself.

If I were able to connect to the server myself, I could play with different requests.
(Reporter)

Comment 32

3 years ago
Dick, would you be able to provide another log of a working connection with Firefox 35?

One difference between FF 35 and FF 36: FF 36 no longer offers ciphers with RC4 to the server. It seems that it doesn't matter, because the server happily selects a non-RC4 cipher from the ones offered by Firefox 36. Maybe we can see another difference in the 35 vs. 36 logs.
(Reporter)

Comment 33

3 years ago
Another suggestion:

With FF 36, go to about:config, and in the search box, type: spdy

Find the preference network.http.spdy.enabled.http2

I assume it's set to "true". Doubleclick the line, and it should switch to "false".

Restart Firefox and try to connect to the 5505 again.
(Reporter)

Comment 34

3 years ago
And if it still doesn't work, find network.http.spdy.enabled and switch it to false and restart and try again.

Comment 35

3 years ago
I will try the "spdy" tests this morning.  

I keep wondering how Firefox has worked for over 15 years with this Cisco hardware box, and the one before it, and now all of a sudden, all these problems?

Comment 36

3 years ago
I changed Firefox 36:

network.http.spdy.enabled.http2 to false and re-started Firefox;  still hangs trying to connect to the VPN box.

Then changed:

network.http.spdy.enabled to false and re-started Firefox;  still hangs trying to connect to the VPN box.

I changed both settings back to true.

What do we try next?

Comment 37

3 years ago
I want to make clear that I appreciate all your help in resolving this regression.  In that light, I make this suggestion:

There can only be so many mods between Firefox 35 and Firefox 36 related to certificate handling.  Why not pull all of those mods from Firefox 36, build a test Firefox, and verify that the regressions is indeed resolved?  This conclusively indicates one of those mods introduced the regression.

While a bit tedious, then start adding back mods one or two at a time until the problem returns.  This gives us the mod responsible for introducing the regression.

Of course, this is only practical when there is a limited number of mods.

I will help with whatever debug course you take.
(Reporter)

Comment 38

3 years ago
Dick, I have another theory, inspired by another bug I've investigated today. I wonder if Firefox is confused because you have added many overrides in the past.

Could you please do the following:

Prepare the terminal.

Quit Firefox.

Find the Firefox profile directory.

Usually, if you have never used the "firefox profile manager" to create additional profiles, your profile should be named "whatever".default and the following command should work (please copy/paste):

cd $HOME/Library/Application\ Support/Firefox/Profiles/*.default

Confirm that you have the expected files:

ls *.db

You should have at least cert8.db, key3.db

If you haven't found the right directory yet, use your OSX file software to find the correct name. Or use 
  cd $HOME/Library/Application\ Support/Firefox/Profiles/
then 
  ls
which should print the name. Run
  cd the-name-that-you-saw-printed

Now list the contents of your certificate database:

certutil -d dbm:. -L

This lists the names of intermediate CA certificates Firefox has cached, and personal certificates you might own, plus certificates that have been stored when you had added overrides in the past.

Do you see entries that are named 134.15.84.97 ? How many of them?

Now that firefox is still closed, I suggest you try the following to backup some files, and then clean up.

mkdir backup
mv -i cert8.db key3.db secmod.db cert_override.txt backup/

(Note this command will reset a lot of the SSL/TLS state your browser has remembered, and all your overrides, and if you own personal certificates, their keys will be moved, too. If you have such data, make sure you have a good backup of these files. If necessary, you can later restore them, by replacing any newer files with the ones from your backup.)

Now start firefox and try again.

Does it work now?
(Reporter)

Comment 39

3 years ago
> Do you see entries that are named 134.15.84.97 ? How many of them?

If you don't have such entries, maybe you could send me the output of the certutil -L command by email?

Feel free to redact the output for privacy, if necessary.

Comment 40

3 years ago
I have two Firefox profiles;  one for the public network the other for the company private network.

1) I tried running from the private profile directory and got:

Riegner-iMac3:4qfu6u20.B_VPN dick$ /Users/dick/Download/nss/nss-3.18-darwin-32bit/bin/certutil -d dbm:. -L
dyld: Library not loaded: @executable_path/libssl3.dylib
  Referenced from: /Users/dick/Download/nss/nss-3.18-darwin-32bit/bin/certutil
  Reason: image not found
Trace/BPT trap: 5

2) And then tried running from the download bin directory and got:

Riegner-iMac3:bin dick$ pwd
/Users/dick/Download/nss/nss-3.18-darwin-32bit/bin
Riegner-iMac3:bin dick$ certutil -d dbm:. -L
certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format.

I moved the files to the backup directory and will restart Firefox on the VPN network.

Comment 41

3 years ago
Firefox 36 still hangs trying to access the VPN box after moving out those files.  All four files have been recreated by Firefox in the VPN profile.

Comment 42

3 years ago
Do you need me to complete more tests or provide more documentation?

Comment 43

3 years ago
Are we still working this problem?  There haven't been any updates from anyone but me for several weeks.  I need to let the IT folks at work know if there will eventually be a Firefox fix for this VPN hardware problem.
(Reporter)

Comment 44

3 years ago
Dick, I don't have further ideas what to do.

Without direct access to the machine it's too complicated to analyze.

I wish we'd see some support from Cisco, but I haven't gotten any response from them.
(Reporter)

Comment 45

3 years ago
Try to run from within the download bin directory, because that seems to be required on your system to get the utlility I made to actually run.

But then the parameters must be different and point to the firefox profile directory.

cd /Users/dick/Download/nss/nss-3.18-darwin-32bit/bin

certutil -d dbm:/full/path/to/your/firefox/profile/directory/ -L

Comment 46

3 years ago
Kai,

I appreciate your help and frustration about not having access to my machine.  I will gladly grant you access, if we can think of a way for you to login to my machine while it is connected to a private VPN network.

I found that certutil was missing needed libraries, so I set this environment variable to point to them:

cf-asa-hw-dickie-3:bin dick$ pwd       
/Users/dick/Download/nss/nss-3.18-darwin-32bit/bin

export DYLD_LIBRARY_PATH=/Users/dick/Download/nss/nss-3.18-darwin-32bit/lib

And now certutil seems to run, but says the database is old:

cf-asa-hw-dickie-3:bin dick$ ./certutil -d dbm:/Users/dick/Library/Application Support/Firefox/Profiles/4qfu6u20.B_VPN/ -L
certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format.
cf-asa-hw-dickie-3:bin dick$ 

Dick

Comment 47

3 years ago
Kai,

Ok, dumb mistake of my part;  I needed to protect the embedded space in the profile path name.  Now we are getting some actual results:

cf-asa-hw-dickie-3:bin dick$ ./certutil -d dbm:"/Users/dick/Library/Application Support/Firefox/Profiles/4qfu6u20.B_VPN/" -L

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

DigiCert Secure Server CA                                    ,,   
DigiCert High Assurance CA-3                                 ,,   
EssentialSSL CA                                              ,,   
DigiCert High Assurance EV CA-1                              ,,   
134.15.84.97                                                 ,,   
DigiCert SHA2 Extended Validation Server CA                  ,,   
GeoTrust SSL CA - G2                                         ,,   
Verizon Akamai SureServer CA G14-SHA1                        ,,   
GeoTrust SSL CA - G3                                         ,,   
Go Daddy Secure Certificate Authority - G2                   ,,   
GeoTrust EV SSL CA - G4                                      ,,   
VeriSign Class 3 Secure Server CA - G3                       ,,   
GeoTrust SSL CA                                              ,,   
thawte SSL CA - G2                                           ,,   
DigiCert SHA2 High Assurance Server CA                       ,,   
Cybertrust Public SureServer SV CA                           ,,   
Entrust Certification Authority - L1C                        ,,   
COMODO High-Assurance Secure Server CA                       ,,   
COMODO RSA Organization Validation Secure Server CA          ,,   
COMODO SSL CA                                                ,,   
Verizon Akamai SureServer CA G14-SHA2                        ,,   
PositiveSSL CA 2                                             ,,   
StartCom Class 2 Primary Intermediate Server CA              ,,   
DigiCert SHA2 Secure Server CA                               ,,   
GeoTrust Global CA                                           ,,   
Symantec Class 3 Secure Server CA - G4                       ,,   
RapidSSL CA                                                  ,,   
GlobalSign Organization Validation CA - G2                   ,,   
Google Internet Authority G2                                 ,,   
GeoTrust DV SSL CA                                           ,,   
COMODO RSA Domain Validation Secure Server CA                ,,   
GlobalSign Domain Validation CA - SHA256 - G2                ,,   
Starfield Secure Certificate Authority - G2                  ,,   
cf-asa-hw-dickie-3:bin dick$

Comment 48

3 years ago
This is the IP address of my Cisco VPN box that shows up in the list:

134.15.84.97

Comment 49

3 years ago
I am guessing that you will want to see the details of the certificate from the VPN box at 134.15.84.97.  So here it is:

cf-asa-hw-dickie-3:bin dick$ ./certutil -d dbm:"/Users/dick/Library/Application Support/Firefox/Profiles/4qfu6u20.B_VPN/" -L -n 134.15.84.97
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 247927381 (0xec71255)
        Signature Algorithm: PKCS #1 SHA-1 With RSA Encryption
        Issuer: "CN=134.15.84.97"
        Validity:
            Not Before: Wed Mar 25 14:32:46 2015
            Not After : Sat Mar 22 14:32:46 2025
        Subject: "CN=134.15.84.97"
        Subject Public Key Info:
            Public Key Algorithm: PKCS #1 RSA Encryption
            RSA Public Key:
                Modulus:
                    be:29:a8:fd:6d:44:8c:48:82:4b:5e:e6:83:b2:f5:63:
                    2c:ca:3c:2e:5d:01:11:f2:9e:30:60:cf:d9:97:d1:42:
                    47:b6:92:27:86:f5:6e:10:41:e6:0e:0f:41:cf:ca:03:
                    50:41:98:b6:53:bd:1a:74:33:dc:34:5c:3b:40:d7:a5:
                    ba:cd:60:e6:78:8b:ef:4f:bc:0b:75:80:ed:ac:07:cc:
                    b3:8e:f9:18:aa:88:ee:6b:f2:99:86:07:ee:39:d3:46:
                    50:b2:a8:d5:4c:5d:24:84:4f:96:69:ad:7d:e6:16:29:
                    eb:ed:9f:49:dd:f3:48:82:67:b9:e1:36:a5:23:f4:eb
                Exponent: 65537 (0x10001)
    Signature Algorithm: PKCS #1 SHA-1 With RSA Encryption
    Signature:
        73:18:0b:ce:92:f3:16:e6:9b:d8:3a:d7:33:ca:ad:df:
        fe:c0:fa:a5:11:5e:9e:4c:1d:5b:d5:50:80:09:14:d1:
        5c:20:4b:c0:e0:dc:aa:3f:25:6e:a0:e7:4e:d8:cc:c7:
        ca:99:52:1c:da:0e:7f:77:59:3f:a2:b6:dc:13:e6:92:
        ae:f4:17:cc:d5:b1:8a:fa:e9:7c:b1:f2:8c:32:ad:1f:
        05:0e:82:6b:ab:46:f8:aa:4f:ec:ac:cc:69:dc:f2:02:
        0d:5d:38:df:a8:0b:b1:bc:9c:56:a8:9f:b7:4d:e8:57:
        6b:dc:8e:5e:a4:e3:c2:03:8f:5b:73:6a:d4:a6:d9:69
    Fingerprint (SHA-256):
        3F:92:EE:87:A4:82:22:0B:B4:82:4B:6B:97:36:AA:01:1C:FD:E1:06:8D:4D:06:45:A6:F1:E3:12:65:2A:E0:F0
    Fingerprint (SHA1):
        77:13:C1:54:6D:DB:A1:72:FF:DD:8C:9C:B4:F2:82:B0:E8:BD:40:10

    Certificate Trust Flags:
        SSL Flags:
        Email Flags:
        Object Signing Flags:

cf-asa-hw-dickie-3:bin dick$
(Reporter)

Comment 50

3 years ago
Thanks for that output. It doesn't show anything unusual.

First, please make a backup of the cert8.db file in your directory, in case we might want to analyze it later.

With Firefox closed, attempt to delete the cached copy of the router certificate:

./certutil -d dbm:"/Users/dick/Library/Application Support/Firefox/Profiles/4qfu6u20.B_VPN/" -D -n 134.15.84.97

Now list again

./certutil -d dbm:"/Users/dick/Library/Application Support/Firefox/Profiles/4qfu6u20.B_VPN/" -L -n 134.15.84.97

Does it still list the entry with the 134.... ? It yes, please continue to use the delete command, until the output of -L no longer shows the 134.... entry.

With all entries gone, attempt to connect to the router again.

Does it work?

If it works now, then the reason might have been bug 1056341.


It still doesn't work? Hmm....

Can you try to update to Firefox 37 ? Does it work or not?


After all of the above fails: Did I already suggest that you try a fresh profile?
(Reporter)

Comment 51

3 years ago
The serial number in comment 49 is different from the serial number in comment 22. Maybe it's really bug 1056341 and deleting all certs helps. I'm crossing fingers.

Comment 52

3 years ago
I backed-up the cert8.db file and then deleted the 134.15.84.97 entry with one run of certutil -D.

I then tried to connect to the VPN box, got all the certificate warnings, but the connection still then hangs.  :-(

I then tried using my other profile and it too hangs trying to connect to the VPN box.  :-(

These tests were run with Firefox 37.0.2.

I then went back to Firefox 35 to verify that the test environment had not changed, and I was able to connect with no problems.

Thanks for the help.
(Reporter)

Comment 53

3 years ago
Dick,

I've created a self-signed certificate that's equivalent to the one you listed in comment 49, and I set up a local test server, and configured it to support SSL 3 and TLS 1.0, only. Then I used Firefox 37 to connect to it. After I added the override, the connection worked fine.

You said, that you are successfully able to add an override for the self signed cert of that Cisco VPN box.

I no longer believe the error is on the TLS protocol / security level.
I suspect the error is on the http protocol level (which runs inside the TLS layer).

There are too many variables, and without being able to connect to our hardware, I cannot provide any more help to analysis this issue for you.

I'm resolving this bug as incomplete.

In order to make progress on this bug, we'd need an unencrypted log of the communication between Firefox and the Cisco box.

Maybe you could learn how to use a MITM proxy utility for the purposes of debugging your connection, but I cannot help you with learning how to use such a tool.

Using an utility that shows all the http protocol connection going back and forth between firefox and the server, and by comparing those logs between working firefox 36 and nonworking firefox 37, we could understand, which side of the connection is guilty of not proceeding with the connection.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → INCOMPLETE

Comment 54

3 years ago
Our IT folks upgraded my Cisco ASA 5505 hardware VPN box to IOS version 8.4(6) and then to version 9.1(6).  Running either version I was able to connect and login to my VPN box using Firefox 38.0.1.

This problem has been resolved for me.
(Reporter)

Comment 55

3 years ago
Thanks for the feedback, so let's conclude the implementation of TLS on older firmware versions was broken in one way or the other.
You need to log in before you can comment on or make changes to this bug.