1101214 - SSL certificate validation problem Error code: sec_error_bad_der

Status: RESOLVED INVALID

(Core :: Security: PSM, defect)

Description

[Russ]

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; InfoPath.3; rv:11.0) like Gecko

Steps to reproduce:

HTTPS inspection (using an imported trusted CA) on a Websense proxy is resulting in the Error code: sec_error_bad_der for clients running Firefox 33 and later.  

Confirmed issue is not present in version 32.

Seems similar to bug 1060929



Actual results:

Page fails to load, error message displayed:

Secure Connection Failed

An error occurred during a connection to www.vmware.com. security library: improperly formatted DER-encoded message. (Error code: sec_error_bad_der)

    The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.
    Please contact the website owners to inform them of this problem.



Expected results:

HTTPS web sites should load as expected with a certificate issued by the Websense proxy.

Comment 1

[:Gijs (he/him)]

Can you provide the public cert info? I expect this is the same as bug 1088140, but without the cert it's impossible to tell. :-)

Comment 2

[Russ]

Attachment: example.cer

Sample proxy generated cert attached.

Comment 3

[:Gijs (he/him)]

Not RSA-PSS then... CC'ing folks in the know. Here's openssl's output to save some steps:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            4d:5c:9d:85:12:5e:f8:de:e1:31:88:37:44:35:87
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, ST=WA, L=Spokane, O=Itron Inc, OU=IT Security, CN=Itron Corporate IT Security/emailAddress=[e-mail]
        Validity
            Not Before: Apr 17 08:15:03 2014 GMT
            Not After : Apr 17 08:15:03 2015 GMT
        Subject: C=US, ST=CALIFORNIA, L=Palo Alto, O=VMware, OU=IT Operations, CN=*.vmware.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
            RSA Public Key: (1024 bit)
                Modulus (1024 bit):
                    00:c0:5c:83:54:70:b1:ec:51:98:b8:32:2f:eb:12:
                    20:a8:0a:e1:7d:a7:9c:8c:42:20:46:7a:18:a2:1a:
                    aa:13:ee:e9:c0:4f:11:bb:82:17:f3:01:d8:e2:07:
                    b1:14:32:a2:cd:fe:c0:dc:61:58:4f:89:88:dc:4d:
                    66:d1:fc:f0:02:6e:be:93:bb:84:7e:eb:21:3f:fe:
                    2d:dd:bb:f5:1d:ea:3b:49:a7:8c:5c:84:79:36:5a:
                    97:b2:54:b7:ee:43:c0:8a:a9:92:cf:eb:0a:43:25:
                    6f:36:1b:ec:f0:4a:bc:9a:9d:c5:5d:34:03:d4:05:
                    b4:11:21:24:76:dc:04:78:3d
                Exponent: 65537 (0x10001)
    Signature Algorithm: sha1WithRSAEncryption
        89:67:1b:76:60:d5:76:4c:0c:ab:95:1d:79:80:04:14:74:e1:
        5b:e0:00:07:05:22:a7:cb:23:be:78:c3:f8:e2:9e:b0:a7:f6:
        8a:40:d7:10:bd:a0:c8:0c:56:04:35:c1:2f:02:5e:2b:ef:0a:
        26:9a:55:b2:1d:52:d7:c7:ef:15:9d:c4:90:90:11:d9:16:45:
        62:ff:23:29:fa:bf:ac:72:13:e3:0d:a9:b3:32:22:1a:bd:dd:
        a1:c0:e2:62:b5:f5:86:f8:e0:c8:11:ee:26:74:7a:ee:7a:24:
        a5:14:95:48:70:c2:cb:26:f9:da:2f:3f:6b:cc:51:92:ff:4e:
        d3:65:bb:b7:c3:70:38:9c:10:8e:98:3f:9c:52:85:10:6f:62:
        e2:81:96:13:2e:d9:ae:d4:5f:fc:7f:75:cc:77:e2:05:2a:16:
        83:23:66:9a:9c:f5:76:02:af:69:c9:5e:b1:ed:af:ec:a6:a3:
        a1:3a:33:9c:4e:e0:f7:9c:16:d1:f4:d3:00:32:72:d8:89:21:
        0f:7e:16:51:72:89:55:47:cd:e3:83:71:fe:80:f0:9e:9f:31:
        2a:a3:db:21:aa:a5:e2:a9:f6:53:5f:a5:a2:23:56:0c:35:f3:
        d8:b2:17:75:d8:a8:f7:ca:15:19:74:7d:2d:f7:bd:08:51:e4:
        de:87:c2:ba

Comment 4

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

The encoding of the serial number has a leading 0 byte but its first data byte doesn't have its highest bit set, which makes it an invalid DER encoding (it would be shorter to represent the value without the leading 0).

Comment 5

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Russ, could you please verify that the software on the Websense proxy is up to date with the latest version? And, if so, could you please open a support ticket with VMWare about this issue, including a link to this bug report? It would be good to have a VMWare representative comment in this bug about their intent to address the issue on their end before we consider doing anything here.

Comment 6

[Russ]

Brian, I have filed a report with Websense and was told that they were escalating it to development.  I did reference this thread so hopefully they will respond here as well.

Comment 7

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Thanks Russ! I really appreciate you doing that. If you're willing to share your Websense bug/case #, please do so. If you don't want to, that's understandable.

Comment 8

[Russ]

Websense has effectively rejected the bug, stating that since it works in other browsers and previous versions of Firefox, and that the issue seems to occur with other proxies besides Websense, that this is a Mozilla issue.

Comment 9

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Thanks Russ. I'm worried that if we start accepting invalid encodings of serial numbers, then we'll run into other nasty consequences later, especially if we try to implement new (not-yet-standardized) revocation checking mechanisms, because we'd have to make a choice of how to compare serial numbers--bytewise vs. numerical--where neither choice is clearly better than the other. Therefore, I'm quite hesitant to make a change on our end, especially since Websense is basically saying that such a change would have to be permanent rather than temporary since they don't want to fix their software.

It looks like Websense is generating a random value for the serial number. Does Websense allow you to force it to generate a new certificate? If so, you might try triggering that mechanism until it generates one that has a valid encoding. If you try this, please let me know if it works.

Also, could you ask Websense to supply contact information so that we can communicate directly with them? I'd like to help them fix this issue on their end.

Thanks!

Comment 10

[Russ]

I have asked Websense to respond to the thread.

Comment 11

[Russ]

FYI, my Firefox is now at version 34.0 and does not exhibit this problem anymore.  I have asked other users to test and they can no longer reproduce on the current version either.

Comment 12

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Russ from comment #11)
> FYI, my Firefox is now at version 34.0 and does not exhibit this problem
> anymore.  I have asked other users to test and they can no longer reproduce
> on the current version either.

Russ, did your Websense server regenerate its certificate? Perhaps it regenerated and the new one now doesn't have the redundant zero.

In any case, I think things are working as intended, so I'm going to close this as INVALID, which means "works as intended." Thank you very much for all your help here Russ!