Closed Bug 1611471 Opened 4 years ago Closed 4 years ago

client certificate not always sent to the server

Categories

(Core :: Security: PSM, defect, P3)

72 Branch
defect

Tracking

()

RESOLVED WORKSFORME

People

(Reporter: lassenov, Unassigned)

Details

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0

Steps to reproduce:

I access different servers in my corporate environment via FF.

Actual results:

To most of the servers I get authenticated with my client cert from OS Windows cert store.
To 2 of them, however, the client certificate I don't get authenticated. I can only presume that for some reason the client cert is not being sent.

Expected results:

I should be authenticated in all the servers, expecting my client certificate.
That used to work with older versions of FF in the times when I was able to export/import my client cert manually into the FF cert store. It also works fine with other browsers.

This is kind of continuation of this bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=1120350
where Dana Keeler requested some more info as follows:

Dana Keeler: It would be good to include things like what kinds of certificates work vs. don't, or if there are any server differences, and if you can, include a packet trace of the working/not working TLS handshakes.

Answer: The certificate is the same as with the other servers. It is a client certificate. What other specifics do you need about it?
The servers are different, yes. But I am afraid that is all I can afford revealing in this regard. Equally I cannot provide packet or network traces as this is my corporate environment. However, if there is a way for me to inspect the TLS handshake in FF I will try providing the differences between the successful and the failing cases. Just let me know how can I do that.

Thanks! Some questions:

  • Are you using the same client certificate for each server?
  • If you look at the TLS handshakes, does Firefox send the client certificate to each server?
  • If you disable the osclientcert module and instead import your client certificate into your profile, does it work then?
Component: Networking: HTTP → Security: PSM
Flags: needinfo?(lassenov)

I would love to have some examples which servers don't work and some steps to reproduce. It may be HTTP/2 and connection coalescing involved (I don't recall now how exactly we handle client certificate authentication in such situations, CC'ing Dragana and Andy to chime in if they know from top of the head, or I have to look into the code to refresh my memory.) Other thing is that some cross-origin resources may be loaded via anonymous requests, which are using a separate connection with explicitly disabled client cert authentication. Depends on what symptoms you see (the description is too vague for me.)

There is also a possibility to collect logs with MOZ_LOG (the module list) set as:

timestamp,nsHttp:5,nsSocketTransport:5,pipnss:5

It may contain private info so send via email to Dana or me.

Thanks.

@Dana

  • Yes, it is the same client certificate.
  • Looking into the TLS handshake is the part that I don't know how to dive into with FF. Looking into Honza's comment I presume there is one possibility. The other one is probably with Wireshark. I will try both these days.
  • I am no longer able to import that client cert manually into the native FF cert store as the private key is no longer exportable from the OS. But in the old times, when it was exportable FF was working just fine.

@Honza
What I see is different for the 2 servers. For one of them I get the standard logon page, which in its turn doesn't allow me to login with basic credentials as that is how the server is configured. For the other server I receive [HTTP 403 - Forbidden] - not certain if it proves that the certificate is not sent in this case.
I will try to collect logs as per your instructions and will inspect them myself and will try to extract the part that seems relevant. If that doesn't help I will try Wireshark as well.
In the meantime, if you have further questions or hints based on the symptoms above, just let me know.

I have remembered this option, favorite of mine, in FF "When a server requests your personal certificate". I always keep it to the value of "Select one automatically". Now I switched it to "Ask you every time". The result is that in all the successful cases I get the popup asking me for the client certificate. However, for the 2 servers where the authentication fails I get no such popup. Hence, my initial impression is now proven.

Now we need to identify why that happens. I collected traces for one successful and one failing case and here is the difference I managed to identify.
For the successful case immediately after my certificates are traced these lines follow:

2020-01-25 06:34:27.064000 UTC - [Parent 2268: Timer]: D/nsSocketTransport STS dispatch [000001F9C3372460]
2020-01-25 06:34:27.064000 UTC - [Parent 2268: Timer]: D/nsSocketTransport PollableEvent::Signal
2020-01-25 06:34:27.064000 UTC - [Parent 2268: Timer]: D/nsSocketTransport PollableEvent::Signal PR_Write

Then normal communication continues with transferring web content.
Whereas for the failing scenario I identified these lines:

2020-01-25 06:28:22.805000 UTC - [Parent 20644: Socket Thread]: V/nsHttp nsHttpConnection::OnSocketWritable 000001662294D000 ReadSegments returned [rv=0 read=0 sock-cond=80470007 again=1]
2020-01-25 06:28:22.805000 UTC - [Parent 20644: Socket Thread]: D/nsSocketTransport nsSocketOutputStream::AsyncWait [this=0000016622949AB0]
2020-01-25 06:28:22.805000 UTC - [Parent 20644: Socket Thread]: D/nsSocketTransport STS dispatch [000001662445A370]
2020-01-25 06:28:22.805000 UTC - [Parent 20644: Socket Thread]: D/nsSocketTransport PollableEvent::Signal
2020-01-25 06:28:22.805000 UTC - [Parent 20644: Socket Thread]: D/nsSocketTransport PollableEvent::Signal PR_Write 1

It is the "ReadSegments returned" line that caught my attention. I hope that helps.
Let me know if it is sufficient or you need more traces.

Lyubomir, a packet trace from wireshark would really help here.

As I already outlined it is my corporate environment and those are productive servers. That is why I am simply not allowed to provide you wireshark traces.
What I can point out again is that GC, IE and Bing work properly and authenticate me successfully to both the servers.
Aren't there any TLS compliance tests which would help identify such a case?
One more detail I remember, when I was able to store the client cert in the FF cert store, FF was able to send the client cert those servers, both of them - if that can be of any help.

Flags: needinfo?(lassenov)

The priority flag is not set for this bug.
:keeler, could you have a look please?

For more information, please visit auto_nag documentation.

Flags: needinfo?(dkeeler)

Can you run Firefox with the environment variable RUST_LOG set to osclientcerts_static=debug, attempt to connect to the server, and attach the output here? (this may help: https://developer.mozilla.org/en-US/docs/Mozilla/Command_Line_Options#Miscellaneous)

Flags: needinfo?(dkeeler) → needinfo?(lassenov)
Priority: -- → P3
Status: UNCONFIRMED → RESOLVED
Closed: 4 years ago
Resolution: --- → INCOMPLETE

Unfortunately the luxury period when I was able to work on this issue is over. Although the issue was not resolved I have no other option but to switch to GC. So, you can close this report.

By the way, I observed another issue with client certificates while testing - if a client cert is imported in the Windows certstore and visible in FF, and then deleted from the Windows certstore it is still visible in FF - very strange. Then if again deleted from within FF it disappears from the FF UI, but after reloading the content it appears there again. However, at the same time it is no longer visible via the Windows native UI from IE or GC. Hope this helps to some more extent.

That is all I could do so far. Farewell productive FF. Hello FF for reading news only.

Flags: needinfo?(lassenov)

Hello FF, just to confirm for your info - with FF 77.0.1 the client cert authentication is back to working state in all of my productive use cases.
Awesome! Way to go!
Thanks, guys, thanks a lot!

Resolution: INCOMPLETE → WORKSFORME
You need to log in before you can comment on or make changes to this bug.