1484751 - [meta] IPC PSM API for network process isolation

Status: NEW

(Core :: Security: PSM, enhancement, P3)

Description

[Honza Bambas (:mayhemer)]

See the URL for details.

The idea is to forward following callback processing asynchronously to a different process (triggered on the socket process with an empty NSS and processed on the parent process with a full NSS):

SSL_AuthCertificateHook 
Responsible for certificate (chain) verification
Can return would-block

SSL_HandshakeCallback
Called after the handshake is done, doesn’t return anything, only collects telemetry and updates some info on the socket

SSL_SetCanFalseStartCallback
No need to proxy this one

SSL_GetClientAuthDataHook (probably for a followup bug, we need to pass the handle serialization around and use a modified soft-token overlay to perform the ops with a sync IPC call)
Can return would-block
The private key is only a handle

SSL_SetPKCS11PinArg
This sets an argument (in PSM case IR hanging of the socket) that is passed to the auth function set globally with PK11_SetPasswordFunc
This is used in a sync matter

Comment 1

[Honza Bambas (:mayhemer)]

Attachment: wip1 (backup)

this builds on win and on top of [1] and tries to see all certs as valid (quick workaround for ssl support).  but I'm getting a number of weird assertions all around the code, probably related to response handling that may be because of some violation of the stream listener contract.

kershaw sees some assertions as well, even w/o this patch.

[1] https://hg.mozilla.org/projects/larch/rev/94a22fd022b9e1d6b78a41081f302f5e8309a80a