Our QA is quite extensive on handling of alert corner cases. Our code that checks if a signature algorithm is supported ignores the role of policy. If SHA1 is turned off by policy, for instance, we only detect that late in the game. This shows up in our test cases as decrypt_alerts rather than illegal_parameter or handshake_error alerts. It also shows up in us apparently accepting a client auth request which only has invalid alerts.
We also don't handle filtering out signature algorithms that are illegal in tls 13 mode.
This patch not only fixes these issues, but also issues where we proposing signature algorithms in server mode that we don't support by policy.
This patch is for reference. It still requires 1) new test cases, and 2) handling of signing algorithm policy, not just hash policies.