1644209 - Intermittent [ FAILED ] Tls13PskTest/Tls13PskTest.ClientVerifyHashType/1, where GetParam() = (1, 4867) (36 ms)

Status: RESOLVED FIXED

(NSS :: Test, defect, P1)

Description

[Treeherder Bug Filer]

Filed by: kjacobs [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=305481936&repo=nss
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/bbpvKSuyRBexxncPtbeVQQ/runs/0/artifacts/public/logs/live_backing.log

---

Version: DTLS 1.3
server: Changing state from INIT to CONNECTING
client: Changing state from INIT to CONNECTING
handshake old: [92] fefd829c0ba069767415a5d1dd422d2d90eacd2a8e4a32c1e1095e0ffcca45c1...
handshake new: [92] fefd829c0ba069767415a5d1dd422d2d90eacd2a8e4a32c1e1095e0ffcca45c1...
record old: [104] 0200005c000000000000005cfefd829c0ba069767415a5d1dd422d2d90eacd2a...
record new: [104] 0200005c000000000000005cfefd829c0ba069767415a5d1dd422d2d90eacd2a...
server: Filtered packet: [269] 16fefd000000000000000000680200005c000000000000005cfefd829c0ba069...
client: Fatal alert sent: 50
../../gtests/ssl_gtest/tls_agent.cc:803: Failure
Expected equality of these values:
  expected
    Which is: '/' (47, 0x2F)
  alert->description
    Which is: '2' (50, 0x32)
client: Handshake failed with error SSL_ERROR_RX_MALFORMED_SERVER_HELLO: SSL received a malformed Server Hello handshake message.
client: Changing state from CONNECTING to ERROR
[  FAILED  ] Tls13PskTest/Tls13PskTest.ClientVerifyHashType/1, where GetParam() = (1, 4867) (36 ms)

This one seems a bit more reproducible than the other DTLS intermittents: my MacBook can reproduce it after a few hundred iterations. It appears the SelectedCipherSuiteReplacer filter is occasionally clobbering the ServerHello, resulting in a decode_error alert originating from one of the highlighted lines in [1]

I've confirmed there is no test failure in stream-variant test runs, and removing the filter causes the test to no longer fail in datagram-variant runs.

[1] https://searchfox.org/mozilla-central/rev/35b97af64a55d1d30caa4d6e9fabc1a7fbabc509/security/nss/lib/ssl/ssl3con.c#6748,6777

Comment 1

[Intermittent Failures Robot]

2 failures in 5211 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• nss: 2

Platform breakdown:

• aarch64: 1
• linux32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1644209&startday=2020-06-08&endday=2020-06-14&tree=all

Comment 2

[Intermittent Failures Robot]

2 failures in 5252 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• nss: 2

Platform breakdown:

• windows2012-32: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1644209&startday=2020-06-22&endday=2020-06-28&tree=all

Comment 3

[Intermittent Failures Robot]

1 failures in 5265 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• nss: 1

Platform breakdown:

• linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1644209&startday=2020-07-06&endday=2020-07-12&tree=all

Comment 4

[Intermittent Failures Robot]

1 failures in 4310 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• nss: 1

Platform breakdown:

• windows2012-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1644209&startday=2020-07-20&endday=2020-07-26&tree=all

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Comment 6

[Intermittent Failures Robot]

2 failures in 4046 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• nss: 2

Platform breakdown:

• linux64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1644209&startday=2020-08-24&endday=2020-08-30&tree=all

Comment 7

[Daiki Ueno [:ueno]]

This still happens in our Fedora builds of 3.56.

Comment 8

[Intermittent Failures Robot]

3 failures in 3694 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• nss: 3

Platform breakdown:

• windows2012-32: 1
• aarch64: 1
• linux32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1644209&startday=2020-10-12&endday=2020-10-18&tree=all

Comment 9

[Kevin Jacobs [:kjacobs]]

The root cause here is that SelectedCipherSuiteReplacer doesn't expect a [legacy_]session_id in TLS 1.3+. Further, the version check itself doesn't normalize for DTLS versions, resulting in DTLS 1.2 > TLS 1.3 (`0xfefd > 0x0304) and a mis-parsed header.

Normally this produces the same error code that the test looks for, so the failure was not apparent. Patch incoming.

Comment 10

[Kevin Jacobs [:kjacobs]]

Attachment: Bug 1644209 - Fix broken SelectedCipherSuiteReplacer filter.

This patch corrects the SelectedCipherSuiteReplacerfilter to always parse the session_id variable (legacy_session_id for TLS 1.3+). The previous code attempted to skip it in 1.3+ but did not account for DTLS wire versions, resulting in intermittent failures.

Comment 11

[Kevin Jacobs [:kjacobs]]

https://hg.mozilla.org/projects/nss/rev/a79d14b06b4a3ca19c169a4b0c1f28d5e2f25b35