1973187 - Coverage CI task fails to run ssl_gtests

Status: RESOLVED FIXED

(NSS :: Test, defect)

Description

[Julien Cristau [:jcristau] (back August 18)]

The Coverage CI job is broken since bug 1966786:

[task 2025-06-19T15:12:34.824+00:00] Run ssl_gtests to get a coverage report...
[task 2025-06-19T15:12:34.824+00:00] Traceback (most recent call last):
[task 2025-06-19T15:12:34.824+00:00] File "/builds/worker/nss/./mach", line 304, in <module>
[task 2025-06-19T15:12:34.824+00:00] main()
[task 2025-06-19T15:12:34.824+00:00] File "/builds/worker/nss/./mach", line 300, in main
[task 2025-06-19T15:12:34.824+00:00] parse_arguments()
[task 2025-06-19T15:12:34.824+00:00] File "/builds/worker/nss/./mach", line 296, in parse_arguments
[task 2025-06-19T15:12:34.825+00:00] return parser.parse_args()
[task 2025-06-19T15:12:34.825+00:00] ^^^^^^^^^^^^^^^^^^^
[task 2025-06-19T15:12:34.825+00:00] File "/usr/lib/python3.12/argparse.py", line 1908, in parse_args
[task 2025-06-19T15:12:34.828+00:00] args, argv = self.parse_known_args(args, namespace)
[task 2025-06-19T15:12:34.828+00:00] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[task 2025-06-19T15:12:34.828+00:00] File "/usr/lib/python3.12/argparse.py", line 1941, in parse_known_args
[task 2025-06-19T15:12:34.828+00:00] namespace, args = self._parse_known_args(args, namespace)
[task 2025-06-19T15:12:34.828+00:00] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[task 2025-06-19T15:12:34.828+00:00] File "/usr/lib/python3.12/argparse.py", line 2144, in _parse_known_args
[task 2025-06-19T15:12:34.829+00:00] positionals_end_index = consume_positionals(start_index)
[task 2025-06-19T15:12:34.829+00:00] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[task 2025-06-19T15:12:34.829+00:00] File "/usr/lib/python3.12/argparse.py", line 2121, in consume_positionals
[task 2025-06-19T15:12:34.829+00:00] take_action(action, args)
[task 2025-06-19T15:12:34.829+00:00] File "/usr/lib/python3.12/argparse.py", line 2017, in take_action
[task 2025-06-19T15:12:34.829+00:00] action(self, namespace, argument_values, option_string)
[task 2025-06-19T15:12:34.829+00:00] File "/usr/lib/python3.12/argparse.py", line 1280, in call
[task 2025-06-19T15:12:34.829+00:00] subnamespace, arg_strings = parser.parse_known_args(arg_strings, None)
[task 2025-06-19T15:12:34.829+00:00] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[task 2025-06-19T15:12:34.829+00:00] File "/usr/lib/python3.12/argparse.py", line 1941, in parse_known_args
[task 2025-06-19T15:12:34.829+00:00] namespace, args = self._parse_known_args(args, namespace)
[task 2025-06-19T15:12:34.829+00:00] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[task 2025-06-19T15:12:34.829+00:00] File "/usr/lib/python3.12/argparse.py", line 2165, in _parse_known_args
[task 2025-06-19T15:12:34.829+00:00] stop_index = consume_positionals(start_index)
[task 2025-06-19T15:12:34.829+00:00] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[task 2025-06-19T15:12:34.829+00:00] File "/usr/lib/python3.12/argparse.py", line 2121, in consume_positionals
[task 2025-06-19T15:12:34.829+00:00] take_action(action, args)
[task 2025-06-19T15:12:34.829+00:00] File "/usr/lib/python3.12/argparse.py", line 2017, in take_action
[task 2025-06-19T15:12:34.829+00:00] action(self, namespace, argument_values, option_string)
[task 2025-06-19T15:12:34.829+00:00] File "/builds/worker/nss/./mach", line 203, in call
[task 2025-06-19T15:12:34.829+00:00] self.runSslGtests(outdir)
[task 2025-06-19T15:12:34.829+00:00] File "/builds/worker/nss/./mach", line 182, in runSslGtests
[task 2025-06-19T15:12:34.829+00:00] run_tests("ssl_gtests", env=env, silent=True)
[task 2025-06-19T15:12:34.829+00:00] File "/builds/worker/nss/./mach", line 41, in run_tests
[task 2025-06-19T15:12:34.829+00:00] subprocess.check_call(command, env=os_env, stdout=stdout, stderr=stderr)
[task 2025-06-19T15:12:34.829+00:00] File "/usr/lib/python3.12/subprocess.py", line 413, in check_call
[task 2025-06-19T15:12:34.829+00:00] raise CalledProcessError(retcode, cmd)
[task 2025-06-19T15:12:34.829+00:00] subprocess.CalledProcessError: Command '/builds/worker/nss/tests/all.sh' returned non-zero exit status 1.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

:jschanck, since you are the author of the regressor, bug 1966786, could you take a look? Also, could you set the severity field?

For more information, please visit BugBot documentation.

Comment 2

[Julien Cristau [:jcristau] (back August 18)]

Attachment: leaksanitizer trace

removing silent=True from the run_tests call turns up an error from asan/leaksanitizer.

Comment 3

[Dennis Jackson]

Attachment: Bug 1973187 - Fix leaking locks when toggling SSL_NO_LOCKS. r=#nss-reviewers

Comment 4

[Dennis Jackson]

This patch fixes the underlying issue. However, there's a remaining mystery for why this wasn't caught in other CI jobs. When I ran the OptionSet test locally with ASAN I saw the same issue reported by the coverage job, but the gtest job with an ASAN build doesn't detect the leak.

Comment 5

[Julien Cristau [:jcristau] (back August 18)]

Attachment: Bug 1973187 - don't silence ssl_gtests output when running with code coverage. r?#nss-reviewers

Comment 6

[Julien Cristau [:jcristau] (back August 18)]

don't silence ssl_gtests output when running with code coverage. r=nss-reviewers,nkulatova
https://hg-edge.mozilla.org/projects/nss/rev/5304763a1847db672055a04fb95697e122b63a16

Comment 7

[Dennis Jackson]

https://hg.mozilla.org/projects/nss/rev/c548e4aa39367e41a46cade6684c54cb4a1b6b36

Comment 8

[Intermittent Failures Robot]

7 failures were associated with this bug in the last 7 days.

Repository breakdown:

• nss: 7

Table

	no_variant
linux-x86/unknown build	7

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1973187&startday=2025-06-23&endday=2025-06-29&tree=all