1592557 - Test scripts appear to be broken

Status: RESOLVED FIXED

(NSS :: Test, defect, P2)

Description

[Franziskus Kiefer [:franziskus]]

The mach tests command is broken in multiple ways. This mostly stems from changes to the test shell scripts that are bad.
mach tests gtests fails because the PRNG test fails (cause is probably somewhere in bug 1496124).

But worse probably is that parsing of results is not working properly. results.html as well as the command line overview claim that tests are passed when they actually failed. This is easy to reproduce by breaking any gtest.
I also see things like gtests.sh: #14: freebl_gtest run successfully - Core file is detected - FAILED in CI logs [1] such that I have doubts that green on CI actually means everything is ok.

Marcus can you have a look at this as you're looking at the test scripts already?

[1] https://taskcluster-artifacts.net/MCgT8hxLRqK_9hI1k8s_1w/0/public/logs/live_backing.log

Comment 1

[Kevin Jacobs [:kjacobs]]

Regarding the cores, we were ignoring them completely before bug 1560593. They're still ignored on debug builds, because death tests (such as [1]) make it hard to determine if a core is actually the result of a failure.

[1] https://searchfox.org/mozilla-central/source/security/nss/gtests/freebl_gtest/mpi_unittest.cc#216

Comment 2

[Franziskus Kiefer [:franziskus]]

Right, I forgot about the death tests. So I guess that one is fine. I'm still a little concerned about the wrong overview though.

Comment 3

[Franziskus Kiefer [:franziskus]]

Attachment: Bug 1592557 - fix prng kat tests, r?jcj

fix for prng kat tests

Comment 4

[J.C. Jones [:jcj] (he/him)]

https://hg.mozilla.org/projects/nss/rev/474334bb790be9fac335cb66bf336fa3d9d15608