Closed Bug 1592557 Opened 2 years ago Closed 2 years ago

Test scripts appear to be broken

Categories

(NSS :: Test, defect, P2)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: franziskus, Assigned: franziskus)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

The mach tests command is broken in multiple ways. This mostly stems from changes to the test shell scripts that are bad.
mach tests gtests fails because the PRNG test fails (cause is probably somewhere in bug 1496124).

But worse probably is that parsing of results is not working properly. results.html as well as the command line overview claim that tests are passed when they actually failed. This is easy to reproduce by breaking any gtest.
I also see things like gtests.sh: #14: freebl_gtest run successfully - Core file is detected - FAILED in CI logs [1] such that I have doubts that green on CI actually means everything is ok.

Marcus can you have a look at this as you're looking at the test scripts already?

[1] https://taskcluster-artifacts.net/MCgT8hxLRqK_9hI1k8s_1w/0/public/logs/live_backing.log

Flags: needinfo?(marcus.apb)

Regarding the cores, we were ignoring them completely before bug 1560593. They're still ignored on debug builds, because death tests (such as [1]) make it hard to determine if a core is actually the result of a failure.

[1] https://searchfox.org/mozilla-central/source/security/nss/gtests/freebl_gtest/mpi_unittest.cc#216

Right, I forgot about the death tests. So I guess that one is fine. I'm still a little concerned about the wrong overview though.

Assignee: nobody → franziskuskiefer
Status: NEW → ASSIGNED
Flags: needinfo?(marcus.apb)
Priority: -- → P2

fix for prng kat tests

Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → 3.48
You need to log in before you can comment on or make changes to this bug.