1560593 - Cleanup.sh script does not set error exit code for tests that "Failed with core"

Status: RESOLVED FIXED

(NSS :: Test, defect, P1)

Description

[Kevin Jacobs [:kjacobs]]

tests/common/cleanup.sh only counts tests non-crashing failures and ASAN errors when determining the exit code: https://searchfox.org/mozilla-central/source/security/nss/tests/common/cleanup.sh#56

This results in taskcluster reporting green for jobs that fail with a core dump. For example, Bogo fails on debug builds but is reported as passing:

#1  0x00007f413f29e801 in __GI_abort () at abort.c:79
#2  0x00007f413f9eb932 in PR_Assert (s=0x55a588471de3 "0", file=0x55a588471b20 "../../lib/ssl/ssl3con.c", ln=7432) at ../../../../pr/src/io/prlog.c:553
#3  0x000055a5883a1748 in ssl3_HandleCertificateRequest (ss=0x55a589ffe8d0, b=0x55a58a01029f 'N' <repeats 200 times>..., length=0) at ../../lib/ssl/ssl3con.c:7432
#4  0x000055a5883ab790 in ssl3_HandlePostHelloHandshakeMessage (ss=0x55a589ffe8d0, b=0x55a58a010294 "\002\001@", length=11) at ../../lib/ssl/ssl3con.c:11898
#5  0x000055a5883ab4a4 in ssl3_HandleHandshakeMessage (ss=0x55a589ffe8d0, b=0x55a58a010294 "\002\001@", length=11, endOfRecord=1) at ../../lib/ssl/ssl3con.c:11822
#6  0x000055a5883abb3a in ssl3_HandleHandshake (ss=0x55a589ffe8d0, origBuf=0x55a589ffeb80) at ../../lib/ssl/ssl3con.c:11996
#7  0x000055a5883acdf0 in ssl3_HandleNonApplicationData (ss=0x55a589ffe8d0, rType=ssl_ct_handshake, epoch=0, seqNum=3, databuf=0x55a589ffeb80)
    at ../../lib/ssl/ssl3con.c:12515
#8  0x000055a5883ad7e6 in ssl3_HandleRecord (ss=0x55a589ffe8d0, cText=0x7ffd35288b10) at ../../lib/ssl/ssl3con.c:12796
#9  0x000055a5883eb41b in ssl3_GatherCompleteHandshake (ss=0x55a589ffe8d0, flags=0) at ../../lib/ssl/ssl3gthr.c:512
#10 0x000055a5883ec067 in ssl_GatherRecord1stHandshake (ss=0x55a589ffe8d0) at ../../lib/ssl/sslcon.c:74
#11 0x000055a5883bde06 in ssl_Do1stHandshake (ss=0x55a589ffe8d0) at ../../lib/ssl/sslsecur.c:41
#12 0x000055a5883bef58 in SSL_ForceHandshake (fd=0x55a589ffe860) at ../../lib/ssl/sslsecur.c:384
#13 0x000055a58838a26a in TestAgent::Handshake (this=0x55a589ffe830) at ../../gtests/nss_bogo_shim/nss_bogo_shim.cc:421
#14 0x000055a58838a5c1 in TestAgent::DoExchange (this=0x55a589ffe830) at ../../gtests/nss_bogo_shim/nss_bogo_shim.cc:491
#15 0x000055a58838bd43 in RunCycle (cfg=std::unique_ptr<const Config> = {...}) at ../../gtests/nss_bogo_shim/nss_bogo_shim.cc:612
#16 0x000055a58838bf6d in main (argc=11, argv=0x7ffd35288fe8) at ../../gtests/nss_bogo_shim/nss_bogo_shim.cc:650```

Comment 1

[J.C. Jones [:jcj] (he/him)]

Failed with core should be handled somehow other than just passing -- we also have the retrigger result for things that may be intermittent, and this might be appropriate here too.

Comment 2

[Kevin Jacobs [:kjacobs]]

Attachment: Bug 1560593 - Cleanup.sh to treat core dumps as test failures on optimized builds. r=jcj

Comment 3

[J.C. Jones [:jcj] (he/him)]

https://hg.mozilla.org/projects/nss/rev/360010725fdb499e0c63f8b37385f9c914808a0b

Comment 4

[Kai Engert (:KaiE:)]

This change introduced a new error:
/home/user/moz/nss/hg/nss/tests/common/cleanup.sh: line 63: [: -eq: unary operator expected

This happens if BUILD_OPT is undefined (empty).

In my build environment variable OBJDIR isn't set.

Comment 5

[J.C. Jones [:jcj] (he/him)]

Oops. That seems like an easy fixup though.

Comment 6

[Kevin Jacobs [:kjacobs]]

Attachment: Bug 1560593 - Check that BUILD_OPT is defined before testing its value. r?jcj

Comment 7

[J.C. Jones [:jcj] (he/him)]

Attachment: Bug 1560593 - Fixup, check BUILD_OPT before using in cleanup.sh r?kjacobs

Comment 8

[J.C. Jones [:jcj] (he/him)]

https://hg.mozilla.org/projects/nss/rev/44aa330de2aa0072d6121d82516d39a00346c1b9

Comment 9

[Kai Engert (:KaiE:)]

thanks!