1292611 - buildbot: log entry with SSL_ERROR_HANDSHAKE_FAILURE_ALERT incorrectly leads to status "failed build"

Status: RESOLVED FIXED

(NSS :: Test, defect)

Description

[Kai Engert (:KaiE:)]

automation/buildbot-slave/build.sh uses
  grep -B50 FAIL ${OUTPUTFILE}

to dump any fragments around failures, and it uses the grep result (lines found) to decide if there were failures.

This worked in the past, but nowadays the log mentions error code SSL_ERROR_HANDSHAKE_FAILURE_ALERT, which apparently is expected, and doesn't mean a failure.

It matches the grep, and we get false negative build results.

Ideally our logfile shouldn't contain uppercase FAIL for expected failures.

In the past, the usual keyword for detecting a failure was FAILED.

I wonder if I had changed that to FAIL because of gtests. I'm not sure. However, just yesterday we changed the gtest to print html_passed or html_failed lines for each individual gtest, which means, we'll a get PASSED or FAILED for each gtest, too.

So, I suggest we change the grep to search for "FAILED".

Comment 1

[Kai Engert (:KaiE:)]

Attachment: 1292611-v1.patch

Comment 2

[Tim Taubert [:ttaubert] (inactive)]

Comment on attachment 8778344 [details] [diff] [review]
1292611-v1.patch

Review of attachment 8778344 [details] [diff] [review]:
-----------------------------------------------------------------

The whole approach seems brittle, but so is having multiple test suites with different output formats :| I'm fine with this if it fixes gtests false-negatives on buildbot.

Comment 3

[Kai Engert (:KaiE:)]

https://hg.mozilla.org/projects/nss/rev/1299323cf18b

Comment 4

[Kai Engert (:KaiE:)]

(In reply to Tim Taubert [:ttaubert] from comment #2)
> I'm fine with this if it fixes gtests false-negatives on buildbot.

It fixed it.

Comment 5

[Tim Taubert [:ttaubert] (inactive)]

(In reply to Kai Engert (:kaie) from comment #4)
> (In reply to Tim Taubert [:ttaubert] from comment #2)
> > I'm fine with this if it fixes gtests false-negatives on buildbot.
> 
> It fixed it.

\o/