1007056 - Match against any TEST-UNEXPECTED-FOO not just TEST-UNEXPECTED-{PASS,FAIL}

Status: RESOLVED FIXED

(Tree Management Graveyard :: TBPL, defect)

Description

[Ed Morley [:emorley]]

<jgraham> edmorley: How hard would it be to fix tbpl so that WARNING - TEST-UNEXPECTED-[^ ]* is marked for the purposes of the summary etc.
<jgraham> See e.g. https://tbpl.mozilla.org/php/getParsedLog.php?id=39140593&tree=Cedar&full=1 which has Summary: empty but clearly has a warning in the logs and has the status correctly set to WARNING
<edmorley> jgraham: looking
<edmorley> jgraham: TBPL currently matches separately against either ERROR|CRITICAL or TEST-UNEXPECTED-(?:PASS|FAIL)
<edmorley> jgraham: adjusting the latter would be easy and not likely to give false positives; the former I'd have to see if anything else has been using warning for things that don't make the job fail
<edmorley> https://hg.mozilla.org/webtools/tbpl/file/a4ecc8160583/php/inc/GeneralErrorFilter.php#l28
<edmorley> jgraham: though there will be other consumers of log output that won't expect !TEST-UNEXPECTED-{FAIL,PASS} - and yeah we really need to sort this out to something consistent and using structured logs etc ettc
<jgraham> edmorley: So I think that web-platform-tests can produce TEST-UNEXPECTED-TIMEOUT, -CRASH and -NOTRUN
<jgraham> Oh and -ERROR
<jgraham> But I will check that's actually what mozharness does
<edmorley> other places that may need changing (some just match on this string, not what follows) eg https://mxr.mozilla.org/build-central/search?string=TEST-UNEXPECTED
<jgraham> Yeah, so my mozharness script basically does TEST-UNEXPECTED-%(status)s
<jgraham> (and the possible values of status are PASS, FAIL, TIMEOUT, NOTRUN, ERROR, CRASH)
<edmorley> jgraham: I don't see any problems with changing TBPL and others to just match against /TEST-UNEXPECTED-/
<edmorley> we're just boxing ourselves into a hardcoded corner otherwise :-)
<jgraham> edmorley: OK. I don't want to break things if these same regexps are being used in some places to actually set the results
<edmorley> well, more than already
<edmorley> We seem to be using that regexp in some places already, I think there's little chance of false positives

Comment 1

[Ed Morley [:emorley]]

Attachment: Match against TEST-UNEXPECTED-* not just TEST-UNEXPECTED-{PASS,FAIL}

Will test in the VM before landing, but I can't see how this will result in false positives.

Comment 2

[Ed Morley [:emorley]]

Haven't had time to test in a VM (takes ages to build after a vagrant destroy & then none of the recent logs will be downloaded and parsed), will check on dev:
remote:   https://hg.mozilla.org/webtools/tbpl/rev/4cd82e3fd793

Comment 3

[Ryan VanderMeulen [:RyanVM]]

In production :)