1847153 - mochitest-a11y-1proc test-failures are getting mistakenly starred to a grab-bag bug, because the log parser can't find the actual test failure

Status: RESOLVED DUPLICATE of bug 1845221

(Core :: Disability Access APIs, defect)

Description

[Daniel Holbert [:dholbert]]

Bug 1538599 is a grab-bag of orange logs where the only thing the parser could find was "exit status 2" (which I think is present at the end of every test failure log -- that or "exit status 1").

One category of test-failures that are getting starred there seem to be mochitest-a11y-1proc logs, where there was some test failure (i.e. we have INFO Failed: 1 in the summary at the bottom), but the test failure itself is not there in the log (at least, not in a way that I can find or the log parser can find).

Here are some examples:
Win11: https://treeherder.mozilla.org/logviewer?job_id=424230733&repo=autoland&lineNumber=13427
macOS: https://treeherder.mozilla.org/logviewer?job_id=424230269&repo=autoland&lineNumber=12468
(You can look at the "orange factor" link at the top of bug 1538599 and search for mochitest-a11y-1proc to find more of these.)

These all seem to end with something like the following:

[task 2023-07-29T04:56:54.666Z] 04:56:54     INFO -  0 INFO TEST-START | Shutdown
[task 2023-07-29T04:56:54.670Z] 04:56:54     INFO -  1 INFO Passed:  38396
[task 2023-07-29T04:56:54.670Z] 04:56:54     INFO -  2 INFO Failed:  1
[task 2023-07-29T04:56:54.670Z] 04:56:54     INFO -  3 INFO Todo:    70
[task 2023-07-29T04:56:54.670Z] 04:56:54     INFO -  4 INFO Mode:    non-e10s
[task 2023-07-29T04:56:54.670Z] 04:56:54     INFO -  5 INFO SimpleTest FINISHED

...where there was exactly 1 subtest failure, but I'm not immediately finding it anywhere in the log above, and the log parser apparently isn't either.

eeejay, perhaps you or someone else from the a11y team could take a look and help find out what's going on here? Maybe there's a test that's misbehaving somehow? Right now this is contributing a fair amount to the failure volume on grab-bag bug 1538599, for what seems to really be a legit test failure that's getting buried somehow. Thanks!

Comment 1

[James Teh [:Jamie]]

This is being investigated in bug 1845221. In short, we don't know why this is happening. The log doesn't show anything useful and we can't reproduce the problem locally. We know it's somehow caused by bug 854796, but no more than that. The current plan is to rewrite the problematic test in a newer framework which should at the very least hopefully make it clearer what the problem is.

Comment 2

[Daniel Holbert [:dholbert]]

Thanks, Jamie!