1784993 - Need a way to differentiate failures in sub tests to avoid being tracked by a single intermittent failure bug

Status: NEW

(Testing :: General, enhancement, P3)

Description

[Hiroyuki Ikezoe (:hiro)]

For example, test_group_hittest-1.html has a bunch of sub tests and each sub test is basically independent and unrelated to others. The test_group_hittest-1.html just opens a new tab for each sub test and runs each sub test respectively. So if test_group_hittest-1.html is disabled because some of the sub tests failed frequently, we we will loose opportunities to run some unrelated sub tests. It would be really nice to have a way to avoid this situation.

Comment 1

[Hiroyuki Ikezoe (:hiro)]

FYI test_printpreview.xhtml is another example :alaskanemily raised in a layout team standup meeting.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

I am looking into what next steps could be taken here, and I see that we report failures like this:
TEST-UNEXPECTED-FAIL | gfx/layers/apz/test/mochitest/test_group_hittest-1.html | helper_hittest_basic.html | active scrollframe - apzaware block hit info - got "VISIBLE", expected "VISIBLE | APZ_AWARE_LISTENERS"

so if the test is outputting this, we already have the subtest embedded in the failure message. Given how we file single tracking bugs, we count on a format of <ERROR-CODE> | TEST-NAME | ERROR-MESSAGE/ETC.

in the case of bug 1776971, the title is Intermittent gfx/layers/apz/test/mochitest/test_group_hittest-1.html | single tracking bug. We need the TEST-NAME to be a file path so we can correlate it to failures, manifests, etc. Possibly we can adjust the failure path to be:
<ERROR-CODE> | TEST-FILENAME :: SUB-FILENAME | ERROR-MESSAGE/ETC..

In general if you are working on investigating an intermittent test failure, won't you be running the test to reproduce the failure locally/try and see the other failure modes as well?

There is a feature built into treeherder intermittent failures where when viewing the failure:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?startday=2023-01-08&endday=2023-02-07&tree=all&failurehash=all&bug=1776971

there are filter text boxes under the header, and for the Log column it defaults to All, click that and a drop down of all the different failure types show up- this gives an opportunity to see ALL failures as a big picture, but drill into a specific failure message.

Comment 3

[Hiroyuki Ikezoe (:hiro)]

Thanks for the tip! I didn't know we can filter them.