1862116 - wpt errorsummary file shows many groups with ERROR, but none are shown in treeherder

Status: RESOLVED FIXED

(Testing :: web-platform-tests, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

looking at central:
https://treeherder.mozilla.org/jobs?repo=mozilla-central&selectedTaskRun=ERcOWB7KR0Cvm3m1qRhVuA.0&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunning%2Cpending%2Crunnable&searchStr=wpt2%2Clinux

the errorsummary log file has so many groups with status ERROR:
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/ERcOWB7KR0Cvm3m1qRhVuA/runs/0/artifacts/public/test_info/wpt_errorsummary.log

but treeherder only shows 1 failure:
TEST-UNEXPECTED-OK | /navigation-api/per-entry-events/dispose-for-full-session-history.tentative.html | expected ERROR

and in the log file it indicates only 1 failure:

[task 2023-10-30T19:04:40.033Z] 19:04:40     INFO - SUITE-END | took 959s
[task 2023-10-30T19:04:40.273Z] 19:04:40     INFO - Got 1 unexpected results, with 1 unexpected passes
[task 2023-10-30T19:04:40.275Z] 19:04:40     INFO - wptserve Stopped WebTransport over HTTP/3 server on 127.0.0.1:11000
[task 2023-10-30T19:04:40.349Z] 19:04:40     INFO - wptserve Close on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8889))
[task 2023-10-30T19:04:40.555Z] 19:04:40     INFO - wptserve Stopped http server on 127.0.0.1:9000
[task 2023-10-30T19:04:40.563Z] 19:04:40     INFO - wptserve Stopped http server on 127.0.0.1:8001
[task 2023-10-30T19:04:40.574Z] 19:04:40     INFO - wptserve Stopped http server on 127.0.0.1:8002
[task 2023-10-30T19:04:40.574Z] 19:04:40     INFO - wptserve Stopped http server on 127.0.0.1:8444
[task 2023-10-30T19:04:40.574Z] 19:04:40     INFO - wptserve Stopped http server on 127.0.0.1:8445
[task 2023-10-30T19:04:40.591Z] 19:04:40     INFO - wptserve Stopped http server on 127.0.0.1:8446
[task 2023-10-30T19:04:40.601Z] 19:04:40     INFO - wptserve Stopped http server on 127.0.0.1:8003
[task 2023-10-30T19:04:40.676Z] 19:04:40     INFO - wptserve Stopped http server on 127.0.0.1:8000
[task 2023-10-30T19:04:40.714Z] 19:04:40     INFO - wptserve Stopped http server on 127.0.0.1:8443
[task 2023-10-30T19:04:40.770Z] 19:04:40     INFO - wptserve Close on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8888))
[task 2023-10-30T19:04:40.841Z] 19:04:40     INFO - Removed font: Ahem.ttf
[task 2023-10-30T19:04:40.913Z] 19:04:40     INFO - Closing logging queue
[task 2023-10-30T19:04:40.913Z] 19:04:40     INFO - queue closed
[task 2023-10-30T19:04:42.258Z] 19:04:42     INFO - Return code: 1
[task 2023-10-30T19:04:42.259Z] 19:04:42  WARNING - Got 1 unexpected statuses

I am confusing why so many groups are status ERROR, only 1 group should have a status of ERROR.

:jgraham, is this intentional? If it isn't can you fix it?

Comment 1

[James Graham [:jgraham]]

It looks to me like a regression from Bug 1631828.

https://searchfox.org/mozilla-central/source/testing/mozbase/mozlog/mozlog/formatters/errorsummary.py#59 only sets the group status to OK if the test has an expected status of either OK or PASS. If there's an expected FAIL or similar we end up setting the group status to ERROR.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

thanks for pointing that out, I see a way to fix this. I have a lot of edge cases to test. unittests are great, but what gets posted from many jobs ends up being different edge cases.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1862116 - adjust group status info for different status vs expected output. r=marco!

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

the problem here is that we have a status that is:
status: FAIL

but we are not passing in expected, so it is not logical to assume group status is OK given that a test start is FAIL.

The logger sees:

action: test_status, status: FAIL
action: test_end, status: OK

what the logger should see is:

action: test_status, status: FAIL, expected: FAIL
action: test_end, status: OK

:jgraham can you fix the wpt harness to send expected with test_status?

Comment 5

[James Graham [:jgraham]]

Unless I'm missing something it does that already: https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wptrunner/wptrunner/testrunner.py#715-722 (and if it didn't absolutely everything would be broken. Expected subtest failures is very much normal in wpt).

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1862116 - Adjust errorsummary group status to work with mozlog rules for status and expected. r=marco!

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f1fb5f0afb58
Adjust errorsummary group status to work with mozlog rules for status and expected. r=marco

Comment 8

[Cristina Horotan [:chorotan]]

https://hg.mozilla.org/mozilla-central/rev/f1fb5f0afb58