1890026 - Add Fenix/Focus UI tests to Push Health Tests

Status: RESOLVED WONTFIX

(Tree Management :: Push Health, defect, P2)

Description

[Aaron Train [:aaronmt]]

Currently, e.g, Try run

Push Health does not recognize failures coming in from ui-test-apk and android-startup-test under /taskcluster/ci/ and is currently reporting them as success.

(Please move this bug category if need be)

Comment 1

[Johan Lorenzo [:jlorenzo]]

UI Tests used to be visible on Github Pull Requests and would put an explicit ❌ if they failed. Let's get this issue resolved before we finish the Android repo migration. Marking ticket as P2.

:jmaher, what would be needed to take these jobs into account in the push health metric? Is it something we can change in-tree?

Comment 2

[Aaron Train [:aaronmt]]

Aside, if it's the case where because ui-test-apk tasks are Tier 2 that would be a seperate discussion

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Push Health is not accurate and not maintained. I would not trust it. Our focus has been on upgrading and operational performance with treeherder, so we have been putting off fixing or killing push health for the last 1.5 years.

I believe push health does a variety of queries and might use certain patterns on job names or some other metadata to determine test vs build vs lint.

Comment 4

[Joe M [:jmahon]]

Aaron, could you set a severity rating for this bug, please?

Comment 5

[Polly [:polly]]

from comment 3, it sounds like this is not worth pursuing, so i am closing this.
If i have misinterpreted this, please reach out and let me know!