1283265 - Canceled tasks (eg. due to deadline exceeded) are not reported to Treeherder

Status: RESOLVED WORKSFORME

(Firefox Build System :: Task Configuration, task)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Today I wanted to check the test results of the in-tree firefox-ui-tests as executed via Taskcluster for the changeset used to create the Nightly build. But sadly our tests weren't run at all for all flavors on Linux64. The same is even true for any other kind of test executed via TC. And that for all platforms we currently support running tests via TC.

Here the appropriate changeset in Treeherder:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=e45890951ce77c3df05575bd54072b9f300d77b0&filter-searchStr=linux%20x64

While talking to Greg on IRC he mentioned that the underlying issue is that all test jobs were canceled due to `deadline-exceeded`. You can see that here:

https://tools.taskcluster.net/task-group-inspector/#VSHpeGM8Q1yEERDwb-pgBg/A0PbVx43QmmuWrR6gUeCDw/0

In case of at least this issue Taskcluster doesn't report ANY test state to Treeherder. Which means sheriffs and release drivers will not notice when particular tests have not been run. In this case it is obvious but keep in mind that only some tests could fail.

I assume the same could also happen on the beta and release branch. So I'm afraid that this could be a gateway to broken releases - in case the changeset we selected for the release brought in a regression, and no tests were run which could have indicated that problem.

As Greg also mentioned the current behavior of not reporting to Treeherder is something what sheriffs wanted to have. Maybe we have to think about that decision again and find a compromise.

Comment 1

[Greg Arndt [:garndt]]

Thinking about how we could report this but hide it from most people, it is possible for TC jobs to specify a tier level.  What if rather than not reporting it, we report it as tier 3 so only those with the filter enabled will see it.  Although that does overload the meaning of the tiers.

Comment 2

[Chris Cooper [:coop] (he/him)]

Found in triage.

There are two different relevant cases where a task could exceed a deadline:

1. Task is scheduled -> enters pending state -> exceeds deadline
2. Task is scheduled -> exceeds deadline

#1 is the one I think we actually care about, because a pending task is expected to eventually run, whereas there are cases where #2 will never enter the pending state, e.g. test jobs won't run for a build that fails.

We would need Treeherder to distinguish between those two cases in order to display the one we care about.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Brian, is this something we could fix in tc-treeherder?

Comment 4

[Brian Stack [:bstack]]

iirc it would take a bit of collaboration with th itself, but I think we could.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

This might be something that we can improve while looking at streamlining the handling of pulse inputs to Treeherder (and perhaps eliminating tc-treeherder in the process).

Comment 6

[Andrew Halberstadt [:ahal]]

Pretty sure this works now.