Today I wanted to check the test results of the in-tree firefox-ui-tests as executed via Taskcluster for the changeset used to create the Nightly build. But sadly our tests weren't run at all for all flavors on Linux64. The same is even true for any other kind of test executed via TC. And that for all platforms we currently support running tests via TC. Here the appropriate changeset in Treeherder: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=e45890951ce77c3df05575bd54072b9f300d77b0&filter-searchStr=linux%20x64 While talking to Greg on IRC he mentioned that the underlying issue is that all test jobs were canceled due to `deadline-exceeded`. You can see that here: https://tools.taskcluster.net/task-group-inspector/#VSHpeGM8Q1yEERDwb-pgBg/A0PbVx43QmmuWrR6gUeCDw/0 In case of at least this issue Taskcluster doesn't report ANY test state to Treeherder. Which means sheriffs and release drivers will not notice when particular tests have not been run. In this case it is obvious but keep in mind that only some tests could fail. I assume the same could also happen on the beta and release branch. So I'm afraid that this could be a gateway to broken releases - in case the changeset we selected for the release brought in a regression, and no tests were run which could have indicated that problem. As Greg also mentioned the current behavior of not reporting to Treeherder is something what sheriffs wanted to have. Maybe we have to think about that decision again and find a compromise.
Thinking about how we could report this but hide it from most people, it is possible for TC jobs to specify a tier level. What if rather than not reporting it, we report it as tier 3 so only those with the filter enabled will see it. Although that does overload the meaning of the tiers.