1221903 - Have a way to determine if all necessary jobs have been run for a changeset

Status: RESOLVED DUPLICATE of bug 1377528

(Tree Management :: Treeherder, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The problem as described here came up a couple of days ago in one of our team meetings and it looks like there is no solution existent yet. Cameron suggested that we file this as a bug to get at least a discussion and/or planning started.

The current behavior of Treeherder is that it simply displays a list of recent changesets of a specified branch and all of its related run jobs including their status. Currently there is no way to easily identify if all neccessary jobs have been run. For example our Firefox UI Update jobs rely on a Pulse message which triggers the tests. If this message is not being sent no tests will be executed and finally no results will be present in Treeherder. If you do not exactly know which jobs have to be run you can miss that and be under the assumption that everything is green (or has known issues).

Given that we may not run every job for each changeset we might have to use a couple of changesets to check if specific jobs have been run or not. In case of a job hasn't been seen for a while the tool might be able to insert a fake job entry and mark it with a new state `missed`?

I'm cc'ing some sheriffs who might also have some good input.

Comment 1

[Jonathan Griffin (:jgriffin)]

This is really tricky, since the number of jobs that theoretically can run changes over time, and buildbot does not provide a good API for querying this. I wonder if this will be easier in TaskCluster?

Comment 2

[Ed Morley [:emorley]]

So this is a subset of bug 1052397 (ETAs to completion) & I guess also related to bug 1194830 (trying to figure out all possible jobs in order to provide a UI for requesting additional jobs eg on a try run).

The problem is that we don't have a full graph of what is supposed to run. Bug 1194830 tries to do that - but in a buildbot-specific way.

That said - what's the main use case here? From the example given in comment 0, it sounds like at least one of the use cases is "check that at least some jobs from submitter X were scheduled on each push, so that total breakage can be spotted". Solving that would be much easier than trying to track the graph of all jobs run.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

So in our case it would be the jobs inside the Ff and Fu groups. Right now those are always the same jobs so it might be easy to check automatically. But in the future we will have a rotation script for locales, so that we test at least each locale once a week. So comparing each job for changesets might not be good, but we should have at least both groups present. This was not the case lately when TaskCluster no longer pushed the funsize update messages to pulse.

Comment 4

[Ed Morley [:emorley]]

The problem is at the moment Treeherder has no concept of which job names belong to which submitter, so we'd pretty much have to hardcode these.

In addition, whilst "always the same job types" sounds easy, is that on every platform, only some of them? On all repos (including try or twig repos) or only some of them?

We can probably come up with a solution, but it may (not unlike the autoclassification work recently) not be as straightforwards as it first appears :-)

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

We don't need that right now. It was just something we noticed lately. So totally no hurry, but thanks for offering an interim solution.