1353715 - Group test jobs run against the same revision after a threshold is reached

Status: RESOLVED INCOMPLETE

(Tree Management :: Treeherder: Frontend, defect)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Some Firefox Test Engineering jobs run regularly against repositories that do not frequently change. As an example, snippets-tests [1] runs every 10 minutes against both production and staging environments, which will result in almost 300 jobs a day. There were just 3 commits to this repository in March, and since the last commit there would have been over 6,000 jobs submitted to Treeherder.

Based on my observations of the existing FxAPOM jobs [2], which run once a day, I suspect the Treeherder UI will be difficult to use with so many jobs per revision. I'm aware that it's possible to filter out the passing jobs, and that there is already a feature for grouping jobs. Does it make sense for us to use groups even if there's only ever going to be one job type in it, and would the existing grouping feature scale to thousands of jobs?

[1] https://github.com/mozilla/snippets-tests
[2] https://treeherder.allizom.org/#/jobs?repo=fxapom

Comment 1

[Ed Morley [:emorley]]

Treeherder being able to handle thousands of jobs per push is something we need for eg hyper-chunking etc too. There is job group collapsing already, but I suspect perhaps it only operates on jobs within a group, rather than those at the top level (ie don't have a group defined). Cameron wrote the implementation, so will be able to say for sure.

That said, Treeherder is very much designed to a system to map job results to pushes, and so is never going to satisfy the case of wanting to test the same revision across time particularly well. For minute-to-minute monitoring it sounds like https://newrelic.com/synthetics or similar might be a better bet?

Comment 2

[Ed Morley [:emorley]]

(In reply to Ed Morley [:emorley] from comment #1)
> That said, Treeherder is very much designed to a system to map job results
> to pushes, and so is never going to satisfy the case of wanting to test the
> same revision across time particularly well. For minute-to-minute monitoring
> it sounds like https://newrelic.com/synthetics or similar might be a better
> bet?

To be clearer:

I think there are two/three very different sets of testing that should be performed against a web app:
1) On commit CI tests using test data / mocks
2a) On deploy tests using real data
2b) Regular site monitoring using real data

Treeherder is great for managing the results for (1) and perhaps also (2a), however I think the data model/UI required for solving (2b) is quite different, and so would require significant alternations to Treeherder that would stretch its scope too wide. 2a/2b also have quite a number of other requirements such as paging notifications/escalations etc, that don't fit into typical CI.

However you are of course welcome to submit to Treeherder and we'll do the best we can to make it usable (so long as it doesn't scope creep it too much) :-)

Comment 3

[Cameron Dawson [:camd]]

(In reply to Ed Morley [:emorley] from comment #1)
> Treeherder being able to handle thousands of jobs per push is something we
> need for eg hyper-chunking etc too. There is job group collapsing already,
> but I suspect perhaps it only operates on jobs within a group, rather than
> those at the top level (ie don't have a group defined). Cameron wrote the
> implementation, so will be able to say for sure.
> 
> That said, Treeherder is very much designed to a system to map job results
> to pushes, and so is never going to satisfy the case of wanting to test the
> same revision across time particularly well. For minute-to-minute monitoring
> it sounds like https://newrelic.com/synthetics or similar might be a better
> bet?

Yes, within groups, we will aggregate them down to counts by result (pass/fail/etc).  So this would be the way to go.  There's also a button on the toolbar where you can have it show "Duplicates" without un-aggregating everything on the page, fwiw.  Not sure that's of any use, but mentioned it anyway.  :)

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Thanks for your feedback. I've raised https://github.com/mozilla/fxtest-jenkins-pipeline/issues/13 to default to grouping these results in a similar way to TaskCluster for now. I agree that it would make sense to limit the number of monitoring jobs submitting to Treeherder, so this is something we'll revisit.

Comment 5

[Ed Morley [:emorley]]

Marking incomplete since the use case here is not one we're actively supporting, and a related one for hyper-chunking may be implemented differently.