1172434 - docker-worker: generic-worker: Custom task results meta data

Status: RESOLVED WONTFIX

(Taskcluster :: General, defect)

Description

[Pete Moore [:pmoore][:pete]]

The result of a task is failure/error/pass.

It occurs to me that often there is other meta data that would be highly useful to capture. A very good example of this is that if we run a test suite, it would be useful to know: a) which tests ran, b) what the result of each test was.

Currently I believe the way we discover which test(s) failed is to browse the log file(s) by hand.

I think it would be useful if we had a generic mechanism to post the overall result together with custom (task-specific) meta data.

In this case, when a test suite runs, we could post a dictionary of results (maybe a simple artifact) that would allow downstream services (such as treeherder) to track test results across branches. A good use for this might be for treeherder to present a matrix view of test results across changesets (at least for non-passing tests). Other than visual benefits, it also allows for tools to track metrics on tests, discover patterns, discover bad tests, predict code quality, etc.

Tests are the main use case I see for custom result metadata, but there are other use cases, such as capturing custom task-specific performance metrics, defined at task level.

Currently tasks can post custom metrics as artifacts, but I'm not sure what guarantees there are that these get uploaded in the case of e.g. tests failing. Maybe we need to introduce something like a post-run meta data hook that allows a custom command to be run to generate the meta data artifact after the main task has completed? Agreed that could be manually achieved in the current state by having a task that runs the given job, and then runs the meta data generation as a second step - so maybe also not required. I guess even if all of this is currently achievable, what would be nice is a best-practice design of how to implement such a thing consistently. My gut feeling though is that if we had some design around the concept we'd be likely to find gains over such a thing being implemented arbitrarily and inconsistently with existing design.

So this bug really is a call for people's thoughts on whether a concept of task meta data appeals and might be useful. For example do we agree capturing granular test results in a structured format might be useful? Is there other data that we typically parse from logs, which would be useful to report in a structured way? Sheriffs are probably most familiar with the types of information they scrape from log files...

Comment 1

[Pete Moore [:pmoore][:pete]]

Maybe this is a load of nonsense, and simple publishing of a test results artifact is enough, e.g. with a dictionary of test name to test result. It would of course require a convention, but maybe it is better to engineer this outside of taskcluster, and leave taskcluster to manage high level job execution... I can't decide for myself whether this is a good idea or not.

Comment 2

[Pete Moore [:pmoore][:pete]]

The more I think about it, the more I think this doesn't really offer a lot of value. I'm killing it!

Comment 3

[Pete Moore [:pmoore][:pete]]

(basically since any task can publish meta data as a simple artifact already)

Comment 4

[Jonas Finnemann Jensen (:jonasfj)]

Yeah, I think it should be in the test harness to publish a small artifact that TreeHerder can consume,
or that our TreeHerder integration can consume.

Comment 5

[Pete Moore [:pmoore][:pete]]

Resetting Version and Target Milestone that accidentally got changed...