1822129 - when treeherder detects a failure, consider and run if possible test-verify mode

Status: NEW

(Tree Management :: Treeherder, task)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

in bug 1822127 an action task will be created for running test-verify as needed. There will be some edge cases where test-verify mode will not be possible to run.

This bug will:

• create a way when a test fails to run the action task given the failing task as input (this might require some authentication) - or maybe we determine this isn't for treeherder
• determine a set of criteria that would indicate we cannot run verify mode:
  • infra failure ? (maybe we do run)
  • this failing task is a verify task
  • limited hardware, might take a few hours in the queue or flood the queue
  • missing errorsummary.json (or no failures in there at the test level)
  • other criteria TBD
• handle multiple failures in a single errorsummary.json
• handle multiple errorsummary.json files for a given task

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

currently we are not planning on treeherder executing the task, right now the path is to use mozci- this could change as the plumbing we have in taskgraph and tooling will support various models.

a few models to consider:

1. when a task fails, it calls taskgraph to trigger the -cf task. This would require the ability to determine if the failure is NEW in order to reduce load/frequency of -cf tasks
2. mozci runs on a cron job, this can analyze and use other logic to determine if we need to retrigger/confirm-failure or ignore
3. have treeherder detect things and trigger an action task

currently we are going with #2.

There is still a need to fix the NEW annotations to be more persistent and given that we could end up with #1 having full support if #2 isn't as reliable as we would like.