1405023 - Add a way to push a test to Try and keep retriggering it until it fails

Status: NEW

(Developer Infrastructure :: Try, enhancement, P3)

Description

[Botond Ballo [:botond]]

The current workflow for debugging intermittent test failures, which involves pushing a test to Try with added logging or an attempt at a fix, and running it until the failure reproduces or you're convinced that it's fixed, could use some improvements.

Two specific pain points are:

  - Figuring out which test chunk currently contains the 
    test in question

  - Having to manually retrigger the chunk until the test
    in question fails

It seems to me like these tasks are good candidates for being automated away.

What I'd like to see ultimately is a command along the following lines:

  $ mach run-on-try-until-failure <test-name> <platform-name>

This command would:

  - Figure out which test chunk contains <test-name>

  - Push the chunk in question to try on the indicated platforms

      - For selecting the platform, instead of specifying the
        platform string on the command line, it could also pop
        you into a fuzzy platform-selector UI.

  - Instruct taskcluster to keep retriggering the chunk until
    <test-name> fails, or some maximum limit (that's preferably
    configurable) has been reached.

I know this might be asking for a lot, but I think something like this would make the process for debugging and fixing intermittent failures much more manageable.

Comment 1

[Geoff Brown [:gbrown]]

test-verify -- landing today, coincidentally? -- provides some of this functionality.

By tomorrow, you could use test-verify on try by:
 - modifying the test of interest, and;
 - pushing to try with -p linux64 -u test-verify-e10s.

Bug 1400895 hopes to make the push to try process easier.

test-verify only runs on linux64 currently, but I hope to get other platforms up soon.

test-verify only runs one, or perhaps a few, test(s) -- not the whole chunk/directory/manifest. There's probably some incremental value in those alternatives.