1822172 - document a rollout plan for automatic test-verify

Status: NEW

(Testing :: General, task)

Description

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

once we have new test-verify working for most tests and treeherder automatically starting it and using the results for value add, we need a plan for rolling this out.

Considerations:

• try vs autoland vs other release branches vs reviewbot
• limited platforms / configs
• limited harnesses
• limited users

The considerations should be for reducing confusion, to prove out metrics are doing what we think (i.e. providing value and not doubling cost and end to end time).

This rollout plan could be 2 or more stages, either alpha/beta/release, or some numbered phase.

Please file bugs for the series of stages and what we expect to be a result (# tasks, # bugs, metric X==Y, etc.)

We can roll out each phase and this can be interleaved with fixing edge cases, etc.