799728 - testing strategy for talos

Status: RESOLVED FIXED

(Testing :: Talos, defect)

Description

[Jeff Hammel]

We don't have a documented way of testing talos changes before
committing.  I suggest the following:

* we have a Testing section in Buildbot/Talos wiki
* we document the procedure, maybe have a few paths:
  1. test locally: what should you test locally?
  2a. Minimal try run: if the change looks minor, what should you push
      to try?
  2b. Full try run: if the change is major, what should you push to try?

* when reviewed, the reviewer should denote what tests should be run
  (some subset of (None, 1, 2a, 2b))

* [optional but nice] we check in a script somewhere that makes 2a and
  2b and conceivably 1 easy

This can't cover 100% of scenarios, but will probably cover most cases

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

https://wiki.mozilla.org/Buildbot/Talos#I_have_a_patch_for_Talos.2C_what_tests_to_I_run.3F

Comment 2

[Jeff Hammel]

Attachment: recommended changes to documented policy

I think we should probably give the try server syntax for each of the methods . Also, it would be nice to tell more what "test locally" means, otherwise it is completely open to interpretation.  I would like to start getting in the habit of having the reviewer call out what sort of local testing is necessary, perhaps with a (partial) command line or a YAML/JSON configuration file

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 670148 [details]
recommended changes to documented policy

I am glad to see these instructions in here with trychooser syntax.  I didn't specify mac because there is almost no difference between linux and mac in terms of what we run and what we do.  There is a python version difference between windows and everybody else.

Comment 4

[Jeff Hammel]

I've updated https://wiki.mozilla.org/Buildbot/Talos#I_have_a_patch_for_Talos.2C_what_tests_to_I_run.3F with the patch.  Please feel free to do future edits!

I would really like to add a script to make this a little more automatic. But....I can't think of what exactly that script would do for the time being, so I will hold off ticketing for now.

Should we close this bug?

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

lgtm, thanks for updating this.