1394418 - Add code pointer(s) + wiki page to encourage lots of testing before adding new test jobs / platforms

Status: RESOLVED FIXED

(Testing :: General, enhancement)

Description

[Geoff Brown [:gbrown]]

I am sometimes disappointed to see new tier 1/2 test jobs -- or whole new test platforms -- added to taskcluster without try pushes, or without significant retries and without commentary on intermittent failure rates.

I'd like to add a comment to tests.yml (or similar) saying something like:

# Before adding new tier 1/tier 2 tests, please run through the checklist at ...

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

ideally if a push changes taskcluster to have new jobs we should warn people and point them here :)

Comment 2

[Geoff Brown [:gbrown]]

(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #1)
> ideally if a push changes taskcluster to have new jobs we should warn people
> and point them here :)

That sounds good, but I don't know how to automate that.

Comment 3

[Geoff Brown [:gbrown]]

https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy might be a good resource to point to, although it doesn't directly address the issue of adding a platform without sufficient testing.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

:ahal- would you have ideas on how to determine if a changeset adds new jobs and we could find a way to turn the decision task orange, etc.?

Comment 5

[Andrew Halberstadt [:ahal]]

I don't think there's any easy way to do this :/. You'd need to compare the output from the previous push's decision task. This will likely be hard to get right. Plus if it slowed the decision task down by more than ~1-2 seconds, I'd be opposed to it.

Comment 6

[Geoff Brown [:gbrown]]

Attachment: add comment to tests.yml

Worth a try? I don't know how many people will notice this, but at least it's *something*.

Change suggestions/edits of the wiki page are also welcome!

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8902921 [details] [diff] [review]
add comment to tests.yml

Review of attachment 8902921 [details] [diff] [review]:
-----------------------------------------------------------------

nice and simple

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/359ddcdaff63
Add reminder of job visibility policy to tests.yml; r=jmaher

Comment 9

[Ryan VanderMeulen [:RyanVM]]

Possibly worth considering adding a commit hook that requires review from a certain list of peers if tests.yml is touched?

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

are there examples of how to add a commit hook?

Comment 11

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/hgcustom/version-control-tools/file/default/hghooks/mozhghooks/

Look at some of the "prevent" ones to get an idea of what I'm proposing. Something similar to prevent_webidl_changes.py.

Comment 12

[Geoff Brown [:gbrown]]

Thanks! Filed bug 1395696 for the commit hook idea.

Comment 13

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/359ddcdaff63