995118 - Devise a standard way for mozharness to pass down a "your timeout should be less than mine" to test harnesses

Status: RESOLVED WONTFIX

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Ed Morley [:emorley]]

Broken out from bug 994830:

(In reply to Ted Mielczarek [:ted.mielczarek] from bug 994830 comment #7)
> Comment on attachment 8404919 [details] [diff] [review]
> Change individual test timeout from 1200 to 900 seconds
>
> Oops. We should figure out some standard way for mozharness to pass down
> "your timeout setting should be less than mine" to test harnesses.

We've also had this happen in several other bugs; it would be good if we had a way to avoid this footgun :-)

Comment 1

[Aki Sasaki (not active)]

I think this is also easily doable.  :jgriffin mentioned we should have a global timeout, which I agree with, though I don't know how much fallout there will be.

I'm currently thinking:

--max-output-timeout TIMEOUT_SECONDS

If max_output_timeout is not None, run_command() and get_output_from_command() would take the smaller number from the max_output_timeout and output_timeout (which could also be None) values.  Setting max_output_timeout would have the side effect of adding output timeouts to every run_command() and get_output_from_command() that doesn't currently have one; this ideally has zero negative side effects, but in reality it may burn a number of jobs.

We could then set a default-but-overridable max_output_timeout... either at the global level or at, say, the script or TestingMixin level.

a) At the script level, we know exactly what we're affecting, but this involves a lot of edits and new scripts don't automatically get this.

b) At the TestingMixin level, we would be affecting only test scripts.  We would likely need to change TestingMixin to TestingScript due to the inability of mixins to override already-existing methods like __init__().  We could set the best max_output_timeout for most scripts, and override via commandline or at script-level for any scripts that need longer |max_output_timeout|s.

c) At the global level, we would affect everything, but this is the largest amount of unknown fallout.  This also would affect non-buildbot non-tbpl scripts like vcs-sync.

Currently leaning towards (b).