1302650 - Investigate stretched timeouts based on type of build or cpu load

Status: NEW

(Testing :: Marionette Client and Harness, defect, P5)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

It feels awkward to have to set high timeouts for specific Wait.until() calls because only debug and asan builds will require those to not fail tests. The downside is that for opt builds we loose valuable time while running our tests.

Cannot we do it better by using a somewhat stretched timeout based on the build type or even the mean value of the recent cpu load? A change like that should not affect users of Selenium because I think they will always work with opt builds, where we shouldn't make modifications. But for our custom builds in CI, we could easily do it.

Proposed steps:

1. Figure out the build type based on GetBuildConfiguration():

https://dxr.mozilla.org/mozilla-central/source/js/src/builtin/TestingFunctions.cpp?q=getBuildConfiguration&redirect_type=direct#89

We could return the build type as a capability to the client:

> var { debug, release, asan } = Components.utils.getJSTestingFunctions().getBuildConfiguration()

With that we can differentiate between release, nightly, opt, debug, asan, and even others if necessary.

2. Update the client code to use a varying stretch value based on the build type. Here just an example:

release, opt = 1x
debug = 2x
asan = 4x

3. (Optional and may be harder) Analyze the cpu load and calculate a current stretch value. It will ensure to always use the best as possible timeout independently from the build type, which would also include bad behavior of other applications running on the same host taken away valuable cpu time.

What do you think about such a proposal?

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

this seems reasonable- would this be in a main library, or per test?  It does seem  that small changes to the browser or environment seem to affect timing often- this might reduce the pain.

Comment 2

[Ryan VanderMeulen [:RyanVM]]

As mentioned on IRC yesterday, "release" is useless to you here and you should avoid using it unless you want to end up very confused. In this context, release corresponds to the RELEASE_BUILD ifdef, which is defined based on the *version number* in the tree, not opt/debug status. So unless you want to detect whether you're on Beta/Release, avoid using it.