1291234 - Large number of low-frequency intermittent timeouts

Status: RESOLVED WORKSFORME

Description

[James Graham [:jgraham] (away until 2018-07-01)]

Starting sometime around 20/7-25/7 web-platform-tests started experiencing a large number of new low-frequency intermittent timeouts. There are no obvious wpt changes in that time period [1], but I do notice that it seems to correspond to dustin's configuration changes for taskcluster, although not every new intermittent occurs on TC, and I can't obviously see any relevant configuration changes.

The only plausible-looking marionette change I can see that might affect things is [2], but it's hard to see how because I believe that code is only called from the python navigate() method, which we only use to load the initial harness page, which isn't being reloaded before the failing tests.

So I am back to wondering if something could have changed in the test environment that might affect the worst-case performance of the tests. Otherwise I suppose a gecko change might have had some effect…

[1] http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?startdate=2016-07-19&enddate=2016-07-25
[2] http://hg.mozilla.org/integration/mozilla-inbound/rev/b2e910bb1d72

Comment 1

[Dustin J. Mitchell [:dustin] pronoun: he]

The configuration changes that I made were tested to ensure that they produced the same task description (modulo things like treeherder attributes which wouldn't affect pass/fail status).  That the failures are occurring on Buildbot as well is also strong evidence these changes are unrelated.

The TC test environment is also defined in-tree in testing/docker/desktop-test, so you can look for changes there.  But again, Buildbot is completely different so given that the failures span TC and BB, I doubt the test environment is related.

Comment 2

[OrangeFactor Robot]

6 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 4
* mozilla-inbound: 1
* fx-team: 1

Platform breakdown:
* linux64: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1291234&startday=2016-08-15&endday=2016-08-21&tree=all

Comment 3

[Jon Coppeard (:jonco)]

This should hopefully be fixed by bug 1288579.

Comment 4

[Jon Coppeard (:jonco)]

Closing given bug 1295840 comment 6.