1301922 - Windows 7 opt/PGO mochitest-webgl-2 rarely even runs to completion

Status: RESOLVED DUPLICATE of bug 1297965

(Core :: Graphics: CanvasWebGL, defect)

Description

[Phil Ringnalda (:philor)]

Of course the failure rate is massively over what's acceptable according to the Visibility Policy, but even worse than that is the fact that it only rarely even manages to run all the tests: the mochitest harness treats four test timeouts as a sign that you've lost and kills the rest of the run, and between that and the driver crashes and the 330 seconds without output hangs, of the last 20 opt runs on mozilla-central only 12 managed to run all the tests, and while PGO (which includes nightly builds) was a bit better at 14 of 20 running all the tests, hardly anybody on try runs PGO. So if someone's patch causes bustage in a test toward the end of that chunk, even with the automatic retrigger of a failing first run they still have pretty fair odds of getting two runs which will just have known intermittents, without even running the test they cause to fail.

Comment 1

[Phil Ringnalda (:philor)]

Lest I forget this yet again and go chasing after the cause of these failures: the cause is either just running them in three chunks instead of just one, or is something completely hidden by the fact that for a while after switching to three chunks we ran three chunks which ran exactly zero tests each, so we didn't have anything to show us if a push broke them.

Comment 2

[Phil Ringnalda (:philor)]

Actually, the 330 seconds without output hangs appear to be part of a different pattern of failure: doesn't help that they exist, but all the ones I looked at, while willing to happen on Win7 opt, don't restrict themselves to just Win7 opt.

Comment 3

[Phil Ringnalda (:philor)]

Oh, and not to diminish the whole Visibility Policy/overall failure rate thing, PGO and opt were each actually green just 4 of their last 20 runs, so with an 80% failure rate their days of running visible are numbered, and not numbered very high.

Comment 4

[Phil Ringnalda (:philor)]

1. To my surprise, just disabling those seven tests gets it pure green - https://treeherder.mozilla.org/#/jobs?repo=try&revision=4c09f6a72e93&group_state=expanded. That's just a quick and brutal disable, not for checkin, obviously, but you can quite easily disable your way out of my intention to hide it.

2. JFC, do we really have to trip over our clownshoes while walking toward our other clownshoes to put on our clownshoes not realizing that we're already wearing a third pair of clownshoes? My hypothetical Try user who would get two runs that didn't run the test he broke doesn't exist because WE DON'T RUN IT ON TRY. I had to manually add those with the treeherder Add new Jobs UI, because no matter what you ask for in the way of tests, you will not get Win7 mochitest-gl.

Comment 5

[Phil Ringnalda (:philor)]

(Workaround: -u all[Windows 7] and thus I presume -u mochitest-gl[Windows 7] will schedule the jobs that not having the platform restriction does not schedule.)

Comment 6

[Phil Ringnalda (:philor)]

Well played, looks like what I meant was "hey, we should probably use ANGLE for WebGL2!"