843738 - Extremely frequent Android ARMv6 Reftest-2 TEST-UNEXPECTED-FAIL | <some test> | application ran for longer than allowed maximum time

Status: RESOLVED WORKSFORME

(Core :: Graphics: CanvasWebGL, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

The landing of bug 716859 has led to Android ARMv6 Reftest-2 failing with roughly 80% frequency due to running for longer than the harness permits. This suite contains the WebGL tests among other things.

From my looking into this, it appears that the suite has become bimodal with the passing runs typically finishing in around 60 minutes (normal for this suite) and the orange runs being killed around 80 minutes. There are occasionally green runs that finish just short of 80 minutes.

There are a few possible options for where we go from here:
1.) Change the harness to work around this. We can increase the timeout threshold or change the chunking.
2.) Back out bug 716859.
3.) Disable the webgl reftests on Android (not sure if we can disable on just ARMv6).

I would prefer option 2 as it's an obvious regression from the code that landed in that bug. However, Jeff and Vlad have voiced their extreme displeasure with the idea of backing this out (again).

I'm personally opposed to making changes to the test harness to work around this as it seems like we're papering over a real regression and it removes any real incentive to investigate and fix whatever regressed.

Therefore, I'm proposing we go with Option 3 for now until the cause of the regression can be located and fixed.

Comment 1

[Ryan VanderMeulen [:RyanVM]]

In bug 843728, jmaher proposed landing the temporary increase for now with a 2-26 backout deadline.

Comment 2

[Ed Morley [:emorley]]

I'm strongly against increasing the timeout.

Comment 3

[Jeff Muizelaar [:jrmuizel]]

I've relanded bug 716859 with webgl disabled on ARMv6

Comment 4

[Ed Morley [:emorley]]

Resolving WFM keyword:intermittent-failure bugs last modified >3 months ago, whose whiteboard contains none of:
{random,disabled,marked,fuzzy,todo,fails,failing,annotated,time-bomb,leave open}

There will inevitably be some false positives; for that (and the bugspam) I apologise. Filter on orangewfm.