Extremely frequent Android ARMv6 Reftest-2 TEST-UNEXPECTED-FAIL | <some test> | application ran for longer than allowed maximum time

RESOLVED WORKSFORME

Status

()

defect
--
critical
RESOLVED WORKSFORME
6 years ago
6 years ago

People

(Reporter: RyanVM, Unassigned)

Tracking

({intermittent-failure})

Trunk
ARM
Android
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

6 years ago
The landing of bug 716859 has led to Android ARMv6 Reftest-2 failing with roughly 80% frequency due to running for longer than the harness permits. This suite contains the WebGL tests among other things.

From my looking into this, it appears that the suite has become bimodal with the passing runs typically finishing in around 60 minutes (normal for this suite) and the orange runs being killed around 80 minutes. There are occasionally green runs that finish just short of 80 minutes.

There are a few possible options for where we go from here:
1.) Change the harness to work around this. We can increase the timeout threshold or change the chunking.
2.) Back out bug 716859.
3.) Disable the webgl reftests on Android (not sure if we can disable on just ARMv6).

I would prefer option 2 as it's an obvious regression from the code that landed in that bug. However, Jeff and Vlad have voiced their extreme displeasure with the idea of backing this out (again).

I'm personally opposed to making changes to the test harness to work around this as it seems like we're papering over a real regression and it removes any real incentive to investigate and fix whatever regressed.

Therefore, I'm proposing we go with Option 3 for now until the cause of the regression can be located and fixed.
(Reporter)

Updated

6 years ago
Depends on: 843733
(Reporter)

Updated

6 years ago
Depends on: 843728
No longer depends on: 843733
(Reporter)

Comment 1

6 years ago
In bug 843728, jmaher proposed landing the temporary increase for now with a 2-26 backout deadline.
I'm strongly against increasing the timeout.
I've relanded bug 716859 with webgl disabled on ARMv6
Resolving WFM keyword:intermittent-failure bugs last modified >3 months ago, whose whiteboard contains none of:
{random,disabled,marked,fuzzy,todo,fails,failing,annotated,time-bomb,leave open}

There will inevitably be some false positives; for that (and the bugspam) I apologise. Filter on orangewfm.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.