808647 - Disable failing sanity tests for B2G reftest

Status: RESOLVED FIXED

(Testing :: Reftest, defect)

Comment 1

[Jonathan Griffin (:jgriffin)]

Attachment: Disable failing sanity reftests for B2G,

Comment 2

[Jonathan Griffin (:jgriffin)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/1e3b0555498f

Comment 3

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/1e3b0555498f

Comment 4

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Why are these skip-if rather than fails-if?

Comment 5

[Andrew Halberstadt [:ahal]]

(In reply to David Baron [:dbaron] from comment #4)
> Why are these skip-if rather than fails-if?

Tests run about 10 times slower on the emulator, and there are currently thousands of tests that fail/are random oranges for various reasons (yes there are some problems either with B2G or the automation that needs to be solved). In the meantime I don't want to have thousands of failing/random tests taking up test slave time while these issues are being fixed.

Comment 6

[Andrew Halberstadt [:ahal]]

This patch is only 5 tests, but when the full set of passing tests get enabled there will be a lot more skip-if's than 5

Comment 7

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

I think skip-if was really a misnomer; I should perhaps have called it crashes-or-hangs-if.  Marking tests as skipped means they're probably never going to be revisited, even if what's causing them to fail gets fixed.

Comment 8

[Jonathan Griffin (:jgriffin)]

Pushed to aurora so we can turn reftests on there:  https://hg.mozilla.org/releases/mozilla-aurora/rev/df349536978c

Comment 9

[Andrew Halberstadt [:ahal]]

(In reply to David Baron [:dbaron] from comment #7)
> I think skip-if was really a misnomer; I should perhaps have called it
> crashes-or-hangs-if.  Marking tests as skipped means they're probably never
> going to be revisited, even if what's causing them to fail gets fixed.

I don't know whether the tests I've triaged are random oranges or perma oranges. So I can only mark tests as skip-if or random-if.

I filed bug 773482 awhile ago (the tests that got disabled in this patch are covered in some of 773482's dependencies). I will be working to re-enable the tests that get skipped in the initial triage. When it gets to the point where we have a more manageable set of failing tests I'll get a patch to re-enable everything and use a project branch to re-triage the failures properly.