Closed Bug 808647 Opened 9 years ago Closed 9 years ago

Disable failing sanity tests for B2G reftest

Categories

(Testing :: Reftest, defect)

defect
Not set
normal

Tracking

(firefox18 fixed, firefox19 fixed)

RESOLVED FIXED
mozilla19
Tracking Status
firefox18 --- fixed
firefox19 --- fixed

People

(Reporter: jgriffin, Assigned: jgriffin)

References

Details

Attachments

(1 file)

No description provided.
https://hg.mozilla.org/mozilla-central/rev/1e3b0555498f
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla19
Why are these skip-if rather than fails-if?
(In reply to David Baron [:dbaron] from comment #4)
> Why are these skip-if rather than fails-if?

Tests run about 10 times slower on the emulator, and there are currently thousands of tests that fail/are random oranges for various reasons (yes there are some problems either with B2G or the automation that needs to be solved). In the meantime I don't want to have thousands of failing/random tests taking up test slave time while these issues are being fixed.
This patch is only 5 tests, but when the full set of passing tests get enabled there will be a lot more skip-if's than 5
I think skip-if was really a misnomer; I should perhaps have called it crashes-or-hangs-if.  Marking tests as skipped means they're probably never going to be revisited, even if what's causing them to fail gets fixed.
Pushed to aurora so we can turn reftests on there:  https://hg.mozilla.org/releases/mozilla-aurora/rev/df349536978c
(In reply to David Baron [:dbaron] from comment #7)
> I think skip-if was really a misnomer; I should perhaps have called it
> crashes-or-hangs-if.  Marking tests as skipped means they're probably never
> going to be revisited, even if what's causing them to fail gets fixed.

I don't know whether the tests I've triaged are random oranges or perma oranges. So I can only mark tests as skip-if or random-if.

I filed bug 773482 awhile ago (the tests that got disabled in this patch are covered in some of 773482's dependencies). I will be working to re-enable the tests that get skipped in the initial triage. When it gets to the point where we have a more manageable set of failing tests I'll get a patch to re-enable everything and use a project branch to re-triage the failures properly.
You need to log in before you can comment on or make changes to this bug.