861024 - fix or disable perma-orange B2G test suites on mozilla-central

Status: RESOLVED DUPLICATE of bug 853024

(Testing :: General, defect)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

per discussion in RelEng, we think the following test suites are perma-orange and perma-hidden on mozilla-central. 


b2g_ics_armv7a_gecko_emulator mozilla-central opt test crashtest-1
b2g_ics_armv7a_gecko_emulator mozilla-central opt test crashtest-2
b2g_ics_armv7a_gecko_emulator mozilla-central opt test crashtest-3
b2g_ics_armv7a_gecko_emulator mozilla-central opt test reftest-1
b2g_ics_armv7a_gecko_emulator mozilla-central opt test reftest-2
b2g_ics_armv7a_gecko_emulator mozilla-central opt test reftest-3
b2g_ics_armv7a_gecko_emulator mozilla-central opt test reftest-4
b2g_ics_armv7a_gecko_emulator mozilla-central opt test reftest-5
b2g_ics_armv7a_gecko_emulator mozilla-central opt test reftest-6
b2g_ics_armv7a_gecko_emulator mozilla-central opt test reftest-7
b2g_ics_armv7a_gecko_emulator mozilla-central opt test reftest-8
b2g_ics_armv7a_gecko_emulator mozilla-central opt test reftest-9
b2g_ics_armv7a_gecko_emulator mozilla-central opt test reftest-10
b2g_panda mozilla-central opt test gaia-ui-test 


These are perma-hidden by sheriffs, so not providing any goodness for developers, yet are chewing up CPU time every checkin. Lets get these suites fixed (if that makes sense) or disabled (if the test suites running on mozilla-b2g18 is enough).

These same tests suites are typically working fine on mozilla-b2g18, so maybe this is because mozilla-central is gecko23?

Regardless, burning scarce CPU cycles like this is not ok.

Comment 1

[Ryan VanderMeulen [:RyanVM]]

Ultimately, the B2G team is going to have to decide if they can spare the resources to fix up these tests on m-c or not. Agreed that they're a waste of resources running hidden by default (and in the case of reftests/crashtests, perma-orange since the layers refactoring landed).

Comment 2

[cmtalbert]

I heard from ahal today that the gfx team is actively working to fix the perma orange they created when they landed the layers refactor.  So there is hope that the reftests will be fixed soon.

For mochitests, they are indeed largely hidden because we need an owner from development to help track down and fix why the tests are failing.

Comment 3

[Martijn Wargers (dead)]

(In reply to Clint Talbert ( :ctalbert ) from comment #2)
> For mochitests, they are indeed largely hidden because we need an owner from
> development to help track down and fix why the tests are failing.

According to bug 853024, comment 5, they were unhidden on 2013-04-01. I can see them on https://tbpl.mozilla.org/?showall=1 (as I can with the b2g reftests). The last 4 runs of b2g mochitests were green there.

Comment 4

[Andrew Halberstadt [:ahal]]

Clint and Martjin are both right. The graphics team has already fixed the perma-orange (as of this morning) and mochitests have been unhidden for awhile.

Comment 5

[Ed Morley [:emorley]]

(In reply to Martijn Wargers [:mw22] (QA - IRC nick: mw22) from comment #3)
> According to bug 853024, comment 5, they were unhidden on 2013-04-01. I can
> see them on https://tbpl.mozilla.org/?showall=1 (as I can with the b2g
> reftests).

.../?showall=1 shows all the jobs, which is why they are visible there. If you look at https://tbpl.mozilla.org/?jobname=b2g you can see that the reftests and crashtests are still hidden (ie comment 0 here is accurate). However, this whole bug is a dupe of bug 853024, which was already filed to track getting reliability increased and the jobs show in the default view.