1102225 - Break Android 4.0 tests into two sets: needs-hardware and emulator-only

Status: RESOLVED WORKSFORME

(Firefox for Android Graveyard :: Testing, defect)

Description

[Laura Thomson :laura]

AIUI, Android 4.0 tests currently run on both hardware (Pandas at present) and emumlators. In the interest of reducing our hardware footprint, we'd like to split the tests in two, and only run on hardware when we have to.

(We talked about this in the Mobile testing meeting this week, and I said I'd file a bug for snorp, so here it is.)

Comment 1

[Mark Finkle (:mfinkle) (use needinfo?)]

It might be nice to gather some real data, but as I have been looking into some intermittent failure recently I notice that the majority of our failures happen on emulators (Android 2.3 and Android 4 x86).

Comment 2

[Geoff Brown [:gbrown]]

We should think about how we want this to affect Debug tests: We currently run some tests on Pandas against Debug builds; we run no Debug tests on emulators.

Comment 3

[Geoff Brown [:gbrown]]

We have (basically) retired Pandaboards now. We run all tests on emulators, and a select few on hardware via autophone. We probably did not go through the intentional process suggested in this bug, but rather chose some obvious tests that needed hardware and plunged onward. I think that's good enough for now; the set of tests run on autophone will continue to evolve naturally.