1174072 - testSettingsMenuItem fails on all try pushes, wasting developers' time

Status: RESOLVED DUPLICATE of bug 1171789

(Firefox for Android Graveyard :: Testing, defect)

Description

[Robert O'Callahan (:roc) (email my personal email if necessary)]

I was trying to figure out why Android 4.0 rc5 was failing on my try push in testSettingsMenuItems. I spent a few hours poking around before I finally realized that testSettingsMenuItems is disabled on Android 2.3 and 4.3, and it's hidden on inbound, so it probably doesn't pass anywhere. In particular it looks for the text "Queue links for later instead of switching to" which isn't present in the UI anymore in any configuration --- the test is the only occurrence of that text under mobile/android :-(.

Tests that always fail on try should be hidden or disabled on try as well as inbound. Otherwise we're just wasting developers' time.

Comment 1

[Geoff Brown [:gbrown]]

We are in an unusual situation with the whole Android 4.0 platform. Android 4.0 is being retired, with 4.0 tests being replaced with corresponding Android 4.3 tests. On mozilla-central, nearly all tests have been migrated (Debug reftests being the remaining exception). But changes are riding the trains, so many Android 4.0 tests are still running on mozilla-aurora and mozilla-beta. On try, all Android 4.0 and Android 4.3 tests still run, all unhidden. Android 4.0 try is useful for "try"-ing aurora+ changes, but confusing for central -- a green push on inbound can cause a 4.0 try failure.

This particular failure was also noted in bug 1171789.