Closed Bug 1171789 Opened 8 years ago Closed 8 years ago

`TEST-UNEXPECTED-FAIL | testSettingsMenuItems | Waiting for settings item default ^Queue links for later` permafail on Android 4.0

Categories

(Firefox for Android Graveyard :: General, defect)

Product:
Firefox for Android Graveyard
Component:
General
Platform:
All
Android
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox41 fixed)

Status:
RESOLVED FIXED
Milestone:
Firefox 41
Tracking Flags:
Tracking Status
firefox41 --- fixed

People

(Reporter: mcomella, Assigned: mcomella)

References

Details

Attachments

(3 files)

Failure output
682.83 KB, text/plain
Details
disable testSettingsMenuItems
1.42 KB, patch
mcomella
: review-
Details | Diff | Splinter Review
MozReview Request: Bug 1171789 - Add updated summary to tab queue preference item in test. r?gbrown
40 bytes, text/x-review-board-request
liuche
: review+
Details
Attached file Failure output — 
It looks like robocop is not running on fx-team for Android 4.0 builds (Geoff, known issue?) and because of that we have a permafail result:

16:11:44 INFO - TEST-PASS | testSettingsMenuItems | Waiting for settings item choice ^Never$ in section ^Customize$ - The ^Never$ choice is present in section ^Customize$
16:11:44 INFO - TEST-PASS | testSettingsMenuItems | Section-item - Each item must at least have a title
16:11:44 INFO - TEST-PASS | testSettingsMenuItems | Waiting for settings item ^Open multiple links$ in section ^Customize$ - The ^Open multiple links$ option is present in section ^Customize$
16:11:44 INFO - waitForText timeout on ^Queue links for later instead of switching to (Fennec|Nightly|Aurora|Firefox Beta|Firefox) each time$
16:11:44 WARNING - TEST-UNEXPECTED-FAIL | testSettingsMenuItems | Waiting for settings item default ^Queue links for later instead of switching to (Fennec|Nightly|Aurora|Firefox Beta|Firefox) each time$ in section ^Customize$ - The ^Queue links for later instead of switching to (Fennec|Nightly|Aurora|Firefox Beta|Firefox) each time$ default is present in section ^Customize$
16:11:44 INFO - 0 ERROR Exception caught during test! - junit.framework.AssertionFailedError: TEST-UNEXPECTED-FAIL | testSettingsMenuItems | Waiting for settings item default ^Queue links for later instead of switching to (Fennec|Nightly|Aurora|Firefox Beta|Firefox) each time$ in section ^Customize$ - The ^Queue links for later instead of switching to (Fennec|Nightly|Aurora|Firefox Beta|Firefox) each time$ default is present in section ^Customize$
16:11:44 INFO - at junit.framework.Assert.fail(Assert.java:47)
16:11:44 INFO - at org.mozilla.gecko.FennecMochitestAssert._logMochitestResult(FennecMochitestAssert.java:128)
16:11:44 INFO - at org.mozilla.gecko.FennecMochitestAssert.ok(FennecMochitestAssert.java:150)
16:11:44 INFO - at org.mozilla.gecko.tests.testSettingsMenuItems.checkMenuHierarchy(testSettingsMenuItems.java:264)
16:11:44 INFO - at org.mozilla.gecko.tests.testSettingsMenuItems.testSettingsMenuItems(testSettingsMenuItems.java:160) 

via https://treeherder.mozilla.org/#/jobs?repo=try&revision=b5294696b21c

Additionally, https://bugzilla.mozilla.org/show_bug.cgi?id=1168808#c7
Flags: needinfo?(gbrown)
I suspect bug 1133755 for not updating the tests (which means they haven't been running for a while). Perhaps this test doesn't run on emulators for some reason?
Blocks: 1133755
We are in the process of "retiring" the Pandaboards and the Android 4.0 tests that run on them. Android 4.3 tests are replacing the corresponding 4.0 tests: When Android 4.3 Opt mochitest-1 started running on mozilla-central, Android 4.0 Opt mochitest-1 was turned off. Changes are "riding the trains", so most Android 4.0 tests are still running on mozilla-beta, for instance. try is a special case, generally running both Android 4.3 and Android 4.0 (handy if you want to "try" a beta patch, for instance). Once 4.3 is running on all trees, I expect Android 4.0 will be turned off on try as well.

Android 4.0 Opt tests no longer run on mozilla-central or any of the integration branches (m-i, fxteam, b2g-i) and won't ever again. So you can ignore Android 4.0 Opt failures on try without fear of causing breakage on mozilla-central.
Flags: needinfo?(gbrown)
So we can ignore this failure specifically, but we should investigate why this doesn't fail on the emulators (because I'm pretty sure it should).
It is disabled on both 2.3 and 4.3:

[testSettingsMenuItems.java]
# disabled on Android 2.3, bug 979552; on 4.3, bug 1144898
skip-if = android_version == "10" || android_version == "18"
Martyn, do you want to fix this test on 4.0 and then we'll fix it later for the emulators? It'd be a shame to lose this test too. :\
I should not have suggested ignoring the failure -- that's just going to cause more confusion for others pushing to try. Let's disable it now, fix it as soon as we can.
Attachment #8621565 - Flags: review?(mhaigh)
Comment on attachment 8621565 [details] [diff] [review]
disable testSettingsMenuItems

Review of attachment 8621565 [details] [diff] [review]:
-----------------------------------------------------------------

Actually, the fix shouldn't be too bad – let's just fix it now so it's easier to restore on 2.3 & 4.3.

Patch inbound...
Attachment #8621565 - Flags: review+ → review-
Blocks: 1136477
Assignee: gbrown → michael.l.comella
Comment on attachment 8621717 [details]
MozReview Request: Bug 1171789 - Add updated summary to tab queue preference item in test. r?gbrown

https://reviewboard.mozilla.org/r/11031/#review9627

Ship It!
Attachment #8621717 - Flags: review?(liuche) → review+
https://hg.mozilla.org/mozilla-central/rev/414fa69645a8
Status: NEW → RESOLVED
Closed: 8 years ago
status-firefox41: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → Firefox 41
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in before you can comment on or make changes to this bug.