1171789 - `TEST-UNEXPECTED-FAIL | testSettingsMenuItems | Waiting for settings item default ^Queue links for later` permafail on Android 4.0

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: General, defect)

Description

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Attachment: Failure output

It looks like robocop is not running on fx-team for Android 4.0 builds (Geoff, known issue?) and because of that we have a permafail result:

16:11:44 INFO - TEST-PASS | testSettingsMenuItems | Waiting for settings item choice ^Never$ in section ^Customize$ - The ^Never$ choice is present in section ^Customize$
16:11:44 INFO - TEST-PASS | testSettingsMenuItems | Section-item - Each item must at least have a title
16:11:44 INFO - TEST-PASS | testSettingsMenuItems | Waiting for settings item ^Open multiple links$ in section ^Customize$ - The ^Open multiple links$ option is present in section ^Customize$
16:11:44 INFO - waitForText timeout on ^Queue links for later instead of switching to (Fennec|Nightly|Aurora|Firefox Beta|Firefox) each time$
16:11:44 WARNING - TEST-UNEXPECTED-FAIL | testSettingsMenuItems | Waiting for settings item default ^Queue links for later instead of switching to (Fennec|Nightly|Aurora|Firefox Beta|Firefox) each time$ in section ^Customize$ - The ^Queue links for later instead of switching to (Fennec|Nightly|Aurora|Firefox Beta|Firefox) each time$ default is present in section ^Customize$
16:11:44 INFO - 0 ERROR Exception caught during test! - junit.framework.AssertionFailedError: TEST-UNEXPECTED-FAIL | testSettingsMenuItems | Waiting for settings item default ^Queue links for later instead of switching to (Fennec|Nightly|Aurora|Firefox Beta|Firefox) each time$ in section ^Customize$ - The ^Queue links for later instead of switching to (Fennec|Nightly|Aurora|Firefox Beta|Firefox) each time$ default is present in section ^Customize$
16:11:44 INFO - at junit.framework.Assert.fail(Assert.java:47)
16:11:44 INFO - at org.mozilla.gecko.FennecMochitestAssert._logMochitestResult(FennecMochitestAssert.java:128)
16:11:44 INFO - at org.mozilla.gecko.FennecMochitestAssert.ok(FennecMochitestAssert.java:150)
16:11:44 INFO - at org.mozilla.gecko.tests.testSettingsMenuItems.checkMenuHierarchy(testSettingsMenuItems.java:264)
16:11:44 INFO - at org.mozilla.gecko.tests.testSettingsMenuItems.testSettingsMenuItems(testSettingsMenuItems.java:160) 

via https://treeherder.mozilla.org/#/jobs?repo=try&revision=b5294696b21c

Additionally, https://bugzilla.mozilla.org/show_bug.cgi?id=1168808#c7

Comment 1

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

I suspect bug 1133755 for not updating the tests (which means they haven't been running for a while). Perhaps this test doesn't run on emulators for some reason?

Comment 2

[Geoff Brown [:gbrown]]

We are in the process of "retiring" the Pandaboards and the Android 4.0 tests that run on them. Android 4.3 tests are replacing the corresponding 4.0 tests: When Android 4.3 Opt mochitest-1 started running on mozilla-central, Android 4.0 Opt mochitest-1 was turned off. Changes are "riding the trains", so most Android 4.0 tests are still running on mozilla-beta, for instance. try is a special case, generally running both Android 4.3 and Android 4.0 (handy if you want to "try" a beta patch, for instance). Once 4.3 is running on all trees, I expect Android 4.0 will be turned off on try as well.

Android 4.0 Opt tests no longer run on mozilla-central or any of the integration branches (m-i, fxteam, b2g-i) and won't ever again. So you can ignore Android 4.0 Opt failures on try without fear of causing breakage on mozilla-central.

Comment 3

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

So we can ignore this failure specifically, but we should investigate why this doesn't fail on the emulators (because I'm pretty sure it should).

Comment 4

[Geoff Brown [:gbrown]]

It is disabled on both 2.3 and 4.3:

[testSettingsMenuItems.java]
# disabled on Android 2.3, bug 979552; on 4.3, bug 1144898
skip-if = android_version == "10" || android_version == "18"

Comment 5

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Martyn, do you want to fix this test on 4.0 and then we'll fix it later for the emulators? It'd be a shame to lose this test too. :\

Comment 6

[Geoff Brown [:gbrown]]

Attachment: disable testSettingsMenuItems

I should not have suggested ignoring the failure -- that's just going to cause more confusion for others pushing to try. Let's disable it now, fix it as soon as we can.

Comment 8

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Comment on attachment 8621565 [details] [diff] [review]
disable testSettingsMenuItems

Review of attachment 8621565 [details] [diff] [review]:
-----------------------------------------------------------------

Actually, the fix shouldn't be too bad – let's just fix it now so it's easier to restore on 2.3 & 4.3.

Patch inbound...

Comment 9

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Attachment: MozReview Request: Bug 1171789 - Add updated summary to tab queue preference item in test. r?gbrown

Bug 1171789 - Add updated summary to tab queue preference item in test. r?gbrown

Comment 10

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Given review to liuche so she can unblock bug 1136477.

Comment 11

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Tests pass for me locally.

Comment 12

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=baccdc76ec9b

Comment 13

[Chenxia Liu [:liuche]]

Comment on attachment 8621717 [details]
MozReview Request: Bug 1171789 - Add updated summary to tab queue preference item in test. r?gbrown

https://reviewboard.mozilla.org/r/11031/#review9627

Ship It!

Comment 14

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

https://hg.mozilla.org/integration/fx-team/rev/414fa69645a8

Comment 15

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/414fa69645a8

Comment 16

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=try&job_id=9125046
repository: try
start_time: 2015-07-06T18:20:33
who: m[dot]lopiccolo3[at]gmail[dot]com
machine: panda-0277
buildname: Android 4.0 armv7 API 11+ try opt test robocop-5
revision: fa0fdf234565

TEST-UNEXPECTED-FAIL | testSettingsMenuItems | Waiting for settings item default ^Queue links for later instead of switching to (Fennec|Nightly|Aurora|Firefox Beta|Firefox) each time$ in section ^Customize$ - The ^Queue links for later instead of switching to (Fennec|Nightly|Aurora|Firefox Beta|Firefox) each time$ default is present in section ^Customize$
0 ERROR Exception caught during test! - junit.framework.AssertionFailedError: TEST-UNEXPECTED-FAIL | testSettingsMenuItems | Waiting for settings item default ^Queue links for later instead of switching to (Fennec|Nightly|Aurora|Firefox Beta|Firefox) each time$ in section ^Customize$ - The ^Queue links for later instead of switching to (Fennec|Nightly|Aurora|Firefox Beta|Firefox) each time$ default is present in section ^Customize$
TEST-UNEXPECTED-FAIL | testSettingsMenuItems | Exception caught - junit.framework.AssertionFailedError: TEST-UNEXPECTED-FAIL | testSettingsMenuItems | Waiting for settings item default ^Queue links for later instead of switching to (Fennec|Nightly|Aurora|Firefox Beta|Firefox) each time$ in section ^Customize$ - The ^Queue links for later instead of switching to (Fennec|Nightly|Aurora|Firefox Beta|Firefox) each time$ default is present in section ^Customize$
07-06 18:45:25.179 E/GeckoLinker( 3509): /data/data/com.adobe.flashplayer/lib/libysshared.so: Text relocations are not supported
Return code: 1
07-06 18:45:25.179 E/GeckoLinker( 3509): /data/data/com.adobe.flashplayer/lib/libysshared.so: Text relocations are not supported