Closed Bug 912519 Opened 6 years ago Closed 6 years ago

Robocop testSettingsMenuItems fails on x86 emulator

Categories

(Firefox for Android :: General, defect)

x86
Android
defect
Not set

Tracking

()

RESOLVED FIXED
Firefox 26

People

(Reporter: gbrown, Assigned: gbrown)

References

Details

Attachments

(3 files)

testSettingsMenuItems is failing again on the x86 emulator:

    18:33:41     INFO -  INFO | automation.py | Launching: /home/cltbld/tests/scripts/scripts/build/hostutils/bin/xpcshell -g /home/cltb
    ld/tests/scripts/scripts/build/hostutils/xre -v 170 -f /home/cltbld/tests/scripts/scripts/build/hostutils/bin/components/httpd.js -e
     "const _PROFILE_PATH = '/tmp/tmpty1jlT'; const _SERVER_PORT = '8888'; const _SERVER_ADDR = '10.0.2.2'; const _TEST_PREFIX = undefin
    ed; const _DISPLAY_RESULTS = false;" -f ./server.js
    18:33:41     INFO -  Mochitest INFO | runtests.py | Server pid: 5690
    18:33:42     INFO -  Mochitest INFO | runtests.py | Running tests: start.
    18:34:51     INFO -  Robocop derived process name: org.mozilla.fennec
    18:34:51     INFO -  INFO | automation.py | Application pid: 0
    18:34:51     INFO -  0 INFO SimpleTest START
    18:34:51     INFO -  1 INFO TEST-START | testSettingsMenuItems
    18:34:51     INFO -  EventExpecter: no longer listening for Gecko:Ready
    18:34:51     INFO -  2 INFO TEST-PASS | testSettingsMenuItems | Browser toolbar URL stayed the same - about:home should equal about:
    home
    18:34:51     INFO -  3 INFO TEST-PASS | testSettingsMenuItems | Waiting for Sync option - The Sync option is present
    18:34:51     INFO -  4 INFO TEST-PASS | testSettingsMenuItems | Section-item - Each item must at least have a title
    18:34:51     INFO -  5 INFO TEST-PASS | testSettingsMenuItems | Waiting for settings item ^About (Fennec|Nightly|Aurora|Firefox|Fire
    fox Beta)$ in section ^Mozilla$ - The ^About (Fennec|Nightly|Aurora|Firefox|Firefox Beta)$ option is present in section ^Mozilla$
    18:34:51     INFO -  6 INFO TEST-PASS | testSettingsMenuItems | Section-item - Each item must at least have a title
    18:34:51     INFO -  7 INFO TEST-PASS | testSettingsMenuItems | Waiting for settings item ^FAQs$ in section ^Mozilla$ - The ^FAQs$ o
    ption is present in section ^Mozilla$
    18:34:51     INFO -  8 INFO TEST-PASS | testSettingsMenuItems | Section-item - Each item must at least have a title
    18:34:51     INFO -  9 INFO TEST-PASS | testSettingsMenuItems | Waiting for settings item ^Give feedback$ in section ^Mozilla$ - The
     ^Give feedback$ option is present in section ^Mozilla$
    18:34:51     INFO -  10 INFO TEST-PASS | testSettingsMenuItems | Section-item - Each item must at least have a title
    18:34:51     INFO -  11 INFO TEST-PASS | testSettingsMenuItems | Waiting for settings item ^Show product announcements$ in section ^
    Mozilla$ - The ^Show product announcements$ option is present in section ^Mozilla$
    18:34:51     INFO -  12 INFO TEST-PASS | testSettingsMenuItems | Section-item - Each item must at least have a title
    18:34:51     INFO -  13 INFO TEST-PASS | testSettingsMenuItems | Waiting for settings item ^Data choices$ in section ^Mozilla$ - The
     ^Data choices$ option is present in section ^Mozilla$
    18:34:51     INFO -  14 INFO TEST-PASS | testSettingsMenuItems | Section-item - Each item must at least have a title
    18:34:51     INFO -  15 INFO TEST-PASS | testSettingsMenuItems | Waiting for settings item ^(Fennec|Nightly|Aurora|Firefox|Firefox B
    eta) Health Report$ in section ^Mozilla$ - The ^(Fennec|Nightly|Aurora|Firefox|Firefox Beta) Health Report$ option is present in sec
    tion ^Mozilla$
    18:34:51     INFO -  16 INFO TEST-UNEXPECTED-FAIL | testSettingsMenuItems | Waiting for settings item default ^Shares data with Mozi
    lla about your browser health and helps you understand your browser performance$ in section ^Mozilla$ - The ^Shares data with Mozill
    a about your browser health and helps you understand your browser performance$ default is present in section ^Mozilla$
    18:34:51     INFO -  Exception caught during test!
    18:34:51     INFO -  junit.framework.AssertionFailedError: 16 INFO TEST-UNEXPECTED-FAIL | testSettingsMenuItems | Waiting for settin
    gs item default ^Shares data with Mozilla about your browser health and helps you understand your browser performance$ in section ^M
    ozilla$ - The ^Shares data with Mozilla about your browser health and helps you understand your browser performance$ default is pres
    ent in section ^Mozilla$
    18:34:51     INFO -     at junit.framework.Assert.fail(Assert.java:50)
    18:34:51     INFO -     at org.mozilla.fennec.FennecMochitestAssert._logMochitestResult(FennecMochitestAssert.java:111)
    18:34:51     INFO -     at org.mozilla.fennec.FennecMochitestAssert.ok(FennecMochitestAssert.java:140)
    18:34:51     INFO -     at org.mozilla.fennec.tests.testSettingsMenuItems.checkMenuHierarchy(testSettingsMenuItems.java:224)
    18:34:51     INFO -     at org.mozilla.fennec.tests.testSettingsMenuItems.testSettingsMenuItems(testSettingsMenuItems.java:118)
    18:34:51     INFO -     at java.lang.reflect.Method.invokeNative(Native Method)
    18:34:51     INFO -     at java.lang.reflect.Method.invoke(Method.java:511)
    18:34:51     INFO -     at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
    18:34:51     INFO -     at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
    18:34:51     INFO -     at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192)
    18:34:51     INFO -     at org.mozilla.fennec.tests.BaseTest.runTest(BaseTest.java:161)
    18:34:51     INFO -     at junit.framework.TestCase.runBare(TestCase.java:134)
    18:34:51     INFO -     at junit.framework.TestResult$1.protect(TestResult.java:115)
    18:34:51     INFO -     at junit.framework.TestResult.runProtected(TestResult.java:133)
    18:34:51     INFO -     at junit.framework.TestResult.run(TestResult.java:118)
    18:34:51     INFO -     at junit.framework.TestCase.run(TestCase.java:124)
    18:34:51     INFO -     at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:190)
    18:34:51     INFO -     at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:175)
    18:34:51     INFO -     at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:555)
    18:34:51     INFO -     at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1661)
    18:34:51     INFO -  17 INFO TEST-UNEXPECTED-FAIL | testSettingsMenuItems | Exception caught - junit.framework.AssertionFailedError:
     16 INFO TEST-UNEXPECTED-FAIL | testSettingsMenuItems | Waiting for settings item default ^Shares data with Mozilla about your brows
    er health and helps you understand your browser performance$ in section ^Mozilla$ - The ^Shares data with Mozilla about your browser
     health and helps you understand your browser performance$ default is present in section ^Mozilla$
    18:34:51     INFO -  18 INFO TEST-END | testSettingsMenuItems | finished in 60825ms
    18:34:51     INFO -  19 INFO TEST-START | Shutdown
    18:34:51     INFO -  20 INFO Passed: 14
    18:34:51  WARNING -  21 INFO Failed: 2
    18:34:51  WARNING -  One or more unittests failed.
    18:34:51     INFO -  22 INFO Todo: 0
    18:34:51     INFO -  23 INFO SimpleTest FINISHED
    18:34:51     INFO -  INFO | automation.py | Application ran for: 0:01:08.626905 


Submit a correction or amendment below (click here to make a fresh posting)
After submitting an amendment, you'll be able to view the differences between the old and new posts easily.

Syntax highlighting:

To highlight particular lines, prefix each line with ~~
18:33:41 INFO - INFO | automation.py | Launching: /home/cltbld/tests/scripts/scripts/build/hostutils/bin/xpcshell -g /home/cltb ld/tests/scripts/scripts/build/hostutils/xre -v 170 -f /home/cltbld/tests/scripts/scripts/build/hostutils/bin/components/httpd.js -e "const _PROFILE_PATH = '/tmp/tmpty1jlT'; const _SERVER_PORT = '8888'; const _SERVER_ADDR = '10.0.2.2'; const _TEST_PREFIX = undefin ed; const _DISPLAY_RESULTS = false;" -f ./server.js 18:33:41 INFO - Mochitest INFO | runtests.py | Server pid: 5690 18:33:42 INFO - Mochitest INFO | runtests.py | Running tests: start. 18:34:51 INFO - Robocop derived process name: org.mozilla.fennec 18:34:51 INFO - INFO | automation.py | Application pid: 0 18:34:51 INFO - 0 INFO SimpleTest START 18:34:51 INFO - 1 INFO TEST-START | testSettingsMenuItems 18:34:51 INFO - EventExpecter: no longer listening for Gecko:Ready 18:34:51 INFO - 2 INFO TEST-PASS | testSettingsMenuItems | Browser toolbar URL stayed the same - about:home should equal about: home 18:34:51 INFO - 3 INFO TEST-PASS | testSettingsMenuItems | Waiting for Sync option - The Sync option is present 18:34:51 INFO - 4 INFO TEST-PASS | testSettingsMenuItems | Section-item - Each item must at least have a title 18:34:51 INFO - 5 INFO TEST-PASS | testSettingsMenuItems | Waiting for settings item ^About (Fennec|Nightly|Aurora|Firefox|Fire fox Beta)$ in section ^Mozilla$ - The ^About (Fennec|Nightly|Aurora|Firefox|Firefox Beta)$ option is present in section ^Mozilla$ 18:34:51 INFO - 6 INFO TEST-PASS | testSettingsMenuItems | Section-item - Each item must at least have a title 18:34:51 INFO - 7 INFO TEST-PASS | testSettingsMenuItems | Waiting for settings item ^FAQs$ in section ^Mozilla$ - The ^FAQs$ o ption is present in section ^Mozilla$ 18:34:51 INFO - 8 INFO TEST-PASS | testSettingsMenuItems | Section-item - Each item must at least have a title 18:34:51 INFO - 9 INFO TEST-PASS | testSettingsMenuItems | Waiting for settings item ^Give feedback$ in section ^Mozilla$ - The ^Give feedback$ option is present in section ^Mozilla$ 18:34:51 INFO - 10 INFO TEST-PASS | testSettingsMenuItems | Section-item - Each item must at least have a title 18:34:51 INFO - 11 INFO TEST-PASS | testSettingsMenuItems | Waiting for settings item ^Show product announcements$ in section ^ Mozilla$ - The ^Show product announcements$ option is present in section ^Mozilla$ 18:34:51 INFO - 12 INFO TEST-PASS | testSettingsMenuItems | Section-item - Each item must at least have a title 18:34:51 INFO - 13 INFO TEST-PASS | testSettingsMenuItems | Waiting for settings item ^Data choices$ in section ^Mozilla$ - The ^Data choices$ option is present in section ^Mozilla$ 18:34:51 INFO - 14 INFO TEST-PASS | testSettingsMenuItems | Section-item - Each item must at least have a title 18:34:51 INFO - 15 INFO TEST-PASS | testSettingsMenuItems | Waiting for settings item ^(Fennec|Nightly|Aurora|Firefox|Firefox B eta) Health Report$ in section ^Mozilla$ - The ^(Fennec|Nightly|Aurora|Firefox|Firefox Beta) Health Report$ option is present in sec tion ^Mozilla$ 18:34:51 INFO - 16 INFO TEST-UNEXPECTED-FAIL | testSettingsMenuItems | Waiting for settings item default ^Shares data with Mozi lla about your browser health and helps you understand your browser performance$ in section ^Mozilla$ - The ^Shares data with Mozill a about your browser health and helps you understand your browser performance$ default is present in section ^Mozilla$ 18:34:51 INFO - Exception caught during test! 18:34:51 INFO - junit.framework.AssertionFailedError: 16 INFO TEST-UNEXPECTED-FAIL | testSettingsMenuItems | Waiting for settin gs item default ^Shares data with Mozilla about your browser health and helps you understand your browser performance$ in section ^M ozilla$ - The ^Shares data with Mozilla about your browser health and helps you understand your browser performance$ default is pres ent in section ^Mozilla$ 18:34:51 INFO - at junit.framework.Assert.fail(Assert.java:50) 18:34:51 INFO - at org.mozilla.fennec.FennecMochitestAssert._logMochitestResult(FennecMochitestAssert.java:111) 18:34:51 INFO - at org.mozilla.fennec.FennecMochitestAssert.ok(FennecMochitestAssert.java:140) 18:34:51 INFO - at org.mozilla.fennec.tests.testSettingsMenuItems.checkMenuHierarchy(testSettingsMenuItems.java:224) 18:34:51 INFO - at org.mozilla.fennec.tests.testSettingsMenuItems.testSettingsMenuItems(testSettingsMenuItems.java:118) 18:34:51 INFO - at java.lang.reflect.Method.invokeNative(Native Method) 18:34:51 INFO - at java.lang.reflect.Method.invoke(Method.java:511) 18:34:51 INFO - at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214) 18:34:51 INFO - at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199) 18:34:51 INFO - at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192) 18:34:51 INFO - at org.mozilla.fennec.tests.BaseTest.runTest(BaseTest.java:161) 18:34:51 INFO - at junit.framework.TestCase.runBare(TestCase.java:134) 18:34:51 INFO - at junit.framework.TestResult$1.protect(TestResult.java:115) 18:34:51 INFO - at junit.framework.TestResult.runProtected(TestResult.java:133) 18:34:51 INFO - at junit.framework.TestResult.run(TestResult.java:118) 18:34:51 INFO - at junit.framework.TestCase.run(TestCase.java:124) 18:34:51 INFO - at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:190) 18:34:51 INFO - at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:175) 18:34:51 INFO - at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:555) 18:34:51 INFO - at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1661) 18:34:51 INFO - 17 INFO TEST-UNEXPECTED-FAIL | testSettingsMenuItems | Exception caught - junit.framework.AssertionFailedError: 16 INFO TEST-UNEXPECTED-FAIL | testSettingsMenuItems | Waiting for settings item default ^Shares data with Mozilla about your brows er health and helps you understand your browser performance$ in section ^Mozilla$ - The ^Shares data with Mozilla about your browser health and helps you understand your browser performance$ default is present in section ^Mozilla$ 18:34:51 INFO - 18 INFO TEST-END | testSettingsMenuItems | finished in 60825ms 18:34:51 INFO - 19 INFO TEST-START | Shutdown 18:34:51 INFO - 20 INFO Passed: 14 18:34:51 WARNING - 21 INFO Failed: 2 18:34:51 WARNING - One or more unittests failed. 18:34:51 INFO - 22 INFO Todo: 0 18:34:51 INFO - 23 INFO SimpleTest FINISHED 18:34:51 INFO - INFO | automation.py | Application ran for: 0:01:08.626905
From the screenshot, I wonder if the problem is that the title is visible but the item text is scrolled off screen.
Assignee: nobody → gbrown
To resolve the x86 emulator failure, I just needed to add another scrollDown/retry case.

I noticed that the test currently calls waitForText twice, where once would be enough. Eliminating the unnecessary waits reduces the total test time from about 4 minutes to about 3 minutes.

Finally, I couldn't resist some cleanup: 

 - s/midHeight/mMidHeight/
 - correct some 2-space indentation
Attachment #799877 - Flags: review?(liuche)
Blocks: 892688
Comment on attachment 799877 [details] [diff] [review]
(1) scroll down in one more case, in case menu item is partially obscured

Review of attachment 799877 [details] [diff] [review]:
-----------------------------------------------------------------

The API actually has a waitForText that take in a toScroll arg, and presumably this does the scrolling for us. Let's use that instead.
Attachment #799877 - Flags: review?(liuche) → review-
(In reply to Chenxia Liu [:liuche] from comment #4)
> The API actually has a waitForText that take in a toScroll arg, and
> presumably this does the scrolling for us. Let's use that instead.

BaseTest.waitForText calls Solo.waitForText(string) which calls Waiter.waitForText(string) which already calls Waiter.waitForText(..., scroll=true)

https://github.com/jayway/robotium/blob/master/robotium-solo/src/main/java/com/jayway/android/robotium/solo/Solo.java#L199

https://github.com/jayway/robotium/blob/master/robotium-solo/src/main/java/com/jayway/android/robotium/solo/Waiter.java#L374

The scroll parameter does not seem to work in this case.
Attachment #799877 - Flags: review- → review?(liuche)
Comment on attachment 799877 [details] [diff] [review]
(1) scroll down in one more case, in case menu item is partially obscured

Review of attachment 799877 [details] [diff] [review]:
-----------------------------------------------------------------

Sorry for the slow review - I was looking at different ways to do scrolling that weren't hard-coded to one additional scroll. mSolo.scrollDown should return a boolean depending on whether the content was scrolled (as opposed to at an edge) but just doesn't work on dialogs. Short of implementing a check for whether we've hit an edge in our scrollDown() method, this seems to be okay.

The scrolling-twice code is repeated multiple times, so I'd kind of like to have that pulled out into a method so it's clear that we're doing something a little hacky, and can be replaced more easily if robotium eventually supports scrolling in dialogs.
Attachment #799877 - Flags: review?(liuche) → review+
(In reply to Chenxia Liu [:liuche] from comment #6)
> The scrolling-twice code is repeated multiple times, so I'd kind of like to
> have that pulled out into a method so it's clear that we're doing something
> a little hacky, and can be replaced more easily if robotium eventually
> supports scrolling in dialogs.

Good idea. 

I'm anxious to land a fix for x86, so I'll proceed with this patch and put together a follow-up patch soon.
Attachment #802641 - Flags: review?(liuche)
Attachment #799877 - Attachment description: scroll down in one more case, in case menu item is partially obscured → (1) scroll down in one more case, in case menu item is partially obscured
Whiteboard: [leave open]
Comment on attachment 802641 [details] [diff] [review]
(2) simple refactoring  follow-up

Review of attachment 802641 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good to me!
Attachment #802641 - Flags: review?(liuche) → review+
https://hg.mozilla.org/mozilla-central/rev/380a028c1a60
https://hg.mozilla.org/mozilla-central/rev/c856007f844e
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 26
Blocks: 936226
You need to log in before you can comment on or make changes to this bug.