897970 - Robocop testBookmark fails consistently on Android x86 emulators

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: General, defect)

Description

[Geoff Brown [:gbrown]]

Robocop's testBookmark fails consistently when run on the x86 emulator setup (see bug 891959). It also fails locally when run on a Motorola Razr i (XT890).

This is likely a fails-on-Jellybean rather than a fails-on-x86 problem, but it is the effort to get tests running on x86 that is bringing about this bug.

Locally, I see the first call to toggleBookmark() bring up the menu, but then fail to tap on the bookmark "star" icon.

Comment 1

[Geoff Brown [:gbrown]]

0 INFO SimpleTest START
1 INFO TEST-START | testBookmark
EventExpecter: no longer listening for Gecko:Ready
2 INFO TEST-PASS | testBookmark | checking that bookmarks list exists - bookmarks list exists and has 4 children (the default bookmarks)
3 INFO TEST-PASS | testBookmark | Find default bookmark - Default bookmark for about:firefox found
4 INFO TEST-PASS | testBookmark | Find default bookmark - Default bookmark for about:home found
5 INFO TEST-PASS | testBookmark | Find default bookmark - Default bookmark for https://addons.mozilla.org/en-US/android/ found
6 INFO TEST-PASS | testBookmark | Find default bookmark - Default bookmark for http://support.mozilla.org/en-US/products/mobile found
7 INFO TEST-PASS | testBookmark | bookmarks list has 5 children (the default bookmarks and the new one) - 5 should equal 5
8 INFO TEST-PASS | testBookmark | list item can be retrieved - android.widget.RelativeLayout@215cff98
PaintExpecter: no longer listening for events
9 INFO TEST-PASS | testBookmark | Page is not bookmarked initially - false should equal false
10 INFO TEST-PASS | testBookmark | Awesomebar URL typed properly - http://mochi.test:8888/tests/robocop/robocop_blank_01.html should equal http://mochi.test:8888/tests/robocop/robocop_blank_01.html
EventExpecter: no longer listening for DOMContentLoaded
PaintExpecter: no longer listening for events
waitForText timeout on Bookmark
11 INFO TEST-UNEXPECTED-FAIL | testBookmark | intentional fail - intentional fail
Exception caught during test!
junit.framework.AssertionFailedError: 11 INFO TEST-UNEXPECTED-FAIL | testBookmark | intentional fail - intentional fail
        at junit.framework.Assert.fail(Assert.java:50)
        at org.mozilla.fennec.FennecMochitestAssert._logMochitestResult(FennecMochitestAssert.java:107)
        at org.mozilla.fennec.FennecMochitestAssert.ok(FennecMochitestAssert.java:136)
        at org.mozilla.fennec.tests.testBookmark.toggleBookmark(testBookmark.java:156)
        at org.mozilla.fennec.tests.testBookmark.setUpBookmark(testBookmark.java:163)
        at org.mozilla.fennec.tests.testBookmark.runMenuTest(testBookmark.java:55)
        at org.mozilla.fennec.tests.testBookmark.testBookmark(testBookmark.java:48)
        at java.lang.reflect.Method.invokeNative(Native Method)
        at java.lang.reflect.Method.invoke(Method.java:511)
        at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
        at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
        at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192)
        at org.mozilla.fennec.tests.BaseTest.runTest(BaseTest.java:133)
        at junit.framework.TestCase.runBare(TestCase.java:134)
        at junit.framework.TestResult$1.protect(TestResult.java:115)
        at junit.framework.TestResult.runProtected(TestResult.java:133)
        at junit.framework.TestResult.run(TestResult.java:118)
        at junit.framework.TestCase.run(TestCase.java:124)
        at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:190)
        at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:175)
        at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:555)
        at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1584)
12 INFO TEST-UNEXPECTED-FAIL | testBookmark | Exception caught - junit.framework.AssertionFailedError: 11 INFO TEST-UNEXPECTED-FAIL | testBookmark | intentional fail - intentional fail
13 INFO TEST-END | testBookmark | finished in 41181ms
14 INFO TEST-START | Shutdown
15 INFO Passed: 9
16 INFO Failed: 2
17 INFO Todo: 0
18 INFO SimpleTest FINISHED
INFO | automation.py | Application ran for: 0:00:45.109988

Comment 2

[Geoff Brown [:gbrown]]

testBookmark uses its own toggleBookmark() function to select the bookmark menu item. It seems too simplistic, looking for "Bookmark" text in the menu...how can that work on ICS+, where only the icon is shown? 

There is more complicated code in BaseTest's Navigation.bookmark(), which accounts for the differences between phones/tablets and different android versions. But it is only used by the disabled test testWebContentContextMenu. I tried using that code in testBookmark, but it fails also.

There is also another approach used in testHistoryTab -- that works great!

Comment 3

[Geoff Brown [:gbrown]]

I pasted the wrong log in Comment 1.

0 INFO SimpleTest START
1 INFO TEST-START | testBookmark
EventExpecter: no longer listening for Gecko:Ready
2 INFO TEST-PASS | testBookmark | checking that bookmarks list exists - bookmarks list exists and has 4 children (the default bookmarks)
3 INFO TEST-PASS | testBookmark | Find default bookmark - Default bookmark for about:firefox found
4 INFO TEST-PASS | testBookmark | Find default bookmark - Default bookmark for about:home found
5 INFO TEST-PASS | testBookmark | Find default bookmark - Default bookmark for https://addons.mozilla.org/en-US/android/ found
6 INFO TEST-PASS | testBookmark | Find default bookmark - Default bookmark for http://support.mozilla.org/en-US/products/mobile found
7 INFO TEST-PASS | testBookmark | bookmarks list has 5 children (the default bookmarks and the new one) - 5 should equal 5
8 INFO TEST-PASS | testBookmark | list item can be retrieved - android.widget.RelativeLayout@21571158
PaintExpecter: no longer listening for events
9 INFO TEST-PASS | testBookmark | Page is not bookmarked initially - false should equal false
10 INFO TEST-PASS | testBookmark | Awesomebar URL typed properly - http://mochi.test:8888/tests/robocop/robocop_blank_01.html should equal http://mochi.test:8888/tests/robocop/robocop_blank_01.html
EventExpecter: no longer listening for DOMContentLoaded
PaintExpecter: no longer listening for events
waitForText timeout on Bookmark
Exception caught during test!
junit.framework.AssertionFailedError: The text: Bookmark is not found!
	at junit.framework.Assert.fail(Assert.java:50)
	at junit.framework.Assert.assertTrue(Assert.java:20)
	at com.jayway.android.robotium.solo.Clicker.clickOnText(Clicker.java:330)
	at com.jayway.android.robotium.solo.Solo.clickOnText(Solo.java:839)
	at org.mozilla.fennec.tests.testBookmark.toggleBookmark(testBookmark.java:154)
	at org.mozilla.fennec.tests.testBookmark.setUpBookmark(testBookmark.java:160)
	at org.mozilla.fennec.tests.testBookmark.runMenuTest(testBookmark.java:55)
	at org.mozilla.fennec.tests.testBookmark.testBookmark(testBookmark.java:48)
	at java.lang.reflect.Method.invokeNative(Native Method)
	at java.lang.reflect.Method.invoke(Method.java:511)
	at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
	at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
	at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192)
	at org.mozilla.fennec.tests.BaseTest.runTest(BaseTest.java:133)
	at junit.framework.TestCase.runBare(TestCase.java:134)
	at junit.framework.TestResult$1.protect(TestResult.java:115)
	at junit.framework.TestResult.runProtected(TestResult.java:133)
	at junit.framework.TestResult.run(TestResult.java:118)
	at junit.framework.TestCase.run(TestCase.java:124)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:190)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:175)
	at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:555)
	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1584)
11 INFO TEST-UNEXPECTED-FAIL | testBookmark | Exception caught - junit.framework.AssertionFailedError: The text: Bookmark is not found!
12 INFO TEST-END | testBookmark | finished in 51167ms
13 INFO TEST-START | Shutdown
14 INFO Passed: 9
15 INFO Failed: 1
16 INFO Todo: 0
17 INFO SimpleTest FINISHED
INFO | automation.py | Application ran for: 0:00:55.118882

Comment 4

[Geoff Brown [:gbrown]]

Attachment: re-use code from testHistoryTab

This moves the bookmarking code from testHistoryTab to BaseTest.toggleBookmark, and then calls toggleBookmark from both testHistoryTab and testBookmark. 

I have marked Navigation.bookmark() as deprecated -- we can re-visit that when we try to enable testWebContentContextMenu.

Try run at https://tbpl.mozilla.org/?tree=Try&rev=5251432f7fc0.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 780980 [details] [diff] [review]
re-use code from testHistoryTab

Review of attachment 780980 [details] [diff] [review]:
-----------------------------------------------------------------

eventually basetest will have everything!

Comment 6

[Geoff Brown [:gbrown]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/277c1e376c3c

Comment 7

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/277c1e376c3c