817970 - Robocop: Add tests for Bookmark remove and edit from context menu

Status: RESOLVED FIXED

Description

[Adrian Tamas (:AdrianT)]

This bug will track the progress for adding Robocop test for the remove bookmark and edit bookmarks from content menu tests that are part of the Firefox for Android Smoke test suite.

Remove bookmark from context menu: https://moztrap.mozilla.org/manage/case/316/
Edit bookmark from context menu: https://moztrap.mozilla.org/manage/case/315/

Comment 1

[Adrian Tamas (:AdrianT)]

Attachment: Edit and Remove bookmark from the context menu v1

Comment 2

[Joel Maher ( :jmaher ) (UTC-4)]

Comment on attachment 689703 [details] [diff] [review]
Edit and Remove bookmark from the context menu v1

Review of attachment 689703 [details] [diff] [review]:
-----------------------------------------------------------------

additional tests would be to test invalid characters and the flow of error messages.

::: mobile/android/base/tests/testBookmarksTab.java.in
@@ +198,2 @@
>          // back again to exit the awesomebar
>          mActions.sendSpecialKey(Actions.SpecialKey.BACK);

is this back still valid?  Also are we still in the awesomebar?

@@ +237,5 @@
> +        } else {
> +           mSolo.clickOnText("Cancel");
> +           return false;
> +        }
> +    }

both of these functions would make a good library function for other tests to use (not needed for this bug, but please file a followup bug)

Comment 3

[Adrian Tamas (:AdrianT)]

(In reply to Joel Maher (:jmaher) from comment #2)
> Comment on attachment 689703 [details] [diff] [review]
> Edit and Remove bookmark from the context menu v1
> 
> Review of attachment 689703 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> additional tests would be to test invalid characters and the flow of error
> messages.
> 
> ::: mobile/android/base/tests/testBookmarksTab.java.in
> @@ +198,2 @@
> >          // back again to exit the awesomebar
> >          mActions.sendSpecialKey(Actions.SpecialKey.BACK);
> 
> is this back still valid?  Also are we still in the awesomebar?
The BACK from above this just closes the Custom Menu which was opened to test the bookmark title. This closes the Awesomebar although the test ends after this so this may not be necessary but it is part of the original test.

> 
> @@ +237,5 @@
> > +        } else {
> > +           mSolo.clickOnText("Cancel");
> > +           return false;
> > +        }
> > +    }
> 
> both of these functions would make a good library function for other tests
> to use (not needed for this bug, but please file a followup bug)

Follow-up bug logged - bug 820276


Please check in the patch as I do not have the permissions to do that.

Comment 4

[Joel Maher ( :jmaher ) (UTC-4)]

I was not able to run this test locally:
0 INFO SimpleTest START
1 INFO TEST-START | testBookmarksTab
2 INFO TEST-PASS | testBookmarksTab | Inserted at:  - content://org.mozilla.fennec_jmaher.db.browser/bookmarks/10?profile=default
3 INFO TEST-PASS | testBookmarksTab | checking that bookmarks list exists - android.widget.ListView@443830f0
4 INFO TEST-UNEXPECTED-FAIL | testBookmarksTab | bookmarks list has 5 children (defaults + a folder) - got 0, expected 5
Exception caught during test!
junit.framework.AssertionFailedError: 4 INFO TEST-UNEXPECTED-FAIL | testBookmarksTab | bookmarks list has 5 children (defaults + a folder) - got 0, expected 5
	at junit.framework.Assert.fail(Assert.java:47)
	at org.mozilla.fennec_jmaher.FennecMochitestAssert._logMochitestResult(FennecMochitestAssert.java:107)
	at org.mozilla.fennec_jmaher.FennecMochitestAssert.ok(FennecMochitestAssert.java:136)
	at org.mozilla.fennec_jmaher.FennecMochitestAssert.is(FennecMochitestAssert.java:142)
	at org.mozilla.fennec_jmaher.tests.testBookmarksTab.testList(testBookmarksTab.java:65)
	at org.mozilla.fennec_jmaher.tests.testBookmarksTab.testBookmarksTab(testBookmarksTab.java:52)
	at java.lang.reflect.Method.invokeNative(Native Method)
	at java.lang.reflect.Method.invoke(Method.java:521)
	at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:204)
	at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:194)
	at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:186)
	at org.mozilla.fennec_jmaher.tests.BaseTest.runTest(BaseTest.java:124)
	at junit.framework.TestCase.runBare(TestCase.java:127)
	at junit.framework.TestResult$1.protect(TestResult.java:106)
	at junit.framework.TestResult.runProtected(TestResult.java:124)
	at junit.framework.TestResult.run(TestResult.java:109)
	at junit.framework.TestCase.run(TestCase.java:118)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:169)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:154)
	at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:520)
	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1447)
5 INFO TEST-UNEXPECTED-FAIL | testBookmarksTab | Exception caught - junit.framework.AssertionFailedError: 4 INFO TEST-UNEXPECTED-FAIL | testBookmarksTab | bookmarks list has 5 children (defaults + a folder) - got 0, expected 5
6 INFO TEST-END | testBookmarksTab | finished in 80041ms
7 INFO TEST-START | Shutdown
8 INFO Passed: 2
9 INFO Failed: 2
10 INFO Todo: 0

Comment 5

[Adrian Tamas (:AdrianT)]

I did not modify that part of the original bookmark test. I believe that Bug 817414 already covers this. Should I try and look into this?

Comment 6

[Joel Maher ( :jmaher ) (UTC-4)]

After a dozen runs I couldn't get this to pass locally or on try server, so I can't land it quite yet.  We need to get bug 817414 fixed first.

Comment 7

[Adrian Tamas (:AdrianT)]

Joel, can you please retest this now that bug 817414 has been fixed?

Comment 8

[Joel Maher ( :jmaher ) (UTC-4)]

this is what I get now:
FIRE PROC: '"MOZ_CRASHREPORTER=1,XPCOM_DEBUG_BREAK=stack,MOZ_HIDE_RESULTS_TABLE=1,MOZ_CRASHREPORTER_NO_REPORT=1,NO_EM_RESTART=1,MOZ_PROCESS_LOG=/tmp/tmpkzErS_pidlog,XPCOM_MEM_BLOAT_LOG=/tmp/tmpt0w6Lz/runtests_leaks.log" am instrument -w -e deviceroot /mnt/sdcard/tests -e class org.mozilla.fennec_jmaher.tests.testBookmarksTab org.mozilla.roboexample.test/org.mozilla.fennec_jmaher.FennecInstrumentationTestRunner'
Robocop derived process name: org.mozilla.fennec_jmaher
INFO | automation.py | Application pid: 0
0 INFO SimpleTest START
1 INFO TEST-START | testBookmarksTab
2 INFO TEST-PASS | testBookmarksTab | Inserted at:  - content://org.mozilla.fennec_jmaher.db.browser/bookmarks/10?profile=default
3 INFO TEST-PASS | testBookmarksTab | checking that bookmarks list exists - android.widget.ListView@44306820 should not equal null
4 INFO TEST-PASS | testBookmarksTab | bookmarks list has 5 children (defaults + a folder) - 5 should equal 5
5 INFO TEST-PASS | testBookmarksTab | TextView is filled in - Firefox: Support
6 INFO TEST-PASS | testBookmarksTab | TextView is filled in - http://support.mozilla.org/en-US/mobile
7 INFO TEST-PASS | testBookmarksTab | Correct number of ImageViews visible - 1 should equal 1
8 INFO TEST-PASS | testBookmarksTab | Correct number of TextViews visible - 2 should equal 2
9 INFO TEST-PASS | testBookmarksTab | TextView is filled in - Firefox: Customize with add-ons
10 INFO TEST-PASS | testBookmarksTab | TextView is filled in - https://addons.mozilla.org/en-US/android/
11 INFO TEST-PASS | testBookmarksTab | Correct number of ImageViews visible - 1 should equal 1
12 INFO TEST-PASS | testBookmarksTab | Correct number of TextViews visible - 2 should equal 2
13 INFO TEST-PASS | testBookmarksTab | TextView is filled in - Firefox Start
14 INFO TEST-PASS | testBookmarksTab | TextView is filled in - about:home
15 INFO TEST-PASS | testBookmarksTab | Correct number of ImageViews visible - 1 should equal 1
16 INFO TEST-PASS | testBookmarksTab | Correct number of TextViews visible - 2 should equal 2
17 INFO TEST-PASS | testBookmarksTab | TextView is filled in - Firefox: About your browser
18 INFO TEST-PASS | testBookmarksTab | TextView is filled in - about:firefox
19 INFO TEST-PASS | testBookmarksTab | Correct number of ImageViews visible - 1 should equal 1
20 INFO TEST-PASS | testBookmarksTab | Correct number of TextViews visible - 2 should equal 2
21 INFO TEST-PASS | testBookmarksTab | TextView is filled in - Desktop Bookmarks
22 INFO TEST-PASS | testBookmarksTab | Correct number of ImageViews visible - 1 should equal 1
23 INFO TEST-PASS | testBookmarksTab | Correct number of TextViews visible - 1 should equal 1
24 INFO TEST-PASS | testBookmarksTab | desktop folder has correct number of children - 4 should equal 4
25 INFO TEST-PASS | testBookmarksTab | TextView is filled in - Unsorted Bookmarks
26 INFO TEST-PASS | testBookmarksTab | Correct number of ImageViews visible - 1 should equal 1
27 INFO TEST-PASS | testBookmarksTab | Correct number of TextViews visible - 1 should equal 1
28 INFO TEST-PASS | testBookmarksTab | TextView is filled in - Bookmarks Menu
29 INFO TEST-PASS | testBookmarksTab | Correct number of ImageViews visible - 1 should equal 1
30 INFO TEST-PASS | testBookmarksTab | Correct number of TextViews visible - 1 should equal 1
31 INFO TEST-PASS | testBookmarksTab | TextView is filled in - Bookmarks Toolbar
32 INFO TEST-PASS | testBookmarksTab | Correct number of ImageViews visible - 1 should equal 1
33 INFO TEST-PASS | testBookmarksTab | Correct number of TextViews visible - 1 should equal 1
34 INFO TEST-PASS | testBookmarksTab | TextView is filled in - Desktop Bookmarks
35 INFO TEST-PASS | testBookmarksTab | Correct number of ImageViews visible - 0 should equal 0
36 INFO TEST-PASS | testBookmarksTab | Correct number of TextViews visible - 1 should equal 1
37 INFO TEST-PASS | testBookmarksTab | toolbar folder has correct number of children - 2 should equal 2
38 INFO TEST-PASS | testBookmarksTab | TextView is filled in - BOOKMARK_TITLE
39 INFO TEST-PASS | testBookmarksTab | TextView is filled in - http://www.example.com
40 INFO TEST-PASS | testBookmarksTab | Correct number of ImageViews visible - 1 should equal 1
41 INFO TEST-PASS | testBookmarksTab | Correct number of TextViews visible - 2 should equal 2
42 INFO TEST-PASS | testBookmarksTab | TextView is filled in - Bookmarks Toolbar
43 INFO TEST-PASS | testBookmarksTab | Correct number of ImageViews visible - 0 should equal 0
44 INFO TEST-PASS | testBookmarksTab | Correct number of TextViews visible - 1 should equal 1
45 INFO TEST-PASS | testBookmarksTab | Back moved up one level - 
46 INFO TEST-PASS | testBookmarksTab | Back moved up one level - 
47 INFO TEST-PASS | testBookmarksTab | found TextView: - Firefox: About your browser
48 INFO TEST-PASS | testBookmarksTab | found TextView: - about:firefox
clickLongOnView: android.widget.RelativeLayout@44433708
49 INFO TEST-PASS | testBookmarksTab | Context menu has New Tab option - Open in New Tab
50 INFO TEST-PASS | testBookmarksTab | Context menu has Share option - Share
51 INFO TEST-PASS | testBookmarksTab | Context menu has Edit option - Edit
52 INFO TEST-PASS | testBookmarksTab | Context menu has Remove option - Remove
53 INFO TEST-PASS | testBookmarksTab | Context menu has Add to Home Screen option - Add to Home Screen
54 INFO TEST-PASS | testBookmarksTab | Folders have no context menu - false should equal false
Exception caught during test!
junit.framework.AssertionFailedError: The text: Edit is not found!
	at junit.framework.Assert.fail(Assert.java:47)
	at junit.framework.Assert.assertTrue(Assert.java:20)
	at com.jayway.android.robotium.solo.Clicker.clickOnText(Clicker.java:291)
	at com.jayway.android.robotium.solo.Solo.clickOnText(Solo.java:823)
	at org.mozilla.fennec_jmaher.tests.testBookmarksTab.editBookmark(testBookmarksTab.java:207)
	at org.mozilla.fennec_jmaher.tests.testBookmarksTab.testContextMenu(testBookmarksTab.java:160)
	at org.mozilla.fennec_jmaher.tests.testBookmarksTab.testBookmarksTab(testBookmarksTab.java:52)
	at java.lang.reflect.Method.invokeNative(Native Method)
	at java.lang.reflect.Method.invoke(Method.java:521)
	at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:204)
	at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:194)
	at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:186)
	at org.mozilla.fennec_jmaher.tests.BaseTest.runTest(BaseTest.java:130)
	at junit.framework.TestCase.runBare(TestCase.java:127)
	at junit.framework.TestResult$1.protect(TestResult.java:106)
	at junit.framework.TestResult.runProtected(TestResult.java:124)
	at junit.framework.TestResult.run(TestResult.java:109)
	at junit.framework.TestCase.run(TestCase.java:118)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:169)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:154)
	at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:520)
	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1447)
55 INFO TEST-UNEXPECTED-FAIL | testBookmarksTab | Exception caught - junit.framework.AssertionFailedError: The text: Edit is not found!
56 INFO TEST-END | testBookmarksTab | finished in 122209ms
57 INFO TEST-START | Shutdown
58 INFO Passed: 53
59 INFO Failed: 1
60 INFO Todo: 0
61 INFO SimpleTest FINISHED
INFO | automation.py | Application ran for: 0:02:06.885968
INFO | automation.py | Reading PID log: /tmp/tmpkzErS_pidlog
getting files in '/mnt/sdcard/tests/profile/minidumps/'

really close!

Comment 9

[Adrian Tamas (:AdrianT)]

Attachment: Edit and Remove bookmark from the context menu v2

Removed the calls to press "Back" in order to close the vkb since now when the user enters the Bookmarks tab the vkb is dismissed by default.
Also changed the patch to get the Bookmarks list once since when building the ListView it seems that the order is not always the same and getChildAt(1) would select different bookmarks not always the first one.

Comment 10

[Joel Maher ( :jmaher ) (UTC-4)]

Comment on attachment 701822 [details] [diff] [review]
Edit and Remove bookmark from the context menu v2

Review of attachment 701822 [details] [diff] [review]:
-----------------------------------------------------------------

this looks good, I will send it to try server as and land it if it works great.  Locally it passed.

Comment 11

[Joel Maher ( :jmaher ) (UTC-4)]

this has failures on try server:
https://tbpl.mozilla.org/?tree=Try&rev=e0b581d23864

53 INFO TEST-PASS | testBookmarksTab | Context menu has Add to Home Screen option - Add to Home Screen
54 INFO TEST-PASS | testBookmarksTab | Folders have no context menu - false should equal false
55 INFO TEST-UNEXPECTED-FAIL | testBookmarksTab | Bookmark Name was changed - got false, expected true
Exception caught during test!
junit.framework.AssertionFailedError: 55 INFO TEST-UNEXPECTED-FAIL | testBookmarksTab | Bookmark Name was changed - got false, expected true
	at junit.framework.Assert.fail(Assert.java:47)
	at org.mozilla.fennec.FennecMochitestAssert._logMochitestResult(FennecMochitestAssert.java:107)
	at org.mozilla.fennec.FennecMochitestAssert.ok(FennecMochitestAssert.java:136)
	at org.mozilla.fennec.FennecMochitestAssert.is(FennecMochitestAssert.java:142)
	at org.mozilla.fennec.tests.testBookmarksTab.testContextMenu(testBookmarksTab.java:157)
	at org.mozilla.fennec.tests.testBookmarksTab.testBookmarksTab(testBookmarksTab.java:46)
	at java.lang.reflect.Method.invokeNative(Native Method)
	at java.lang.reflect.Method.invoke(Method.java:521)
	at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:204)
	at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:194)
	at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:186)
	at org.mozilla.fennec.tests.BaseTest.runTest(BaseTest.java:129)
	at junit.framework.TestCase.runBare(TestCase.java:127)
	at junit.framework.TestResult$1.protect(TestResult.java:106)
	at junit.framework.TestResult.runProtected(TestResult.java:124)
	at junit.framework.TestResult.run(TestResult.java:109)
	at junit.framework.TestCase.run(TestCase.java:118)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:169)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:154)
	at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:520)
	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1447)
56 INFO TEST-UNEXPECTED-FAIL | testBookmarksTab | Exception caught - junit.framework.AssertionFailedError: 55 INFO TEST-UNEXPECTED-FAIL | testBookmarksTab | Bookmark Name was changed - got false, expected true
57 INFO TEST-END | testBookmarksTab | finished in 83371ms
58 INFO TEST-START | Shutdown
59 INFO Passed: 53
60 INFO Failed: 2
61 INFO Todo: 0

Comment 12

[Adrian Tamas (:AdrianT)]

Attachment: Edit and Remove bookmark from the context menu v2.1

I am unable to reproduce the fail at all in about 20+ tries on an HTC Desire running Android 2.2.
Best guess at a fix for this: making sure the Edit Bookmark pop-up is opened before checking the text and closing the vkb after exiting Bookmark Edit pop-up to ensure that the hole bookmark list is visible.

Comment 13

[Joel Maher ( :jmaher ) (UTC-4)]

Comment on attachment 702319 [details] [diff] [review]
Edit and Remove bookmark from the context menu v2.1

sorry to say this, but using the latest patch still causes me to fail on a production tegra.

Comment 14

[Adrian Tamas (:AdrianT)]

Attachment: Edit and Remove bookmark from the context menu v3

I believe I have identified the issue. On Gingerbread the keyboard is opened when entering the text in the edit bookmark pop-up but on ICS it's not so I added an OS specific restriction.
Also I am clearing the fields before adding the text in order to make sure there are no issues with the text.
Tested this version of the patch on the HTC Desire (Android 2.2), Samsung Galaxy Tab 2 7.0 (Android 4.0.4) and I managed to get a Kindle Fire for half an hour to test on. The Kindle has a custom Android build but it is based on the 2.3 version so it may be somewhat relevant and comparable to the Tegra boards.

Comment 15

[Joel Maher ( :jmaher ) (UTC-4)]

Comment on attachment 704526 [details] [diff] [review]
Edit and Remove bookmark from the context menu v3

Review of attachment 704526 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks for adding this.  Looks great.

Comment 16

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/0770079f15c7