855978 - Robocop: upgrade to robotium-solo-4.0.jar

Status: RESOLVED WONTFIX

(Firefox for Android Graveyard :: General, defect)

Description

[Cristian Nicolae (:xti)]

Robotium 4.0 was released and it has full support for Web!

New functionality includes: waitForWebElement(By by), clickOnWebElement(WebElement webElement), enterTextInWebElement(By by, String text), typeTextInWebElement(By by, String text), clearTextInWebElement(By by), getWebElement(By by, int index), getCurrentViews(Class classToFilterBy), takeScreenshot(String name, int quality) and more.

We should upgrate it from 3.6 to 4.0.

Note: http://www.jayway.com/2013/03/04/robotium-4-0-web-support/

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

can you guys please put a patch together along with a try server run.  I would be happy to see this upgraded.

Comment 2

[Adrian Tamas (:AdrianT)]

Attachment: Add robotium 4.0 and change needed tests

Due to the removal of specific getCurrentX methods I changed all the failures in the tests that used the methods. 
Replaced:
 mSolo.getCurrentListViews with mSolo.getCurrentViews(ListView class)
 mSolo.getCurrentCheckBoxes with mSolo.getCurrentViews(CheckBox class)
 mSolo.getCurrentTextViews(parrent) with mSolo.getCurrentViews(TextView class, parrent)

Tryserver run: https://tbpl.mozilla.org/?tree=Try&rev=27f19cc25cd7

There is only a known intermittent fail on testDoorHangers which I will try to debug separatly and add a fix in bug 832659

Comment 3

[Kartikaya Gupta (email:[e-mail])]

(In reply to Cristian Nicolae (:xti) from comment #0)
> Robotium 4.0 was released and it has full support for Web!
> 
> New functionality includes: waitForWebElement(By by),
> clickOnWebElement(WebElement webElement), enterTextInWebElement(By by,
> String text), typeTextInWebElement(By by, String text),
> clearTextInWebElement(By by), getWebElement(By by, int index),
> getCurrentViews(Class classToFilterBy), takeScreenshot(String name, int
> quality) and more.
> 
> We should upgrate it from 3.6 to 4.0.
> 
> Note: http://www.jayway.com/2013/03/04/robotium-4-0-web-support/

Note that this doesn't help us even a little bit, because it assumes that the app being tested is using an android WebView. We are not.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 731130 [details] [diff] [review]
Add robotium 4.0 and change needed tests

Review of attachment 731130 [details] [diff] [review]:
-----------------------------------------------------------------

the changes here look fine, running some stuff locally it looks great.  I would like to ensure this is a solid green on try server (2 of each test type for tegra and panda) before landing this.

Comment 5

[Adrian Tamas (:AdrianT)]

second run with a fix attempt for the testDoorHanger test causes the fail to move to a different point. Continuing to debug: https://tbpl.mozilla.org/?tree=Try&rev=c272805f4ccd

Comment 6

[Adrian Tamas (:AdrianT)]

Attachment: Add robotium 4.0 and change needed tests v2

For some reason with robotium 4.0 clickOnButton in the doorhanger notifications does not work correctly. It seems that for some reason it does not find "Share" - the first button of the doorhanger, "Don't Allow" - the second button in the doorhanger and "Save" - the first button in the doorhanger. since there is no obvious pattern to the fails, I substituted all the text with button indexes which makes the test pass locally on panda and Froyo/Gingerbread devices. Try run can be found here but I am still awaiting results: https://tbpl.mozilla.org/?tree=Try&rev=a4820d42fb5c

Another solution might be looking for the exact button test. I didn't try this yet but it may work. Also we might want to CC sriram since he changed the doorhangers in bug 847578 and check with him what is the difference between pop-up buttons(since there does not seem to be any issues with those) and doorhanger buttons in order to see if this is a robotium bug and what exactly causes it.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 731869 [details] [diff] [review]
Add robotium 4.0 and change needed tests v2

Review of attachment 731869 [details] [diff] [review]:
-----------------------------------------------------------------

panda board rc1 never turns green.  I have seen it turn green normally at least one out of 4 jobs, so this patch seems to be affecting that.

Comment 8

[Adrian Tamas (:AdrianT)]

On this run https://tbpl.mozilla.org/?tree=Try&rev=27f19cc25cd7 the tests passed rc1 but not rc2 because of the doorhanger fails. On https://tbpl.mozilla.org/?tree=Try&rev=a4820d42fb5c the only change is in the doorhangers test in the second run which should not affect the rc1 run.

Either robotium 4.0 is very unstable which would account for a green run and then fails on the second or the fails on the pandas are an unfortunate coincidence. Although I've seen some discussion about fails to connect to pandas on IRC but I don't know details

Comment 9

[Geoff Brown [:gbrown]]

robotium 4.1 is available now.

Comment 10

[Adrian Tamas (:AdrianT)]

Attachment: Add robotium 4.1 and change needed tests

Did a new patch with robotium 4.1. Now it passes on tegra and also there are no more issues with the doorhangers test but now the intermittent fails on testBookmarksTab seem consistent on pandas:
https://tbpl.mozilla.org/?tree=Try&rev=b6cf1e56c31b

Comment 11

[Joel Maher ( :jmaher ) (UTC -8)]

We should this on hold until we have better control over our robocop failure rates on panda boards (android 4.0).  I would think in another week we will have a better understanding of what we are seeing and possible a fix.

Comment 12

[Adrian Tamas (:AdrianT)]

Tryrun:https://tbpl.mozilla.org/?tree=Try&rev=b4040d53ecc9. 
I see constant fail on Pandas rc1. It seems that there still are issues with robotium 4.1. Should we close this as won't fix?

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

We have enough other issues, our current 3.6 provides us enough functionality, we should approach this topic again in a couple months, maybe 4.2 will be out :)