854872 - Robocop: Verifying if the checkboxes checked/unchecked fails in landscape mode on ICS

Status: RESOLVED INCOMPLETE

(Firefox for Android Graveyard :: General, defect)

Description

[Cristian Nicolae (:xti)]

On tablets running Android 4.x and Panda boards checking the state of the checkboxes from the Settings menu fails if the device is in the landscape orientation. This doesn't happen on Gingerbread devices or on ICS/JB devices in portrait orientation

Running the attached proof of concept Robocop test the following can be observed:

- the first few checkboxes are correctly verified
- the last 2-3 checkboxes are always displayed checked or unchecked or cannot be found at all.

The test verfies the state of each checkbox, clicks it to change its state and then verifies the checkbox again.

Please see the filtered log output of the test:

I/Robocop (23007): 2 INFO TEST-PASS | testCheckBoxes | CheckBox List =  - [Pinch to reflow text, Tell sites not to track me, Remember passwords, Use master password, Show search suggestions, Show product announcements]
I/Robocop (23007): 3 INFO TEST-PASS | testCheckBoxes | Number of checkboxes is:  - 6
I/Robocop (23007): 4 INFO TEST-PASS | testCheckBoxes | Pinch to reflow text is checked =  - false
I/Robocop (23007): 5 INFO TEST-PASS | testCheckBoxes | Pinch to reflow text is checked =  - true
I/Robocop (23007): 6 INFO TEST-PASS | testCheckBoxes | Tell sites not to track me is checked =  - false
I/Robocop (23007): 7 INFO TEST-PASS | testCheckBoxes | Tell sites not to track me is checked =  - true
I/Robocop (23007): 8 INFO TEST-PASS | testCheckBoxes | Remember passwords is checked =  - true
I/Robocop (23007): 9 INFO TEST-PASS | testCheckBoxes | Remember passwords is checked =  - false
I/Robocop (23007): 10 INFO TEST-PASS | testCheckBoxes | Use master password is checked =  - false
I/Robocop (23007): 11 INFO TEST-PASS | testCheckBoxes | Use master password is checked =  - true
I/Robocop (23007): 12 INFO TEST-PASS | testCheckBoxes | Show search suggestions is checked =  - false
I/Robocop (23007): 13 INFO TEST-PASS | testCheckBoxes | Show search suggestions is checked =  - false
I/Robocop (23007): Exception caught during test!
I/Robocop (23007): 	at org.mozilla.fennec_nicolaecristian.tests.testCheckBoxes.testCheckBoxes(testCheckBoxes.java:99)
I/Robocop (23007): 	at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
I/Robocop (23007): 	at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
I/Robocop (23007): 	at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192)
I/Robocop (23007): 	at org.mozilla.fennec_nicolaecristian.tests.BaseTest.runTest(BaseTest.java:128)
I/Robocop (23007): 	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:190)
I/Robocop (23007): 	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:175)
I/Robocop (23007): 	at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:555)
I/Robocop (23007): 14 INFO TEST-UNEXPECTED-FAIL | testCheckBoxes | Exception caught - junit.framework.AssertionFailedError: CheckBox with index 5 is not available!
I/Robocop (23007): 15 INFO TEST-END | testCheckBoxes | finished in 41288ms

Comment 1

[Cristian Nicolae (:xti)]

Attachment: Proof of concept test

Comment 2

[Cristian Nicolae (:xti)]

Here is the filtered test log output in portrait mode:

I/Robocop (23784): 2 INFO TEST-PASS | testCheckBoxes | CheckBox List =  - [Pinch to reflow text, Tell sites not to track me, Remember passwords, Use master password, Show search suggestions, Show product announcements]
I/Robocop (23784): 3 INFO TEST-PASS | testCheckBoxes | Number of checkboxes is:  - 6
I/Robocop (23784): 4 INFO TEST-PASS | testCheckBoxes | Pinch to reflow text is checked =  - false
I/Robocop (23784): 5 INFO TEST-PASS | testCheckBoxes | Pinch to reflow text is checked =  - true
I/Robocop (23784): 6 INFO TEST-PASS | testCheckBoxes | Tell sites not to track me is checked =  - false
I/Robocop (23784): 7 INFO TEST-PASS | testCheckBoxes | Tell sites not to track me is checked =  - true
I/Robocop (23784): 8 INFO TEST-PASS | testCheckBoxes | Remember passwords is checked =  - true
I/Robocop (23784): 9 INFO TEST-PASS | testCheckBoxes | Remember passwords is checked =  - false
I/Robocop (23784): 10 INFO TEST-PASS | testCheckBoxes | Use master password is checked =  - false
I/Robocop (23784): 11 INFO TEST-PASS | testCheckBoxes | Use master password is checked =  - true
I/Robocop (23784): 12 INFO TEST-PASS | testCheckBoxes | Show search suggestions is checked =  - true
I/Robocop (23784): 13 INFO TEST-PASS | testCheckBoxes | Show search suggestions is checked =  - false
I/Robocop (23784): 14 INFO TEST-PASS | testCheckBoxes | Show product announcements is checked =  - false
I/Robocop (23784): 15 INFO TEST-PASS | testCheckBoxes | Show product announcements is checked =  - true
I/Robocop (23784): 16 INFO TEST-END | testCheckBoxes | finished in 32277ms

Comment 3

[Geoff Brown [:gbrown]]

I suspect the issue here is view visibility. In my experience, if a view is not displayed, it may not be available, or it may not be fully populated (not have the expected text, etc). This seems to be a characteristic of the Android framework: presumably the framework is managing/caching views to reduce memory or improve performance, and so a view is only really guaranteed to be available when it is actually displayed.

For the Settings list, not all of the items can be displayed on many devices, even in portrait; but the situation is worse -- fewer items are displayed -- in landscape, increasing the chance of this type of failure.

Robotium has support for automatically scrolling when searching for a view -- many of the Solo functions have versions with a "scroll" parameter, and those without the "scroll" parameter generally default to scroll=true. I had a quick look at the Robotium source, and it looked to me like isCheckBoxChecked would scroll if needed. Maybe that scrolling doesn't work? ...doesn't work in our Settings list? ...??

Comment 4

[Cristian Nicolae (:xti)]

Indeed, mSolo.isCheckBoxChecked() will scroll the page if it's needed and it works as expected. On a 7" tablet, there are checkboxes, both in portrait and landscape, that there aren't in view when the test starts.

However when a checkbox is verified, it is always in view, so I cannot explain what is happening in landscape orientation that makes the test to fail. I recorded a video while I run the test in landscape mode: http://youtu.be/MSRvCdmtjTw

Comment 5

[Geoff Brown [:gbrown]]

Thanks for the video!

Might the problem be related to the special handling of Master Password? If you comment out the Master Password code, does that help? I'm just guessing...

Comment 6

[Cristian Nicolae (:xti)]

(In reply to Geoff Brown [:gbrown] from comment #5)
> Thanks for the video!
> 
> Might the problem be related to the special handling of Master Password? If
> you comment out the Master Password code, does that help? I'm just
> guessing...

I commented the part where I enable the Master Password pref and here is the log output:

I/Robocop ( 9050): 2 INFO TEST-PASS | testCheckBoxes | CheckBox List =  - [Pinch to reflow text, Tell sites not to track me, Remember passwords, Use master password, Show search suggestions, Show product announcements]
I/Robocop ( 9050): 3 INFO TEST-PASS | testCheckBoxes | Number of checkboxes is:  - 6
I/Robocop ( 9050): 4 INFO TEST-PASS | testCheckBoxes | Pinch to reflow text is checked =  - false
I/Robocop ( 9050): 5 INFO TEST-PASS | testCheckBoxes | Pinch to reflow text is checked =  - true
I/Robocop ( 9050): 6 INFO TEST-PASS | testCheckBoxes | Tell sites not to track me is checked =  - false
I/Robocop ( 9050): 7 INFO TEST-PASS | testCheckBoxes | Tell sites not to track me is checked =  - true
I/Robocop ( 9050): 8 INFO TEST-PASS | testCheckBoxes | Remember passwords is checked =  - true
I/Robocop ( 9050): 9 INFO TEST-PASS | testCheckBoxes | Remember passwords is checked =  - false
I/Robocop ( 9050): 10 INFO TEST-PASS | testCheckBoxes | Use master password is checked =  - false
I/Robocop ( 9050): 11 INFO TEST-PASS | testCheckBoxes | Show search suggestions is checked =  - true
I/Robocop ( 9050): 12 INFO TEST-PASS | testCheckBoxes | Show search suggestions is checked =  - true
I/Robocop ( 9050): Exception caught during test!
I/Robocop ( 9050): 	at org.mozilla.fennec_nicolaecristian.tests.testCheckBoxes.testCheckBoxes(testCheckBoxes.java:52)
I/Robocop ( 9050): 	at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
I/Robocop ( 9050): 	at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
I/Robocop ( 9050): 	at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192)
I/Robocop ( 9050): 	at org.mozilla.fennec_nicolaecristian.tests.BaseTest.runTest(BaseTest.java:128)
I/Robocop ( 9050): 	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:190)
I/Robocop ( 9050): 	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:175)
I/Robocop ( 9050): 	at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:555)
I/Robocop ( 9050): 13 INFO TEST-UNEXPECTED-FAIL | testCheckBoxes | Exception caught - junit.framework.AssertionFailedError: CheckBox with index 5 is not available!
I/Robocop ( 9050): 14 INFO TEST-END | testCheckBoxes | finished in 37224ms


Unfortunately, seems to be the same result as in comment #0

Comment 7

[BMO Automation]

We have completed our launch of our new Firefox on Android. The development of the new versions use GitHub for issue tracking. If the bug report still reproduces in a current version of [Firefox on Android nightly](https://play.google.com/store/apps/details?id=org.mozilla.fenix) an issue can be reported at the [Fenix GitHub project](https://github.com/mozilla-mobile/fenix/). If you want to discuss your report please use [Mozilla's chat](https://wiki.mozilla.org/Matrix#Connect_to_Matrix) server https://chat.mozilla.org and join the [#fenix](https://chat.mozilla.org/#/room/#fenix:mozilla.org) channel.