837318 - testVkbOverlap - Found 0 green pixels after tapping; expected 2427 -- on Pandaboard

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: General, defect)

Description

[Geoff Brown [:gbrown]]

Robocop's testVkbOverlap fails consistently on pandaboards:

0 INFO SimpleTest START
1 INFO TEST-START | testVkbOverlap
2 INFO TEST-PASS | testVkbOverlap | Awesomebar URL typed properly - http://mochi.test:8888/tests/robocop/robocop_input.html should equal http://mochi.test:8888/tests/robocop/robocop_input.html
3 INFO TEST-PASS | testVkbOverlap | testInputVisible - Found 2427 green pixels after scrolling
4 INFO TEST-UNEXPECTED-FAIL | testVkbOverlap | testVkbOverlap - Found 0 green pixels after tapping; expected 2427
Exception caught during test!
junit.framework.AssertionFailedError: 4 INFO TEST-UNEXPECTED-FAIL | testVkbOverlap | testVkbOverlap - Found 0 green pixels after tapping; expected 2427
	at junit.framework.Assert.fail(Assert.java:47)
	at org.mozilla.fennec.FennecMochitestAssert._logMochitestResult(FennecMochitestAssert.java:107)
	at org.mozilla.fennec.FennecMochitestAssert.ok(FennecMochitestAssert.java:136)
	at org.mozilla.fennec.tests.testVkbOverlap.testVkbOverlap(testVkbOverlap.java:68)
	at java.lang.reflect.Method.invokeNative(Native Method)
	at java.lang.reflect.Method.invoke(Method.java:511)
	at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
	at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
	at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192)
	at org.mozilla.fennec.tests.BaseTest.runTest(BaseTest.java:132)
	at junit.framework.TestCase.runBare(TestCase.java:127)
	at junit.framework.TestResult$1.protect(TestResult.java:106)
	at junit.framework.TestResult.runProtected(TestResult.java:124)
	at junit.framework.TestResult.run(TestResult.java:109)
	at junit.framework.TestCase.run(TestCase.java:118)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:169)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:154)
	at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:545)
	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1551)
5 INFO TEST-UNEXPECTED-FAIL | testVkbOverlap | Exception caught - junit.framework.AssertionFailedError: 4 INFO TEST-UNEXPECTED-FAIL | testVkbOverlap | testVkbOverlap - Found 0 green pixels after tapping; expected 2427
6 INFO TEST-END | testVkbOverlap | finished in 29508ms

For an example, see https://tbpl.mozilla.org/php/getParsedLog.php?id=19365119&tree=Try&full=1#error3

and consider this segment of logcat:

2-01 12:06:51.296 I/Robocop ( 4717): 3 INFO TEST-PASS | testVkbOverlap | testInputVisible - Found 2427 green pixels after scrolling
02-01 12:06:51.296 D/RobocopMotionEventHelper( 4717): Triggering down at (5.0,611.0)
02-01 12:06:51.296 D/RobocopMotionEventHelper( 4717): Triggering up at (5.0,611.0)
02-01 12:06:51.312 D/Robocop ( 4717): Received drawFinished notification
02-01 12:06:51.664 D/Robocop ( 4717): Received drawFinished notification
...
02-01 12:06:52.429 D/GeckoLayerClient( 4717): Window-size changed to (1280,236)
02-01 12:06:52.429 D/Robocop ( 4717): Received drawFinished notification
02-01 12:06:52.492 D/Robocop ( 4717): Received drawFinished notification
...
02-01 12:06:58.109 I/Robocop ( 4717): 4 INFO TEST-UNEXPECTED-FAIL | testVkbOverlap | testVkbOverlap - Found 0 green pixels after tapping; expected 2427

Why would this happen:

   Window-size changed to (1280,236)

??


(There was some discussion of this failure in bug 823165, and there is a tegra bug, similar but apparently different cause at bug 814282.)

Comment 1

[Geoff Brown [:gbrown]]

This fails the same on my local pandaboard, and logcat shows the same window size change to 1280x236. I added logging and checked the dimensions of the PaintedSurface in countGreenPixels; the first time the dimensions are 616x1280 (success/2427 green pixels) but the second time the dimensions are 236x1280 (failure/0 green pixels).

Comment 2

[Geoff Brown [:gbrown]]

Attachment: screenshot after initial loadAndPaint

Comment 3

[Geoff Brown [:gbrown]]

Attachment: screenshot after meh.dragSync() + waitForPaint()

Comment 4

[Geoff Brown [:gbrown]]

Attachment: screenshot at time of failure (after meh.tap(), sleep, and blockUntilClear)

These don't look right to me at all!

Comment 5

[Geoff Brown [:gbrown]]

(In reply to Geoff Brown [:gbrown] from comment #0)
> Why would this happen:
> 
>    Window-size changed to (1280,236)
> 
> ??

Running on a Galaxy Tab, in landscape mode, very similar behavior can be observed visually. There is a similar window size change, caused by the VKB coming up. It fully obscures the input field, causing the same test failure.

Comment 6

[Kartikaya Gupta (email:[e-mail])]

Your screenshots don't look right at all, agreed. How are you grabbing those? If you have the panda hooked up a physical display, is that what you see as well?

The window height shrinking to 236 is expected since the VKB is supposed to pop up. According to the log from comment 0 everything seems to be working fine from the test side; looks like a legit bug in Fennec.

Comment 7

[Geoff Brown [:gbrown]]

I'm taking the screenshots with mSolo.takeScreenshot(). Maybe they cannot be trusted...I haven't used that before on a pandaboard. I have never connected a physical display to a panda.

Comment 8

[Kartikaya Gupta (email:[e-mail])]

Sorry; I posted through a mid-air collision on my last comment when I shouldn't have. And yeah, I think robotium can't take a screenshot of the GL surface properly, which is why it just shows black. If it's reproducible on a galaxy tab though it should be debuggable. I can take a look next week if you don't figure it out by then.

Comment 9

[Geoff Brown [:gbrown]]

I repeated the test on a Galaxy Nexus in portrait mode: the input field was repositioned when the VKB came up and the test passed.

I repeated the test on a Galaxy Nexus in *landscape* mode: the input field was NOT repositioned when the VKB came up and the test failed.

Aha!

Comment 10

[Geoff Brown [:gbrown]]

With bugs 839199/830760 fixed, this test is working much better on Pandas...but is not quite passing yet!

https://tbpl.mozilla.org/php/getParsedLog.php?id=19752362&tree=Mozilla-Inbound&full=1 seems typical: 

0 INFO SimpleTest START
1 INFO TEST-START | testVkbOverlap
2 INFO TEST-PASS | testVkbOverlap | Awesomebar URL typed properly - http://mochi.test:8888/tests/robocop/test_viewport.sjs?metadata=initial-scale%3D1.0%2C%20user-scalable%3Dno should equal http://mochi.test:8888/tests/robocop/test_viewport.sjs?metadata=initial-scale%3D1.0%2C%20user-scalable%3Dno
3 INFO TEST-PASS | testVkbOverlap | testInputVisible - Found 2427 green pixels after scrolling
4 INFO TEST-PASS | testVkbOverlap | testVkbOverlap - Found 2278 green pixels after tapping; expected 2427
5 INFO TEST-PASS | testVkbOverlap | Awesomebar URL typed properly - http://mochi.test:8888/tests/robocop/test_viewport.sjs?metadata=initial-scale%3D1.0 should equal http://mochi.test:8888/tests/robocop/test_viewport.sjs?metadata=initial-scale%3D1.0
6 INFO TEST-PASS | testVkbOverlap | testInputVisible - Found 2427 green pixels after scrolling
7 INFO TEST-PASS | testVkbOverlap | testVkbOverlap - Found 2278 green pixels after tapping; expected 2427
8 INFO TEST-PASS | testVkbOverlap | Awesomebar URL typed properly - http://mochi.test:8888/tests/robocop/test_viewport.sjs?metadata= should equal http://mochi.test:8888/tests/robocop/test_viewport.sjs?metadata=
9 INFO TEST-PASS | testVkbOverlap | testInputVisible - Found 4115 green pixels after scrolling
10 INFO TEST-UNEXPECTED-FAIL | testVkbOverlap | testVkbOverlap - Found 3972 green pixels after tapping; expected 4115
Exception caught during test!
junit.framework.AssertionFailedError: 10 INFO TEST-UNEXPECTED-FAIL | testVkbOverlap | testVkbOverlap - Found 3972 green pixels after tapping; expected 4115
	at junit.framework.Assert.fail(Assert.java:47)
	at org.mozilla.fennec.FennecMochitestAssert._logMochitestResult(FennecMochitestAssert.java:107)
	at org.mozilla.fennec.FennecMochitestAssert.ok(FennecMochitestAssert.java:136)
	at org.mozilla.fennec.tests.testVkbOverlap.testSetup(testVkbOverlap.java:77)
	at org.mozilla.fennec.tests.testVkbOverlap.testVkbOverlap(testVkbOverlap.java:28)
	at java.lang.reflect.Method.invokeNative(Native Method)
	at java.lang.reflect.Method.invoke(Method.java:511)
	at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
	at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
	at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192)
	at org.mozilla.fennec.tests.BaseTest.runTest(BaseTest.java:130)
	at junit.framework.TestCase.runBare(TestCase.java:127)
	at junit.framework.TestResult$1.protect(TestResult.java:106)
	at junit.framework.TestResult.runProtected(TestResult.java:124)
	at junit.framework.TestResult.run(TestResult.java:109)
	at junit.framework.TestCase.run(TestCase.java:118)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:169)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:154)
	at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:545)
	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1551)
11 INFO TEST-UNEXPECTED-FAIL | testVkbOverlap | Exception caught - junit.framework.AssertionFailedError: 10 INFO TEST-UNEXPECTED-FAIL | testVkbOverlap | testVkbOverlap - Found 3972 green pixels after tapping; expected 4115
12 INFO TEST-END | testVkbOverlap | finished in 86478ms
13 INFO TEST-START | Shutdown
14 INFO Passed: 8
15 INFO Failed: 2
16 INFO Todo: 0
17 INFO SimpleTest FINISHED

Comment 11

[Geoff Brown [:gbrown]]

On the tegras we get

9 INFO TEST-PASS | testVkbOverlap | testInputVisible - Found 2696 green pixels after scrolling
10 INFO TEST-PASS | testVkbOverlap | testVkbOverlap - Found 37948 green pixels after tapping; expected 2696

Are we not zooming in on the pandas?

Comment 12

[Wesley Johnston (:wesj)]

Do they identify themselves as Tablets? I should probably fix the tests to know that...

Comment 13

[Geoff Brown [:gbrown]]

I bet that's it...I'll try.

Comment 14

[Geoff Brown [:gbrown]]

As seen here, updating the test to not expect zooming-in on tablets fixes the test on pandaboard, but breaks it on tegras! https://tbpl.mozilla.org/?tree=Try&rev=28c305e5eb21&noignore=1

The robocop notion of a tablet is not the same as Fennec's...I'll sort that out.

Comment 15

[Geoff Brown [:gbrown]]

Attachment: update testVkbOverlap and testTabHistory to account for tablets

I changed the way that tablets are detected in BaseTest.Device to use Fennec's code and updated testVkbOverlap to not expect zooming on tablets -- that fixed testVkbOverlap on Pandas and allowed it to continue working on Tegras. But, the Device changes broke testTabHistory, because that test had been treating the Panda as a phone, and its code for tablets was broken.

With the final patch, all tests pass on Tegra and Panda: https://tbpl.mozilla.org/?tree=Try&rev=1faef6660602&noignore=1.

Comment 16

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 714971 [details] [diff] [review]
update testVkbOverlap and testTabHistory to account for tablets

Review of attachment 714971 [details] [diff] [review]:
-----------------------------------------------------------------

thanks for working on this!

::: mobile/android/base/tests/BaseTest.java.in
@@ +643,4 @@
>                      Element reloadBtn = mDriver.findElement(getActivity(), "reload");
>                      reloadBtn.click();
> +                } else {
> +                    mSolo.clickOnText("^Reload$");

why did we switch from Forward to Reload?

::: mobile/android/base/tests/testTabHistory.java.in
@@ +33,5 @@
> +        Navigation nav = new Navigation(device);
> +        mAsserter.dumpLog("device type: "+device.type);
> +        mAsserter.dumpLog("device version: "+device.version);
> +        mAsserter.dumpLog("device width: "+device.width);
> +        mAsserter.dumpLog("device height: "+device.height);

is this just debugging left in?

Comment 17

[Geoff Brown [:gbrown]]

(In reply to Joel Maher (:jmaher) from comment #16)
> Comment on attachment 714971 [details] [diff] [review]
> > +                    mSolo.clickOnText("^Reload$");
> 
> why did we switch from Forward to Reload?

I believe the use of 'Forward' in the reload() function was a copy/paste error dating back to the original implementation of reload(); it was not being exercised previously because of other errors. testTabHistory will fail if we click on "Forward" in reload() -- it goes to page 3, rather than re-loading page 2.

> is this just debugging left in?

Yes, but intentionally. I have found myself wondering "is this device considered a phone or a tablet?" on more than one occasion -- this was a convenient location to log out all of this info.

Comment 18

[Geoff Brown [:gbrown]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/16c9ccf0e50b

Comment 19

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/16c9ccf0e50b