Closed Bug 1144874 Opened 11 years ago Closed 10 years ago

Robocop PixelTests fail on Android 4.3 emulator

Categories

(Firefox for Android Graveyard :: Testing, defect)

Product:
Firefox for Android Graveyard
Component:
Testing
Platform:
x86_64
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox48 fixed)

Status:
RESOLVED FIXED
Milestone:
Firefox 48
Tracking Flags:
Tracking Status
firefox48 --- fixed

People

(Reporter: gbrown, Assigned: gbrown)

References

Details

On the Android 4.3 emulator, all of the robocop PixelTest tests fail consistently. http://ftp.mozilla.org/pub/mozilla.org/mobile/try-builds/gbrown@mozilla.com-e80c0fdd1ef6/try-android-api-9/try_ubuntu64_vm_mobile_test-robocop-1-bm114-tests1-linux64-build7.txt.gz 18:07:28 INFO - TEST-START | testFlingCorrectness 18:07:28 INFO - TEST-PASS | testFlingCorrectness | Robocop tests need the test device screen to be powered on. - 18:07:28 INFO - TEST-PASS | testFlingCorrectness | Given message occurred for registered event: {"type":"Gecko:Ready"} - Gecko:Ready should equal Gecko:Ready 18:07:28 INFO - EventExpecter: no longer listening for Gecko:Ready 18:07:28 INFO - TEST-PASS | testFlingCorrectness | waiting for urlbar text to gain focus - urlbar text gained focus 18:07:28 INFO - TEST-PASS | testFlingCorrectness | URL typed properly - http://mochi.test:8888/tests/robocop/robocop_boxes.html should equal http://mochi.test:8888/tests/robocop/robocop_boxes.html 18:07:28 INFO - TEST-PASS | testFlingCorrectness | Given message occurred for registered event: {"errorType":"","bgColor":"transparent","type":"DOMContentLoaded","tabID":0} - DOMContentLoaded should equal DOMContentLoaded 18:07:28 INFO - EventExpecter: no longer listening for DOMContentLoaded 18:07:28 INFO - PaintExpecter: no longer listening for events 18:07:28 INFO - TEST-PASS | testFlingCorrectness | Pixel at 0, 0 - Color rgba(0,0,0,255) close enough to expected rgb(0,0,0) 18:07:28 INFO - TEST-PASS | testFlingCorrectness | Pixel at 100, 0 - Color rgba(32,100,0,255) close enough to expected rgb(32,100,0) 18:07:28 INFO - TEST-PASS | testFlingCorrectness | Pixel at 0, 100 - Color rgba(0,100,32,255) close enough to expected rgb(0,100,32) 18:07:28 INFO - TEST-PASS | testFlingCorrectness | Pixel at 100, 100 - Color rgba(32,200,32,255) close enough to expected rgb(32,200,32) 18:07:28 INFO - PaintExpecter: no longer listening for events 18:07:28 WARNING - TEST-UNEXPECTED-FAIL | testFlingCorrectness | Pixel at 0, 0 - Color rgba(0,100,32,255) not close enough to expected rgb(0,200,64) 18:07:28 INFO - 7 ERROR Exception caught during test! - junit.framework.AssertionFailedError: TEST-UNEXPECTED-FAIL | testFlingCorrectness | Pixel at 0, 0 - Color rgba(0,100,32,255) not close enough to expected rgb(0,200,64) 18:07:28 INFO - at junit.framework.Assert.fail(Assert.java:50) 18:07:28 INFO - at org.mozilla.gecko.FennecMochitestAssert._logMochitestResult(FennecMochitestAssert.java:128) 18:07:28 INFO - at org.mozilla.gecko.FennecMochitestAssert.ok(FennecMochitestAssert.java:150) 18:07:28 INFO - at org.mozilla.gecko.FennecMochitestAssert.ispixel(FennecMochitestAssert.java:170) 18:07:28 INFO - at org.mozilla.gecko.tests.PixelTest.checkScrollWithBoxes(PixelTest.java:111) 18:07:28 INFO - at org.mozilla.gecko.tests.testFlingCorrectness.testFlingCorrectness(testFlingCorrectness.java:35) 18:07:28 INFO - at java.lang.reflect.Method.invokeNative(Native Method) 18:07:28 INFO - at java.lang.reflect.Method.invoke(Method.java:525) 18:07:28 INFO - at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214) 18:07:28 INFO - at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199) 18:07:28 INFO - at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192) 18:07:28 INFO - at org.mozilla.gecko.tests.BaseTest.runTest(BaseTest.java:132) 18:07:28 INFO - at junit.framework.TestCase.runBare(TestCase.java:134) 18:07:28 INFO - at junit.framework.TestResult$1.protect(TestResult.java:115) 18:07:28 INFO - at junit.framework.TestResult.runProtected(TestResult.java:133) 18:07:28 INFO - at junit.framework.TestResult.run(TestResult.java:118) 18:07:28 INFO - at junit.framework.TestCase.run(TestCase.java:124) 18:07:28 INFO - at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:191) 18:07:28 INFO - at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:176) 18:07:28 INFO - at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:554) 18:07:28 INFO - at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1701) 18:07:28 INFO - 18:07:28 WARNING - TEST-UNEXPECTED-FAIL | testFlingCorrectness | Exception caught - junit.framework.AssertionFailedError: TEST-UNEXPECTED-FAIL | testFlingCorrectness | Pixel at 0, 0 - Color rgba(0,100,32,255) not close enough to expected rgb(0,200,64) 18:07:28 INFO - TEST-OK | testFlingCorrectness | took 143203ms 18:07:28 INFO - TEST-START | Shutdown 18:07:28 INFO - Passed: 9 18:07:28 INFO - Failed: 2
Whiteboard: [test disabled on android 4.3]
http://ftp.mozilla.org/pub/mozilla.org/mobile/try-builds/gbrown@mozilla.com-e80c0fdd1ef6/try-android-api-9/try_ubuntu64_vm_mobile_test-robocop-2-bm67-tests1-linux64-build73.txt.gz 17:55:43 INFO - TEST-START | testPanCorrectness 17:55:43 INFO - TEST-PASS | testPanCorrectness | Robocop tests need the test device screen to be powered on. - 17:55:43 INFO - TEST-PASS | testPanCorrectness | Given message occurred for registered event: {"type":"Gecko:Ready"} - Gecko:Ready should equal Gecko:Ready 17:55:43 INFO - EventExpecter: no longer listening for Gecko:Ready 17:55:43 INFO - TEST-PASS | testPanCorrectness | waiting for urlbar text to gain focus - urlbar text gained focus 17:55:43 INFO - TEST-PASS | testPanCorrectness | URL typed properly - http://mochi.test:8888/tests/robocop/robocop_boxes.html should equal http://mochi.test:8888/tests/robocop/robocop_boxes.html 17:55:43 INFO - TEST-PASS | testPanCorrectness | Given message occurred for registered event: {"errorType":"","bgColor":"transparent","type":"DOMContentLoaded","tabID":0} - DOMContentLoaded should equal DOMContentLoaded 17:55:43 INFO - EventExpecter: no longer listening for DOMContentLoaded 17:55:43 INFO - PaintExpecter: no longer listening for events 17:55:43 INFO - TEST-PASS | testPanCorrectness | Pixel at 0, 0 - Color rgba(0,0,0,255) close enough to expected rgb(0,0,0) 17:55:43 INFO - TEST-PASS | testPanCorrectness | Pixel at 100, 0 - Color rgba(32,100,0,255) close enough to expected rgb(32,100,0) 17:55:43 INFO - TEST-PASS | testPanCorrectness | Pixel at 0, 100 - Color rgba(0,100,32,255) close enough to expected rgb(0,100,32) 17:55:43 INFO - TEST-PASS | testPanCorrectness | Pixel at 100, 100 - Color rgba(32,200,32,255) close enough to expected rgb(32,200,32) 17:55:43 INFO - PaintExpecter: no longer listening for events 17:55:43 WARNING - TEST-UNEXPECTED-FAIL | testPanCorrectness | Pixel at 0, 0 - Color rgba(0,0,0,255) not close enough to expected rgb(0,100,32) 17:55:43 INFO - 3 ERROR Exception caught during test! - junit.framework.AssertionFailedError: TEST-UNEXPECTED-FAIL | testPanCorrectness | Pixel at 0, 0 - Color rgba(0,0,0,255) not close enough to expected rgb(0,100,32) 17:55:43 INFO - at junit.framework.Assert.fail(Assert.java:50) 17:55:43 INFO - at org.mozilla.gecko.FennecMochitestAssert._logMochitestResult(FennecMochitestAssert.java:128) 17:55:43 INFO - at org.mozilla.gecko.FennecMochitestAssert.ok(FennecMochitestAssert.java:150) 17:55:43 INFO - at org.mozilla.gecko.FennecMochitestAssert.ispixel(FennecMochitestAssert.java:170) 17:55:43 INFO - at org.mozilla.gecko.tests.PixelTest.checkScrollWithBoxes(PixelTest.java:111) 17:55:43 INFO - at org.mozilla.gecko.tests.testPanCorrectness.testPanCorrectness(testPanCorrectness.java:33) 17:55:43 INFO - at java.lang.reflect.Method.invokeNative(Native Method) 17:55:43 INFO - at java.lang.reflect.Method.invoke(Method.java:525) 17:55:43 INFO - at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214) 17:55:43 INFO - at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199) 17:55:43 INFO - at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192) 17:55:43 INFO - at org.mozilla.gecko.tests.BaseTest.runTest(BaseTest.java:132) 17:55:43 INFO - at junit.framework.TestCase.runBare(TestCase.java:134) 17:55:43 INFO - at junit.framework.TestResult$1.protect(TestResult.java:115) 17:55:43 INFO - at junit.framework.TestResult.runProtected(TestResult.java:133) 17:55:43 INFO - at junit.framework.TestResult.run(TestResult.java:118) 17:55:43 INFO - at junit.framework.TestCase.run(TestCase.java:124) 17:55:43 INFO - at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:191) 17:55:43 INFO - at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:176) 17:55:43 INFO - at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:554) 17:55:43 INFO - at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1701) 17:55:43 INFO - 17:55:43 WARNING - TEST-UNEXPECTED-FAIL | testPanCorrectness | Exception caught - junit.framework.AssertionFailedError: TEST-UNEXPECTED-FAIL | testPanCorrectness | Pixel at 0, 0 - Color rgba(0,0,0,255) not close enough to expected rgb(0,100,32) 17:55:43 INFO - TEST-OK | testPanCorrectness | took 129909ms 17:55:43 INFO - TEST-START | Shutdown 17:55:43 INFO - Passed: 9 17:55:43 INFO - Failed: 2
http://ftp.mozilla.org/pub/mozilla.org/mobile/try-builds/gbrown@mozilla.com-e80c0fdd1ef6/try-android-api-9/try_ubuntu64_vm_mobile_test-robocop-1-bm114-tests1-linux64-build7.txt.gz 10:42:49 INFO - TEST-START | testAxisLocking 10:42:49 INFO - TEST-PASS | testAxisLocking | Robocop tests need the test device screen to be powered on. - 10:42:49 INFO - TEST-PASS | testAxisLocking | Given message occurred for registered event: {"type":"Gecko:Ready"} - Gecko:Ready should equal Gecko:Ready 10:42:49 INFO - EventExpecter: no longer listening for Gecko:Ready 10:42:49 INFO - TEST-PASS | testAxisLocking | waiting for urlbar text to gain focus - urlbar text gained focus 10:42:49 INFO - TEST-PASS | testAxisLocking | URL typed properly - http://mochi.test:8888/tests/robocop/robocop_boxes.html should equal http://mochi.test:8888/tests/robocop/robocop_boxes.html 10:42:49 INFO - TEST-PASS | testAxisLocking | Given message occurred for registered event: {"errorType":"","bgColor":"transparent","type":"DOMContentLoaded","tabID":0} - DOMContentLoaded should equal DOMContentLoaded 10:42:49 INFO - EventExpecter: no longer listening for DOMContentLoaded 10:42:49 INFO - PaintExpecter: no longer listening for events 10:42:49 INFO - TEST-PASS | testAxisLocking | Pixel at 0, 0 - Color rgba(0,0,0,255) close enough to expected rgb(0,0,0) 10:42:49 INFO - TEST-PASS | testAxisLocking | Pixel at 100, 0 - Color rgba(32,100,0,255) close enough to expected rgb(32,100,0) 10:42:49 INFO - TEST-PASS | testAxisLocking | Pixel at 0, 100 - Color rgba(0,100,32,255) close enough to expected rgb(0,100,32) 10:42:49 INFO - TEST-PASS | testAxisLocking | Pixel at 100, 100 - Color rgba(32,200,32,255) close enough to expected rgb(32,200,32) 10:42:49 INFO - PaintExpecter: no longer listening for events 10:42:49 WARNING - TEST-UNEXPECTED-FAIL | testAxisLocking | Pixel at 0, 0 - Color rgba(0,0,0,255) not close enough to expected rgb(0,100,32) 10:42:49 INFO - 3 ERROR Exception caught during test! - junit.framework.AssertionFailedError: TEST-UNEXPECTED-FAIL | testAxisLocking | Pixel at 0, 0 - Color rgba(0,0,0,255) not close enough to expected rgb(0,100,32) 10:42:49 INFO - at junit.framework.Assert.fail(Assert.java:50) 10:42:49 INFO - at org.mozilla.gecko.FennecMochitestAssert._logMochitestResult(FennecMochitestAssert.java:128) 10:42:49 INFO - at org.mozilla.gecko.FennecMochitestAssert.ok(FennecMochitestAssert.java:150) 10:42:49 INFO - at org.mozilla.gecko.FennecMochitestAssert.ispixel(FennecMochitestAssert.java:170) 10:42:49 INFO - at org.mozilla.gecko.tests.PixelTest.checkScrollWithBoxes(PixelTest.java:111) 10:42:49 INFO - at org.mozilla.gecko.tests.testAxisLocking.testAxisLocking(testAxisLocking.java:36) 10:42:49 INFO - at java.lang.reflect.Method.invokeNative(Native Method) 10:42:49 INFO - at java.lang.reflect.Method.invoke(Method.java:525) 10:42:49 INFO - at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214) 10:42:49 INFO - at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199) 10:42:49 INFO - at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192) 10:42:49 INFO - at org.mozilla.gecko.tests.BaseTest.runTest(BaseTest.java:132) 10:42:49 INFO - at junit.framework.TestCase.runBare(TestCase.java:134) 10:42:49 INFO - at junit.framework.TestResult$1.protect(TestResult.java:115) 10:42:49 INFO - at junit.framework.TestResult.runProtected(TestResult.java:133) 10:42:49 INFO - at junit.framework.TestResult.run(TestResult.java:118) 10:42:49 INFO - at junit.framework.TestCase.run(TestCase.java:124) 10:42:49 INFO - at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:191) 10:42:49 INFO - at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:176) 10:42:49 INFO - at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:554) 10:42:49 INFO - at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1701) 10:42:49 INFO - 10:42:49 WARNING - TEST-UNEXPECTED-FAIL | testAxisLocking | Exception caught - junit.framework.AssertionFailedError: TEST-UNEXPECTED-FAIL | testAxisLocking | Pixel at 0, 0 - Color rgba(0,0,0,255) not close enough to expected rgb(0,100,32) 10:42:49 INFO - TEST-OK | testAxisLocking | took 136752ms 10:42:49 INFO - TEST-START | Shutdown 10:42:49 INFO - Passed: 9 10:42:49 INFO - Failed: 2
testLoad seems to pass reliably.
This came up while I was discussing stuff with rbarker today. I suspect the failures on 4.3 are a result of the screen having a higher DPI than the test expects, so the page doesn't scroll as far as the test is expecting.
Assignee: nobody → gbrown
Blocks: 1252961
These don't seem to be failing now: https://treeherder.mozilla.org/#/jobs?repo=try&revision=c85e0f0f718b I don't know why/what has changed.
https://hg.mozilla.org/mozilla-central/rev/154d6073509b
Status: NEW → RESOLVED
Closed: 10 years ago
status-firefox48: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → Firefox 48
Whiteboard: [test disabled on android 4.3]
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in before you can comment on or make changes to this bug.