Closed Bug 1144874 Opened 9 years ago Closed 8 years ago

Robocop PixelTests fail on Android 4.3 emulator

Categories

(Firefox for Android Graveyard :: Testing, defect)

Product:
Firefox for Android Graveyard
Component:
Testing
Platform:
x86_64
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox48 fixed)

Status:
RESOLVED FIXED
Milestone:
Firefox 48
Tracking Flags:
Tracking Status
firefox48 --- fixed

People

(Reporter: gbrown, Assigned: gbrown)

References

Details

On the Android 4.3 emulator, all of the robocop PixelTest tests fail consistently.

http://ftp.mozilla.org/pub/mozilla.org/mobile/try-builds/gbrown@mozilla.com-e80c0fdd1ef6/try-android-api-9/try_ubuntu64_vm_mobile_test-robocop-1-bm114-tests1-linux64-build7.txt.gz

18:07:28     INFO -  TEST-START | testFlingCorrectness
18:07:28     INFO -  TEST-PASS | testFlingCorrectness | Robocop tests need the test device screen to be powered on. - 
18:07:28     INFO -  TEST-PASS | testFlingCorrectness | Given message occurred for registered event: {"type":"Gecko:Ready"} - Gecko:Ready should equal Gecko:Ready
18:07:28     INFO -  EventExpecter: no longer listening for Gecko:Ready
18:07:28     INFO -  TEST-PASS | testFlingCorrectness | waiting for urlbar text to gain focus - urlbar text gained focus
18:07:28     INFO -  TEST-PASS | testFlingCorrectness | URL typed properly - http://mochi.test:8888/tests/robocop/robocop_boxes.html should equal http://mochi.test:8888/tests/robocop/robocop_boxes.html
18:07:28     INFO -  TEST-PASS | testFlingCorrectness | Given message occurred for registered event: {"errorType":"","bgColor":"transparent","type":"DOMContentLoaded","tabID":0} - DOMContentLoaded should equal DOMContentLoaded
18:07:28     INFO -  EventExpecter: no longer listening for DOMContentLoaded
18:07:28     INFO -  PaintExpecter: no longer listening for events
18:07:28     INFO -  TEST-PASS | testFlingCorrectness | Pixel at 0, 0 - Color rgba(0,0,0,255)  close enough to expected rgb(0,0,0)
18:07:28     INFO -  TEST-PASS | testFlingCorrectness | Pixel at 100, 0 - Color rgba(32,100,0,255)  close enough to expected rgb(32,100,0)
18:07:28     INFO -  TEST-PASS | testFlingCorrectness | Pixel at 0, 100 - Color rgba(0,100,32,255)  close enough to expected rgb(0,100,32)
18:07:28     INFO -  TEST-PASS | testFlingCorrectness | Pixel at 100, 100 - Color rgba(32,200,32,255)  close enough to expected rgb(32,200,32)
18:07:28     INFO -  PaintExpecter: no longer listening for events
18:07:28  WARNING -  TEST-UNEXPECTED-FAIL | testFlingCorrectness | Pixel at 0, 0 - Color rgba(0,100,32,255) not close enough to expected rgb(0,200,64)
18:07:28     INFO -  7 ERROR Exception caught during test! - junit.framework.AssertionFailedError: TEST-UNEXPECTED-FAIL | testFlingCorrectness | Pixel at 0, 0 - Color rgba(0,100,32,255) not close enough to expected rgb(0,200,64)
18:07:28     INFO -  	at junit.framework.Assert.fail(Assert.java:50)
18:07:28     INFO -  	at org.mozilla.gecko.FennecMochitestAssert._logMochitestResult(FennecMochitestAssert.java:128)
18:07:28     INFO -  	at org.mozilla.gecko.FennecMochitestAssert.ok(FennecMochitestAssert.java:150)
18:07:28     INFO -  	at org.mozilla.gecko.FennecMochitestAssert.ispixel(FennecMochitestAssert.java:170)
18:07:28     INFO -  	at org.mozilla.gecko.tests.PixelTest.checkScrollWithBoxes(PixelTest.java:111)
18:07:28     INFO -  	at org.mozilla.gecko.tests.testFlingCorrectness.testFlingCorrectness(testFlingCorrectness.java:35)
18:07:28     INFO -  	at java.lang.reflect.Method.invokeNative(Native Method)
18:07:28     INFO -  	at java.lang.reflect.Method.invoke(Method.java:525)
18:07:28     INFO -  	at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
18:07:28     INFO -  	at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
18:07:28     INFO -  	at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192)
18:07:28     INFO -  	at org.mozilla.gecko.tests.BaseTest.runTest(BaseTest.java:132)
18:07:28     INFO -  	at junit.framework.TestCase.runBare(TestCase.java:134)
18:07:28     INFO -  	at junit.framework.TestResult$1.protect(TestResult.java:115)
18:07:28     INFO -  	at junit.framework.TestResult.runProtected(TestResult.java:133)
18:07:28     INFO -  	at junit.framework.TestResult.run(TestResult.java:118)
18:07:28     INFO -  	at junit.framework.TestCase.run(TestCase.java:124)
18:07:28     INFO -  	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:191)
18:07:28     INFO -  	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:176)
18:07:28     INFO -  	at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:554)
18:07:28     INFO -  	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1701)
18:07:28     INFO -  
18:07:28  WARNING -  TEST-UNEXPECTED-FAIL | testFlingCorrectness | Exception caught - junit.framework.AssertionFailedError: TEST-UNEXPECTED-FAIL | testFlingCorrectness | Pixel at 0, 0 - Color rgba(0,100,32,255) not close enough to expected rgb(0,200,64)
18:07:28     INFO -  TEST-OK | testFlingCorrectness | took 143203ms
18:07:28     INFO -  TEST-START | Shutdown
18:07:28     INFO -  Passed: 9
18:07:28     INFO -  Failed: 2
Whiteboard: [test disabled on android 4.3]
http://ftp.mozilla.org/pub/mozilla.org/mobile/try-builds/gbrown@mozilla.com-e80c0fdd1ef6/try-android-api-9/try_ubuntu64_vm_mobile_test-robocop-2-bm67-tests1-linux64-build73.txt.gz

17:55:43     INFO -  TEST-START | testPanCorrectness
17:55:43     INFO -  TEST-PASS | testPanCorrectness | Robocop tests need the test device screen to be powered on. - 
17:55:43     INFO -  TEST-PASS | testPanCorrectness | Given message occurred for registered event: {"type":"Gecko:Ready"} - Gecko:Ready should equal Gecko:Ready
17:55:43     INFO -  EventExpecter: no longer listening for Gecko:Ready
17:55:43     INFO -  TEST-PASS | testPanCorrectness | waiting for urlbar text to gain focus - urlbar text gained focus
17:55:43     INFO -  TEST-PASS | testPanCorrectness | URL typed properly - http://mochi.test:8888/tests/robocop/robocop_boxes.html should equal http://mochi.test:8888/tests/robocop/robocop_boxes.html
17:55:43     INFO -  TEST-PASS | testPanCorrectness | Given message occurred for registered event: {"errorType":"","bgColor":"transparent","type":"DOMContentLoaded","tabID":0} - DOMContentLoaded should equal DOMContentLoaded
17:55:43     INFO -  EventExpecter: no longer listening for DOMContentLoaded
17:55:43     INFO -  PaintExpecter: no longer listening for events
17:55:43     INFO -  TEST-PASS | testPanCorrectness | Pixel at 0, 0 - Color rgba(0,0,0,255)  close enough to expected rgb(0,0,0)
17:55:43     INFO -  TEST-PASS | testPanCorrectness | Pixel at 100, 0 - Color rgba(32,100,0,255)  close enough to expected rgb(32,100,0)
17:55:43     INFO -  TEST-PASS | testPanCorrectness | Pixel at 0, 100 - Color rgba(0,100,32,255)  close enough to expected rgb(0,100,32)
17:55:43     INFO -  TEST-PASS | testPanCorrectness | Pixel at 100, 100 - Color rgba(32,200,32,255)  close enough to expected rgb(32,200,32)
17:55:43     INFO -  PaintExpecter: no longer listening for events
17:55:43  WARNING -  TEST-UNEXPECTED-FAIL | testPanCorrectness | Pixel at 0, 0 - Color rgba(0,0,0,255) not close enough to expected rgb(0,100,32)
17:55:43     INFO -  3 ERROR Exception caught during test! - junit.framework.AssertionFailedError: TEST-UNEXPECTED-FAIL | testPanCorrectness | Pixel at 0, 0 - Color rgba(0,0,0,255) not close enough to expected rgb(0,100,32)
17:55:43     INFO -  	at junit.framework.Assert.fail(Assert.java:50)
17:55:43     INFO -  	at org.mozilla.gecko.FennecMochitestAssert._logMochitestResult(FennecMochitestAssert.java:128)
17:55:43     INFO -  	at org.mozilla.gecko.FennecMochitestAssert.ok(FennecMochitestAssert.java:150)
17:55:43     INFO -  	at org.mozilla.gecko.FennecMochitestAssert.ispixel(FennecMochitestAssert.java:170)
17:55:43     INFO -  	at org.mozilla.gecko.tests.PixelTest.checkScrollWithBoxes(PixelTest.java:111)
17:55:43     INFO -  	at org.mozilla.gecko.tests.testPanCorrectness.testPanCorrectness(testPanCorrectness.java:33)
17:55:43     INFO -  	at java.lang.reflect.Method.invokeNative(Native Method)
17:55:43     INFO -  	at java.lang.reflect.Method.invoke(Method.java:525)
17:55:43     INFO -  	at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
17:55:43     INFO -  	at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
17:55:43     INFO -  	at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192)
17:55:43     INFO -  	at org.mozilla.gecko.tests.BaseTest.runTest(BaseTest.java:132)
17:55:43     INFO -  	at junit.framework.TestCase.runBare(TestCase.java:134)
17:55:43     INFO -  	at junit.framework.TestResult$1.protect(TestResult.java:115)
17:55:43     INFO -  	at junit.framework.TestResult.runProtected(TestResult.java:133)
17:55:43     INFO -  	at junit.framework.TestResult.run(TestResult.java:118)
17:55:43     INFO -  	at junit.framework.TestCase.run(TestCase.java:124)
17:55:43     INFO -  	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:191)
17:55:43     INFO -  	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:176)
17:55:43     INFO -  	at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:554)
17:55:43     INFO -  	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1701)
17:55:43     INFO -  
17:55:43  WARNING -  TEST-UNEXPECTED-FAIL | testPanCorrectness | Exception caught - junit.framework.AssertionFailedError: TEST-UNEXPECTED-FAIL | testPanCorrectness | Pixel at 0, 0 - Color rgba(0,0,0,255) not close enough to expected rgb(0,100,32)
17:55:43     INFO -  TEST-OK | testPanCorrectness | took 129909ms
17:55:43     INFO -  TEST-START | Shutdown
17:55:43     INFO -  Passed: 9
17:55:43     INFO -  Failed: 2
http://ftp.mozilla.org/pub/mozilla.org/mobile/try-builds/gbrown@mozilla.com-e80c0fdd1ef6/try-android-api-9/try_ubuntu64_vm_mobile_test-robocop-1-bm114-tests1-linux64-build7.txt.gz

10:42:49     INFO -  TEST-START | testAxisLocking
10:42:49     INFO -  TEST-PASS | testAxisLocking | Robocop tests need the test device screen to be powered on. - 
10:42:49     INFO -  TEST-PASS | testAxisLocking | Given message occurred for registered event: {"type":"Gecko:Ready"} - Gecko:Ready should equal Gecko:Ready
10:42:49     INFO -  EventExpecter: no longer listening for Gecko:Ready
10:42:49     INFO -  TEST-PASS | testAxisLocking | waiting for urlbar text to gain focus - urlbar text gained focus
10:42:49     INFO -  TEST-PASS | testAxisLocking | URL typed properly - http://mochi.test:8888/tests/robocop/robocop_boxes.html should equal http://mochi.test:8888/tests/robocop/robocop_boxes.html
10:42:49     INFO -  TEST-PASS | testAxisLocking | Given message occurred for registered event: {"errorType":"","bgColor":"transparent","type":"DOMContentLoaded","tabID":0} - DOMContentLoaded should equal DOMContentLoaded
10:42:49     INFO -  EventExpecter: no longer listening for DOMContentLoaded
10:42:49     INFO -  PaintExpecter: no longer listening for events
10:42:49     INFO -  TEST-PASS | testAxisLocking | Pixel at 0, 0 - Color rgba(0,0,0,255)  close enough to expected rgb(0,0,0)
10:42:49     INFO -  TEST-PASS | testAxisLocking | Pixel at 100, 0 - Color rgba(32,100,0,255)  close enough to expected rgb(32,100,0)
10:42:49     INFO -  TEST-PASS | testAxisLocking | Pixel at 0, 100 - Color rgba(0,100,32,255)  close enough to expected rgb(0,100,32)
10:42:49     INFO -  TEST-PASS | testAxisLocking | Pixel at 100, 100 - Color rgba(32,200,32,255)  close enough to expected rgb(32,200,32)
10:42:49     INFO -  PaintExpecter: no longer listening for events
10:42:49  WARNING -  TEST-UNEXPECTED-FAIL | testAxisLocking | Pixel at 0, 0 - Color rgba(0,0,0,255) not close enough to expected rgb(0,100,32)
10:42:49     INFO -  3 ERROR Exception caught during test! - junit.framework.AssertionFailedError: TEST-UNEXPECTED-FAIL | testAxisLocking | Pixel at 0, 0 - Color rgba(0,0,0,255) not close enough to expected rgb(0,100,32)
10:42:49     INFO -  	at junit.framework.Assert.fail(Assert.java:50)
10:42:49     INFO -  	at org.mozilla.gecko.FennecMochitestAssert._logMochitestResult(FennecMochitestAssert.java:128)
10:42:49     INFO -  	at org.mozilla.gecko.FennecMochitestAssert.ok(FennecMochitestAssert.java:150)
10:42:49     INFO -  	at org.mozilla.gecko.FennecMochitestAssert.ispixel(FennecMochitestAssert.java:170)
10:42:49     INFO -  	at org.mozilla.gecko.tests.PixelTest.checkScrollWithBoxes(PixelTest.java:111)
10:42:49     INFO -  	at org.mozilla.gecko.tests.testAxisLocking.testAxisLocking(testAxisLocking.java:36)
10:42:49     INFO -  	at java.lang.reflect.Method.invokeNative(Native Method)
10:42:49     INFO -  	at java.lang.reflect.Method.invoke(Method.java:525)
10:42:49     INFO -  	at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
10:42:49     INFO -  	at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
10:42:49     INFO -  	at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192)
10:42:49     INFO -  	at org.mozilla.gecko.tests.BaseTest.runTest(BaseTest.java:132)
10:42:49     INFO -  	at junit.framework.TestCase.runBare(TestCase.java:134)
10:42:49     INFO -  	at junit.framework.TestResult$1.protect(TestResult.java:115)
10:42:49     INFO -  	at junit.framework.TestResult.runProtected(TestResult.java:133)
10:42:49     INFO -  	at junit.framework.TestResult.run(TestResult.java:118)
10:42:49     INFO -  	at junit.framework.TestCase.run(TestCase.java:124)
10:42:49     INFO -  	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:191)
10:42:49     INFO -  	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:176)
10:42:49     INFO -  	at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:554)
10:42:49     INFO -  	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1701)
10:42:49     INFO -  
10:42:49  WARNING -  TEST-UNEXPECTED-FAIL | testAxisLocking | Exception caught - junit.framework.AssertionFailedError: TEST-UNEXPECTED-FAIL | testAxisLocking | Pixel at 0, 0 - Color rgba(0,0,0,255) not close enough to expected rgb(0,100,32)
10:42:49     INFO -  TEST-OK | testAxisLocking | took 136752ms
10:42:49     INFO -  TEST-START | Shutdown
10:42:49     INFO -  Passed: 9
10:42:49     INFO -  Failed: 2
testLoad seems to pass reliably.
This came up while I was discussing stuff with rbarker today. I suspect the failures on 4.3 are a result of the screen having a higher DPI than the test expects, so the page doesn't scroll as far as the test is expecting.
Assignee: nobody → gbrown
Blocks: 1252961
These don't seem to be failing now:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c85e0f0f718b

I don't know why/what has changed.
https://hg.mozilla.org/mozilla-central/rev/154d6073509b
Status: NEW → RESOLVED
Closed: 8 years ago
status-firefox48: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → Firefox 48
Whiteboard: [test disabled on android 4.3]
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in before you can comment on or make changes to this bug.