898036 - Robocop pixel tests fail consistently on Android x86 emulator and many devices

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: General, defect)

Description

[Geoff Brown [:gbrown]]

The Robocop pixel tests -- testLoad, testAxisLocking, etc -- fail consistently when run on the x86 emulator. They also fail locally when run on a Motorola Razr i (XT890). The failures look the same as the intermittent failures in bug 851861.

Comment 1

[Geoff Brown [:gbrown]]

0 INFO SimpleTest START
1 INFO TEST-START | testLoad
EventExpecter: no longer listening for Gecko:Ready
2 INFO TEST-PASS | testLoad | Awesomebar URL typed properly - http://mochi.test:8888/tests/robocop/robocop_boxes.html should equal http://mochi.test:8888/tests/robocop/robocop_boxes.html
EventExpecter: no longer listening for DOMContentLoaded
PaintExpecter: no longer listening for events
3 INFO TEST-PASS | testLoad | Pixel at 0, 0 - Color rgba(0,0,0,255)  close enough to expected rgb(0,0,0)
4 INFO TEST-UNEXPECTED-FAIL | testLoad | Pixel at 100, 0 - Color rgba(0,0,0,255) not close enough to expected rgb(32,100,0)
Exception caught during test!
junit.framework.AssertionFailedError: 4 INFO TEST-UNEXPECTED-FAIL | testLoad | Pixel at 100, 0 - Color rgba(0,0,0,255) not close enough to expected rgb(32,100,0)
        at junit.framework.Assert.fail(Assert.java:50)
        at org.mozilla.fennec.FennecMochitestAssert._logMochitestResult(FennecMochitestAssert.java:107)
        at org.mozilla.fennec.FennecMochitestAssert.ok(FennecMochitestAssert.java:136)
        at org.mozilla.fennec.FennecMochitestAssert.ispixel(FennecMochitestAssert.java:156)
        at org.mozilla.fennec.tests.PixelTest.checkScrollWithBoxes(PixelTest.java:92)
        at org.mozilla.fennec.tests.PixelTest.loadAndVerifyBoxes(PixelTest.java:107)
        at org.mozilla.fennec.tests.testLoad.testLoad(testLoad.java:21)
        at java.lang.reflect.Method.invokeNative(Native Method)
        at java.lang.reflect.Method.invoke(Method.java:511)
        at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
        at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
        at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192)
        at org.mozilla.fennec.tests.BaseTest.runTest(BaseTest.java:133)
        at junit.framework.TestCase.runBare(TestCase.java:134)
        at junit.framework.TestResult$1.protect(TestResult.java:115)
        at junit.framework.TestResult.runProtected(TestResult.java:133)
        at junit.framework.TestResult.run(TestResult.java:118)
        at junit.framework.TestCase.run(TestCase.java:124)
        at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:190)
        at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:175)
        at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:555)
        at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1584)
5 INFO TEST-UNEXPECTED-FAIL | testLoad | Exception caught - junit.framework.AssertionFailedError: 4 INFO TEST-UNEXPECTED-FAIL | testLoad | Pixel at 100, 0 - Color rgba(0,0,0,255) not close enough to expected rgb(32,100,0)
6 INFO TEST-END | testLoad | finished in 12081ms
7 INFO TEST-START | Shutdown
8 INFO Passed: 2
9 INFO Failed: 2
10 INFO Todo: 0
11 INFO SimpleTest FINISHED

Comment 2

[Geoff Brown [:gbrown]]

Attachment: pixels.map

/mnt/sdcard/tests/pixels.map, from failed testLoad on razr

Comment 3

[Geoff Brown [:gbrown]]

Actually, testLoad fails for me on local runs on any phone...how are these tests passing (intermittently) on tbpl?

Comment 4

[Geoff Brown [:gbrown]]

This looks just like bug 784242, which we resolved by setting:

browser.viewport.scaleRatio=100

The test harness still sets browser.viewport.scaleRatio, but it looks like that pref is no longer used -- changed in bug 803207.

:kats -- how should this work now?

Comment 5

[Kartikaya Gupta (email:[e-mail])]

Ah, sorry, I should have remembered that the tests use that pref. That pref should be replaced with this one:

layout.css.devPixelsPerPx=1.0

Comment 6

[Geoff Brown [:gbrown]]

Thanks :kats!

I originally reported on irc that this made no difference. The problem was that I was creating:

user_pref("layout.css.devPixelsPerPx", 1.0);

instead of:

user_pref("layout.css.devPixelsPerPx", "1.0");

This works great now.

Comment 7

[Geoff Brown [:gbrown]]

Attachment: set pref layout.css.devPixelsPerPx="1.0" during robocop tests

Comment 8

[Kartikaya Gupta (email:[e-mail])]

Comment on attachment 781371 [details] [diff] [review]
set pref layout.css.devPixelsPerPx="1.0" during robocop tests

Review of attachment 781371 [details] [diff] [review]:
-----------------------------------------------------------------

Cool, thanks! I also didn't realize it needed to be quoted but it makes sense in retrospect. All of our float prefs are actually strings, or integers divided by 1000, or some such.

Comment 9

[Geoff Brown [:gbrown]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/9b8b2c5d0d26

Comment 10

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/9b8b2c5d0d26

Comment 11

[Geoff Brown [:gbrown]]

Attachment: testLoad-razr-fixed.png

Screenshot from Motorola razr i, after fix applied (passing test).