Closed Bug 898036 Opened 7 years ago Closed 7 years ago

Robocop pixel tests fail consistently on Android x86 emulator and many devices

Categories

(Firefox for Android :: General, defect)

x86
Android
defect
Not set
normal

Tracking

()

RESOLVED FIXED
Firefox 25

People

(Reporter: gbrown, Assigned: gbrown)

References

Details

Attachments

(3 files)

The Robocop pixel tests -- testLoad, testAxisLocking, etc -- fail consistently when run on the x86 emulator. They also fail locally when run on a Motorola Razr i (XT890). The failures look the same as the intermittent failures in bug 851861.
0 INFO SimpleTest START
1 INFO TEST-START | testLoad
EventExpecter: no longer listening for Gecko:Ready
2 INFO TEST-PASS | testLoad | Awesomebar URL typed properly - http://mochi.test:8888/tests/robocop/robocop_boxes.html should equal http://mochi.test:8888/tests/robocop/robocop_boxes.html
EventExpecter: no longer listening for DOMContentLoaded
PaintExpecter: no longer listening for events
3 INFO TEST-PASS | testLoad | Pixel at 0, 0 - Color rgba(0,0,0,255)  close enough to expected rgb(0,0,0)
4 INFO TEST-UNEXPECTED-FAIL | testLoad | Pixel at 100, 0 - Color rgba(0,0,0,255) not close enough to expected rgb(32,100,0)
Exception caught during test!
junit.framework.AssertionFailedError: 4 INFO TEST-UNEXPECTED-FAIL | testLoad | Pixel at 100, 0 - Color rgba(0,0,0,255) not close enough to expected rgb(32,100,0)
        at junit.framework.Assert.fail(Assert.java:50)
        at org.mozilla.fennec.FennecMochitestAssert._logMochitestResult(FennecMochitestAssert.java:107)
        at org.mozilla.fennec.FennecMochitestAssert.ok(FennecMochitestAssert.java:136)
        at org.mozilla.fennec.FennecMochitestAssert.ispixel(FennecMochitestAssert.java:156)
        at org.mozilla.fennec.tests.PixelTest.checkScrollWithBoxes(PixelTest.java:92)
        at org.mozilla.fennec.tests.PixelTest.loadAndVerifyBoxes(PixelTest.java:107)
        at org.mozilla.fennec.tests.testLoad.testLoad(testLoad.java:21)
        at java.lang.reflect.Method.invokeNative(Native Method)
        at java.lang.reflect.Method.invoke(Method.java:511)
        at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
        at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
        at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192)
        at org.mozilla.fennec.tests.BaseTest.runTest(BaseTest.java:133)
        at junit.framework.TestCase.runBare(TestCase.java:134)
        at junit.framework.TestResult$1.protect(TestResult.java:115)
        at junit.framework.TestResult.runProtected(TestResult.java:133)
        at junit.framework.TestResult.run(TestResult.java:118)
        at junit.framework.TestCase.run(TestCase.java:124)
        at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:190)
        at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:175)
        at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:555)
        at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1584)
5 INFO TEST-UNEXPECTED-FAIL | testLoad | Exception caught - junit.framework.AssertionFailedError: 4 INFO TEST-UNEXPECTED-FAIL | testLoad | Pixel at 100, 0 - Color rgba(0,0,0,255) not close enough to expected rgb(32,100,0)
6 INFO TEST-END | testLoad | finished in 12081ms
7 INFO TEST-START | Shutdown
8 INFO Passed: 2
9 INFO Failed: 2
10 INFO Todo: 0
11 INFO SimpleTest FINISHED
See Also: → 851861
Attached file pixels.map
/mnt/sdcard/tests/pixels.map, from failed testLoad on razr
Actually, testLoad fails for me on local runs on any phone...how are these tests passing (intermittently) on tbpl?
This looks just like bug 784242, which we resolved by setting:

browser.viewport.scaleRatio=100

The test harness still sets browser.viewport.scaleRatio, but it looks like that pref is no longer used -- changed in bug 803207.

:kats -- how should this work now?
Flags: needinfo?(bugmail.mozilla)
Ah, sorry, I should have remembered that the tests use that pref. That pref should be replaced with this one:

layout.css.devPixelsPerPx=1.0
Flags: needinfo?(bugmail.mozilla)
Thanks :kats!

I originally reported on irc that this made no difference. The problem was that I was creating:

user_pref("layout.css.devPixelsPerPx", 1.0);

instead of:

user_pref("layout.css.devPixelsPerPx", "1.0");

This works great now.
Summary: Robocop pixel tests fail consistently on Android x86 emulators → Robocop pixel tests fail consistently on Android x86 emulator and many devices
Comment on attachment 781371 [details] [diff] [review]
set pref layout.css.devPixelsPerPx="1.0" during robocop tests

Review of attachment 781371 [details] [diff] [review]:
-----------------------------------------------------------------

Cool, thanks! I also didn't realize it needed to be quoted but it makes sense in retrospect. All of our float prefs are actually strings, or integers divided by 1000, or some such.
Attachment #781371 - Flags: review?(bugmail.mozilla) → review+
https://hg.mozilla.org/mozilla-central/rev/9b8b2c5d0d26
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 25
Attached image testLoad-razr-fixed.png
Screenshot from Motorola razr i, after fix applied (passing test).
Blocks: 927476
Blocks: 936226
You need to log in before you can comment on or make changes to this bug.