Robocop pixel tests fail consistently on Android x86 emulator and many devices

RESOLVED FIXED in Firefox 25

Status

()

defect
RESOLVED FIXED
6 years ago
6 years ago

People

(Reporter: gbrown, Assigned: gbrown)

Tracking

unspecified
Firefox 25
x86
Android
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(3 attachments)

Assignee

Description

6 years ago
The Robocop pixel tests -- testLoad, testAxisLocking, etc -- fail consistently when run on the x86 emulator. They also fail locally when run on a Motorola Razr i (XT890). The failures look the same as the intermittent failures in bug 851861.
Assignee

Comment 1

6 years ago
0 INFO SimpleTest START
1 INFO TEST-START | testLoad
EventExpecter: no longer listening for Gecko:Ready
2 INFO TEST-PASS | testLoad | Awesomebar URL typed properly - http://mochi.test:8888/tests/robocop/robocop_boxes.html should equal http://mochi.test:8888/tests/robocop/robocop_boxes.html
EventExpecter: no longer listening for DOMContentLoaded
PaintExpecter: no longer listening for events
3 INFO TEST-PASS | testLoad | Pixel at 0, 0 - Color rgba(0,0,0,255)  close enough to expected rgb(0,0,0)
4 INFO TEST-UNEXPECTED-FAIL | testLoad | Pixel at 100, 0 - Color rgba(0,0,0,255) not close enough to expected rgb(32,100,0)
Exception caught during test!
junit.framework.AssertionFailedError: 4 INFO TEST-UNEXPECTED-FAIL | testLoad | Pixel at 100, 0 - Color rgba(0,0,0,255) not close enough to expected rgb(32,100,0)
        at junit.framework.Assert.fail(Assert.java:50)
        at org.mozilla.fennec.FennecMochitestAssert._logMochitestResult(FennecMochitestAssert.java:107)
        at org.mozilla.fennec.FennecMochitestAssert.ok(FennecMochitestAssert.java:136)
        at org.mozilla.fennec.FennecMochitestAssert.ispixel(FennecMochitestAssert.java:156)
        at org.mozilla.fennec.tests.PixelTest.checkScrollWithBoxes(PixelTest.java:92)
        at org.mozilla.fennec.tests.PixelTest.loadAndVerifyBoxes(PixelTest.java:107)
        at org.mozilla.fennec.tests.testLoad.testLoad(testLoad.java:21)
        at java.lang.reflect.Method.invokeNative(Native Method)
        at java.lang.reflect.Method.invoke(Method.java:511)
        at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
        at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
        at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192)
        at org.mozilla.fennec.tests.BaseTest.runTest(BaseTest.java:133)
        at junit.framework.TestCase.runBare(TestCase.java:134)
        at junit.framework.TestResult$1.protect(TestResult.java:115)
        at junit.framework.TestResult.runProtected(TestResult.java:133)
        at junit.framework.TestResult.run(TestResult.java:118)
        at junit.framework.TestCase.run(TestCase.java:124)
        at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:190)
        at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:175)
        at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:555)
        at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1584)
5 INFO TEST-UNEXPECTED-FAIL | testLoad | Exception caught - junit.framework.AssertionFailedError: 4 INFO TEST-UNEXPECTED-FAIL | testLoad | Pixel at 100, 0 - Color rgba(0,0,0,255) not close enough to expected rgb(32,100,0)
6 INFO TEST-END | testLoad | finished in 12081ms
7 INFO TEST-START | Shutdown
8 INFO Passed: 2
9 INFO Failed: 2
10 INFO Todo: 0
11 INFO SimpleTest FINISHED
See Also: → 851861
Assignee

Comment 2

6 years ago
Posted file pixels.map
/mnt/sdcard/tests/pixels.map, from failed testLoad on razr
Assignee

Comment 3

6 years ago
Actually, testLoad fails for me on local runs on any phone...how are these tests passing (intermittently) on tbpl?
Assignee

Comment 4

6 years ago
This looks just like bug 784242, which we resolved by setting:

browser.viewport.scaleRatio=100

The test harness still sets browser.viewport.scaleRatio, but it looks like that pref is no longer used -- changed in bug 803207.

:kats -- how should this work now?
Flags: needinfo?(bugmail.mozilla)
Ah, sorry, I should have remembered that the tests use that pref. That pref should be replaced with this one:

layout.css.devPixelsPerPx=1.0
Flags: needinfo?(bugmail.mozilla)
Assignee

Comment 6

6 years ago
Thanks :kats!

I originally reported on irc that this made no difference. The problem was that I was creating:

user_pref("layout.css.devPixelsPerPx", 1.0);

instead of:

user_pref("layout.css.devPixelsPerPx", "1.0");

This works great now.
Assignee

Updated

6 years ago
Summary: Robocop pixel tests fail consistently on Android x86 emulators → Robocop pixel tests fail consistently on Android x86 emulator and many devices
Comment on attachment 781371 [details] [diff] [review]
set pref layout.css.devPixelsPerPx="1.0" during robocop tests

Review of attachment 781371 [details] [diff] [review]:
-----------------------------------------------------------------

Cool, thanks! I also didn't realize it needed to be quoted but it makes sense in retrospect. All of our float prefs are actually strings, or integers divided by 1000, or some such.
Attachment #781371 - Flags: review?(bugmail.mozilla) → review+
https://hg.mozilla.org/mozilla-central/rev/9b8b2c5d0d26
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 25
Assignee

Comment 11

6 years ago
Screenshot from Motorola razr i, after fix applied (passing test).
Assignee

Updated

6 years ago
Blocks: 927476
Assignee

Updated

6 years ago
Blocks: 936226
You need to log in before you can comment on or make changes to this bug.