478508 - Some reftests fail consistently on my Linux boxes, with a small number of differing pixels

Status: NEW

(Core :: Layout, defect)

Description

[Daniel Holbert [:dholbert]]

Attachment: reftest failures (with data: urls & links to MXR)

There are 9 reftests that fail consistently on two of my Linux boxes.  I'm attaching the failure log (with the test-links fixed to point to MXR instead of my local test files).

These failures are all few-pixel differences -- they're not noticeable to the naked eye, but they can be detected via the ImageMagick "compare" tool.

I'm using this HG revision of mozilla-central:
  http://hg.mozilla.org/mozilla-central/rev/47b4599e95f0

One of my machines is running Ubuntu 8.10 32-bit, and the other is running Ubuntu 8.10 64-bit.

The failures are in these tests:
http://mxr.mozilla.org/mozilla-central/source/layout/reftests/box-shadow/boxshadow-inner-basic.html 
http://mxr.mozilla.org/mozilla-central/source/layout/reftests/bugs/322461-1.xml 
http://mxr.mozilla.org/mozilla-central/source/layout/reftests/bugs/323656-4.html 
http://mxr.mozilla.org/mozilla-central/source/layout/reftests/bugs/371043-1.html 
http://mxr.mozilla.org/mozilla-central/source/layout/reftests/bugs/390318-1a.html 
http://mxr.mozilla.org/mozilla-central/source/layout/reftests/bugs/391140-1.html 
http://mxr.mozilla.org/mozilla-central/source/layout/reftests/bugs/391994-1.html 
http://mxr.mozilla.org/mozilla-central/source/layout/reftests/bugs/393760-2.xml 
http://mxr.mozilla.org/mozilla-central/source/layout/reftests/pagination/border-breaking-000-cols.xhtml

Comment 1

[Daniel Holbert [:dholbert]]

Attachment: "compare" output for 323656-4.html

As an example, here's the result of running the 323656-4.html screenshots (from the already-attached failure log) through the "compare" tool.

There are 4 pixels that differ slightly on the edge of each "A" character.

Comment 2

[Daniel Holbert [:dholbert]]

Strangely, these tests don't seem to fail when I single them out for testing.

i.e. if I create a custom reftest.list file in the "reftests/bugs" directory and include only the bugs-directory tests that I mentioned in comment 0, then they all pass.

After discovering this, I re-ran the full reftest suite again, to make sure I'm not crazy.  Sure enough, the exact same tests failed once again, when run as part of the full suite.

Comment 3

[Daniel Holbert [:dholbert]]

A few more notes:

I tried running all of the "bugs" reftests (but no others).  This introduced four more consistent failures: 201293-1a, 201293-1b, 201293-1c, and 201293-1d.html.  Aside from that, the list of failing tests didn't change.

Then --- this part is REALLY WEIRD -- I tried removing all tests after 393760-2.xml, which is the last test I'd seen fail.  This change actually **consistently fixed failures in two earlier tests** somehow.  Specifically, it "fixed" these previously-failing tests: 322461-1.xml and 323656-4.html.

I confirmed this by re-running the reftests (the two tests were still fixed), and then undoing the modification I'd made to reftest.list (the two tests became broken once more).

So it appears that the contents of the end of a reftest.list file can affect behavior of tests **higher up** in that reftest.list file....  Maybe this is due to an accidental uninitialized memory read, or something like that.

Comment 4

[Daniel Holbert [:dholbert]]

Attachment: reftest failures on 1.9.1

Here's a similar list of failures for a full reftest run in mozilla-1.9.1 (revision 2e75b379e41f)

Note that 371043-1.html, 393760-2.xml, and border-breaking-000-cols.xhtml are in both this list and the list for mozilla-central (with the same data: urls). The rest of the failures are different.

Comment 5

[Timothy Nikkel (:tnikkel)]

On m-c I'm seeing failures on 371043-1.html, 393760-2.xml, and border-breaking-000-cols.xhtml consistently, along with layout/reftests/svg/filters/filter-filterRes-low-01.svg.