558910 - Several reftest tests are failing for Fedora unit tests from antialiasing

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Armen [:armenzg]]

REFTEST TEST-UNEXPECTED-PASS | file:///home/cltbld/talos-slave/mozilla-central-fedora-opt-u-reftest/build/reftest/tests/layout/reftests/bidi/bidi-004.html |
REFTEST TEST-UNEXPECTED-PASS | file:///home/cltbld/talos-slave/mozilla-central-fedora-opt-u-reftest/build/reftest/tests/layout/reftests/bidi/bidi-004-j.html |
REFTEST TEST-UNEXPECTED-PASS | file:///home/cltbld/talos-slave/mozilla-central-fedora-opt-u-reftest/build/reftest/tests/layout/reftests/bidi/mirroring-02.html |
REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/talos-slave/mozilla-central-fedora-opt-u-reftest/build/reftest/tests/layout/reftests/box-shadow/boxshadow-inner-basic.html |
REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/talos-slave/mozilla-central-fedora-opt-u-reftest/build/reftest/tests/layout/reftests/bugs/180085-1.html |
REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/talos-slave/mozilla-central-fedora-opt-u-reftest/build/reftest/tests/layout/reftests/bugs/359903-1.html |
REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/talos-slave/mozilla-central-fedora-opt-u-reftest/build/reftest/tests/layout/reftests/bugs/371043-1.html |
REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/talos-slave/mozilla-central-fedora-opt-u-reftest/build/reftest/tests/layout/reftests/bugs/393760-2.xml |
REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/talos-slave/mozilla-central-fedora-opt-u-reftest/build/reftest/tests/layout/reftests/bugs/449171-1.html |
REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/talos-slave/mozilla-central-fedora-opt-u-reftest/build/reftest/tests/layout/reftests/bugs/456219-1c.html |
REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/talos-slave/mozilla-central-fedora-opt-u-reftest/build/reftest/tests/layout/reftests/bugs/482592-1a.xhtml |
REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/talos-slave/mozilla-central-fedora-opt-u-reftest/build/reftest/tests/layout/reftests/bugs/482592-1b.xhtml |
REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/talos-slave/mozilla-central-fedora-opt-u-reftest/build/reftest/tests/layout/reftests/pagination/border-breaking-000-cols.xhtml |
REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/talos-slave/mozilla-central-fedora-opt-u-reftest/build/reftest/tests/layout/reftests/svg/objectBoundingBox-and-fePointLight-01.svg |
REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/talos-slave/mozilla-central-fedora-opt-u-reftest/build/reftest/tests/layout/reftests/text-decoration/underline-block-propagation-2-standards.html |
PROCESS-CRASH | automation.py | application crashed (minidump found)
Thread 1 (crashed)
PROCESS-CRASH | automation.py | application crashed (minidump found)
Thread 1 (crashed)

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1271106747.1271107786.6273.gz

Comment 1

[Phil Ringnalda (:philor)]

Sliced the first three off as bug 558975. I'd say that leaves you with quite probably eleven separate bugs, across four or five different components, all in an indigestible lump.

Comment 2

[Phil Ringnalda (:philor)]

(In reply to comment #0)
> REFTEST TEST-UNEXPECTED-FAIL |
> file:///home/cltbld/talos-slave/mozilla-central-fedora-opt-u-reftest/build/reftest/tests/layout/reftests/svg/objectBoundingBox-and-fePointLight-01.svg

Filed bug 558977

Comment 3

[Phil Ringnalda (:philor)]

(In reply to comment #0)
> REFTEST TEST-UNEXPECTED-FAIL |
> file:///home/cltbld/talos-slave/mozilla-central-fedora-opt-u-reftest/build/reftest/tests/layout/reftests/text-decoration/underline-block-propagation-2-standards.html

Filed bug 558981

Comment 4

[Phil Ringnalda (:philor)]

Actually, I probably overestimated the number of separate bugs, since there are a whole lot of them that don't seem like they would be related, except the actual differences are 1 or 2 or 21 pixels of difference on the inside of the bottom curve of a t.

Comment 5

[Phil Ringnalda (:philor)]

(In reply to comment #0)
> REFTEST TEST-UNEXPECTED-FAIL |
> file:///home/cltbld/talos-slave/mozilla-central-fedora-opt-u-reftest/build/reftest/tests/layout/reftests/bugs/180085-1.html
> REFTEST TEST-UNEXPECTED-FAIL |
> file:///home/cltbld/talos-slave/mozilla-central-fedora-opt-u-reftest/build/reftest/tests/layout/reftests/bugs/359903-1.html

Filed bug 559125

Comment 6

[Phil Ringnalda (:philor)]

(In reply to comment #0)
> REFTEST TEST-UNEXPECTED-FAIL |
> file:///home/cltbld/talos-slave/mozilla-central-fedora-opt-u-reftest/build/reftest/tests/layout/reftests/box-shadow/boxshadow-inner-basic.html
> |
> REFTEST TEST-UNEXPECTED-FAIL |
> file:///home/cltbld/talos-slave/mozilla-central-fedora-opt-u-reftest/build/reftest/tests/layout/reftests/bugs/456219-1c.html

Filed bug 559426

Comment 7

[Phil Ringnalda (:philor)]

And what that leaves is the ones where the failure is in antialiasing, so I asked roc, and he said "you probably want to turn off subpixel antialiasing on those boxes" which puts the ball back in your court.

Comment 8

[Armen [:armenzg]]

(In reply to comment #7)
> And what that leaves is the ones where the failure is in antialiasing, so I
> asked roc, and he said "you probably want to turn off subpixel antialiasing on
> those boxes" which puts the ball back in your court.

Hi roc,
Would this be sufficient to fix the anti-aliasing problems? I don't understand the implications of having it turned off. Do Linux users have that turned off by default? If our tests pass with anti-aliasing shut off will it still make sure that Firefox will be tested for users with anti-aliasing on?

I just want to make sure that we are doing the right coverage since this change will deploy across all Fedora unit test machines.

[cltbld@talos-r3-fed-001 ~]$ cat .fonts.conf 
<match target="font">
  <edit name="antialias" mode="assign">
   <bool>false</bool>
  </edit>
</match>

Comment 9

[Robert O'Callahan (:roc) (email my personal email if necessary)]

That'll do it.

We should turn it back on at some point, but there are known bugs that cause reftest failures with various kinds of antialiasing.

Comment 10

[Armen [:armenzg]]

(In reply to comment #9)
> That'll do it.
> 
> We should turn it back on at some point, but there are known bugs that cause
> reftest failures with various kinds of antialiasing.

By turning on you refer to removing the file or setting the boolean to true, right?

Do you know why this wasn't happening on the CentOS VMs where we also don't have the .fonts.conf file?

I am trying to understand the differences and if the test can be made more robust instead of deploying the file across the slaves (not that it is difficult but I want to understand this a little better and that we are doing the right choice).

I will soon have a couple of Fedora slaves to loan to devs. I hope this could help with the different reftest oranges.

Comment 11

[Robert O'Callahan (:roc) (email my personal email if necessary)]

(In reply to comment #10)
> By turning on you refer to removing the file or setting the boolean to true,
> right?

Yes.

> Do you know why this wasn't happening on the CentOS VMs where we also don't
> have the .fonts.conf file?

No.

We can reproduce antialiasing reftest failures quite easily. I don't think we need a machine for this. I think we know what the relevant bugs are, it just requires some rework to fix that we haven't had time for yet.

Comment 13

[Armen [:armenzg]]

Attachment: Reftest log with anti-aliasing disabled

By using this:
(In reply to comment #8)
> [cltbld@talos-r3-fed-001 ~]$ cat .fonts.conf 
> <match target="font">
>   <edit name="antialias" mode="assign">
>    <bool>false</bool>
>   </edit>
> </match>

We are down to 2 bugs:
* bug 559426 - reftests/box-shadow/boxshadow-inner-basic.html 
* bug 564274 - 456219-1a.html, 456219-1b.html and 456219-1c.html

[cltbld@talos-r3-fed-002 armenzg]$ grep UNEXPEC reftests.log 
> REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/armenzg/reftest/tests/layout/reftests/box-shadow/boxshadow-inner-basic.html |
> REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/armenzg/reftest/tests/layout/reftests/bugs/456219-1a.html |
> REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/armenzg/reftest/tests/layout/reftests/bugs/456219-1b.html |
> REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/armenzg/reftest/tests/layout/reftests/bugs/456219-1c.html |


[cltbld@talos-r3-fed-002 armenzg_without_fonts.conf]$ grep UNEXPEC reftests.log 
REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/armenzg_without_fonts.conf/reftest/tests/layout/reftests/box-shadow/boxshadow-inner-basic.html |
REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/armenzg_without_fonts.conf/reftest/tests/layout/reftests/bugs/371043-1.html |
REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/armenzg_without_fonts.conf/reftest/tests/layout/reftests/bugs/393760-2.xml |
REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/armenzg_without_fonts.conf/reftest/tests/layout/reftests/bugs/449171-1.html |
REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/armenzg_without_fonts.conf/reftest/tests/layout/reftests/bugs/456219-1a.html |
REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/armenzg_without_fonts.conf/reftest/tests/layout/reftests/bugs/456219-1b.html |
REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/armenzg_without_fonts.conf/reftest/tests/layout/reftests/bugs/456219-1c.html |
REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/armenzg_without_fonts.conf/reftest/tests/layout/reftests/bugs/482592-1a.xhtml |
REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/armenzg_without_fonts.conf/reftest/tests/layout/reftests/bugs/482592-1b.xhtml |
REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/armenzg_without_fonts.conf/reftest/tests/layout/reftests/pagination/border-breaking-000-cols.xhtml |


== setup steps ==
wget --progress=dot:mega -N http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1273170756/firefox-3.7a5pre.en-US.linux-i686.tar.bz2
wget --progress=dot:mega -N http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1273170756/firefox-3.7a5pre.en-US.linux-i686.tests.zip
tar -jxvf firefox-3.7a5pre.en-US.linux-i686.tar.bz2
unzip -o firefox-3.7a5pre.en-US.linux-i686.tests.zip
export DISPLAY=:0
python reftest/runreftest.py --appname=firefox/firefox-bin --utility-path=bin --extra-profile-file=bin/plugins --symbols-path=symbols reftest/tests/layout/reftests/reftest.list | tee reftests.log

Comment 14

[Armen [:armenzg]]

I am going to refer to comment #9 and comment #11.

Do we want to deploy .fonts.conf into the Fedora slaves?
Is this a permanent configuration to have disabled?
Would it be reflecting the reality of end-users?
NOTE that I have a really minimum understanding of reftests and anti-aliasing.
The only thing I don't want is to deploy something that should have not been deployed but instead we should have fixed the code/tests.
I believe there is also the option of filing the "known bugs" (as per comment #9) and mark the tests TEST-KNOWN-FAIL (I might be saying a stupid option for my lack of understanding).
FYI, the day that we won't have perma-oranges we will be stop running unit tests on CentOS machines and we should only expect coverage on Fedora slaves.

roc and dbaron, what is the suggested plan? do my concerns make more or less sense?



> [cltbld@talos-r3-fed-001 ~]$ cat .fonts.conf 
> <match target="font">
>   <edit name="antialias" mode="assign">
>    <bool>false</bool>
>   </edit>
> </match>

(In reply to comment #9)
> That'll do it.
> 
> We should turn it back on at some point, but there are known bugs that cause
> reftest failures with various kinds of antialiasing.

(In reply to comment #11)
> (In reply to comment #10)
> > By turning on you refer to removing the file or setting the boolean to true,
> > right?
> 
> Yes.
> 
> > Do you know why this wasn't happening on the CentOS VMs where we also don't
> > have the .fonts.conf file?
> 
> No.
> 
> We can reproduce antialiasing reftest failures quite easily. I don't think we
> need a machine for this. I think we know what the relevant bugs are, it just
> requires some rework to fix that we haven't had time for yet.

Comment 15

[Armen [:armenzg]]

Attachment: Centos' fonts.conf file

For comparison, here is the fonts.conf file that the CentOS slaves (build/unit test slaves) are using.

Comment 16

[Zack Weinberg (:zwol)]

Karl Tomlinson might know something about this.

Comment 17

[Phil Ringnalda (:philor)]

Attachment: reftest log

Log excerpt for the remaining failures that are this bug.

For those (like me a couple of weeks ago) who don't know what to do with it, open layout/tools/reftest/reftest-analyzer.xhtml from a mozilla-central tree in Firefox, load the file, and click on the links for the tests. You can blink back and forth between Image 1 and Image 2 to see that you can't even tell what the difference is, then check the Circle differences box to show you that it's things like 4 pixels at the corners of an X which is the same color as the background, which the antialiasing makes invisibly different, or the word test in green twice, from two different stylesheets, where the test doesn't care whether every pixel around the curved edges is identical, it only cares whether the text is green or red.

Comment 18

[Jeff Walden [:Waldo]]

Is it reasonable to expect any contributor on Fedora to have to manually twiddle his local system as done here if he wants to be able to run tests on it?  (Not to mention, as noted previously, any end user seeing sites that happen to exercise identical functionality to what the relevant reftests hit.)  I'm not sure it is, but I have no idea what the alternative fix is -- maybe it's something that needs to be upstreamed?

Comment 19

[Zack Weinberg (:zwol)]

Running local tests under xvfb, instead of on the real display, irons out a lot of these problems and also makes things run about twice as fast (for me anyway).  Maybe that ought to be the recommended procedure.

Comment 20

[Armen [:armenzg]]

I would like to bring this up in the dev.platform meeting to be sure what is the right way to go.

Comment 21

[Armen [:armenzg]]

Attachment: disable anti-aliasing on talos fedora slaves (32 and 63 bit)

Comment 22

[[e-mail] (:bhearsum)]

(In reply to comment #18)
> Is it reasonable to expect any contributor on Fedora to have to manually
> twiddle his local system as done here if he wants to be able to run tests on
> it?  (Not to mention, as noted previously, any end user seeing sites that
> happen to exercise identical functionality to what the relevant reftests hit.) 
> I'm not sure it is, but I have no idea what the alternative fix is -- maybe
> it's something that needs to be upstreamed?

Was this ever discussed? It seems like a very valid point to me.

Comment 23

[Armen [:armenzg]]

(In reply to comment #22)
> (In reply to comment #18)
> > Is it reasonable to expect any contributor on Fedora to have to manually
> > twiddle his local system as done here if he wants to be able to run tests on
> > it?  (Not to mention, as noted previously, any end user seeing sites that
> > happen to exercise identical functionality to what the relevant reftests hit.) 
> > I'm not sure it is, but I have no idea what the alternative fix is -- maybe
> > it's something that needs to be upstreamed?
> 
> Was this ever discussed? It seems like a very valid point to me.

I discussed this with roc and we will be disabling anti-aliasing (as per patch) and in the future he will let me know when to enable it back (easy to revert).
I don't know what the plan will be but that I leave in his hands.

Comment 24

[[e-mail] (:bhearsum)]

Comment on attachment 444912 [details] [diff] [review]
disable anti-aliasing on talos fedora slaves (32 and 63 bit)

Based on the last comment this seems fine.

Comment 25

[Robert O'Callahan (:roc) (email my personal email if necessary)]

(In reply to comment #22)
> (In reply to comment #18)
> > Is it reasonable to expect any contributor on Fedora to have to manually
> > twiddle his local system as done here if he wants to be able to run tests on
> > it?  (Not to mention, as noted previously, any end user seeing sites that
> > happen to exercise identical functionality to what the relevant reftests
> > hit.) I'm not sure it is, but I have no idea what the alternative fix is --
> > maybe it's something that needs to be upstreamed?
> 
> Was this ever discussed? It seems like a very valid point to me.

On all platforms, someone running reftests on their own system will often see a few failures that we don't see on Tinderbox. This happens for a variety of reasons, but often due to different antialiasing settings.

This is not a good situation, and we need to fix it, but it's nothing new and shouldn't block getting Fedora unit tests turned on.

Comment 26

[Armen [:armenzg]]

Comment on attachment 444912 [details] [diff] [review]
disable anti-aliasing on talos fedora slaves (32 and 63 bit)

http://hg.mozilla.org/build/puppet-manifests/rev/fd156cd10c40

I was able to sync a couple of production slaves manually as expected.

I had to add the module [talos] as we did on staging on /etc/puppet/fileserver.conf.

Comment 27

[Armen [:armenzg]]

[root@production-puppet linux]# cvs add fonts.conf 
cvs add: scheduling file `fonts.conf' for addition
cvs add: use 'cvs commit' to add this file permanently
[root@production-puppet linux]# cvs commit -m "Bug 558910. fonts.conf to disable anti-aliasing on Fedora testings slaves. r=bhearsum"
cvs commit: Examining .
cvs commit: Examining RPMS
cvs commit: Examining RPMS/i686
cvs commit: Examining RPMS/x86_64
RCS file: /mofo/puppet-files/talos/linux/fonts.conf,v
done
Checking in fonts.conf;
/mofo/puppet-files/talos/linux/fonts.conf,v  <--  fonts.conf
initial revision: 1.1
done

Comment 28

[Armen [:armenzg]]

All reftests looking green.