Open Bug 589573 Opened 14 years ago Updated 12 years ago

[SeaMonkey 2.1, reftest, Linux] 11 TEST-UNEXPECTED-*

Categories

(SeaMonkey :: Testing Infrastructure, defect)

Product:
SeaMonkey
Component:
Testing Infrastructure
Platform:
x86
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
REOPENED

People

(Reporter: kairo, Unassigned)

References

(Blocks 1 open bug)

Details

(Whiteboard: [sm-perma][cc-orange]) 

Those unexpected results started happening recently and permanently on all our SeaMobnky trunk Linux test machines (CentOS):

REFTEST TEST-UNEXPECTED-FAIL | file:///builds/slave/comm-central-trunk-linux-debug-unittest-reftest/build/reftest/tests/layout/reftests/bidi/bidi-004.html |
REFTEST TEST-UNEXPECTED-FAIL | file:///builds/slave/comm-central-trunk-linux-debug-unittest-reftest/build/reftest/tests/layout/reftests/bidi/bidi-004-j.html |
REFTEST TEST-UNEXPECTED-FAIL | file:///builds/slave/comm-central-trunk-linux-debug-unittest-reftest/build/reftest/tests/layout/reftests/bidi/mirroring-02.html |
REFTEST TEST-UNEXPECTED-FAIL | file:///builds/slave/comm-central-trunk-linux-debug-unittest-reftest/build/reftest/tests/layout/reftests/font-features/font-features-turkish.html | (!=)
REFTEST TEST-UNEXPECTED-FAIL | file:///builds/slave/comm-central-trunk-linux-debug-unittest-reftest/build/reftest/tests/layout/reftests/font-features/font-features-turkish.html |
REFTEST TEST-UNEXPECTED-PASS | file:///builds/slave/comm-central-trunk-linux-debug-unittest-reftest/build/reftest/tests/layout/reftests/font-features/font-features-turkish.html |
REFTEST TEST-UNEXPECTED-PASS | file:///builds/slave/comm-central-trunk-linux-debug-unittest-reftest/build/reftest/tests/layout/reftests/font-features/font-features-turkish-override-1.html |
REFTEST TEST-UNEXPECTED-PASS | file:///builds/slave/comm-central-trunk-linux-debug-unittest-reftest/build/reftest/tests/layout/reftests/font-features/font-features-turkish-override-2.html |
REFTEST TEST-UNEXPECTED-PASS | file:///builds/slave/comm-central-trunk-linux-debug-unittest-reftest/build/reftest/tests/layout/reftests/font-features/font-features-turkish-override-3.html |
REFTEST TEST-UNEXPECTED-PASS | file:///builds/slave/comm-central-trunk-linux-debug-unittest-reftest/build/reftest/tests/layout/reftests/font-features/font-features-turkish-override-4.html |
REFTEST TEST-UNEXPECTED-PASS | file:///builds/slave/comm-central-trunk-linux-debug-unittest-reftest/build/reftest/tests/layout/reftests/font-features/font-features-turkish-override-5.html |
Whiteboard: [sm-perma][orange]
Not that recently:
http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1279787534.1279789246.24091.gz

Last green:
http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1279787534.1279789246.24091.gz
Linux comm-central-trunk debug test reftest on 2010/07/22 01:32:14

First perma-orange:
http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1279804342.1279805734.23632.gz
Linux comm-central-trunk debug test reftest on 2010/07/22 06:12:22
The 8 "font-features-turkish*.html".

Regression timeframes:
http://hg.mozilla.org/comm-central/pushloghtml?fromchange=3061bbc82e7d&tochange=7223e8b0311f
(1 bug)
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=f1bd51cdac68&tochange=9805493631d4
(some bugs, including:) Bug 511339 added these new tests !

(I assume the 3 other tests were added since then...)
Blocks: 511339
The font-features-* tests are known-fail on Linux, as we don't have the necessary harfbuzz-based font backend in place. They should be marked as failing in the reftest list.
(In reply to comment #3)

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1282593076.1282593856.10651.gz&fulltext=1
Rev3 Fedora 12 mozilla-central opt test reftest on 2010/08/23 12:51:16
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1282594799.1282596202.22041.gz&fulltext=1
Rev3 Fedora 12 mozilla-central debug test reftest on 2010/08/23 13:19:59
2 TEST-PASS + 6 TEST-KNOWN-FAIL :-)

http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1282591364.1282593723.10223.gz&fulltext=1
Linux comm-central-trunk debug test reftest on 2010/08/23 12:22:44
2 TEST-UNEXPECTED-FAIL + 6 TEST-UNEXPECTED-PASS :-<

Would SeaMonkey boxes have the font backend which Firefox boxes misses ?
(Could you check the SM image logs.?.)
(In reply to comment #1)
> (I assume the 3 other tests were added since then...)

Bug 569623 "modified" the 3 'bidi/' tests.!.
(mirroring-02.html was added by bug 542777.
The 2 other were there long before.)
Blocks: 569623
(In reply to comment #4)
> Would SeaMonkey boxes have the font backend which Firefox boxes misses ?

No. This isn't a question of what's on "the boxes", it's about Gecko code (thebes/gfxFont*...) that is only implemented for Mac and Windows so far.

> (Could you check the SM image logs.?.)

This is failing:

# check that Turkish language causes a change in rendering (no fi ligature)
# (also works via Pango)
fails-if(winWidget) HTTP(..) != font-features-turkish.html font-features-ref.html

which suggests perhaps the SM boxes have an older version of Pango, so even this test - which works via our old Pango backend in FF - is failing there. (That invalidates most of what follows in the font-features directory.)

Similarly here:

# compare Turkish rendering with reference using ZWNJ to break the ligature
# (also works via Pango)
fails-if(winWidget) HTTP(..) == font-features-turkish.html font-features-turkish-ref.html

the "also works via Pango" doesn't seem to be true for SM tests.

For the UNEXPECTED-PASS cases, note the comment in reftest.list:

# The following should pass even if feature support isn't available,
# because both testcase and reference will have the default rendering,
# though they're not really meaningful unless the tests above passed already.

So these tests are pointless; the features they're testing simply don't exist in your build. They "pass" because they're essentially no-ops; on the FF test boxes a bunch of them fail because Pango gives us a bit of the necessary functionality, but not all of it.
(In reply to comment #5)
> (In reply to comment #4)
> 
> http://mxr.mozilla.org/mozilla-central/source/layout/reftests/font-features/reftest.list

line #3:

# These rely on the Linux Libertine font (loaded via @font-face)

Now it's entirely possible that CentOS 5 doesn't have that installed. I know why I dislike this fact that Firefox is tessting on different systems than us nowadays - and there's no good way out of it unless we get enough Minis that we can set up Talos slaves and run all tests there.
(In reply to comment #7)
> No. This isn't a question of what's on "the boxes"

Well, it probably is about "the boxes" in the end, as we need to run tests on CentOS5 while you are running them on some much newer Fedora version.
(In reply to comment #8)
> (In reply to comment #5)
> > (In reply to comment #4)
> > 
> > http://mxr.mozilla.org/mozilla-central/source/layout/reftests/font-features/reftest.list
> 
> line #3:
> 
> # These rely on the Linux Libertine font (loaded via @font-face)
> 
> Now it's entirely possible that CentOS 5 doesn't have that installed. 

No, we load the font using @font-face (from the reftests/fonts directory) rather than depending on platform fonts, so what fonts are installed on the host platform is irrelevant here.
(In reply to comment #9)
> (In reply to comment #7)
> > No. This isn't a question of what's on "the boxes"
> 
> Well, it probably is about "the boxes" in the end, as we need to run tests on
> CentOS5 while you are running them on some much newer Fedora version.

No; as I said, "it's about Gecko code (thebes/gfxFont*...) that is only implemented for Mac and Windows so far". See bug 569770 for the Linux side of this.
(In reply to comment #11)
> No; as I said, "it's about Gecko code (thebes/gfxFont*...) that is only
> implemented for Mac and Windows so far". See bug 569770 for the Linux side of
> this.

OK, but then, they should be failing on Firefox as well, no?

What happens, though, is that the SeaMonkey machines report unexpected results while the Firefox ones don't, and that's what we need to solve.
(In reply to comment #7)

> No. This isn't a question of what's on "the boxes", it's about Gecko code
> (thebes/gfxFont*...) that is only implemented for Mac and Windows so far.

(Then I'm not sure about what the cause of this bug could likely be...)

> which suggests perhaps the SM boxes have an older version of Pango, so even
> this test - which works via our old Pango backend in FF - is failing there.

Iiuc, this would mean that these tests should only be built/run (on Linux) if some pango version check succeeds, right?
Would this check be missing in m-c, or perhaps not yet "ported" to c-c?
Some of this test failures are also mentioned in bug 586462
(In reply to comment #14)
> Some of this test failures are also mentioned in bug 586462

I'd wonder a lot if we'd have OpenGL enabled on those VMs we're running the tests on.
Serge, is this bug still occurring?
Callek: do have OpenGL enabled on those VMs we're running the tests on?
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → WORKSFORME
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
Whiteboard: [sm-perma][orange] → [sm-perma][cc-orange]
You need to log in before you can comment on or make changes to this bug.