1236086 - 6 of the font-features reftests fail when run on a docker image

Status: RESOLVED FIXED

(Core :: CSS Parsing and Computation, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

we are working to get our tests running inside a docker container.  There have been a handful of bugs filed, this would be identical to bug 1236081 except different tests.  In fact we have 6 tests which all fail and look to fail similarly:

REFTEST TEST-UNEXPECTED-FAIL | file:///home/worker/workspace/build/tests/reftest/tests/layout/reftests/font-features/font-variant-alternates.html | image comparison (==), max difference: 156, number of differing pixels: 16448
REFTEST TEST-UNEXPECTED-FAIL | file:///home/worker/workspace/build/tests/reftest/tests/layout/reftests/font-features/font-variant-caps.html | image comparison (==), max difference: 156, number of differing pixels: 5654
REFTEST TEST-UNEXPECTED-FAIL | file:///home/worker/workspace/build/tests/reftest/tests/layout/reftests/font-features/font-variant-east-asian.html | image comparison (==), max difference: 156, number of differing pixels: 11308
REFTEST TEST-UNEXPECTED-FAIL | file:///home/worker/workspace/build/tests/reftest/tests/layout/reftests/font-features/font-variant-ligatures.html | image comparison (==), max difference: 156, number of differing pixels: 24672
REFTEST TEST-UNEXPECTED-FAIL | file:///home/worker/workspace/build/tests/reftest/tests/layout/reftests/font-features/font-variant-numeric.html | image comparison (==), max difference: 156, number of differing pixels: 16962
REFTEST TEST-UNEXPECTED-FAIL | file:///home/worker/workspace/build/tests/reftest/tests/layout/reftests/font-features/font-variant-position.html | image comparison (==), max difference: 156, number of differing pixels: 1542 

the magic try push is here:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=899f61b37a6b&filter-searchStr=r3&selectedJob=14965115

these consistently fail and in fact they fail on my local development machine which happens to be linux64.  Sadly I did try my trick of increasing the font size on the body tag, this didn't help at all.  Looking a the reftest compare:
http://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/R2xUWAZiR-mXpl04wKk8aw/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

this doesn't look like the type of failures that would be corrected by adjusting the font-size.  Sadly I don't know what to do here- ideally the original author can help out and determine a way forward given the failures and what we can either change in the OS (installing fonts, changing configs, etc.) or in the tests themselves.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

:jtd, can you help out here?

Comment 2

[John Daggett (:jtd)]

The glyphs for the reftests are getting clipped (that's bad, it's more than just a reftest failure!) and the reference case is aligned slightly differently.

> these consistently fail and in fact they fail on my local development
> machine which happens to be linux64.

Which distro are you using?

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

I run ubuntu 14.04 on a lenovo x230 locally.  In docker, we run ubuntu 12.04 (or 12.10, not sure).  Can I get more information for you?  Maybe instructions for how to run tests on the ubuntu image:
docker pull elvis314/desktop-test:0.5.3
docker run elvis314/desktop-test:0.5.3

if you want details, I can give you a small script to go from docker pull -> running tests or hacking.  Otherwise docker run will drop you into a shell and you can query the system.

Comment 4

[John Daggett (:jtd)]

(In reply to Joel Maher (:jmaher) from comment #3)
> I run ubuntu 14.04 on a lenovo x230 locally.  In docker, we run ubuntu 12.04
> (or 12.10, not sure).  Can I get more information for you?

I'm guessing there's an underlying bug here related to frame metrics based on font metrics not getting flushed quite correctly after a downloadable font is loaded. Guessing that this shows up when the metrics of the fallback font are different from the downloadable font, which is why this shows up only sometimes.

It would be helpful to have fontlist/textrun logs for just a single test/reference pair, in whatever environment this is failing in.

Steps:

1. Create a new profile and enable show previously open tabs
2. Load font-variant-numeric.html and font-variant-numeric-ref.html into tabs
3. Quit the browser
4. Enable fontlist/textrun logging:

  export NSPR_LOG_MODULES=textrun:5,fontlist:5
  export NSPR_LOG_FILE=textrun.out

5. Open the browser with the profile from (1)
6. Wait for both pages to render and then quit the browser

If you could attach this that would be helpful. Then I can try and replicate the same conditions to see if I can replicate the test/reference difference locally and then fix it.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: textrun.out from my local computer

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: textrun.out from the docker instance (what we want to use in taskcluster)

Comment 7

[John Daggett (:jtd)]

Joel, could you run this again on your machine? With logging, there's one logfile per process so I'm guessing the content process for the reftests ends up in one of the other logfiles. You should see textruns with a fontlist including "gsub-test".

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: textrun.out from my local computer - child process

this was easy to find!

Comment 9

[John Daggett (:jtd)]

When you run the test locally do you see a bunch of lines with "PASS PASS PASS"? Or do you see a bunch of hexboxes? The test (e.g. font-variant-numeric.html) uses PUA codepoints, while the reference (e.g. font-variant-numeric-ref.html) uses only ASCII. If the font correctly loads, you should see "PASS PASS PASS" for both.

The logfile only reports seeing textruns before the font is loaded:

(textrun-fontmatching) fontgroup: [gsub-test] default: serif lang: en script: 61 [0:3] <null> (list)

There should be a second set of lines with 'gsub-test' instead of <null> above.

I'll see if I can duplicate this with Deja Vu Serif as the fallback font.

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

I see a bunch of P, not PASS on font-variant-numeric-ref.html.  The font-variant-numeric.html is hex boxes.

Let me know if I can help at all.

Comment 11

[John Daggett (:jtd)]

I can now reproduce this using Ubuntu. The placement of glyphs is roughly +/- 1px higher/lower for the test/reference. The only difference in the content is PUA codepoints for the test vs. 'P' for the reference.

I'm beginning to suspect that this might be a Freetype issue, where Freetype is somehow applying autohinting differently for the same glyph based on its Unicode code point (i.e. Latin vs. PUA). That would absolutely suck but this is still pure supposition.

Comment 12

[John Daggett (:jtd)]

(In reply to Joel Maher (:jmaher) from comment #10)
> I see a bunch of P, not PASS on font-variant-numeric-ref.html.  The
> font-variant-numeric.html is hex boxes.
> 
> Let me know if I can help at all.

This is when you explicitly run reftests via something like:

  ./mach reftest layout/reftests/font-features

Seeing 'P' for the reference and hexboxes for the test means the font is failing to load. Note that these tests must be run through an HTTP server, since the ../fonts/xxx font URL's will fail to load when the page is opened as a file.

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

sorry, I was just loading the page from the file:/// url and reporting that.  the pages go by too fast when run with reftest, I can try again.

Comment 14

[Joel Maher ( :jmaher ) (UTC -8)]

I see a lot of PASS when running it through the reftest harness :)

Comment 15

[Jonathan Kew [:jfkthame]]

(In reply to Joel Maher (:jmaher) from comment #13)
> sorry, I was just loading the page from the file:/// url and reporting that.
> the pages go by too fast when run with reftest, I can try again.

Note that you can work around this for local testing purposes by setting the pref security.fileuri.strict_origin_policy to false; then the testcases should load successfully via file:///. (And then you can easily load a testcase and reference pair into two tabs and switch back and forth between them to see what's different, for example.)

Comment 16

[Jonathan Kew [:jfkthame]]

(In reply to John Daggett (:jtd) from comment #11)
> I'm beginning to suspect that this might be a Freetype issue, where Freetype
> is somehow applying autohinting differently for the same glyph based on its
> Unicode code point (i.e. Latin vs. PUA). That would absolutely suck but this
> is still pure supposition.

That sounds very plausible; Freetype definitely uses its knowledge of character properties (script, case, etc) to inform auto-hinting.

If that's the issue, you could avoid it by re-encoding the fonts to use PUA codepoints throughout, I presume.

Comment 17

[Joel Maher ( :jmaher ) (UTC -8)]

can we install fonts to solve this?

Comment 18

[Jonathan Kew [:jfkthame]]

(In reply to Joel Maher (:jmaher) from comment #17)
> can we install fonts to solve this?

I don't think so. It's possible it could be avoided (perhaps) by tweaking fontconfig settings in the OS so as to disable all hinting, but if John's suspicion from comment 11 is correct, it'd be better to update the custom fonts being used in the tests.

John, can you try the suggestion from comment 16 and see if that fixes the inconsistency?

Comment 19

[John Daggett (:jtd)]

(In reply to Jonathan Kew (:jfkthame) from comment #16)
> (In reply to John Daggett (:jtd) from comment #11)
> > I'm beginning to suspect that this might be a Freetype issue, where Freetype
> > is somehow applying autohinting differently for the same glyph based on its
> > Unicode code point (i.e. Latin vs. PUA). That would absolutely suck but this
> > is still pure supposition.
> 
> That sounds very plausible; Freetype definitely uses its knowledge of
> character properties (script, case, etc) to inform auto-hinting.
> 
> If that's the issue, you could avoid it by re-encoding the fonts to use PUA
> codepoints throughout, I presume.

That sounds like something fairly easy to test. I'll tweak the script for generating the fonts and see if that fixes this problem.

Comment 20

[John Daggett (:jtd)]

Attachment: patch, use a PUA codepoint in the reference rendering

Switching to a PUA codepoint (instead of 'P') fixes the problem in my version of Ubuntu.

Joel, could you verify that this fixes the problem for you?

Comment 21

[Jonathan Kew [:jfkthame]]

Comment on attachment 8705481 [details] [diff] [review]
patch, use a PUA codepoint in the reference rendering

Review of attachment 8705481 [details] [diff] [review]:
-----------------------------------------------------------------

I expect this is fine, but shouldn't there be font files (and font-generating script) in this patch, too?

Comment 22

[Jonathan Kew [:jfkthame]]

Comment on attachment 8705481 [details] [diff] [review]
patch, use a PUA codepoint in the reference rendering

Review of attachment 8705481 [details] [diff] [review]:
-----------------------------------------------------------------

Oh, never mind - I see you're using a codepoint that was already present in the fonts. Should be fine, then.

Comment 23

[Joel Maher ( :jmaher ) (UTC -8)]

hey, this works great for me locally!  Thanks for fixing this

Comment 24

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/da327b54c78e

Comment 25

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/da327b54c78e