452870 - Implement tests for @font-face rules

Status: RESOLVED WORKSFORME

(Core :: CSS Parsing and Computation, defect, P2)

Description

[Akira Yoshihara]

User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1a2pre) Gecko/20080828020913 Minefield/3.1a2pre
Build Identifier: 

This bug will contain the reftest test cases for the @font-face rules.

Reproducible: Always

Comment 1

[Akira Yoshihara]

Attachment: Test case for @font-face rules v.0.1

Comment 2

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

I don't think we can check Verdana (assuming that's what Verdana.ttf is) into our tree -- we'd need to use fonts that can be tri-licensed (i.e., are licensed with it or something more liberal -- e.g., I think we're probably OK on BSD).

Comment 3

[cmtalbert]

(In reply to comment #2)
> I don't think we can check Verdana (assuming that's what Verdana.ttf is) into
> our tree -- we'd need to use fonts that can be tri-licensed (i.e., are licensed
> with it or something more liberal -- e.g., I think we're probably OK on BSD).

Yes, true.  I'm working on figuring out what fonts we can use instead.  We just wanted to back up Akira's work to date.

Comment 4

[Akira Yoshihara]

Attachment: Test case for @font-face rules v.0.5

Test cases for @font-face rules.

Now test cases are reduced in number to around 15. To execute these test cases, you need the following font files which I did not include in this diff file due to size restriction.

Bitstream Vera Sans Roman, Italic
- install the font into your system. Also put the font files in the same directory with test cases. Please use the following names: vera.ttf, verait.ttf

Sazanami Gothic Regular
- install the font into your system. Also put the font files in the same directory with test cases. Please use the following name: sazanami-gothic.ttf

Some tests still fail. You can see the details about each test on the wiki page.
https://wiki.mozilla.org/QA/Firefox3.1/FontFace_TestStatus

Comment 5

[Akira Yoshihara]

Attachment: Test case for @font-face rules v.0.6

Added some test cases, and added comment on every test cases.

To execute these test cases, you need the following font file which I did not include in this diff file.

BKM-cmb10
- install the font into your system. Also put the font files in the same
directory with test cases. Please use the following names: cmb10.otf

Comment 6

[John Daggett (:jtd)]

Fonts need to run these tests:

Bakoma Computer Modern

1. Download:

  http://www.ctan.org/get/fonts/cm/ps-type1/bakoma.zip

2. Copy the folks below to the layout/reftests/fonts directory:

  cmb10.otf

Bitstream Vera fonts

1. Download:

  http://ftp.gnome.org/pub/GNOME/sources/ttf-bitstream-vera/1.10/ttf-bitstream-vera-1.10.zip

2. Copy the folks below to the layout/reftests/fonts directory:

  Vera.ttf
  VeraIt.ttf  

Sazanami Japanese fonts

1. Download:

  http://sourceforge.jp/projects/efont/downloads/9934/sazanami-20040618.tar.bz2

2. Copy the folks below to the layout/reftests/fonts directory:

  sazanami-gothic.ttf

All of these fonts *also* need to be among the system fonts.  On Windows, copy them to the system fonts folder.  On Mac OS X, make a folder "reftest fonts" in ~/Library/Fonts and copy the fonts there.

Comment 7

[John Daggett (:jtd)]

Scheherazade Arabic font

1. Download:

  http://scripts.sil.org/cms/scripts/render_download.php?site_id=nrsi&format=file&media_id=scheherazade_opentype&filename=ScheherazadeRegOT_ttf.zip

2. Copy the font below to the layout/reftests/fonts directory:

  ScheherazadeRegOT.ttf

This also needs to be copied to the appropriate system/user fonts directory.

Comment 8

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Could we use something with fixed rendering, like Ahem, and compare to a non-text reference? We can't have reftests requiring special fonts to be installed on the tester's system.

Comment 9

[John Daggett (:jtd)]

Attachment: testcases for @font-face rules, v.0.7

Slight revisions, use case-sensitive font filenames, and use javascript-generated load test instead of one based on perl scripts.

Getting download failures on a couple of these tests, need to track this down:

WARNING: ATSFontGetPostScriptName err = -984: file /builds/mozcentral/gfx/thebes/src/gfxQuartzFontCache.mm, line 160

The downloadable-font-markupdifference test is probably not a valid test, since it's comparing real italics to fake italics.  But it seems to be illustrating a bug in italic handling.

Comment 10

[John Daggett (:jtd)]

(In reply to comment #8)
> Could we use something with fixed rendering, like Ahem, and compare to a
> non-text reference? We can't have reftests requiring special fonts to be
> installed on the tester's system.

Yeah, I don't think tests that use src: local(xxx) should rely on additional fonts being installed.  But some of these tests do show that for some reason layout is different when using the same font that is also installed locally.  We probably just want to keep around a separate reftest list for these, one that isn't run as part of automated coverage tests.

Comment 11

[cmtalbert]

Let me know if there are any of these fonts that we *don't* need to check into the tree.  I have open bugs to get the licenses reviewed for these fonts and was going to start leaning hard on the lawyers to get that done so we can get these tests in.

Comment 12

[Robert O'Callahan (:roc) (email my personal email if necessary)]

I think we should figure out what the bugs are that cause rendering of downloaded fonts to differ from local fonts, then figure out if we can make testcases for those bugs using Ahem or something like it --- testcases whose references are non-text. I assume creating a simple test font shouldn't be too hard (and soon we may have access to a contributor with experience in complex font design).

Comment 13

[Robert O'Callahan (:roc) (email my personal email if necessary)]

What I'd really like to have are a few *small* test fonts like Ahem that we can use to test @font-face and also kerning, ligaturization, font selection etc in predictable ways on all platforms.

This might be something that Jonathan could help with. If we design them ourselves then there are no legal issues, we can release them to the public domain like Ahem.

Comment 14

[Jonathan Kew [:jfkthame]]

Rather than starting from sratch and releasing fonts into the public domain, would it be acceptable to use fonts licensed under the Open Font License (see http://scripts.sil.org/OFL for details)? The easiest way for me to create some small test fonts would be to extract subsets from SIL's OFL releases such as Charis or Gentium - just enough to test the features we want to look at. That means our test fonts would be derivates of SIL's, and would therefore have to be OFL'd themselves.

Comment 15

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

For what it's worth, I'm working on a few reftests for bug 457821; for that, I've generated some extremely simple fonts that gave a single glyph (I'm currently working with 2 different glyphs), and varied which Unicode codepoint gets that glyph (I'm currently using 4 codepoints, capital A-D).  That's plenty for testing the CSS side of things (what overrides what, which @font-face rules are used when), but has nothing to do with testing fancy font features.

Comment 16

[Robert O'Callahan (:roc) (email my personal email if necessary)]

David, OK cool. They would be enough for this bug here too, right? And bug 451426? Jonathan, could you try to write a reftest for bug 451426 using the fonts in bug 457821 "patch 5"?

If that works, then we don't need anything special here and we should file a new bug for shaping tests.

Comment 17

[Jonathan Kew [:jfkthame]]

Attachment: patch for Japanese test in testcases v0.7

This patch modifies the downloadable-font-japanese test to rely on markA.ttf (from bug 457821 patch 7) instead of requiring the Vera Sans font. This test is relevant to bug 451426; it fails until the patch for that bug is applied, then passes.

The (apparently spurious) large Latin character in the test is a workaround to prevent the baseline varying, which would cause the test to fail. That is a separate problem from bug 451426, and is shown in other reftests (downloadable-font-layouttest and -loadtest), so I have chosen to suppress the failure in this case so that this test focuses only on the Asian character font fallbacks.

Comment 18

[Jonathan Kew [:jfkthame]]

Attachment: workaround for occasional activation failure on Mac OS X

On Mac OS X, when many fonts are downloaded/activated "simultaneously", we get occasional failures as mentioned in comment #9. This appears to happen when the ATS font database generation number changes between the activation and the call to ATSFontGetPostScriptName (during initialization of the font entry).

This manifests as occasional load failures in downloadable-font-loadtest.html; of the 50 lines in that test, I typically see about 3 failures (although it is the same font file in all 50 cases). The exact places where the failures happen varies. Reloading the page generally fixes the bad lines, though reloading multiple times in quick succession usually leads to a failure again.

I suspect there may be an underlying Mac OS X bug that is causing this problem, though it may be difficult to prove.

The proposed patch works around the problem by checking for a changed ATSGeneration value if the font entry appears invalid; in this case, it will retry the activation and font initialization process (up to a small number of times - we don't want to risk retrying infinitely). In my testing with the downloadable-font-loadtest.html file, the first retry has always succeeded each time there is a failure, and so all 50 lines of the test are rendered in the proper font.

Comment 19

[Jonathan Kew [:jfkthame]]

Shortly after posting #18 and associated patch, I found bug 458861 which addresses the same issue. As noted there, deleting the ATS font caches resolves the issue with occasional, somewhat random load failures.

However, I suggest this patch should still be applied, as ATS font corruption is a fairly common occurrence on OS X 10.5, and may be caused by other applications or fonts on the system, not just by Mozilla trying to load a bad font. The checks provided by the fix for bug 458861 are a good thing, and may help to reduce the frequency of problems, but they cannot prevent the issue altogether. So this patch, which makes us more robust in the face of a damaged ATS font cache, still seems like a worthwhile addition.

Comment 20

[John Daggett (:jtd)]

Comment on attachment 346907 [details] [diff] [review]
workaround for occasional activation failure on Mac OS X

Looks like you caught a bug of my making, using ATSGeneration instead of ATSGetGeneration, doh.  But I think it would be good to make a new bug for this and to come up with a patch that doesn't use goto's.

Comment 21

[Jonathan Kew [:jfkthame]]

OK, can do. I have a new version ready - just re-testing.

Actually, would it make most sense to re-open bug 458861 and include the new patch there? That bug is about precisely this issue, and the existing fix only really addresses one aspect of it - it tries to ensure that we don't cause font cache problems by loading bad fonts, but it doesn't address the failures we get if the cache is already flaky. That's where this new patch helps.

So shall I make a new bug or re-open the old one? I don't mind... someone with more experience, please let me know the preferred approach.

Comment 22

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Normally we'd only reopen an old bug if the fix was backed out or completely failed to do what it was supposed to do. For a more comprehensive fix, even for the same issue, we probably want a new bug.

Comment 23

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

A few comments on the patch here:

* I think we're better off using reftests/fonts/ as a directory that contains fonts and putting actual tests elsewhere.  (That was sort of my assumption based on Ahem.ttf already being checked in there; I just put some more fonts in fonts/ and some tests in font-face/ .)

* Using setTimeout(..., 2000) to detect font load completion is bad for two reasons:
  1. it slows down the running of reftest most of the time
  2. in some situations (e.g., heavily loaded tinderbox machines) the font might not be loaded after 2 seconds.

I think for static tests we shouldn't need that anyway, since the font loading should delay 'onload' (and it seems like it does now).  For dynamic tests, see the approach I took in bug 457821.

Comment 24

[John Daggett (:jtd)]

(In reply to comment #23)

> * Using setTimeout(..., 2000) to detect font load completion is bad for two

This shouldn't be needed now.  Akira originally had to use this because the font loader wasn't included in a load group, so the onload event would fire before fonts had been downloaded.

Comment 25

[Jonathan Kew [:jfkthame]]

Comment on attachment 346907 [details] [diff] [review]
workaround for occasional activation failure on Mac OS X

Marking attachment 346907 [details] [diff] [review] as obsolete; this is superseded by bug 463806 and an updated patch is attached there.

Comment 26

[John Daggett (:jtd)]

This is an old bug, we now have a number of @font-face related tests.  Marking as WORKSFORME.