770591 - Continuous-testing HarfBuzz

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect, P3)

Description

[Behdad Esfahbod]

[Not sure what the right component is.]

Would be nice if we can get continuous-testing infrastructure for HarfBuzz.  I don't currently have a test runner infrastructure, but we have most tools needed to put one together.  Would be nice if these can be deployed to test continuously, most commits, and monitor memory, speed, and test results.

Requirements are fairly simple, Linux builds are fine, but any system should do.

Comment 1

[:Atoll]

harfbuzz appears to be some sort of opentype layout thing, which implies "a server-side tinderbox-like product for client team", so I'm guessing this is destined for the IT triage queue.

Comment 2

[(no longer active)]

The RelEng team is the appropriate owner for this project.

Behdad, how does one currently run tests on harfbuzz?

Comment 3

[Behdad Esfahbod]

(In reply to Ehsan Akhgari [:ehsan] from comment #2)
> Behdad, how does one currently run tests on harfbuzz?

Right now we do manually.  In essence:

  * You build the HarfBuzz repo, and the util/hb-shape binary is your main tool.  You give it font and text, and it gives you textual output.

We have extensive word lists (with frequency data) extracted from wikipedias for 63 different languages.  That data lives here:

  http://code.google.com/p/harfbuzz-testing-wikipedia/downloads/list

It's in the order of a million words per language for most.

We need to compile a set of fonts we care about.  Some are free, but most (Windows) ones are not.  So that may mean that we need to host this on Windows, or otherwise access those files in whatever way RelEng is comfortable doing.

  * For all given wordlist + font combinations we care about, we need to generate expected-output files.  Initially we do this by recording what Uniscribe generates.  From there on, tweaking manually.

So, the pieces are there, not the scripts to run and summarize/visualize the output.  I'm trying to get those together too, but thought I can get some help from people who have already done this in other contexts.

Comment 4

[(no longer active)]

(In reply to comment #3)
> (In reply to Ehsan Akhgari [:ehsan] from comment #2)
> > Behdad, how does one currently run tests on harfbuzz?
> 
> Right now we do manually.  In essence:
> 
>   * You build the HarfBuzz repo, and the util/hb-shape binary is your main
> tool.  You give it font and text, and it gives you textual output.

Is there a higher level script which runs this binary for all of the fonts to be tested?  I think that may be a requirement for this.

> We have extensive word lists (with frequency data) extracted from wikipedias
> for 63 different languages.  That data lives here:
> 
>   http://code.google.com/p/harfbuzz-testing-wikipedia/downloads/list
> 
> It's in the order of a million words per language for most.

Is it reasonable/possible for this data to live in the harfbuzz repo?

> We need to compile a set of fonts we care about.  Some are free, but most
> (Windows) ones are not.  So that may mean that we need to host this on Windows,

With all else being equal, would you rather only test this on Linux, or on all platforms?

> or otherwise access those files in whatever way RelEng is comfortable doing.

I think we can use tooltool to grab those fonts (provided that you can provide RelEng with an initial copy.)  tooltool is a system which serves you packages identified by a sha512, designed for this kind of task.

>   * For all given wordlist + font combinations we care about, we need to
> generate expected-output files.  Initially we do this by recording what
> Uniscribe generates.  From there on, tweaking manually.
> 
> So, the pieces are there, not the scripts to run and summarize/visualize the
> output.  I'm trying to get those together too, but thought I can get some help
> from people who have already done this in other contexts.

I see.  I don't know a lot about the specifics of what needs to be tested, but one way to do things which we use for the main mozilla tests is to put known tokens in the stdout output of the test script (like TEST-UNEXPECTED-FAIL, TEST-KNOWN-FAIL, etc.) and then parse those out in the buildbot to see what tests failed, etc.  But I don't know if that is the preferred way of doing things any more.  Chris?

Comment 5

[[e-mail] (:bhearsum)]

I'm not sure if this is still a thing, but this can be self served with docs.taskcluster.net now!

Comment 6

[Behdad Esfahbod]

This is actually not fixed, and probably won't be covered by self-serve, but Jonathan can decide what to do with it.  There's probably no point in keeping a bug open for it though.