# implement better support for MathML stretchy fallback in gfx

NEW
Unassigned
(NeedInfo from)

Core
Graphics: Text
NEW
3 years ago
7 months ago

## Tracking

### (Depends on: 1 bug, Blocks: 1 bug)

Trunk
---
Points:
---
Dependency tree / graph

## Details

(Reporter)

### Description

3 years ago
While working on removing the string used to store lists of fonts in nsFont (bug 280443), I realized that the fontlist manipulation code in nsMathMLChar.cpp would probably be better implemented in gfx.  Not sure exactly where the right division line is yet but the swizzling of fontlists within nsMathMLChar::StretchInternal is probably way too complex for what it's trying to do.  For example, running the roughly 300 mathml reftests, the SetFontFamily method is called 132,000 times (!!!).  I know we're not so concerned about MathML speed but the complexity of this code impedes progress on gfx text features so I think it would be worthwhile to restructure this a bit.

There's a discussion on bug 947654 as to what the precise font fallback behavior should be for MathML elements.  I think this bug should cover the work in gfx to support whatever behavior it is that MathML code needs.

Right now my hunch is that the right line is "given the fontlist for this element, give me the first available math-table font that supports this character" and "does this font support this set of characters". I don't think the actual fallback for stretchy characters (e.g. replace char X with parts chars A B C) should live in gfx.

### Comment 1

3 years ago
Now that the mstyle stuff has been cleaned up, I believe the stretchy code is currently the least efficient part of MathML and could be optimized.

After the changes to bug 663740, I wonder whether:
- SetFontFamily is actually still needed (perhaps only for the generic Unicode table?)
- We could replace EnumerateFamilies with just checking the list of fonts in the gfxFontGroup (again, I'm not sure about the generic Unicode table)

I didn't found time recently to read in details the discussion about the fallback, John's patches and to think about that in general, though.

### Comment 2

3 years ago
(In reply to Frédéric Wang (:fredw) from comment #1)
> After the changes to bug 663740, I wonder whether:
> - SetFontFamily is actually still needed (perhaps only for the generic
> Unicode table?)
> - We could replace EnumerateFamilies with just checking the list of fonts in
> the gfxFontGroup (again, I'm not sure about the generic Unicode table)

More precisely and as I recall, before bug 663740 nsMathMLChar had a nsRenderingContext and SetFontFamily was supposed to set the current font-family on that rendering context, and guess that the font exists on success. Also, EnumerateFamilies was needed because we worked with font-family / nsFont instead of gfxFont*.

Now I suspect that we could essentially browse the list of fonts in the gfxFontGroup, obtained from the current font-family on the <mo>. However, I'm not sure about the edge cases with fallback fonts and generic font-family...
(Reporter)

### Comment 3

3 years ago
(In reply to Frédéric Wang (:fredw) from comment #2)
> Now I suspect that we could essentially browse the list of fonts in the
> gfxFontGroup, obtained from the current font-family on the <mo>. However,
> I'm not sure about the edge cases with fallback fonts and generic
> font-family...

This is a bad idea, the abstraction needs to be one level up. MathML code should not be doing random-access on the fontlist, it should be querying gfxFontGroup to select a font that's best for supporting a given character or set of characters.  With downloadable fonts, managing the fontlist is tricky because the state can change as fonts are pulled in.  MathML code really shouldn't have to know or understand that sort of thing.
(Reporter)

### Updated

3 years ago
Depends on: 280443
(Reporter)

### Comment 4

3 years ago
I think what we need to figure out is (1) what are the parameters used to pick a font for stretchy chars and (2) what does this code need to use it for.  Whatever the various methods used in nsMathMLChar.cpp are for, it looks to be doing it *very* inefficiently, even for a single stretchy character, there are lots of textruns created, lots of fonts being experimented with *repeatedly* and lots of calls to SetFontFamily.  It really seems like we should be able to structure this such that it's a one-pass process over <fontlist> + <math fallbacks> and avoid shaping the same textrun with the same font repeatedly.

To view textruns generated by MathML code:
export NSPR_LOG_MODULES=textrun:5
(Reporter)

### Comment 5

3 years ago
Just to provide a quick sample of the inefficiencies with the MathMLChar code, take the example below, from the existing opentype-stretchy.html reftest:

<math displaystyle="true">
<mrow><mo>&#x2A1B;</mo></mrow>
[/itex]

This causes a *huge* number of textruns to be created.  Some rebuilding of textruns can be written down as due to the use of a downloadable font, but not the amount here.  Here are the textruns built *just* for the content above. First column is the count of the number of times a line appeared in the log:

15 (textrun) fontgroup: [Cambria Math] default: none lang: en-jp script: 0 len 1 weight: 400 width: 0 style: normal size:  20.00 2-byte TEXTRUN [⨛] ENDTEXTRUN
15 (textrun) fontgroup: [STIXIntegralsD] default: none lang: en-jp script: 0 len 1 weight: 400 width: 0 style: normal size:  20.00 2-byte TEXTRUN [⨛] ENDTEXTRUN
15 (textrun) fontgroup: [XITS Math] default: none lang: en-jp script: 0 len 1 weight: 400 width: 0 style: normal size:  20.00 2-byte TEXTRUN [⨛] ENDTEXTRUN
15 (textrun) fontgroup: [stretchy,serif] default: none lang: en-jp script: 0 len 1 weight: 400 width: 0 style: normal size:  20.00 2-byte TEXTRUN [⨛] ENDTEXTRUN
5 (textrun) fontgroup: [stretchy] default: none lang: en-jp script: 0 len 1 weight: 400 width: 0 style: normal size:  20.00 2-byte TEXTRUN [⨛] ENDTEXTRUN
3 (textrun) fontgroup: [stretchy] default: serif lang: en-jp script: 0 len 1 weight: 400 width: 0 style: normal size:  20.00 2-byte TEXTRUN [−] ENDTEXTRUN
2 (textrun) fontgroup: [stretchy] default: serif lang: en-jp script: 0 len 1 weight: 400 width: 0 style: normal size:  20.00 2-byte TEXTRUN [⨛] ENDTEXTRUN

So the simple string [⨛] is shaped with Cambria Math, STIXIntegralsD, XITS Math and the downloadable font 15 times for *each* font!!!

### Comment 6

3 years ago
Regarding the text runs, IIRC they are used to:

- get a glyph index from a character. There was some functions like GetGlyphID but they were not implemented on all platforms. Moreover, for bug 945183 we need to get the glyph ID of the mirrored glyph after application of the Unicode mirroring + ltrm feature and the later didn't seem possible without shaping with a textrun.

- measure a glyph (given its glyph index or char unicode value) on all platforms. Karl said this could finally become possible with the Azure project, but suggested to use gfxTextRun for now.

- paint the glyph. That's possible without a textrun but since we store them on the nsMathMLChar class, we just reuse that.

- handle both glyph by index & character. This is mostly because we handle old fonts and Unicode-based constructions. I think we could move to glyph index only once we can rely exclusively on MATH fonts.

Note that if the user has math fonts installed with the appropriate glyphs, the textruns will only be created for the first font (and perhaps again when web fonts arrive). Here the open-stretchy testcase intentionally uses an exotic glyph (integral with a top bar) to be sure that it is not available in installed math fonts, so the code is particularly inefficient since all the fonts are tested.

I believe the old code was even worse because we had to call all the TryVariants/TryParts routines for the various fonts everywhere in nsMathMLChar (both measuring phase and painting phase). Now this is only done during the textrun creation phase. But of course, I'd really like to see this improved...

### Comment 7

3 years ago
Shaping is designed to be efficient for multiple instances of the same word on a page, so inefficiency from multiple text should be coming from multiple allocations and copies rather than repeated font work.

Some of the logic in nsMathMLChar is more complex than it needs to be.  Making the code as simple as possible while still doing the right thing is more important than performance here.  Welcome performance improvements often come with simplification too.
(Reporter)

### Comment 8

3 years ago
(In reply to Karl Tomlinson (needinfo?:karlt) from comment #7)
> Shaping is designed to be efficient for multiple instances of the same word
> on a page, so inefficiency from multiple text should be coming from multiple
> allocations and copies rather than repeated font work.
>
> Some of the logic in nsMathMLChar is more complex than it needs to be.
> Making the code as simple as possible while still doing the right thing is
> more important than performance here.  Welcome performance improvements
> often come with simplification too.

I think we're on the same page here, I think this code needs to be simpler.  The fact that *so* many identical textruns are being recreated is to me more an indication that something needs to be thought through a bit better. The complexity here makes changing code in other parts of Gecko *much* harder, which is definitely what I experienced working on bug 280443.

For this bug, as described in the third paragraph of the description, I think code that's sniffing through a list of fonts should live in gfx. Both for efficiency and code simplicity. Plus the loading behavior associated with downloadable fonts means that iterating over a list of fonts is not a simple task that should be exposed through some form of iterator-like API.

### Comment 9

3 years ago
As I said 947654 comment 43, one of the issue is that the nsMathMLChar code is really just doing some kind of text shaping so it should really mostly belong to gfx (and perhaps other parts like e.g. text-shadow & background selection support should try to be in nsMathMLmoFrame and/or shared with nsTextFrame). Since it currently lives in layout/ we try or want to use gfx API or reinvent the wheel via textrun manipulations. It's possible that if we move most of the code to gfx, such APIs won't really be necessary...

I believe the first step would be to remove support for old fonts + implementing the Unicode fallback differently (perhaps by creating a fake OpenType MATH table or nsOpenTypeGlyphTable instance). This would already simplify the code considerably, avoid the need for external *.properties files, and help migration to gfx. Currently, we handle both the old and new fonts, which sometimes do not behave exactly the same. I wonder how long we want to wait before removing support for these old fonts?

(There is another problem that the MathML code is maintained by volunteers and so does not evolve as fast as other parts of Gecko. It was broken 8 years ago until Karl repaired it and hadn't evolved much until I found time to finally work on the OpenType MATH support recently. I suspect it would have been even harder to switch to arrays of font-family before the recent changes, because the font families were use everywhere in both measuring and painting phase. So seeing the glass half full, it was not as difficult as it could have been to do bug 280443 for MathML)

### Comment 10

3 years ago
FYI, some people at MathUI said that recent versions of Firefox are slower to display MathML for very large pages. I wonder if that has something to do with bug 663740.
(Reporter)

### Comment 11

3 years ago
(In reply to Frédéric Wang (:fredw) from comment #10)
> FYI, some people at MathUI said that recent versions of Firefox are slower
> to display MathML for very large pages. I wonder if that has something to do
> with bug 663740.

This should be relatively easy to test. Can you come up with a set of pages/content to test with? Given the structure of the existing code I think there's probably ample room for perf improvements. :)

### Comment 12

3 years ago
IIUC, this happened for large pages generated with LaTeXML, so checking the XHTML papers of ArXiv would do:

http://arxmliv.kwarc.info/

I haven't had time to check exactly, but just wanted to raise the issue to motivate further improvements and before I forget.

3 years ago
Blocks: 1043358

### Updated

3 years ago
No longer blocks: 1043358

### Comment 13

3 years ago
Testing in bug 1043358 shows a significant decrease in rendering time when the stretch performing <mo> elements are replaced with <mtext> elements.

### Comment 14

3 years ago
(In reply to James Kitchener from comment #13)
> Testing in bug 1043358 shows a significant decrease in rendering time when
> the stretch performing <mo> elements are replaced with <mtext> elements.

In that context, is there any testing I can do the help?

### Comment 15

3 years ago
Some numbers:

bug 1043358 comment 46
bug 1061349 comment 4

### Comment 16

3 years ago
In bug 1139709, after the improvement patches there I observed the testcase spending >50% of its load time in nsMathMLChar::GetMaxWidth (mostly in StretchInternal).

2 years ago
Blocks: 1162409

### Updated

2 years ago
Depends on: 1007090

### Comment 17

a year ago
@Karl: I started implementing the access to math values as well as the shaping of math stretchy operator in Harfbuzz, see https://github.com/behdad/harfbuzz/pull/241. Can you please check the proposed API (hb-ot-layout.h and hb-ot-shape.h) and indicate if you think that would be enough to replace nsMathMLChar completely? Thanks!
Flags: needinfo?(karlt)

### Comment 18

a year ago
Is there a separate API for largeop?

nsMathMLChar.h has some other flags also.

LARGER and SMALLER might be the only ones that are really necessary.  Perhaps
they can be replaced with scaling when drawing the buffer, assuming a new
buffer is used for the character?

Would be nice to pass an enum or flag, instead of 0/1, for horizontal/vertical,
but that's something for hb developers to consider.  hb_direction_t is too
different to overload I guess.
Flags: needinfo?(karlt)

### Comment 19

a year ago
(In reply to Karl Tomlinson (ni?:karlt) from comment #18)
> Is there a separate API for largeop?

I initially proposed one, but Khaled pointed out that we can just ask normal stretching to the HB_OT_MATH_CONSTANT_DISPLAY_OPERATOR_MIN_HEIGHT size (or any additional workaround we have for integral etc). Getting the "max orthogonal advance" is not needed for large operator.

> LARGER and SMALLER might be the only ones that are really necessary.  Perhaps
> they can be replaced with scaling when drawing the buffer, assuming a new
> buffer is used for the character?

I think the MathML spec asks LARGER, which is how I currently implemented it. I don't know exactly in which situation SMALLER is needed but that should not be hard to implement.

hb-ot-shape-math-stretchy reuses the same buffer.

>
> Would be nice to pass an enum or flag, instead of 0/1, for
> horizontal/vertical,
> but that's something for hb developers to consider.  hb_direction_t is too
> different to overload I guess.

Yes, I had the same thought.

Other points to discuss are:
- Do we want the Unicode table in Harfbuzz or keep doing it ourselves or just remove it?
- Will that be enough to implement the font selection (this bug)? Maybe we want to avoid mixing stretchy operators from different fonts and so always use the font of the base glyph and hence stop at the first successful call to hb-ot-shape-math-stretchy? Or otherwise, we'll have to call hb-ot-shape-math-stretchy several times and select the best size which may not really improve performance here.

### Comment 20

a year ago
(In reply to Frédéric Wang (:fredw) from comment #19)
> I think the MathML spec asks LARGER, which is how I currently implemented
> it. I don't know exactly in which situation SMALLER is needed but that
> should not be hard to implement.

With a quick look through Gecko, it seems normal or nearer are usually used,
with larger for roots, and smaller only for the maximum width.

Would be interesting to know what other rendering engines do re nearest or
larger.  Should nested parentheses always get larger?

> Other points to discuss are:
> - Do we want the Unicode table in Harfbuzz or keep doing it ourselves or
> just remove it?

If we can expect a better MATH font to be available, then we can remove Unicode table support, but I suspect we still have many systems without a better MATH font.

If there are some MATH fonts on the system, then it is less clear whether to
use the Unicode table with the author specified font or to look for a MATH
font.  The MATH font will presumably give more size options.  Some experimentation, may answer this question.

It seems the Unicode table fallback could be done in Harfbuzz (though I'm not
familiar with the details of implementation there), but if it is eventually
going to be phased out, then there's probably not much point adding it to
harfbuzz.

> - Will that be enough to implement the font selection (this bug)? Maybe we
> want to avoid mixing stretchy operators from different fonts and so always
> use the font of the base glyph and hence stop at the first successful call
> to hb-ot-shape-math-stretchy? Or otherwise, we'll have to call
> hb-ot-shape-math-stretchy several times and select the best size which may
> not really improve performance here.

I don't know exactly what the proposal was for this bug.  Harfbuzz support
provides the stretching, but not font selection.  I'd guess always using the
font of the base glyph would be a bit restrictive as the base glyph may be in
a font that does not have a MATH table.  Stopping at the first successful call
to hb-ot-shape-math-stretchy seems reasonable, but, if it provides Unicode
table fallback, finding a MATH font may be better.  I don't think it is
necessary to try several MATH fonts.

It won't be practical to call hb-ot-shape-math-stretchy, or even check for a
MATH table, on every font for fallback, so this will need to be restricted to
the set of author-specified fonts + some known MATH fonts.

If searching for a font to shape with the Unicode table, then I guess
calling hb-ot-shape-math-stretchy on each font will be slow.  The optimal way
would be to search for a font that supports all the Unicode characters
required and then shape.

### Comment 21

a year ago
(In reply to Karl Tomlinson (ni?:karlt) from comment #20)
Regarding the target size, the MathML spec says that "operators should stretchy to cover the target size" which means NS_STRETCH_LARGER (which is actually larger or equal). This is how it is implemented in WebKit. For the preferred width, WebKit takes the maximum of all possible constructions for the current font for vertical operators (we do not know the target size) and simply the target size for vertical operators.

The MATH table only describes how to extend delimiter until "that is enough to achieve the size goal" and does not have equivalent parameters, so I'm not sure if they really specify a rule.

The TeX book in the chapter 17 mentioned in NS_STRETCH_NEARER comment describes stretching of |\left| and~|\right| which are equivalent to stretching of symmetric fences in MathML. However, they allow the size to be smaller: "It is usually best not to cover the formula completely, however, but just to come close; so \TeX\ allows you to specify two parameters [...] The minimum delimiter size is taken to be at least $y\cdot f/1000$, and at least $y-\delta$. Appendix~B sets $f=901$ and $\delta=5\pt$". Appendix G rule 19 describes the same rule. Note that with the TeX definition, NS_STRETCH_NEARER is actually NS_STRETCH_LARGER with a smaller size so it might not be needed to have a separate parameter for that.

More generally, I think all the parameter could be implemented by specifying a minsize and maxsize (which exists in MathML):
- NS_STRETCH_LARGER: minsize = target size, maxsize = infity
- NS_STRETCH_NORMAL: minsize = target size - 10%, maxsize = target size + 10%
- NS_STRETCH_NEARER: minsize = max(target size - 10%, target size - 5pt), maxsize = infity (or target size + something if we do not follow TeX rules)
- NS_STRETCH_SMALLER: minsize = 0, maxsize = target size

What do you think?

### Comment 22

a year ago
(In reply to Frédéric Wang (:fredw) from comment #21)
> - NS_STRETCH_SMALLER: minsize = 0, maxsize = target size

Well that's probably not very good for that case. However, I wonder whether we really need this for the preferred max width.

Also, I suspect NS_STRETCH_NORMAL and NS_STRETCH_NEARER could just be interpreted as NS_STRETCH_LARGER with a smaller target size. In the same font, the size variants should in theory grow in the order they are tested and finally we can try the assembly. The "+ something" seems only useful if we are testing different fonts where we want to get the best match over all the fonts. But I'm not sure it's really worth it if we move to the situation where we try only the first MATH font.

### Comment 23

a year ago
So I think the rule of thumb is that a font with a MATH table will give the best math rendering in general not only for stretchy operators but also for unicode coverage, math values, font features etc So we want to try and move to that situation. Doing math layout with Unicode-only font will only be useful for basic equations or for people with low expectation on rendering quality. Ideally, more fonts with a MATH table should be designed to accompany popular text fonts and should be available by default on systems.

1) We DOM/CSS context is
- the base Unicode character (+ its CSS direction)
- the font-family specified by author + some known fallback MATH fonts.
- a target size

2) We do the following attempts to select a font from the list:

a) We try and find a MATH font with a given glyph for the base character.

b) If there is a Unicode construction for that base character, find the first font that has glyphs for all the Unicode characters involved in the construction.

c) Otherwise, find the first font with a glyph for the base character.

For a), the proposed Harfbuzz API allows to check whether a font has a MATH table (just a non-null pointer check) but not whether it has a MathGlyphConstruction for some glyph. This can be added if necessary but that will probably make the font selection slower. Also, using different MATH fonts can make rendering style inconsistent with constructions coming from a different than the base size.

For b), we can probably start simplifying our Unicode fallback by only using glyph assembly (not size variants) and finding constructions by binary search in a hardcoded table rather than loading the *.properties file. This is what I started to do in Harfbuzz https://github.com/behdad/harfbuzz/pull/241/files#diff-db2d1e0cef191d004e282d35a21f44aeR215 but as you said it's probably not worth doing it if that's going to go away and that would allow to simplify/improve the Harfbuzz proposal.

3) Once we have a font then

a) If we have a MATH font, use Harfbuzz API to do
hb_ot_shape_math_stretchy_max_orthogonal_advance (for preferred width computation) and hb_ot_shape_math_stretchy (for selecting the construction when we know the target size and doing the shaping).

b) If we have a font with Unicode construction, do the same with our own
implementation. We only have limited glyph and no overlap so that can be
simpler.

c) Otherwise just use the base character (maybe with the scale fallback).

As I previously say, assuming that the constructions in a given MATH font have increasing sizes and with the simplification of using only glyph assembly for the Unicode fallback the stretching in a) and b) can be reduced to a "NS_STRETCH_LARGER".

### Comment 24

a year ago
Comment 23 sounds sensible for stretchy characters.

I'm not sure whether you were intending that for other characters also, but
preferring MATH fonts for over author-specified fonts for regular characters
would be quite a departure from the normal CSS font selection preferring
author-specified fonts and then looking for a language-supporting font in
fallback.  Perhaps that is not a bad departure.  I often wish Firefox would
use my preferred fonts instead of author-specified fonts.

I'm not sure what to think about minsize/target/maxsize.
It corresponds with minsize/maxsize in MathML, but we don't want
hb_ot_shape_math_stretchy to fail when there is nothing between minsize and
maxsize.  We want the nearest size available.

What might differ in different situations is what is considered "nearest".
But if this is always "larger or equal to", then there is no need to provide
other variations.  A target size alone is sufficient.

The difficulty with "larger or equal to" requirements is that often "larger or roughly equal to" is what is really meant, and then "roughly" becomes vague and really depends on how much larger the other option is.

I think we actually always have a preferred/target size.  My personal opinion is that finding the nearest glyph and scaling to the preferred size is best, but I know we have different preferences with scaling.  "nearest" may be in a geometric / log scale sense where 2/3 and 3/2 are equidistant from 1.  I'm happy to leave the choice to you.

### Comment 25

a year ago
(In reply to Karl Tomlinson (back 26 Apr, ni?:karlt) from comment #24)
> Comment 23 sounds sensible for stretchy characters.
>
> I'm not sure whether you were intending that for other characters also

No, the present bug entry is only for stretchy operators. Although ideally people will use MATH fonts for the math text too so that math layout constants and rendering style makes sense.

> I'm not sure what to think about minsize/target/maxsize.
> What might differ in different situations is what is considered "nearest".

Since there can be several definitions of "nearest", I believe the best for the Harfbuzz API would probably be to pass a function pointer (maybe NULL)

hb_bool_t isNearer(hb_position_t a, hb_position_b)

so that Harfbuzz can find the nearest option among the possible output sizes.

One of my problem was that we have two different kinds of stretching: either use a size variant or use a glyph assembly, and my choice of "larger" was probably influenced by the latter. For the size variant, it is obvious how to find the "nearest" size from a list. For glyph assembly, the MATH spec defines an iterative process to progressively grow the size that would allow to do that to:

> To ensure that the width/height is distributed equally and the symmetry of the shape is preserved, following steps can be used by math handling client.
>
>    Assemble all parts by overlapping connectors by maximum amount, and removing all extenders. This gives the smallest possible result.
>    Determine how much extra width/height can be distributed into all connections between neighboring parts. If that is enough to achieve the size goal, extend each connection equally by changing overlaps of connectors to finish the job.
>    If all connections have been extended to minimum overlap and further growth is needed, add one of each extender, and repeat the process from the first step.

However, I adding extender one by one did not sound efficient so I directly implemented the glyph assembly stretching using some direct computation to find the repetition of extenders and overlap that gives the best upper approximation of target size:

https://github.com/fred-wang/harfbuzz/blob/MATH-2/src/hb-ot-shape-math.cc#L303

It was less clear how to make the algorithm more general, but now I think this calculation can be followed by some decrement of repeatCountExt and increase of connectorOverlap to find a better result with respect to some definition of "nearest". I could also calculate a lower bound for repeatCountExt so that I have an interval [repeatCountExt-, repeatCountExt+] to test with a binary search...
Flags: needinfo?(karlt)

### Comment 26

a year ago
(In reply to Frédéric Wang (:fredw) from comment #25)
> Since there can be several definitions of "nearest", I believe the best for
> the Harfbuzz API would probably be to pass a function pointer (maybe NULL)
>
> hb_bool_t isNearer(hb_position_t a, hb_position_b)
>
> so that Harfbuzz can find the nearest option among the possible output sizes.

So thinking again to this, something like IsSizeOK is not needed with the proposal of comment 23 because once we have a font selected we always try and find the best match for the request.

So we only want IsSizeBetter which essentially has three cases: least upper bound (LARGER, LARGEOP, INTEGRAL), greatest lower bound (SMALLER) or nearest (NEARER and NORMAL). So I guess Harfbuzz should just receive an enum parameter with these three cases (+ maybe a special flag to distinguish LINEAR or LOG scale).

> It was less clear how to make the algorithm more general, but now I think
> this calculation can be followed by some decrement of repeatCountExt and
> increase of connectorOverlap to find a better result with respect to some
> definition of "nearest". I could also calculate a lower bound for
> repeatCountExt so that I have an interval [repeatCountExt-, repeatCountExt+]
> to test with a binary search...

Thinking again to this, there are actually only two repeatCountExt to check so that should be fine.

### Comment 27

a year ago
(In reply to Frédéric Wang (:fredw) from comment #26)
> (In reply to Frédéric Wang (:fredw) from comment #25)
> > Since there can be several definitions of "nearest", I believe the best for
> > the Harfbuzz API would probably be to pass a function pointer (maybe NULL)
> >
> > hb_bool_t isNearer(hb_position_t a, hb_position_b)
> >
> > so that Harfbuzz can find the nearest option among the possible output sizes.
>
> So thinking again to this, something like IsSizeOK is not needed with the
> proposal of comment 23 because once we have a font selected we always try
> and find the best match for the request.
>
> So we only want IsSizeBetter which essentially has three cases: least upper
> bound (LARGER, LARGEOP, INTEGRAL), greatest lower bound (SMALLER) or nearest
> (NEARER and NORMAL). So I guess Harfbuzz should just receive an enum
> parameter with these three cases (+ maybe a special flag to distinguish
> LINEAR or LOG scale).

I agree harfbuzz doesn't need IsSizeOK.

I don't have a strong opinion re the choice between enum and function.
The function is versatile.  It is less convenient to use than an enum, but I guess the default (NULL) is usually appropriate, so that's not a problem.  If using that approach, it needs a target size or closure parameter, to know which of a or b is nearer.
Flags: needinfo?(karlt)

### Comment 28

a year ago
(In reply to Karl Tomlinson (back 26 Apr, ni?:karlt) from comment #27)
> I don't have a strong opinion re the choice between enum and function.
> The function is versatile.  It is less convenient to use than an enum, but I
> guess the default (NULL) is usually appropriate, so that's not a problem.

I think the enum will be more convenient for the implementation of the glyph assembly stretching since you can basically try to approximate the target size and then adjust a bit the number of parts / overlap to get the best size around that first approximation. Whereas a IsSizeBetter function would be too general and would not provide a logical order to test the size. I didn't check what we do, though.

Note that Harfbuzz public API is in C so actually we can not have optional parameters and all HarfBuzz users will have to explicitly specify a function or an enum. But actually do we really need all these stretchy options or is it just a legacy of old implementation that we can get rid of? In the past, I never really tried to think about what we do and just tried to preserve the current behavior...

I mean, as I see the MathML/MATH spec seems to only require the LARGER and there is only this TeXBook suggestion to reduce a bit the target size for fences. The NEARER and NORMAL allowing an interval around the target size seems specific to our implementation (and an extension of the TeXBook suggestion). At least I didn't use that when I implemented stretching in WebKit and Khaled thinks this is not needed by other math engines.

Preferred width computation seems specific to web engines (or at least engines aware of CSS linebreaking). But I'm not clear about the use of SMALLER for that. If we choose to overestimate the width (as we do elsewhere in the MathML code) then for vertical stretching, we basically only want the maximum width for all size variants & pieces of the assembly and for horizontal stretching, we can just use LARGER on the target size (WebKit actually uses the target size itself).

> If using that approach, it needs a target size or closure parameter, to know
> which of a or b is nearer.

Yes, that was obviously a mistake on my side.
Flags: needinfo?(karlt)

### Comment 29

a year ago
For stretching, Gecko uses only nearest and larger really.  NEARER only makes a difference if other fonts are going to be considered.

Gecko usually uses nearest (or similar).

If an expression includes a mixture of different fences ||{}[](), then these may have slight differences in height.  The larger (or equal) approach could lead to the outer fences being considerably larger than the inner fences.  This sounds undesirable in textstyle at least IMO.  Maybe displaystyle too. I'm not sure.

If an implementation wants an operator of exact size, then it will need to scale, and I guess the best glyph to scale is the nearest.  Again this seems useful in textstyle at least.

It seems that there is reason to think that clients may have different concepts of best glyph in different situations, but nearest and larger are probably enough to cover that.

Variant glyphs are reasonably simple to handle I guess?  Perhaps the choice here can be left to the client to determine.  Would it make sense for harfbuzz to focus on APIs for glyph assembly and its minimum size?

Am I overlooking complexities with variant glyphs that harfbuzz would hide because it knows the target size?

SMALLER is for the case where the maximum size of the stretch is known,
through a maxsize attribute I guess, and so the width of only smaller glyphs
needs to be considered.  I think it is not actually used right now because the
maxsize attribute is not handled.  SMALLER is only a stretch flag because of
the implementation.  It would make more sense to have a maxsize a parameter to
hb_ot_shape_math_stretchy_max_orthogonal_advance().  But if variants are easy
enough to query then perhaps harfbuzz should just provide an API for the size
of the glyph assembly?
Flags: needinfo?(karlt)

### Comment 30

a year ago
(In reply to Karl Tomlinson (back 26 Apr, ni?:karlt) from comment #29)
> If an expression includes a mixture of different fences ||{}[](), then these
> may have slight differences in height.  The larger (or equal) approach could
> lead to the outer fences being considerably larger than the inner fences.
> This sounds undesirable in textstyle at least IMO.  Maybe displaystyle too.
> I'm not sure.

I'm not sure I understand this, I see two cases:

1) <mrow><mo>[</mo><mo>(</mo><mo>{</mo> SOMECONTENT <mo>}</mo><mo>)</mo><mo>]</mo></mrow>
2) <mrow><mo>[</mo><mrow><mo>(</mo><mrow><mo>{</mo> SOMECONTENT <mo>}</mo></mrow><mo>)</mo></mrow><mo>]</mo></mrow>

In 1), per the MathML spec the target size is the max height of the non-stretchy element in the <mrow>, that is SOMECONTENT (I'm not sure that we only consider non-stretchy elements btw). All the fences will then use the same target size.

In 2), braces will use target size the height of SOMECONTENT, parenthesis the same + some delta, brackets parenthesis + some delta + some other delta. Indeed, that can lead to parenthesis being much bigger, but I'm guessing that a) the TeX rule of using reducing a bit the target size could help to reduce the increase. b) Fences in the same font provide similar size, so they won't really get bigger after the inner stretch that is some other delta = 0. Of course we would have to try...

> If an implementation wants an operator of exact size, then it will need to
> scale, and I guess the best glyph to scale is the nearest.  Again this seems
> useful in textstyle at least.

I'm not really sure that considering the nearest upper instead of just the nearest is really problematic. And I think MATH-aware engines won't do the scale anyway.

> Variant glyphs are reasonably simple to handle I guess?  Perhaps the choice
> here can be left to the client to determine.  Would it make sense for
> harfbuzz to focus on APIs for glyph assembly and its minimum size?

I think so. Khaled already suggested to exclude the base size as well as other hacks we do for display operator size from the stretch API so it makes sense to only keep glyph assembly.

> Am I overlooking complexities with variant glyphs that harfbuzz would hide
> because it knows the target size?

No, it's only a loop testing the size and finding the largest one. The only difference with our implementation is that it uses the AdvanceMeasurement value from MathGlyphVariantRecord Table which may lead to different values that the one provided by standard functions to retrieve glyph metrics (and hopefully closer to the designer's expectation).

And the shaping is essentially a no-op.

> SMALLER is for the case where the maximum size of the stretch is known,
> through a maxsize attribute I guess, and so the width of only smaller glyphs
> needs to be considered.  I think it is not actually used right now because
> the
> maxsize attribute is not handled.  SMALLER is only a stretch flag because of
> the implementation.

OK, maxsize is specific to the MathML implementation. But actually I'm not sure it's really used in practice so the gain we obtain for the preferred width implementation is not clear.

> It would make more sense to have a maxsize a parameter
> to
> hb_ot_shape_math_stretchy_max_orthogonal_advance().  But if variants are easy
> enough to query then perhaps harfbuzz should just provide an API for the size
> of the glyph assembly?

OK, we could just have hb_ot_shape_math_stretchy_max_orthogonal_advance for the glyph assembly.

Khaled, what do you think about exposing:

1) variant count
2) variant id, variant advance (probably in the same function)
3) glyph assembly shaping
4) glyph assembly advance and italic correction (probably in the same function)
Flags: needinfo?(khaledhosny)

### Comment 31

a year ago
(In reply to Frédéric Wang (:fredw) from comment #30)
> 3) glyph assembly shaping
> 4) glyph assembly advance and italic correction (probably in the same
> function)

I meant "max_orthogonal_advance", but actually this is not really needed. We can just use 3) with a target size 0 and then the x_advance or y_advance of the last glyph in the output buffer will give what you want.

Also, I believe I can easily add some boolean to 3) in order to check whether we get better result with the lower size o = o_min and r = r_min - 1 (using notations from http://www.maths-informatique-jeux.com/blog/frederic/?post/2016/04/16/OpenType-MATH-in-HarfBuzz)

### Updated

11 months ago
Flags: needinfo?(khaledhosny)

### Updated

11 months ago
Depends on: 1305977

### Comment 32

7 months ago
So coming back to the initial idea without trying to rewrite the whole stretch code for now:

Currently, StretchInternal browses the list of font family names and call StretchEnumContext::EnumCallback on it. For each name, we search whether the font has a MATH table, or a non-Unicode *.properties table, or fallback to the Unicode table. For a generic font name, we actually always use the Unicode table.

After bug 1007090, I think we could just try and get the fm->GetThebesFontGroup()->GetFirstMathFont() and use it for an nsOpenTypeTable. Otherwise, just fallback to the Unicode table. The only difference is that the Unicode table will no longer be tested for each single font name but that's probably not a serious issue. I believe in that case, we no longer need to append the "font.name.serif.x-math" list, as it is done automatically during style resolution?

Karl, WDYT?
Flags: needinfo?(karlt)
You need to log in before you can comment on or make changes to this bug.