1673510 - U+301C vertical glyph substitution may fail as the only character of text node

Status: RESOLVED FIXED

(Core :: Layout: Text and Fonts, defect)

Description

[747.neutron]

Attachment: bug-vertical.zip

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:82.0) Gecko/20100101 Firefox/82.0

Steps to reproduce:

OS: Windows 10 1909
Unzip the attached ZIP into any folder and open the HTML on Firefox (make sure the font is loaded).

Actual results:

The last paragraph contains a single character 〜 (WAVE DASH; U+301C) sandwiched by two span elements styled with text-combine-upright: all;. It is rendered in weird way as if just has its horizontal glyph 90-degree turned, which is the flipped image of the correct shape.

If you remove the either span's class or style, or putting any non-punctuation letter next to it, the glyph will be immediately displayed in the correct shape. It also turns out that the wrong glyph shows up generally when U+301C is the only character in the text node with the attached font.

Expected results:

Always shows the correct glyph in vertical writing modes.

Apparently, the font has something to do with this issue, because most CJK typefaces do not exhibit such a bug (you can try with Noto Serif JP, cited in the attached CSS). However, I didn't see any obvious issue in the font on FontForge, and it actually renders correctly on Chrome.

Comment 1

[747.neutron]

Attachment: スクリーンショット 2020-10-27 073233.png

U+301C in wrong shape

Comment 2

[747.neutron]

Attachment: スクリーンショット 2020-10-27 073141.png

U+310C in correct shape, when text-combine-upright: all removed from the span before it

Comment 3

[747.neutron]

Attachment: スクリーンショット 2020-10-27 075041.png

Rendering on Chrome 86 (same font, correct glyph)

Comment 4

[Jonathan Kew [:jfkthame]]

This turns out to be a combination of script-run resolution behavior and details of the font's OpenType tables.

The character U+301C has a Script_Extensions property of Bopo Hang Hani Hira Kana. When there are other letters in the run, it'll resolve to the same script as its neighboring characters, but when it occurs by itself, we (arbitrarily) default to the first of the scripts listed. And unfortunately, although the GSUB table in this font lists the bopo script tag, its feature list does not include the vert feature.

I guess that if Bopomofo is written vertically, it would be appropriate to use the vertical alternates of punctuation, so this is arguably a bug in the font. It should include the vert feature in the feature list for the bopo script.

We could modify the script run itemizer so that it would prefer to tag an isolated punctuation mark like this as Han rather than Bopomofo, in the hope that it is more likely to be supported by fonts; but the issue would still occur if the character occurred surrounded by Bopomofo letters.

Curiously, in Chrome the wavy-dash renders its vert alternate even if surrounded by Bopomofo letters, which suggests they are resolving its script for shaping purposes as Han (or maybe Kana) even in this context. I'm not sure what their basis for that is (or whether it's even a deliberate feature or just an accident of the implementation).

Comment 5

[747.neutron]

It totally makes sense, thank you. Injecting vert into the bopo entry works out nicely. I'd contact the font author to correct the table.

By the way, if you have extra time, could you enlighten me about a small thing? The said glyph apparently handles the substitution via vrt2 instead of vert in that font. But I could only make it work by adding vert in GSUB, not vrt2. Is it due to some OpenType spec, or another font-specific implementation? (or possibly my misunderstanding on ttx program.)

Comment 6

[Jonathan Kew [:jfkthame]]

(In reply to 747.neutron from comment #5)

It totally makes sense, thank you. Injecting vert into the bopo entry works out nicely. I'd contact the font author to correct the table.

By the way, if you have extra time, could you enlighten me about a small thing? The said glyph apparently handles the substitution via vrt2 instead of vert in that font. But I could only make it work by adding vert in GSUB, not vrt2. Is it due to some OpenType spec, or another font-specific implementation? (or possibly my misunderstanding on ttx program.)

According to the CSS Writing Modes spec, the vert feature is used to perform glyph substitutions for upright-vertical text, but vrt2 is explicitly not used.

(It looks to me like the font has that substitution, in GSUB lookup index 14, as part of both the vert and vrt2 features. But browsers following the CSS spec here will only be applying vert.)

Comment 7

[747.neutron]

(In reply to Jonathan Kew (:jfkthame) from comment #6)

According to the CSS Writing Modes spec, the vert feature is used to perform glyph substitutions for upright-vertical text, but vrt2 is explicitly not used.

Oh, I didn't imagine that the CSS spec explicitly oversees it!

Meanwhile, sorry for extending the thread, but as I have tested further, U+301C outputs a wrong glyph with that font in that context, but U+3001 (、) and U+3002 (。) don't. They all have vrt2 substitution and are associated with bopo in the font. What occurs to me is that Firefox does some smart things based on General Category (Pd vs Po) or Vertical Orientation (Tr vs Tu), though if the latter, failure in resolving feature should logically result in wrong upright glyphs too. What is actually happening?

(In reply to Jonathan Kew (:jfkthame) from comment #4)

Curiously, in Chrome the wavy-dash renders its vert alternate even if surrounded by Bopomofo letters, which suggests they are resolving its script for shaping purposes as Han (or maybe Kana) even in this context. I'm not sure what their basis for that is (or whether it's even a deliberate feature or just an accident of the implementation).

I found a 2015 topic in the Chromium bug tracker that may be related to this: Vertical alternate glyphs are not used in complex path for TypeSquare Japanese fonts, though I'm not sure if it helps... (Their final solution is just to remove the logic and leave everything to HarfBuzz.)

Comment 8

[Jonathan Kew [:jfkthame]]

Firefox does treat Tu and Tr characters somewhat differently: if a character has orientation=Tr, and we don't find an alternate glyph for it (using the vert feature for the resolved script, or as a Unicode-encoded vertical presentation form), then it will fall back to Rotated rendering as per the header comments in https://www.unicode.org/Public/UCD/latest/ucd/VerticalOrientation.txt. That's what is happening to U+301C in isolation, because we resolve the script to Bopomofo and there's no vert alternate provided.

For characters like U+3001 with orientation=Tu, rendering will always be in vertical-upright mode. And in this case, harfbuzz will apply a vert feature even if the script/language doesn't match what the font provides, https://github.com/harfbuzz/harfbuzz/issues/63. So that's what is making U+3001/U+3002 work. It doesn't help for U+301C because Firefox determined that it should get Rotated rendering.

So this implies that a possible fix here would be to modify the check we do for characters with orientation=Tr, to give them Upright rendering if the font has any vert feature, regardless of the script. This would more closely match what harfbuzz does internally as a workaround for fonts with incomplete OpenType support.

Comment 9

[Jonathan Kew [:jfkthame]]

Attachment: Bug 1673510 - Improve text-orientation run itemization to better match harfbuzz heuristics for handling vertical alternates. r=lsalzman

Comment 10

[747.neutron]

Thank you for the fix. Regarding the font file in my test bundle, it is an impromptu subset of this font. I should have changed the font name in order to redistribute it, but you can also use the original one (OFL 1.1) if you see fit.

Additionally, I found that the font's behavior of unrotated Bopomofo is correct according to the Taiwanese standard (https://www.unicode.org/L2/L2018/18020-std-var-seq-bopomofo-i.pdf), which most likely continues to be supported by font features in future (https://www.unicode.org/L2/L2018/18052-bopomofo-cmt.pdf).

Also we may need to keep an eye on the outcome of this discussion on bopomofo tone marks AFAIK already worked on by UTC experts (https://www.unicode.org/L2/L2020/20175-utc164-properties-rec.pdf; Section F5), that whether it will turn out rightly Vertical Orientation U and whether it will involve Script Extension Hani.

Comment 11

[Jonathan Kew [:jfkthame]]

(In reply to 747.neutron from comment #10)

Thank you for the fix. Regarding the font file in my test bundle, it is an impromptu subset of this font. I should have changed the font name in order to redistribute it, but you can also use the original one (OFL 1.1) if you see fit.

Thanks for the pointer; I did a brief search for the font name but wasn't confident I found the right source. I've created a minimal testcase using a tiny subset font.

Comment 12

[Jonathan Kew [:jfkthame]]

Attachment: Bug 1673510 - Add reftest for vertical alternate of wavy dash alone or in bopomofo context. r=lsalzman

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/07e4783806b0
Improve text-orientation run itemization to better match harfbuzz heuristics for handling vertical alternates. r=lsalzman
https://hg.mozilla.org/integration/autoland/rev/afdc113c9906
Add reftest for vertical alternate of wavy dash alone or in bopomofo context. r=lsalzman

Comment 14

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/07e4783806b0
https://hg.mozilla.org/mozilla-central/rev/afdc113c9906