1927458 - Don't replace newline between IVS and next han (kanji) with space. Just remove it

Status: VERIFIED FIXED

(Core :: Layout: Text and Fonts, defect)

Description

[Tatsunori Uchino]

Attachment: スクリーンショット 2024-10-28 224334.png

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:131.0) Gecko/20100101 Firefox/131.0

Steps to reproduce:

https://codepen.io/tats-u/pen/PoMQEyQ

Render the following text as HTML

葛󠄀
葛󠄀
葛󠄀
葛
葛
葛
󠄀```

Only the first 3 have IVS (U+E0100).


Actual results:

Spaces are inserted after the first 3 characters

葛󠄀 葛󠄀 葛󠄀 葛葛葛


Expected results:

No spaces are inserted

葛󠄀葛󠄀葛󠄀葛葛葛

Comment 1

[Ardelean Oana, Desktop Test Engineering [:oardelean]]

Hello, thank you for the bug report!

Managed to reproduce this issue on:

• Firefox Nightly 134.0a1;
• Firefox 133.0b1;
• Firefox 132.0;

Tested and reproduced on:

• macOS 14;
• Ubuntu 22;
• Windows 10;

Setting as NEW so the developing team can have a look.

Comment 2

[Jonathan Kew [:jfkthame]]

Yes, this is a flaw in the segment break transformation behavior implemented in TransformWhiteSpaces: it looks only at the single Unicode codepoints immediately preceding and following the segment break, and checks their EastAsianWidth to determine the behavior.

However, if there's a variation selector immediately before the break, we should really look back to the base character preceding it. And I guess we should skip other "default-ignorable" characters, too, if they happen to be present either before or after the segment break that we're considering.

(Most of them are highly unlikely to occur in any real text where we want segment-break transformation, but the most logical thing to do with default-ignorables in unexpected places is.... well.... to ignore them.)

Comment 3

[Jonathan Kew [:jfkthame]]

Attachment: Bug 1927458 - Skip past any default-ignorable codepoints when evaluating the context of a segment break. r=#layout

Comment 4

[Tatsunori Uchino]

IVS (U+E0100 is dedicated to han unlike SVS (U+FE0x), so we don't need to see its previous code point.

葛︀
葛󠇠

These examples are not ideal.

For SVS example, I raise 社︀ (U+793E U+FE00 (= U+FA4C)), 福︀ (U+798F U+FE00 (=U+FA1B)). 葛 is a typical example only for IVS.

Japanese fonts that support IVS normally cover the following IVS forms:
https://wakufactory.jp/densho/font/ivs_aj.html

For another famous example of IVS is "竈門禰󠄀豆子". (禰󠄀 = U+79B0 U+E0100)

Comment 5

[Tatsunori Uchino]

IVS (U+E0100

↓

IVS (U+E0100–U+E01EF)

Comment 6

[Tatsunori Uchino]

while (IsDefaultIgnorable(ucs4before) && pos > 0);

I have not seen the concrete implementation of IsDefaultIgnorable, but we should be careful about U+FE0E/U+FE0F (EVS).
For example, U+3299 U+FE0F (㊙️) is definitely an emoji, but U+3299 U+FE0E (㊙︎) is definitely just a CJK symbol.
Firefox treats U+3299 without such EVSes always treats an emoji, but it's not compatible with Unicode standard 1 2.

Comment 7

[Tatsunori Uchino]

There are some other emojis that are derived from CJK text symbols, e.g. 〽︎.

Comment 8

[Jonathan Kew [:jfkthame]]

(In reply to Tatsunori Uchino from comment #4)

IVS (U+E0100 is dedicated to han unlike SVS (U+FE0x), so we don't need to see its previous code point.

I think we should still check the previous code point, because it's possible one of these IVS codes could be (incorrectly) present after a non-Han character. In such a case, we should ignore it (as it's a default-ignorable code), and let the processing of the segment break be determined by the preceding character instead.

葛︀
葛󠇠

These examples are not ideal.

I realize they're not actual variation sequences that are expected to produce variant glyphs; the idea was that the variation selectors should have no effect, and so we can compare the text with a version without variation selectors. But I'll try with "real" examples. That may be a bit dependent on font availability... I'll run a tryserver job to see how it goes. Thanks for your suggestions!

Comment 9

[Tatsunori Uchino]

I think we should still check the previous code point, because it's possible one of these IVS codes could be (incorrectly) present after a non-Han character. In such a case, we should ignore it (as it's a default-ignorable code), and let the processing of the segment break be determined by the preceding character instead.

If the software can't check the 2 previous character by design, my suggestion will work well. However, since you made it possible for Firefox to check it, the suggested implementation is sufficient. My suggestion became just a fast path and you declined it. It's fine, though.

That may be a bit dependent on font availability...

I believe you don't have to care about it. Yu Gothic (UI) and Source Han Sans (Noto Sans CJK) support them (and I think Hiragino Sans / Hiragino Kaku Gothic (macOS) supports them too), so if we use <html lang="ja">, we can expect sans-serif and system-ui are mapped to fonts with IVS/SVS support.

Comment 10

[Tatsunori Uchino]

塚 (U+585A) with U+FE00 and U+E0101 have almost same form in some Japanese fonts.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1bac07077edd
Skip past any default-ignorable codepoints when evaluating the context of a segment break. r=layout-reviewers,emilio

Comment 12

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/48879 for changes under testing/web-platform/tests

Comment 13

[amarc]

https://hg.mozilla.org/mozilla-central/rev/1bac07077edd

Comment 14

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR merged by moz-wptsync-bot

Comment 15

[Tatsunori Uchino]

Confirm the fix in Nightly. Thank you.

Comment 16

[Tatsunori Uchino]

Cinfirm → Confirmed

Comment 17

[BugBot [:suhaib / :marco]]

The patch landed in nightly and beta is affected.
:jfkthame, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
• If no, please set status-firefox133 to wontfix.

For more information, please visit BugBot documentation.

Comment 18

[Ardelean Oana, Desktop Test Engineering [:oardelean]]

Reproducible on a 2024-10-28 Nightly build on Windows 10.
Verified as fixed on Firefox Nightly 135.0a1 and Firefox 134.0b2 on Windows 10, macOS 14, Ubuntu 22.