1890007 - Intl.Segmenter incorrectly breaks up some flag emoji (grapheme clusters)

Status: RESOLVED DUPLICATE of bug 1899411

(Core :: JavaScript: Internationalization API, defect, P3)

Description

[nick.gard.dev]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:126.0) Gecko/20100101 Firefox/126.0

Steps to reproduce:

const segmenter = new Intl.Segmenter({ granularity: "word" });
Array.from(segmenter.segment('🇺🇸🏴󠁧󠁢󠁥󠁮󠁧󠁿')).map(
({ segment }) => segment
);

Actual results:

It resulted in

[ "🇺", "🇸", "🏴󠁧󠁢󠁥󠁮󠁧󠁿" ]

Expected results:

It should have resulted in

['🇺🇸', '🏴󠁧󠁢󠁥󠁮󠁧󠁿']

Comment 1

[nick.gard.dev]

(In reply to nick.gard.dev from comment #0)
Small mistake in the code pasted above. It does not require the { granularity: 'word' } option.

const segmenter = new Intl.Segmenter();
Array.from(segmenter.segment('🇺🇸🏴󠁧󠁢󠁥󠁮󠁧󠁿')).map(
({ segment }) => segment
);

Comment 2

[BugBot [:suhaib / :marco]]

The Bugbug bot thinks this bug should belong to the 'Core::Internationalization' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 3

[André Bargull [:anba]]

Thanks for the report! I've forwarded this issue to https://github.com/unicode-org/icu4x/issues/4780.

Comment 4

[Ting-Yu Lin [:TYLin] (PST, UTC-7)]

The issue has been fixed in the upstream by https://github.com/unicode-org/icu4x/pull/4536. The patch is large, so it probably cannot be cherry-pick into Firefox, so we'll need to wait until ICU4X segmenter 1.5 release.

Comment 5

[André Bargull [:anba]]

(In reply to Ting-Yu Lin [:TYLin] (PDT, UTC-7) from comment #4)

The issue has been fixed in the upstream by https://github.com/unicode-org/icu4x/pull/4536.

Good to hear that this is already fixed upstream!

The patch is large, so it probably cannot be cherry-pick into Firefox, so we'll need to wait until ICU4X segmenter 1.5 release.

Sounds reasonable to wait for the next release.

Comment 6

[Tom S. (please needinfo tschuster)]

Does this also fix ങ്ക from this reddit post?

Comment 7

[Ting-Yu Lin [:TYLin] (PST, UTC-7)]

(In reply to Tom S [:evilpie] from comment #6)

Does this also fix ങ്ക from this reddit post?

Yes. I test ങ്ക via modifying this test as the following, and ങ്ക produces only two grapheme cluster breaks (begin and end).

#[test]
fn emoji_flags() {
    // https://github.com/unicode-org/icu4x/issues/4780
    let segmenter = GraphemeClusterSegmenter::new();
    let breaks: Vec<usize> = segmenter.segment_str("ങ്ക").collect();
    assert_eq!(breaks, [0, 9]);
}

Comment 8

[Makoto Kato [:m_kato]]

Fixed by ICU4X 1.5