1512647 - Incomplete Unicode support for detecting *marked-up* words in plain text email

Status: RESOLVED FIXED

(Core :: Networking, enhancement, P5)

Description

[Jonathan Kew [:jfkthame]]

+++ This bug was initially created as a clone of Bug #1505911 +++

Steps to reproduce:

Use Thunderbird 52.9.1 and mark up the Deseret-script string *𐐔𐐯𐑅𐐨𐑉𐐯𐐻* with asterisks.
Also try supplementary-plane Chinese _𠜎𠜱𠝹𠱓𠱸_ and /áçčẻñțëḍ/ Latin using combining marks (rather than precomposed characters).


Actual results:

Sending a plain text message with these strings, Thunderbird does not apply the expected bold/underline/italic formatting.


Expected results:

These strings should be formatted just as similarly-marked ASCII text would be.


In bug 1505911, mozTXTToHTMLConv.cpp was patched to extend the definition of "alphabetic" characters used when scanning for such marked strings. However, the Unicode support is still incomplete, as it just looks at individual char16_t code units. Therefore, it fails to recognize supplementary-plane characters that should be treated as letters.

It also doesn't handle combining marks; they are regarded as non-letters, rather than being treated as part of a single cluster that has the category of its base character.

We can improve the behavior here if we iterate over the text using mozilla::unicode::ClusterIterator, and check for surrogate pairs when looking up character types.

Comment 1

[Jonathan Kew [:jfkthame]]

Attachment: Use unicode::ClusterIterator and decode surrogates when scanning text for marked-up words

This should fix the examples mentioned, afaict.

Comment 2

[Jonathan Kew [:jfkthame]]

Attachment: Add mozTXTToHTMLConv testcases including supplementary-plane letters and combining marks

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/8097285ea1c0
Use unicode::ClusterIterator and decode surrogates when scanning text for marked-up words. r=valentin
https://hg.mozilla.org/integration/mozilla-inbound/rev/1bb44497f6c0
Add mozTXTToHTMLConv testcases including supplementary-plane letters and combining marks. r=valentin

Comment 4

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/8097285ea1c0
https://hg.mozilla.org/mozilla-central/rev/1bb44497f6c0