1900708 - Flag emoji such as 🏳️‍🌈 🏳️‍⚧️ 🏴‍☠️ containing zero-width joiners are broken up in subject lines

Status: NEW

(Thunderbird :: Message Reader UI, defect)

Description

[duxovni]

Steps to reproduce:

Send myself an email with 🏳️‍🌈 (rainbow flag emoji), 🏳️‍⚧️ (trans flag emoji), and/or 🏴‍☠️ (pirate flag emoji) in the subject line.

Actual results:

The email subject is displayed as 🏳 🌈 🏳 ⚧ 🏴 ☠ in the message list, and in the subject line when opening the message.

Expected results:

The flag emoji should be displayed normally in their combined forms as 🏳️‍🌈 🏳️‍⚧️ 🏴‍☠️.

Comment 1

[Andre Klapper]

On which exact operating system and version?

Comment 2

[Magnus Melin [:mkmelin]]

Also, please attach a sample message as .eml

Comment 3

[duxovni]

Attachment: Sample email with flag emoji in subject line

Comment 4

[duxovni]

NixOS, 115.11.0

Comment 5

[duxovni]

(NixOS unstable 24.11, Thunderbird 115.11.0)

Comment 6

[Alfred Peters [:infofrommozilla]]

I see the problem here too.

The bug also shows up in the composer.
STR:

1. start a new e-mail.
2. copy and past e.g. the rainbow flag 🏳️‍🌈 into the subject and the body.
3. save to draft.

In the body of the draft message, the flag is correctly coded with the following 4 code points as:
<u+1F3F3><u+FE0F><u+200D><u+1F308>

www.unicode.org/emoji/charts/full-emoji-list.html#1f3f3_fe0f_200d_1f308
(Please be patient. The page needs some time to load.)

However, the subject only contains 2 code points:
<u+1F3F3> <u+1F308>

Comment 7

[Francesco]

It's a feature, they are removed here:
https://searchfox.org/comm-central/rev/76b0f85a58d580310018ffd06fd87c1963febf7f/mailnews/mime/jsmime/jsmime.mjs#336,359
From bug 1617370:
https://hg.mozilla.org/comm-central/rev/84249ce5084a8b7a0925b5d35c00331870a56a0c#l1.45
https://hg.mozilla.org/comm-central/rev/84249ce5084a8b7a0925b5d35c00331870a56a0c#l1.68

Looks like the intended sanitisation damages those emojis.

Comment 8

[Alfred Peters [:infofrommozilla]]

Attachment: Bug 1506587 - From Email Address with ZWJ and VS16.eml

Example with rainbow flag and several ZERO WIDTH JOINERs as well as several VARIATION SELECTORs in the From header

Comment 9

[Alfred Peters [:infofrommozilla]]

Attachment: Screenshot 2024-06-09 182419.png

Neither the ZERO WIDTH JOINER nor the VARIATION SELECTOR 16 are spaces in the true sense of the word. They therefore do not create empty space and are therefore not relevant for Bug 1506587 in my opinion.
I have therefore removed them from cleanToken().
In the picture you can see the correct rainbow flag without an empty space appearing after the “ZWF:” or the “VS16:”.

Or is this more of a wontfix?

Comment 10

[duxovni]

Thanks for finding that!

Looking at those bug reports, it sounds like the goal of removing those characters was to prevent a malicious sender breaking up a series of consecutive spaces like <space><zwj><space><variation selector 16><space> to avoid detection. But if that's the goal, then the regex should only remove those characters when they're preceded by actual space characters. Also, if this is about spoofing senders, then this extra-strict sanitisation should only be applied to the From header, not other headers like Subject.

Comment 11

[duxovni]

It seems like the correct sanitisation behavior would be specifically to replace "a space character followed by a run of only spaces and zero-width codepoints" with a single space.