1829502 - compose reply (and edit as new) Subject corrupting Unicode two-byte characters

Status: VERIFIED FIXED

(Thunderbird :: Message Compose Window, defect)

Description

[Calum Mackay]

When replying to a message that contains a two-byte Unicode character in the Subject, the character is corrupted in the compose window's Subject.

Note that body text is not affected.

This affects: Edit as New, Reply

But does not affect: Forward, nor composing a new message and adding that character to the Subject

The character used in my tests is:

U+2014 EM DASH
UTF-8: e2 80 94 UTF-16BE: 2014 Decimal: — Octal: \020024
—
Category: Pd (Punctuation, Dash); East Asian width: A (ambiguous)
Unicode block: 2000..206F; General Punctuation
Bidi: ON (Other Neutrals)

In the attached screenshots:

TB 1 Subject and body of original email, containing EM DASH

TB 2 Edit as New compose window, showing corrupted char in Subject, but OK in body

TB 3 email from TB 2, after delivery; corruption still there

TB 4 message pane entries for original, and TB 2 emails

The problem first appeared in 0418:

good 114.0a1 (2023-04-17) (64-bit)

bad 114.0a1 (2023-04-18) (64-bit)

MacOS Ventura 13.3.1

Comment 1

[Calum Mackay]

Attachment: TB 1.png

Comment 2

[Calum Mackay]

Attachment: TB 2.png

Comment 3

[Calum Mackay]

Attachment: TB 3.png

Comment 4

[Calum Mackay]

Attachment: TB 4.png

Comment 5

[Calum Mackay]

The screenshots were taken whilst composing in plain text.

I thought that might explain why the body text was not affected. But no, the same thing happens when composing in HTML: Subject corrupted, HTML body fine.

Comment 6

[Wayne Mery (:wsmwk)]

Do you think it is a recent regression?

Comment 7

[Calum Mackay]

Yes, as I said above: The problem first appeared in 0418 😊

Comment 8

[Alfred Peters [:infofrommozilla]]

I bet you have (PGP?) encryption enabled?

I just noticed that when I reply to an email with an encrypted subject, in the reply the umlauts are destroyed.
When I reply to an unencrypted e-mail everything is still OK.

And yes, I can also confirm the bug with the draft folder.
If I disable: Account Settings -> Ent-To-End Encryption
=>[ ] Store draft messages in encrypted format

this is fine again.

Comment 9

[Alfred Peters [:infofrommozilla]]

I can't say when the bug started to appear. But 18.04. would indicate Bug 1819499.

Comment 10

[Calum Mackay]

Thanks.

I do have PGP enabled, but the email to which I'm replying was not encrypted, nor is the email I compose. They are both signed, however.

Comment 11

[MM]

Looks like bug 1819499 didn't cover:
https://searchfox.org/comm-central/rev/04176849c949facaabc16deb07dd2e42b81b5173/mail/extensions/openpgp/content/ui/enigmailMsgComposeOverlay.js#565
Likely the cause since the subject passed in comes from the db header which is where the IDL change happened.

This also needs to be checked:
https://searchfox.org/comm-central/rev/04176849c949facaabc16deb07dd2e42b81b5173/mail/extensions/openpgp/content/modules/mimeDecrypt.jsm#863

Comment 12

[MM]

Attachment: 1829502.patch

Comment 13

[Magnus Melin [:mkmelin] (away until July 8)]

Comment on attachment 9329880 [details] [diff] [review]
1829502.patch

Review of attachment 9329880 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good, thanks! r=mkmelin

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/fca355829b44
Follow-up to bug 1819499: Don't treat subject from DB header as raw UTF-8. r=mkmelin

Comment 15

[Calum Mackay]

confirmed fix for me in: 114.0a1 (2023-04-24) (64-bit)

thanks very much indeed.

Comment 16

[Magnus Melin [:mkmelin] (away until July 8)]

Comment on attachment 9329880 [details] [diff] [review]
1829502.patch

[Approval Request Comment]
Regression fix for bug 1819499

Comment 17

[Wayne Mery (:wsmwk)]

Comment on attachment 9329880 [details] [diff] [review]
1829502.patch

[Triage Comment]
Approved for beta

Comment 18

[Rob Lemley [:rjl]]

Thunderbird 113.0b5:
https://hg.mozilla.org/releases/comm-beta/rev/3cbc46f0ec09