1911227 - External encryption: email with inner message in CTE 8bit renders non-ASCII characters (diacritics, accents, Chinese, etc.) as mojibake

Status: RESOLVED FIXED

(MailNews Core :: Security: OpenPGP, defect, P1)

Description

[super.dukefb1]

Attachment: 20240802-001.jpg

Steps to reproduce:

1. I compose a new mail with Traditional Chinese text in subject and body.
2. Enable the OpenPGP Encrypt, Digitally Sign.
3. Send the mail.

ps. This bug was reported in https://thunderbird.topicbox.com/groups/e2ee/T87cabaa23b8a0245-M1639634330e30957a267cd37

Actual results:

After I received the mail and open it. The message's body is random symbols without the Chinese text.

Expected results:

My received mail should have Chinese text as what I composed it.

Additionally, if I only enable OpenPGP encrypt, then there is no problem.
And if I only enable OpenPGP Digitally Sign, then there is also no problem.

Comment 1

[Francesco]

This now only happens with Chinese characters, it happens with any non-ASCII text (like diacritics or accented characters in Western languages) if TB choses 8bit as CTE for the inner message. With mail.strictly_mime set, the inner message gets CTE QP (non-flowed) or base64 (flowed) and the problem doesn't occur. Looking at the encoding failure for accented characters, it appears that binary JS string containing raw UTF-8, like two bytes for an accented character, is interpreted as "normal" JS string and encoded again to UTF-8, resulting in four bytes of data.
Example: ä (0xC3 0xA4) becomes ä.

Comment 2

[Francesco]

I tried to reproduce this and got no failure. This is the result after gpg -d:

--------------eQg0o7fbxyo4uroa00DryRSj
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

test with some ä

Reporter, are you using external encryption by any chance?

Comment 3

[super.dukefb1]

I have Gpg4win installed and when I compose OpenPGP email with Digitally Sign enabled, Gpg4win will pop-up for the private key password.

I think this is not external encryption. The encryption is done inside Thunderbird and encryption needs recipient's public key only.

By the way, I used to compose Chinese text mails with OpenPGP Encrypt & Digitally sign for years, this is first time I encounter this problem.

Do you want me disable Gpg4win and reproduce it ? (But I don't know how to disable it for now, I need some research.)

Comment 4

[super.dukefb1]

I open my email account's End-to-End Encryption and then remove the external key and import my personal key to the Thunderbird and select it.

I reproduce again, there indeed no problem with my Chinese text email with OpenPGP Encrypt & Digitally sign.

So, the problem is Thunderbird + Gpg4win. Why it had no problem before (Thunderbird previous versions) ?

Comment 5

[Francesco]

Attachment: external-key.png

Sounds like external encryption. In the UI there is a setting "Use your external key through GnuPG (e.g. from a smartcard)". That setting only shows if pref mail.openpgp.allow_external_gnupg is set to true. Is that pref set for you? Did you ever import the key into TB's own key manager?

Comment 6

[Francesco]

Why it had no problem before (Thunderbird previous versions) ?

Something in the system changed.

Comment 7

[Alfred Peters [:infofrommozilla]]

Probably regressed by Bug 1899110?

Comment 8

[Magnus Melin [:mkmelin]]

Maybe bug 1908113 fixes this?

Comment 9

[Magnus Melin [:mkmelin]]

If not, please send me a test message (mkmelin at thunderbird.net, find the key on the keyserver) showing this issue

Comment 10

[Francesco]

(In reply to Magnus Melin [:mkmelin] from comment #8)

Maybe bug 1908113 fixes this?

According to the comments, this is for signed-only messages. I'm not using external encryption, so the reporter would have to send you such a message. Bug 1908113 will ship in 128.1.0 soon, otherwise a release candidate can be used:
http://ftp.mozilla.org/pub/thunderbird/candidates/128.1.0esr-candidates/build1/

Comment 11

[super.dukefb1]

As I said in the report, I can see Chinese text normally if I only enable OpenPGP digitally sign. Of course I mean I use Gpg4win (AKA the term "external encryption").

But I just found that I have another trouble that Thunderbird said that this OpenPGP Digitally Sign message has "a technical error was detected". Is this another bug ? Should I file another bug report ?
(I don't know how to insert a screenshot picture in the comment. But you know what I said.)

There is no problem If I use Thunderbird built-in encryption.

Comment 12

[Magnus Melin [:mkmelin]]

That would be something for another bug report.

Please test a build from comment 10. And if needed, send me a test message.

Comment 13

[super.dukefb1]

(In reply to Magnus Melin [:mkmelin] from comment #12)

That would be something for another bug report.

Please test a build from comment 10. And if needed, send me a test message.

I installed the 128.1.0esr build1 and my problem persist. When I enable both OpenPGP Encrypt and Digitally Sign, the mail would be random symbols.

Do I need to send you a test mail ? If yes, then I can find your email from the profile which is m***+m****@i**.**, could you send me your public key ?

Good news is, there is NO "a technical error was detected" problem as stated in Comment 11

Comment 14

[Magnus Melin [:mkmelin]]

Please find the key for mkmelin at thunderbird.net on the keyserver, and send me a message. (reference this bug)

Comment 15

[Magnus Melin [:mkmelin]]

Thanks for the message. It displays exactly the same with bug 1899110 backed out.

I also tried an 115.10.0 and it's too the same.

But maybe this is all bug 1906903? Which version were you sending with? Bug 115.14.0 should have that fixed.

Comment 16

[Francesco]

Please note the information in https://thunderbird.topicbox.com/groups/e2ee/T87cabaa23b8a0245-M8efe6de47470962ee9f5211f/openpgp-signedencrypted-email-renders-chinese-characters-as-random-symbols summarised in comment #1:
When the reporter decrypts a message produced with TB 128 using gpg -d, he notices that the inner payload has an encoding problem when CTE 8bit is used, there is no encoding issue when base64 is used. This indicates that TB 128 has an encoding issue when encoding the payload into CTE 8bit. Once the message has been badly encoded (and later encrypted), no program will be able to decrypt and decode it correctly. In fact, comment #1 states that the fault seems to be that a JS string is converted twice into UTF-8, first the payload in a JS string is converted to a raw binary JS string, and then that buffer is converted a second time, so one for example, one initial diacritic ä is finally encoded as four raw bytes instead of two (0xC3 0xA4).

Comment 17

[Magnus Melin [:mkmelin]]

Thanks, sounds like it could be from bug 1906903 - https://hg.mozilla.org/comm-central/rev/163c80476e09#l1.47 touches the area of encryptAndOrSign for external keys

Comment 18

[Francesco]

Reporter, can you please test whether the issue exists in TB 115.14.0. That version also contains the fix for bug 1906903. It can already be checked with a message to yourself. After doing the gpg -d, please check that the inner message indeed uses CTE 8bit.

Comment 19

[super.dukefb1]

(In reply to Magnus Melin [:mkmelin] from comment #15)

But maybe this is all bug 1906903? Which version were you sending with? Bug 115.14.0 should have that fixed.

The mail was sent by Thunderbird 128.1.0esr build1 as Comment 13 said.

(In reply to Francesco from comment #18)

Reporter, can you please test whether the issue exists in TB 115.14.0. That version also contains the fix for bug 1906903. It can already be checked with a message to yourself. After doing the gpg -d, please check that the inner message indeed uses CTE 8bit.

I can see the Chinese text correctly in the mail which was sent by TB 115.14.0 as you requested. But the Content-Transfer-Encoding is not 8bit

--------------jMTAtF7xZUOgU66eFhJbyTu5
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: base64

5oiR55So5q2j6auU5Lit5paHDQoNCg0K
--------------jMTAtF7xZUOgU66eFhJbyTu5

By the way, I installed the TB 115.14.0 to another directory. So this TB 115.14.0 is fresh and the only changes are add an account and change mail.openpgp.allow_external_gnupg to true.

I also checked mail.strictly_mime and found it's the default value -- false.

Comment 20

[Francesco]

Thanks for that. Whether TB picks CTE 8bit or base64 depends on various factors. With mail.strictly_mime at default false, using 115.14.0 could you send a message with this content: This is a short message with only one diacritic character ä. TB should pick CTE 8bit for the message.

Comment 21

[super.dukefb1]

(In reply to Francesco from comment #20)

Thanks for that. Whether TB picks CTE 8bit or base64 depends on various factors. With mail.strictly_mime at default false, using 115.14.0 could you send a message with this content: This is a short message with only one diacritic character ä. TB should pick CTE 8bit for the message.

The Content-Transfer-Encoding is still base64 (mail sent by TB 115.14.0)

--------------lyr0gRUHrfoYnBpS1VTcUt5Q
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: base64

VGhpcyBpcyBhIHNob3J0IG1lc3NhZ2Ugd2l0aCBvbmx5IG9uZSBkaWFjcml0aWMgY2hhcmFj
dGVyIMOkLiBUQiBzaG91bGQgDQpwaWNrIENURSA4Yml0IGZvciB0aGUgbWVzc2FnZS4NCg==

--------------lyr0gRUHrfoYnBpS1VTcUt5Q

Both Comment 19 and here, the mails sent by the TB 115.14.0 were plain text and OpenPGP Encrypt & Digitally Sign; and with external encryption (Gpg4win). Just to make sure we are on the same page.

Comment 22

[Francesco]

OK, thanks for testing. So the observed behaviour is that 115 doesn't use CTE 8bit and hence there is no encoding issue. 128 does use CTE 8bit (unless pref mail.strictly_mime is set) and then the encoding issue happens.

Comment 23

[Kai Engert [:KaiE:]]

I traced the data flow in detail.

The mistake happens when processing a correctly encoded UTF-8 stream of bytes with gpgme_op_sign().
The data returned by GPGME changes the UTF-8 bytes.

I found no way to prevent that. I found gpgme_data_set_encoding(), but it isn't offering a UTF-8 like encoding, it only offers settings like none (default == guess), binary, mime... None of the ones I tried produced different results.

The only fix I found is to also do base64-encoding of the inner payload we give to GPGME for signing,
using the same approach that we used to fix bug 1908113.

This was also regressed by 1814893 (just like bug 1908113).
My fix for bug 1908113 did only partially restore the previous logic, because I thought we can same some space by limiting the number of scenarios in which we need the base64 encoding for an inner signed payload. I was wrong, this bug here is another scenario in which it's necessary.

Comment 24

[Kai Engert [:KaiE:]]

Attachment: Bug 1911227 - Always use base64 encoding for inner signed payload when using external GPGME signing. r=mkmelin

Comment 25

[Magnus Melin [:mkmelin]]

Thanks Francesco for the analysis!

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/3d6eec79b621
Always use base64 encoding for inner signed payload when using external GPGME signing. r=mkmelin

Comment 27

[Kai Engert [:KaiE:]]

Comment on attachment 9418212 [details]
Bug 1911227 - Always use base64 encoding for inner signed payload when using external GPGME signing. r=mkmelin

[Approval Request Comment]
Regression caused by (bug #): 1814893
User impact if declined: broken signed messages
Testing completed (on c-c, etc.):
Risk to taking this patch (and alternatives if risky): low, re-uses existing code to enable a wrapper layer in an additional scenario

Comment 28

[Corey Bryant]

Kai, can you update the "Testing completed" section? I assume you tested this.

Comment 29

[Corey Bryant]

Comment on attachment 9418212 [details]
Bug 1911227 - Always use base64 encoding for inner signed payload when using external GPGME signing. r=mkmelin

[Triage Comment]
Approved for beta

Comment 30

[Kai Engert [:KaiE:]]

High level test instructions:

• configure your thunderbird email account for using an external GnuPG key for OpenPGP
  https://wiki.mozilla.org/Thunderbird:OpenPGP:Smartcards
• once your setup is complete, send an encrypted and signed email to yourself,
  using the subject "中文信" and the message body "我用正體中文"
• receive the message, and click it
• Thunderbird should decrypt the message and show it as signed and encrypted, using status icons
• with the broken version, the message body is shown with garbled text,
  with the fix, the message body shows the expected text 我用正體中文

Comment 31

[Corey Bryant]

Comment on attachment 9418212 [details]
Bug 1911227 - Always use base64 encoding for inner signed payload when using external GPGME signing. r=mkmelin

[Triage Comment]
Approved for esr128

Comment 32

[Rob Lemley [:rjl]]

Thunderbird 130.0b2:
https://hg.mozilla.org/releases/comm-beta/rev/4f3558ccfab7

Comment 35

[Rob Lemley [:rjl]]

Thunderbird 128.1.1esr:
https://hg.mozilla.org/releases/comm-esr128/rev/191c774ef0bd