1745458 - Messages incorrectly signed using S/MIME when special characters are present (because amavisd-new changed the message)

Status: RESOLVED INVALID

(Thunderbird :: Untriaged, defect)

Description

[Marcin Wilk]

Attachment: thunderbird-smime-problems.zip

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:95.0) Gecko/20100101 Firefox/95.0

Steps to reproduce:

Create a message containing special characters, sign it with S/MIME, send it.

Actual results:

In Thunderbird and other mail clients (like Outlook, AquaMail, Roundcube), there is a message that is incorrectly digitally signed.

Expected results:

Message should have information that it's signed correctly.
In the attachment i add messages from TB that are bad according to TB and many other email clients).
I'm using newest stable Thunderbird 91.4.0 on Win10 computer.

Comment 1

[Magnus Melin [:mkmelin]]

Bug 1731529 I think.

Comment 2

[Marcin Wilk]

(In reply to Magnus Melin [:mkmelin] from comment #1)

Bug 1731529 I think.

No. This one was fixed at 95.x.beta. I tried that, and 96.0b1 and problem is still there. I noticed also that v 78 didn't have that problem (also there was no problem with new line too). There was also another report about this problem, but not here: https://thunderbird.topicbox.com/groups/e2ee/T353bca1a5919e3e8/umlauts-break-s-mime-signature

I tested more configuration (with Thunderbird beta with fixed newline problem) and here is the full reult:
Thunderbird 78.14.0 (text, image, attachment, special characters) -> AquaMail Pro 1.32.1-091 - Good
Thunderbird 78.14.0 (text, image, attachment, special characters) -> Outlook 2016 - Good
Thunderbird 78.14.0 (text, image, attachment, special characters) -> Roundcube 1.5.1 (rc_smime plugin) - Good
Thunderbird 78.14.0 (text, image, attachment, special characters) -> Thunderbird 78.14.0 - Good
Thunderbird 78.14.0 (text, image, attachment, special characters) -> Thunderbird 96.0b1 (rc_smime plugin) - Good

Thunderbird 91.4.0 (text) -> AquaMail Pro 1.32.1-091 - Good
Thunderbird 91.4.0 (text, image) -> AquaMail Pro 1.32.1-091 - Good
Thunderbird 91.4.0 (text, attachment) -> AquaMail Pro 1.32.1-091 - Good
Thunderbird 91.4.0 (text, image, attachment) -> AquaMail Pro 1.32.1-091 - Good
Thunderbird 91.4.0 (text, image, attachment, special characters) -> AquaMail Pro 1.32.1-091 - Error
Thunderbird 91.4.0 (text, special characters) -> AquaMail Pro 1.32.1-091 - Error
Thunderbird 91.4.0 (text, attachment with special characters in filename) -> AquaMail Pro 1.32.1-091 - Good

Thunderbird 91.4.0 (text) -> Outlook 2016 - Good
Thunderbird 91.4.0 (text, image) -> Outlook 2016 - Error
Thunderbird 91.4.0 (text, attachment) -> Outlook 2016 - Error
Thunderbird 91.4.0 (text, image, attachment) -> Outlook 2016 - Error
Thunderbird 91.4.0 (text, image, attachment, special characters) -> Outlook 2016 - Error
Thunderbird 91.4.0 (text, special characters) -> Outlook 2016 - Error
Thunderbird 91.4.0 (text, attachment with special characters in filename) -> Outlook 2016 - Error

Thunderbird 91.4.0 (text) -> Roundcube 1.5.1 (rc_smime plugin) - Good
Thunderbird 91.4.0 (text, image) -> Roundcube 1.5.1 (rc_smime plugin) - Good
Thunderbird 91.4.0 (text, attachment) -> Roundcube 1.5.1 (rc_smime plugin) - Good
Thunderbird 91.4.0 (text, image, attachment) -> Roundcube 1.5.1 (rc_smime plugin) - Good
Thunderbird 91.4.0 (text, image, attachment, special characters) ->Roundcube 1.5.1 (rc_smime plugin) - Error
Thunderbird 91.4.0 (text, special characters) -> Roundcube 1.5.1 (rc_smime plugin) - Error
Thunderbird 91.4.0 (text, attachment with special characters in filename) -> Roundcube 1.5.1 (rc_smime plugin) - Good

Thunderbird 91.4.0 (text) -> Thunderbird 91.4.0 - Good
Thunderbird 91.4.0 (text, image) -> Thunderbird 91.4.0 - Good
Thunderbird 91.4.0 (text, attachment) -> Thunderbird 91.4.0 - Good
Thunderbird 91.4.0 (text, image, attachment) -> Thunderbird 91.4.0 - Good
Thunderbird 91.4.0 (text, image, attachment, special characters) -> Thunderbird 91.4.0 - Error
Thunderbird 91.4.0 (text, special characters) -> Thunderbird 91.4.0 - Error
Thunderbird 91.4.0 (text, attachment with special characters in filename) -> Thunderbird 91.4.0 - Good

Thunderbird 96.0b1 (text) -> Outlook 2016 - Good
Thunderbird 96.0b1 (text, image) -> Outlook 2016 - Good
Thunderbird 96.0b1 (text, attachment) -> Outlook 2016 - Good
Thunderbird 96.0b1 (text, image, attachment) -> Outlook 2016 - Good
Thunderbird 96.0b1 (text, image, attachment, special characters) -> Outlook 2016 - Error
Thunderbird 96.0b1 (text, special characters) -> Outlook 2016 - Error
Thunderbird 96.0b1 (text, attachment with special characters in filename) -> Outlook 2016 - Good

Thunderbird 96.0b1 (text) -> Thunderbird 96.0b1 - Good
Thunderbird 96.0b1 (text, image) -> Thunderbird 96.0b1 - Good
Thunderbird 96.0b1 (text, attachment) -> Thunderbird 96.0b1 - Good
Thunderbird 96.0b1 (text, image, attachment) -> Thunderbird 96.0b1 - Good
Thunderbird 96.0b1 (text, image, attachment, special characters) -> Thunderbird 96.0b1 - Error
Thunderbird 96.0b1 (text, special characters) -> Thunderbird 96.0b1 - Error
Thunderbird 96.0b1 (text, attachment with special characters in filename) -> Thunderbird 96.0b1 - Good

Comment 3

[newsfan]

At a guess, it will work in TB 91 if you switch mailnews.send.jsmodule to false, and maybe also mailnews.smtp.jsmodule as well. Likely the bug was introduced in the new JS modules. At a further wild guess, likely the signing happens on the "original" message before encoding it into UTF-8.

"Special" characters here means any non-ASCII character, right? So any Russian, Greek, Hebrew, Japanese, Chinese, Korean, Thai, etc. will fail 100%.
Try with this: これはShift_JISのテキストファイルです。

Comment 4

[Marcin Wilk]

(In reply to newsfan from comment #3)

At a guess, it will work in TB 91 if you switch mailnews.send.jsmodule to false, and maybe also mailnews.smtp.jsmodule as well. Likely the bug was introduced in the new JS modules. At a further wild guess, likely the signing happens on the "original" message before encoding it into UTF-8.

"Special" characters here means any non-ASCII character, right? So any Russian, Greek, Hebrew, Japanese, Chinese, Korean, Thai, etc. will fail 100%.
Try with this: これはShift_JISのテキストファイルです。

As special characters i was testing only Polish letters (i used strong with all of them ę€óąśłżźćń but i saw the problem for the first time using some Sweden letters (i had text: Med vänlig hälsning in my signature).

Yes, in 96.0b1 "これはShift_JISのテキストファイルです。" give same S/MIME signature error.

And You were right. Disabling in config editor mailnews.send.jsmodule made message sent from TB 91.4.0 digital signature correct (also with attachment with special characters in filename, and image inside message). So this must be reason for this problem.

Comment 5

[Ping Chen [:rnons] (away)]

I can't reproduce this, in your test file From TB - no image, no attachment, with special characters - 2021-12-10 1608.eml, there is

Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: quoted-printable

 From TB - no image, no attachment, with special characters
=C4=99=C3=B3=C4=85=C5=9B=C5=82=C5=BC=C5=BA=C4=87=C5=84

But we have code to not use quoted-printable when format=flowed, I don't know how it happened. I tried to edit your eml as new, sign with smime, and send out, the signature is valid.

Comment 6

[Marcin Wilk]

Ok, so how can i check what is going on?
Here is the video with every click i make to create email: https://love.itos.pl/s/32oQdGHsByk8Say
And here is the email sent on that video (in eml format): https://love.itos.pl/s/Rc3K2M74cPPs8bN
I tried even installing clear English language version of Thunderbird, it is always same problem.

Comment 7

[Ping Chen [:rnons] (away)]

Do you have some addons installed? It looks like your SMTP server or something modifies the message when sending. Compare the eml files you just linked

RECEIVED.eml

--------------ms050802080002040205040606
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: quoted-printable

=C4=99=E2=82=AC=C3=B3=C4=85=C5=9B=C5=82=C5=BC=C5=BA=C4=87=C5=84

SENT.eml

--------------ms050802080002040205040606
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

ę€óąśłżźćń

Comment 8

[Marcin Wilk]

Well. For my tests i used clear installation of Thunderbird (no addons, no any configuration except adding account itself ans certs).
But do You think postfix or anything else could change message? I noticed difference there. I will check with different email servers now, but it looks strange to me.

Comment 9

[Ping Chen [:rnons] (away)]

It is strange, but possible https://mailing.postfix.users.narkive.com/Oi3Mbmle/8bit-to-quoted-printable.

Can you set mailnews.smtp.loglevel to All, send a mail and see if there is 8BITMIME in the EHLO response?

Also as a test, you may set mail.strictly_mime to true, so that base64 should be used for SENT.eml, and see if the signature is valid when receiving.

Comment 10

[Marcin Wilk]

(In reply to Ping Chen (:rnons) from comment #9)

It is strange, but possible https://mailing.postfix.users.narkive.com/Oi3Mbmle/8bit-to-quoted-printable.

Can you set mailnews.smtp.loglevel to All, send a mail and see if there is 8BITMIME in the EHLO response?

Also as a test, you may set mail.strictly_mime to true, so that base64 should be used for SENT.eml, and see if the signature is valid when receiving.

For now i must admit, that it is mail server fault. Using different one to send s/mime signed message works well! Strange is TB 78 did work well so i was sure it must be TB problem! My fault i think.

Here is the log from console:

mailnews.smtp: S: 250-system.itos.pl
250-PIPELINING
250-SIZE
250-ETRN
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
mailnews.smtp: C: STARTTLS SmtpClient.jsm:578:19
mailnews.smtp: S: 220 2.0.0 Ready to start TLS
mailnews.smtp: C: EHLO [192.168.50.5] SmtpClient.jsm:578:19
mailnews.smtp: S: 250-system.itos.pl
250-PIPELINING
250-SIZE
250-ETRN
250-AUTH PLAIN LOGIN
250-AUTH=PLAIN LOGIN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
mailnews.smtp: Possible auth methods: PLAIN,LOGIN SmtpClient.jsm:898:17
mailnews.smtp: Current auth method: PLAIN SmtpClient.jsm:616:17
mailnews.smtp: Authentication via AUTH PLAIN SmtpClient.jsm:631:21
mailnews.smtp: C: Logging suppressed (it probably contained auth information) SmtpClient.jsm:574:19
mailnews.smtp: S: 235 2.7.0 Authentication successful
mailnews.smtp: Authentication successful. SmtpClient.jsm:1088:17
mailnews.smtp: C: MAIL FROM:<[e-mail]> BODY=8BITMIME SIZE=7172 SmtpClient.jsm:578:19
mailnews.smtp: S: 250 2.1.0 Ok
mailnews.smtp: MAIL FROM successful, proceeding with 1 recipients SmtpClient.jsm:1122:17
mailnews.smtp: Adding recipient... SmtpClient.jsm:1127:17
mailnews.smtp: C: RCPT TO:<[e-mail]> SmtpClient.jsm:578:19
mailnews.smtp: S: 250 2.1.5 Ok
mailnews.smtp: RCPT TO done, proceeding with payload SmtpClient.jsm:1187:19
mailnews.smtp: C: DATA SmtpClient.jsm:578:19
mailnews.smtp: S: 354 End data with <CR><LF>.<CR><LF>
mailnews.smtp: Sending 7172 bytes of payload SmtpClient.jsm:548:17
mailnews.smtp: S: 250 2.0.0 Ok: queued as 13CC016E00F0
mailnews.smtp: Message sent successfully. SmtpClient.jsm:1246:21
mailnews.smtp: C: QUIT SmtpClient.jsm:578:19
mailnews.smtp: S: 221 2.0.0 Bye
mailnews.smtp: Closing connection... SmtpClient.jsm:152:17
mailnews.smtp: Socket closed.

Sending with mail.strictly_mime set to true also is working fine with the bad server. Message inside looks like that:

Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-256; boundary="------------ms090703070504090403010809"

This is a cryptographically signed message in MIME format.

--------------ms090703070504090403010809
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: base64

VGVzdCDEmeKCrMOzxIXFm8WCxbzFusSHxYQNCg0K

--------------ms090703070504090403010809
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: Kryptograficzna sygnatura S/MIME

So it must be some server fault. I will investigate the server itself then. Thank You for taking so much time!

Comment 11

[Marcin Wilk]

Ok. So it wasn't Postfix itself. It was amavisd-new on the server. It was configured like that for better DKIM handling with this:

force MTA conversion to 7-bit (e.g. before DKIM signing)
smtpd_discard_ehlo_keywords => ['8BITMIME']

TB 78 and Outlook didn't have those problems, because they made messages with "Content-Transfer-Encoding: quoted-printable" even before sending data to server.

Thanks again for helping solving this. Without You i would never guess what is happening.

Comment 12

[Magnus Melin [:mkmelin]]

Try 91.4.1 once that's out.

Comment 13

[newsfan]

(In reply to Ping Chen (:rnons) from comment #5)

But we have code to not use quoted-printable when format=flowed, I don't know how it happened. I tried to edit your eml as new, sign with smime, and send out, the signature is valid.

Just out of interest: It also works when format=flowed is switched off? mailnews.send_plaintext_flowed set to false.

Comment 14

[osrsl]

TB 78 and Outlook didn't have those problems, because they made messages with "Content-Transfer-Encoding: quoted-printable" even before sending data to server. https://modprince.com/