1741362 - Mime signature fails with umlaut text - due to server changing signed 8bit text to using transfer-encoding

Status: RESOLVED FIXED

(MailNews Core :: Security: S/MIME, defect, P1)

Description

[Patrick Spendrin]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:94.0) Gecko/20100101 Firefox/94.0

Steps to reproduce:

Write an email with S/MIME signature
containing only the word 'täst'

Actual results:

The email signature is displayed as invalid

Expected results:

The email signature should be displayed as valid
The counter test with the word "test" in it works.

Comment 1

[Patrick Spendrin]

Maybe related bug report:
https://bugzilla.mozilla.org/show_bug.cgi?id=1731529

Comment 2

[Juergen]

This is still the behaviour in TB Beta 102.0b7 Build ID 20220617170556.

You need only one character like "ä" in the message body and the S/MIME signed email is invalid. I reproduced it using TB as the sending and receiving client and tested it also with different email addresses for sender and recipient.

The emails sent with TB Beta 102.0b7 are marked as valid in the sent folder, but always invalid independent of the email client of the recipient.

If the body contains no umlauts (e.g. only "ae"), it works with Content-Transfer-Encoding: 7bit. If it contains an umlaut like "ä" only, it is marked as invalid by the receiver.

It seems, that it happened first time with TB 86.0b1, see https://www.thunderbird-mail.de/forum/thread/88403-s-mime-signierte-emails-werden-von-outlook2016-nicht-korrekt-verifiziert/

„Thunderbird 86.0b1 is using a new implementation of the message sending code that is easier to maintain and more flexible. As with any major change, bugs are bound to happen. The old behavior can be restored by setting the pref "mailnews.send.jsmodule" to false. “

This workaround is fine also for TB 91.10.0, but with TB Beta 102.0b7 the old send module seems to be removed. It is not possible to use the workaround anymore, but the bug is still there.

Does anybody have an idea how to analyze it in detail or how to enable a suitable workaround?
Otherwise we are going to get in trouble because no email containig umlauts can be signed correctly with TB in the future.

Comment 3

[Juergen]

Found a workaround in a c9 in bug 1745458:

... you may set mail.strictly_mime to true, so that base64 should be used for SENT.eml, and see if the signature is valid when receiving.

I tested this with TB Beta 102.0b7, so this is a suitable workaround for me. :-)

I don't understand the details, but I would suggest, that it should be investigated somehow to have a clear understandig, if it happens in TB or on the web server.

Comment 4

[Juergen]

... or on the mail server.

Comment 5

[Patrick Spendrin]

(In reply to Juergen from comment #3)

Found a workaround in a c9 in bug 1745458:

... you may set mail.strictly_mime to true, so that base64 should be used for SENT.eml, and see if the signature is valid when receiving.

I tested this with TB Beta 102.0b7, so this is a suitable workaround for me. :-)

I don't understand the details, but I would suggest, that it should be investigated somehow to have a clear understandig, if it happens in TB or on the web server.

I can confirm that the above workaround works for me too. There is no server required, this is purely a problem of thunderbird.

Comment 6

[Kai Engert (:KaiE:)]

Patrick Brunschwig: The old Enigmail code recommended to keep mail.strictly_mime disabled (do not use quoted printable encoding).

Do you remember what kind of problems you saw?

Comment 7

[Patrick Brunschwig]

(In reply to Kai Engert (:KaiE:) from comment #6)

Patrick Brunschwig: The old Enigmail code recommended to keep mail.strictly_mime disabled (do not use quoted printable encoding).

Do you remember what kind of problems you saw?

The only case was the following: sending inline-PGP signed mails will result in invalid signatures in ca. 95% of all cases if mail.strictly_mime is enabled.

Comment 8

[cgonzalez]

(In reply to Juergen from comment #2)

This is still the behaviour in TB Beta 102.0b7 Build ID 20220617170556.

You need only one character like "ä" in the message body and the S/MIME signed email is invalid. I reproduced it using TB as the sending and receiving client and tested it also with different email addresses for sender and recipient.

The emails sent with TB Beta 102.0b7 are marked as valid in the sent folder, but always invalid independent of the email client of the recipient.

If the body contains no umlauts (e.g. only "ae"), it works with Content-Transfer-Encoding: 7bit. If it contains an umlaut like "ä" only, it is marked as invalid by the receiver.

It seems, that it happened first time with TB 86.0b1, see https://www.thunderbird-mail.de/forum/thread/88403-s-mime-signierte-emails-werden-von-outlook2016-nicht-korrekt-verifiziert/

„Thunderbird 86.0b1 is using a new implementation of the message sending code that is easier to maintain and more flexible. As with any major change, bugs are bound to happen. The old behavior can be restored by setting the pref "mailnews.send.jsmodule" to false. “

This workaround is fine also for TB 91.10.0, but with TB Beta 102.0b7 the old send module seems to be removed. It is not possible to use the workaround anymore, but the bug is still there.

Does anybody have an idea how to analyze it in detail or how to enable a suitable workaround?
Otherwise we are going to get in trouble because no email containig umlauts can be signed correctly with TB in the future.

Problem persist in Thunderbird 102.
Fail happens also when characters with tildes or similar are present in the message.
Workaround as mail.strictly_mime TRUE works

Comment 9

[cgonzalez]

Problem persist in Thunderbird 102.
Fail happens also when characters with tildes or similar are present in the message.
Workaround as mail.strictly_mime TRUE works

Comment 10

[Kai Engert (:KaiE:)]

Magnus, we no longer support sending inline OpenPGP messages.

Do you think we could change mail.strictly_mime by default to true on comm-central?

And the second question is, if that works out on c-c without issues, could we consider to change the default on stable esr102 ?

Comment 11

[Kai Engert (:KaiE:)]

If we don't want to change the default for all emails, the alternative could be: Find a way to enable the mail.strictly_mime=true behavior for all signed email (regardless of the pref value)

Comment 12

[Magnus Melin [:mkmelin]]

mail.strictly_mime will send Quoted-Printable, which is kind of terrible :( I don't think we want it on by default.

For OpenPGP we already do send encoded though (Base64) - IIRC the spec recommends/mandates it, though as we saw e.g. with the topicbox issues, it's not necessarily helping any...

This bug is about S/MIME. If needed it could do what OpenPGP does.

Comment 13

[Kai Engert (:KaiE:)]

(In reply to Magnus Melin [:mkmelin] from comment #12)

mail.strictly_mime will send Quoted-Printable, which is kind of terrible :( I don't think we want it on by default.

Can you please elaborate? What exactly is terrible? Why would using QP be worse than this bug?

We could possibly use a different S/MIME encoding, instead of "clear text signing" we could use "opaque signing". As a consequence users would no longer be able to use "view source" of an email to read its contents (and being able to use view source, for an email that is simply signed, not encrypted, is a nice functionality).

Comment 15

[Kai Engert (:KaiE:)]

(In reply to Kai Engert (:KaiE:) from comment #13)

(In reply to Magnus Melin [:mkmelin] from comment #12)

mail.strictly_mime will send Quoted-Printable, which is kind of terrible :( I don't think we want it on by default.

Can you please elaborate? What exactly is terrible? Why would using QP be worse than this bug?

I still would like to know, out of curiosity, why you consider QP terrible.

We could possibly use a different S/MIME encoding, instead of "clear text signing" we could use "opaque signing". As a consequence users would no longer be able to use "view source" of an email to read its contents (and being able to use view source, for an email that is simply signed, not encrypted, is a nice functionality).

I no longer consider this an option. The reason is, signed messages could no longer be displayed by recipients who use an email client that doesn't support S/MIME.

Comment 16

[Kai Engert (:KaiE:)]

(In reply to Magnus Melin [:mkmelin] from comment #12)

For OpenPGP we already do send encoded though (Base64) - IIRC the spec recommends/mandates it, though as we saw e.g. with the topicbox issues, it's not necessarily helping any...

This bug is about S/MIME. If needed it could do what OpenPGP does.

I think we should avoid sending out 8bit encoding, at least when using S/MIME signing.
Using base64 is acceptable I think.

(Should we avoid 8bit encoding in general? Should we file a spearate bug for that?)

Comment 17

[Kai Engert (:KaiE:)]

Magnus, do you remember (or can find easily) where enforce the use of base64 for OpenPGP?
(I couldn't find it.)

(I also noticed I have unfinished related work in bug 1731600. Which might not make sense. Instead of trying to add a default header for 7bit encoding, it's probably to ensure we always use base64?)

Comment 18

[Kai Engert (:KaiE:)]

This issue keeps coming up. I think we must solve it now for good, so raising priority.

Comment 19

[Kai Engert (:KaiE:)]

Maybe the right place to fix it is in MimeEncoder.jsm function pickEncoding().

This code looks so familiar... I could swear I had recently started to work on a patch for that code. Cannot find it right now.

Comment 20

[Kai Engert (:KaiE:)]

We set msgCompFields.forceMsgEncoding = true;
in enigmailMsgComposeOverlay.js prepareSendMsg(),
that might be the cause for base64 encoding of OpenPGP.

Comment 21

[Kai Engert (:KaiE:)]

(In reply to Kai Engert (:KaiE:) from comment #19)

Maybe the right place to fix it is in MimeEncoder.jsm function pickEncoding().

This code looks so familiar... I could swear I had recently started to work on a patch for that code. Cannot find it right now.

Found it:
https://phabricator.services.mozilla.com/D197243?id=803930#change-VNvLbo0AD6Mc

Comment 22

[Magnus Melin [:mkmelin]]

Will take a quick look.

Comment 23

[Kai Engert (:KaiE:)]

Attachment: Bug 1741362 - Enforce message encoding (non-8bit) for signed, unencrypted messages. r=mkmelin

Comment 24

[Magnus Melin [:mkmelin]]

Attachment: Bug 1741362 - Use transfer encoding for S/MIME signed only messages. r=kaie

Comment 25

[inetuka]

Do you need examples how Evolution is handling this? Sorry for my misconception ASCII = "Reachable via keyboard". My mail servers do not reencode content, so IONOS must be doing something fishy here. Best regards from Hamburg.

Comment 26

[Magnus Melin [:mkmelin]]

Attachment: Bug 1741362 - Modernize compose-send-message and fix some documentation. r=kaie

Depends on D203257

Comment 27

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/41a714e1fd6a
Use transfer encoding for S/MIME signed only messages. r=kaie
https://hg.mozilla.org/comm-central/rev/b4c568b6d7d0
Modernize compose-send-message and fix some documentation. r=kaie

Comment 28

[Kai Engert (:KaiE:)]

Comment on attachment 9388640 [details]
Bug 1741362 - Use transfer encoding for S/MIME signed only messages. r=kaie

[Approval Request Comment]
Regression caused by (bug #): No
User impact if declined: Some signed S/MIME emails will be corrupted on transport through MTAs
Testing completed (on c-c, etc.):
Risk to taking this patch (and alternatives if risky): low, we simply turn on a commonly used base64 encoding for signed S/MIME email

Comment 29

[Wayne Mery (:wsmwk)]

Comment on attachment 9388640 [details]
Bug 1741362 - Use transfer encoding for S/MIME signed only messages. r=kaie

[Triage Comment]
Approved for beta

Comment 30

[Daniel Darnell [:dandarnell]]

Thunderbird 124.0b4:
https://hg.mozilla.org/releases/comm-beta/rev/51761673b2d3

Comment 31

[Wayne Mery (:wsmwk)]

Comment on attachment 9388640 [details]
Bug 1741362 - Use transfer encoding for S/MIME signed only messages. r=kaie

[Triage Comment]
Approved for esr115

Comment 32

[Kai Engert (:KaiE:)]

Attachment: Backport to esr115, only the usual let/const difference.

Comment 33

[Rob Lemley [:rjl]]

Thunderbird 115.9.0:
https://hg.mozilla.org/releases/comm-esr115/rev/e6498d62412d