Closed Bug 1731529 Opened 4 years ago Closed 3 years ago

Thunderbird 91 creates an additional newline in S/MIME messages that causes the signature to be shown as invalid by Outlook

Categories

(MailNews Core :: Security: S/MIME, defect)

Product:
MailNews Core
Component:
Security: S/MIME
Version:
Thunderbird 91
Type:
defect

Tracking

(thunderbird_esr91+ fixed, thunderbird_esr102 unaffected, thunderbird95 fixed)

Status:
RESOLVED FIXED
Milestone:
96 Branch
Tracking Flags:
Tracking Status
thunderbird_esr91 + fixed
thunderbird_esr102 --- unaffected
thunderbird95 --- fixed

People

(Reporter: hannes.lorenz, Assigned: KaiE)

References

(Regression)

Details

(Keywords: regression)

Attachments

(7 files)

an S/MIME-signed message produced by Thunderbird 78
6.88 KB, text/plain
Details
an S/MIME-signed message produced by Thunderbird 91
6.86 KB, text/plain
Details
bad-message.txt
7.39 KB, text/plain
Details
good-message.txt
7.39 KB, text/plain
Details
bad-message-signed.txt
16.35 KB, text/plain
Details
good-message-signed.txt
16.35 KB, text/plain
Details
Bug 1731529 - Restore TB78 behavior for MIME multipart newlines. r=mkmelin
48 bytes, text/x-phabricator-request
wsmwk
: approval-comm-beta+
wsmwk
: approval-comm-esr91+
Details | Review

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:92.0) Gecko/20100101 Firefox/92.0

Steps to reproduce:

Sending an unencrypted, but signed email results in an invalid signature (my certificate is still valid).

Actual results:

The email signature cannot be verified by the receiver: Digitial Signature is Not Valid.
Error message: This message includes a digital siganture, but the signature is invalid. The signature does not match the message content correctly. The message appears to have been altered after the sender signed it. You should not trust the validity of this message until you verify its contents with the sender.

Note that this is a problem on the sender side. My colleague with Thunderbird 91 has it as well, but others with Thunderbird 78 don't.
Also note that the email signature is valid in the own "Sent" folder, but not by the receiver.

Expected results:

The S/MIME signature should be correct.

I can confirm this with the following addtions:
The problem only occurs when sending a not "Text-only"-mail.

When sending a text-only-mail, the signature ist fine and verified by Outlook on the receiving side.
But as soon as you add a single HTML block (like <b>Test</b>) or any attachment the singning process seems to break (at least for Outlook; Thunderbird still displays the sent mail normally and correctly signed).

Flags: needinfo?(hannes.lorenz)

Yes, I just tried it myself and I can confirm that it only happens when sending HTML emails.

Just to add: The signature also can't be verified on another Thunderbird. The signature is only correct in the "Sent" folder.

Flags: needinfo?(hannes.lorenz)

Please provide a pair of test messages.

In thunderbird on the sending side, go to sent folder, and use menu save as file. Attach that here.
Then on the receiving side, do the same, and attached that received file here, too.

This problem is usually caused by a mail transport agent modifying messages, although RFCs say that multipart/signed MIME parts MUST remain unmodified on transport.

Summary: Invalid S/MIME signature → Invalid S/MIME signature shown by Outlook

This trouble does not appear when Delivery Format is set to Plain Text Only or Rich Text Only.
This can be confirmed when Delivery Format is Plain and Rich, then Content-Type lines are rewritten as follows:

Sending:
Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-256; boundary="------------ms020904090905050006060501"

Receive:
Content-Type: multipart/signed;
protocol="application/pkcs7-signature";
micalg=sha-256;
boundary="------------ms020904090905050006060501"

Sending:
Content-Type: text/plain; charset=UTF-8; format=flowed

Receive:
Content-Type: text/plain;
format=flowed;
charset="UTF-8"

Sending:
Content-Type: text/html; charset=UTF-8

Receive:
Content-Type: text/html;
charset="UTF-8"

and so on.

I am also seeing this. Office 365 complains that S/MIME signatures generated by Thunderbird 91 are invalid (message may have been modified), but can verify the S/MIME signatures of messages generated by Thunderbird 78 successfully.

In both cases, Thunderbird can verify both signatures successfully. So can openssl smime -verify. So this suggests two possibilities:

  1. Thunderbird 91.x does something subtly nonstandard (incorrect) when generating S/MIME signatures that most mail clients tolerate, but not O365.
  2. The S/MIME signatures that Thunderbird 91.x are perfectly valid according to the RFCs, but O365 has a bug that causes it to incorrectly assert the signatures are invalid.

I will attach two S/MIME test messages to this bug, one generated by Thunderbird 78 and one generated by Thunderbird 91. All versions of Thunderbird assert the signatures are valid, as do all versions of openssl smime -verify that I can find. But O365 complains the message generated by Thunderbird 91 has an invalid S/MIME signature.

This is an S/MIME-signed message produced by Thunderbird 78. All mail clients that I have been able to test assert that the S/MIME signature is valid, including O365.

This is an S/MIME-signed message produced by Thunderbird 91. All mail clients that I have been able to test assert that the S/MIME signature is valid, except O365, which asserts the signature is invalid.

I believe I have figured out the problem: it’s a bug with Outlook stripping trailing blank lines when it must not do so, and a behavior change between Thunderbird 78 and 91 is triggering it.

Specifically: between Thunderbird 78 and 91, Thunderbird’s behavior changed when generating multipart/signed messages where the content being signed is a nested MIME structure. Specifically, Thunderbird 91 adds a trailing newline, when Thunderbird 78 did not.

You can see this if you look at my attachments in comment 6 and comment 7. Here’s an excerpt of what Thunderbird 78 produced, looking at the end of the signed data and the beginning of the signature:

--------------20022950750624606FF7C21C--

--------------ms040108050204000101080907
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

And here is Thunderbird 91:

--------------MNLPaPHnVTfEP3cqjzuMqFYk--


--------------ms050501010507010003030208
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

After generating the multipart/alternative content, Thunderbird 91 added a trailing newline before generating the signature.

To prove that the extra newline is the smoking gun, I used Thunderbird 91 to generate a multipart/mixed message with two parts (a text/html part and a small image/png part). I then took that multipart/mixed part and then saved it to bad-message.txt and good-message.txt. For bad-message.txt, I added a trailing newline, the same as Thunderbird 91 does. Then I used openssl smime -sign on each message to create two separate S/MIME-signed messages, bad-message-signed.txt and good-message-signed.txt.

openssl smime -verify claims that both bad-message-signed.txt and good-message-signed.txt have valid S/MIME signatures. And when I send both messages to my O365 address, Thunderbird asserts that both messages have valid S/MIME signatures. But Outlook asserts that bad-message-signed.txt has an invalid signature and good-message-signed.txt has a valid signature.

Almost certainly, Outlook is stripping that trailing newline before verifying the signature. But that it must not do. Quoting RFC1521§7.2.1:

Note that the encapsulation boundary must occur at the beginning of a line, i.e., following a CRLF, and that the initial CRLF is considered to be attached to the encapsulation boundary rather than part of the preceding part.

So, if there are two CRLF pairs before the encapsulation boundary, only second CRLF (the one immediately preceding the encapsulation boundary) is part of the encapsulation boundary. The first CRLF pair is part of the content preceding the encapsulation boundary. And RFC3851§3.4.3.2 is also quite clear: the signature must be generated over the entire content. Both Thunderbird and OpenSSL correctly do that when generating signatures, and correctly do that when extracting the signed data to perform the signature verification. But Outlook does not.

My suspicion is that most mail clients, like Thunderbird 78, do not add a trailing newline to multipart/* message bodies before calculating the S/MIME signature. So this bug has likely been present in Outlook for a long time. It is only the change in Thunderbird 91 that has brought it to light.

The true solution here is for Microsoft to fix the bug in Outlook. But even if Microsoft fixes the bug, it will take years (if not decades) before all broken versions of Outlook are upgraded. And the installed user base for Outlook is massive—to the point that Outlook being unable to correctly verify S/MIME signatures generated by Thunderbird 91 is a dealbreaker for using S/MIME signing in Thunderbird 91.

So, even though Thunderbird is doing nothing that violates any RFCs, in order to work around the Outlook bug, the best solution here is for Thunderbird to revert to the Thunderbird 78 behavior of not including a trailing newline when generating multipart/* messages.

Is this possible/feasible?

Attached file bad-message.txt —

Comment on attachment 9245568 [details]
bad-message.txt

See comment 8.

Attachment #9245568 - Attachment description: a multipart/mixed body with a trailing newline → bad-message.txt
Attached file good-message.txt —

See comment 8.

Attached file bad-message-signed.txt —

See comment 8.

See comment 8.

Would you like to try to apply the patch from bug 1731109 and check if that cleanup would fix the issue with Outlook?

Status: UNCONFIRMED → NEW
Component: Security → Security: S/MIME
Ever confirmed: true
Product: Thunderbird → MailNews Core
Summary: Invalid S/MIME signature shown by Outlook → Invalid S/MIME signature shown by Outlook - the signature is valid (according to RFC), but Outlook think's not because Thunderbird 91 added a trailing newline before generating the signature.

If a signed email is received by Thunderbird 78.14.0, it is also shown as invalid.

(In reply to Kai Engert (:KaiE:) from comment #14)

Would you like to try to apply the patch from bug 1731109 and check if that cleanup would fix the issue with Outlook?

I’ve built thunderbird-91.2.0-1.fc33.0 using the patch from bug 1731109, and will test to see if avoids triggering the bug with Outlook.

When Thunderbird 91.2.0 is built with patch from bug 1731109, it no longer adds an additional blank to the nested MIME structure before calculating the signature. Because there is no additional blank line, there is no blank line for Outlook to (erroneously!) strip, and therefore Outlook calculates the signature correctly and asserts that the signature on the message is valid.

In summary, incorporating the patch from bug 1731109 avoids triggering the Outlook bug and thus resolves this issue.

(Our organization has a support contract with Microsoft, so I have asked our Windows team to report this Outlook bug to Microsoft, but I’m not holding my breath for Microsoft to fix it.)

Looks to be resolved as of 91.3 update released yesterday.

Using Thunderbird 91.3.0, the bug still exists for me when I receive a signed unencrypted email. Sent by Thunderbird, received by Thunderbird. This does not seem to only be an Outlook issue.

Just send yourself an email to verify that it's still there.

Here we have the very similar problem, unfortunately still unresolved by 91.3.0 update.
Every message signed in (and only) TB 91 is read by Outlook 2013 and 2016 as "The message contents may have been altered".

My tech-colleagues have found out that the real problem is on Outlook side but the reason is the change of encoding messages in TB 91 from Content-Transfer-Encoding: quoted-printable to Content-Transfer-Encoding: 8bit (not supported by Outlook).

@Zach_H, have you checked the emails in Outlook 2013 and 2016?

Flags: needinfo?(zach.heise)

No, I'm sorry - we only have Outlook "365" and Outlook 2019 in our office (an advantage of university licenses I suppose but makes testing multiple versions more difficult).

I sent about a dozen messages via TB 91.3 yesterday, all in HTML + S/MIME cert and checked the appearance of each one of them in my outlook 365's "sent" item folder to confirm they were showing up as correct, and my boss confirmed that they looked good to him in Outlook 365's inbox too (he was the one that alerted me to the problem since he exclusively uses Outlook; I only use it for calendar/scheduling and use Thunderbird for all email).

It did seem to be a little too good to be true though, to go from James Ralston's comment of "building with this patch fixes it" to the Thunderbird devs incorporating the bugfix into 91.3 in a day or two without further testing.

Flags: needinfo?(zach.heise)

(In reply to Zach_H from comment #22)

It did seem to be a little too good to be true though, to go from James Ralston's comment of "building with this patch fixes it" to the Thunderbird devs incorporating the bugfix into 91.3 in a day or two without further testing.

To clarify, I only tested Thunderbird 91.2.0 (built with the patch from bug 1731109) against Outlook Microsoft 365 (M365). Because 91.2.0+patch does not add a trailing blank line to the nested MIME part before inserting the signature, there is no trailing blank line for Outlook M365 to erroneously strip, and therefore Outlook M365 reports the signature as valid.

Alas, I do not have the ability to test previous versions of Outlook, as our organization recently migrated to M365.

I downloaded the source for Thunderbird 91.3.0 and examined it. Thunderbird 91.3.0 does not contain the patch from bug 1731109. When testing 91.3.0, it still adds a trailing newline to the nested MIME part before inserting the signature, which triggers the Outlook M365 bug and causes Outlook M365 to assert that the signature is invalid (modified).

(Remember, Thunderbird only seems to generate the trailing newline if the part being signed is a nested MIME structure (e.g., multipart/alternative or multipart/mixed). If the part being signed is a non-leaf part (e.g., text/html), Thunderbird does not add a trailing newline, and thus the bug with Outlook M365 is not triggered.)

(In reply to Matt_Brz from comment #21)

Here we have the very similar problem, unfortunately still unresolved by 91.3.0 update.
Every message signed in (and only) TB 91 is read by Outlook 2013 and 2016 as "The message contents may have been altered".

My tech-colleagues have found out that the real problem is on Outlook side but the reason is the change of encoding messages in TB 91 from Content-Transfer-Encoding: quoted-printable to Content-Transfer-Encoding: 8bit (not supported by Outlook).

Does this happen when using Thunderbird 91.3.0 to send a signed message where the signed content is a leaf part? E.g., does it happen if you send a message and manually set Options → Delivery Format to Plain and Rich (HTML) Text and include no attachments in the message (so that the signed part is text/html)?

(In reply to James Ralston from comment #23)

Does this happen when using Thunderbird 91.3.0 to send a signed message where the signed content is a leaf part? E.g., does it happen if you send a message and manually set Options → Delivery Format to Plain and Rich (HTML) Text and include no attachments in the message (so that the signed part is text/html)?

If I manually set Options → Delivery Format to Plain and Rich (HTML) Text and there is any part of HTML (it can be a content of the message or a company's signature in html) the S/MIME signature is displayed as incorrect.
If I manually set Options → Delivery Format to Plain Text S/MIME signature is ok but company's signature and any messages written by a second person are converted to plain text which looks not so good.

This might be important: Even if delivery format is set to html or text and html but there is only pure text in the message, the signature is correct. Any use of html causes problem.

James, do you use OpenPGP or S/MIME? I do not know if this really matter (as a non-tech I do not know if there is a technical difference in use) but the bugzilla referred to OpenPGP problems. I can see that https://bugzilla.mozilla.org/show_bug.cgi?id=1731109 is split into two parts from which only one was implemented in 91.3.0.

(In reply to Zach_H from comment #22)

Thank you for reply.

We stumbled upon this problem in our company too. The problem occurs when we send S/Mime signed Plain-Text Emails with a attachment (e.g. a pdf file).

We used Thunderbird 91.3.0 to send the mail and Outlook 2016 to read it. A new datapoint is that Apple Mail 14.0 also shows said Mail as not signed correctly.

Same in our company.

We, in the organization, also have this problem at the moment. In the case of business communication, we cannot switch to text.
Editing html message content in extensions (such as ThunderHTMLedit) also seems to have no effect on the result.

Hi there,
I did a bit of research because we had the same problem and I hope this will help to locate the Bug.

We had the Problem that a digital signed eMail with an attachment sent by Thunderbird 91.3.1 could not be verified by Outlook2016 or Outlook2019.
(Not tested yes with Outlook2021)
This eMail can be verified by Thunderbird ,eGroupware and even by Openssl at the command line.
Outlook is the only client that can not verify the mail corectly.
After sone research I found out that this Behaviour has been introduced with Thunderbird 86.0b1.
Thunderbird 85.0b3 was the last version that did not have this behaviour.
Reading the releasenotes of TB 86.0b1 showed
"Thunderbird 86.0b1 is using a new implementation of the message sending code that is easier to maintain and more flexible. As with any major change, bugs are bound to happen. The old behavior can be restored by setting the pref "mailnews.send.jsmodule" to false."
https://www.thunderbird.net/en-US/thunderbird/86.0beta/releasenotes/

So I tried to set the option mailnews.send.jsmodule to false and all digitally signed eMails with Attachments were verified by Outlook correctly again.
@ Sebastian: This could be a workaround for You

So the source of this behaviour may be the new implementation od the sending code introduced in 86.0b1.

Greetings
Marc

Hi,
I could reproduce the above comment of Marc on my Windows 10 system with TB 91.3.1 yesterday. With "mailnews.send.jsmodule"=true the receiving outlook will not accept the email as valid.
When I switched "mailnews.send.jsmodule" to false and sent it again the signed email was shown as valid in Outlook.

This morning there was an update to TB 91.3.2. As I do not have an outlook myselfes, I do not tested this with the new version yet.

In the list of fixes in TB 91.3.2 there ist no entry about this problem

Greetings
Axel

Hi,
there is still this problem with invalid signature in Outlook 2013 and 2016 after TB update to 91.3.2
Axel, where can I change this "mailnews.send.jsmodule"? I could werify if encoding a message parts is not 8-bit (wchich probably causes problems in TB91).

HI,
got to Tools => Preferences => General.
Then scroll down to Config Editor (last Entry)
click on "Config Editor"

insert Text "mailnews.send.jsmodule" (without "") in the search-box.

then you can see this entry and its current position. With click on the toggle Button you can change from true to false.

Axel

Hi,
I should additionally mention that after the procedure Axel described, a restart of thunderbird is required.

Marc

I can confirm that:

  1. the hack suggested by Marc does work;
  2. TB update 91.3.2 does not fix the problem, unless 1) is adopted.

Axel, Marc,

Thank you for hints. I can confirm this workaround works for reading properly signature in Outlook 2013 and 2016.

I have also checked the source of the message. It looks different (actually it is like it was in TB78) and what might be important, there is no 8-bit Content-Transfer-Encoding. Instead the Content-Transfer-Encoding is set to quoted-printable which was previously in TB78.
Somehow it might be connected to bug 1731600

As for now, can you tell me if disabling this mailnews.send.jsmodule can have any negative impact on other features?

Hello
I also confirm that it is working properly. I also tested it in Outlook 2019 and it works fine too.

Hi,
for now I have not much expirience with the workarounce since I use it only for the last frew days.
I already adviced my colleques in our company to use this workaround since having the signatute properly verified by Outlook is very importand for us.
I dont expect any negative impact since setting the parameter to false is like enabling the old method of sending eMails in Thunderbird like it was in Version 78.
https://www.thunderbird.net/en-US/thunderbird/86.0beta/releasenotes/

The only thing I worry about is that this old method would be depricated one day without the new method being able to work correctly.

Greetings
Marc

(In reply to Matt_Brz from comment #34)

...
As for now, can you tell me if disabling this mailnews.send.jsmodule can have any negative impact on other features?

No. But it certainly won't be available in the next version, next summer.
I should think valid bugs would be fixed by then.

Blocks: tb91found
Severity: -- → S3
status-thunderbird_esr91: --- → affected
tracking-thunderbird_esr91: --- → +
Summary: Invalid S/MIME signature shown by Outlook - the signature is valid (according to RFC), but Outlook think's not because Thunderbird 91 added a trailing newline before generating the signature. → Invalid S/MIME signature shown by Outlook/Thunderbird - the signature is valid (according to RFC), but Outlook/Thunderbird think not because Thunderbird 91 added a trailing newline before generating the signature.
Assignee: nobody → kaie
Status: NEW → ASSIGNED

Thanks everyone for the input you have provided.

It seems reasonable to revert to the old behavior of TB78, and avoid the double newline between multipart boundaries, a change which the experimental patch from bug 1731109 included.

We're abandoning most of bug 1731109, but I'd like to take some of the cleanup from it to work around the issue you've reported here.

In addition to the double newline, I'd like to make an additional change. At the end of text/html and text/plain parts, we should have a blank line. This was the TB78 behavior. With TB91, we currently place the multipart boundary immediately after the contents. I think we should restore the old behavior.

Keywords: checkin-needed-tb

Pushed by geoff@darktrojan.net:
https://hg.mozilla.org/comm-central/rev/88fa9d0345ee
Restore TB78 behavior for MIME multipart newlines. r=mkmelin

Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Keywords: checkin-needed-tb
Resolution: --- → FIXED
Target Milestone: --- → 96 Branch
Keywords: regression
Regressed by: 1211292

Comment on attachment 9251873 [details]
Bug 1731529 - Restore TB78 behavior for MIME multipart newlines. r=mkmelin

[Approval Request Comment]
Regression caused by (bug #): 1211292
User impact if declined: S/MIME digital signatures sent by Thunderbird are broken in Outlook
Testing completed (on c-c, etc.):
Risk to taking this patch (and alternatives if risky): low, limited to the meta structure of MIME email, should not affect well behaving applications other than Outlook

Attachment #9251873 - Flags: approval-comm-beta?

Let's wait for a few days in beta, then we should consider uplift to 91.x

Is there interest in an experimental build based on stable Thunderbird 91.x so you could test the fix easily?

Summary: Invalid S/MIME signature shown by Outlook/Thunderbird - the signature is valid (according to RFC), but Outlook/Thunderbird think not because Thunderbird 91 added a trailing newline before generating the signature. → Thunderbird 91 creates an additional newline in S/MIME messages that causes the signature to be shown as invalid by Outlook

Comment on attachment 9251873 [details]
Bug 1731529 - Restore TB78 behavior for MIME multipart newlines. r=mkmelin

[Triage Comment]
Approved for beta

FYI, next beta is next Monday. And then 91.4.0 the week after.

Attachment #9251873 - Flags: approval-comm-beta? → approval-comm-beta+

Comment on attachment 9251873 [details]
Bug 1731529 - Restore TB78 behavior for MIME multipart newlines. r=mkmelin

Let's request approval now, it can be considered once you feel this has been tested sufficiently in beta.

Attachment #9251873 - Flags: approval-comm-esr91?
Whiteboard: [TM91.4.+]

Comment on attachment 9251873 [details]
Bug 1731529 - Restore TB78 behavior for MIME multipart newlines. r=mkmelin

[Triage Comment]
Approved for esr91

Attachment #9251873 - Flags: approval-comm-esr91? → approval-comm-esr91+
Whiteboard: [TM91.4.+]

Just updated to Thunderbird 102.2.2 and the issue is back. The workaround in comment 28 (https://bugzilla.mozilla.org/show_bug.cgi?id=1731529#c28) does not work anymore.

Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Please don't reopen closed bugs. Your issue is unlikely to be the same. Please file a new bug with details.

Status: REOPENED → RESOLVED
Closed: 4 years ago → 3 years ago
Resolution: --- → FIXED

(In reply to hannes.lorenz from comment #50)

Just updated to Thunderbird 102.2.2 and the issue is back. The workaround in comment 28 (https://bugzilla.mozilla.org/show_bug.cgi?id=1731529#c28) does not work anymore.

I just tested 102.2.2, and I cannot reproduce: 102.2.2 avoids adding the additional newline that causes Outlook to erroneously assert the S/MIME signature is invalid.

I agree with Magnus: if you are seeing 102.2.2 generate messages that [other] mail clients claim have invalid S/MIME signatures, that is unlikely to be the same issue.

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: