Closed Bug 1731109 Opened 3 years ago Closed 3 years ago

tracker: OpenPGP signatures broken on Thunderbird mailing lists (topicbox bug, they rewrite the multipart/signed part)

Categories

(Thunderbird :: General, defect)

Product:
Thunderbird
Component:
General
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED INVALID

People

(Reporter: KaiE, Assigned: KaiE)

References

Details

Attachments

(6 files, 1 obsolete file)

without-patch.eml
7.89 KB, message/rfc822
Details
without-patch-topicbox.eml
8.68 KB, message/rfc822
Details
with-armor-wrap.eml
7.81 KB, message/rfc822
Details
with-armor-wrap-topicbox.eml
8.63 KB, message/rfc822
Details
with-all-tweaks.eml
7.97 KB, message/rfc822
Details
with-all-tweaks-topicbox.eml
8.49 KB, message/rfc822
Details

This isn't really a bug in Thunderbird - but I suggest we track the related problem here.

The Thunderbird team currently uses the topicbox.com system for their mailing list infrastructure.

Unfortunately, when sending digitally signed (OpenPGP) message to the mailing lists, the Topicbox servers change the message body in ways that break the signatures - on the receiving side, the message body no longer matches the originally sent message, and therefore Thunderbird reports a broken signature.

In October 2020 we had contacted Topicbox support, and they had acknowledged the situation, and said they are interested in fixing the situation. However, they haven't yet fixed it, the problems are ongoing.

I have asked them today to give us an update.

The topicbox servers do the following:

(a) "They add a Content-Transfer-Encoding: 7bit" in multiple places, in the middle of our nested MIME message. We omit that header. 7bit is the default if absent. Could we always add that header? Could you imagine a good place to add it?

(b) They remove strings which are usually not shown to the user, like "this is a multi-part message in mime format". I guess we would have to omit those strings if we wanted to fix it on our side.

(c) They rewrap the message contents. For lines that contain a "=" at the end, they pull in on additional character from the next line, and append it at the end. So "abcd=\nefg" is turned into "abcde=\nfg"
Sometimes, if the last character is space, they encode it as =20 - so "abcd =\nefgh" is changed to "abcd=20\nefgh"

I guess (c) is the most difficult (and unlikely) part for us to fix. Because in order to avoid the wrapping changes on their side, we'd have to create lines of exactly the length they are happy with. And of course there is probably no perfect length that makes all of the world's mailers happy.
clokep: mkmelin do WE have a bugzilla component for thunderbird infrastructure issues?

(d) they wrap our message in an additional layer, and add an outer multipart/mixed, where the second part is a mailing system footer. This might be ok, I've seen mailman doing that too, and since it is outside the part we verify, it should be ok

Duplicate of bug 1710044?

(In reply to Tom from comment #2)

Duplicate of bug 1710044?

yes agreed. However, I'd like to mark the duplicate the other way round, because this bug here has the detailed technical details what's wrong.

Regarding (b), see also bug 1690603.

This message from the IETF OpenPGP mailing list claims that Topicbox is violating RFC 1847:
https://mailarchive.ietf.org/arch/msg/openpgp/acBRgzxXjv4mDPtYiKK5KxqjMEc/

I spent way more time on this than I should have...

I submitted a fix for bug 1732909, which I had wanted to do anyway.

Then I tried to further clean up how Thunderbird creates outgoing messages, but I'm not sure if we should take the following changes. Maybe they are helpful cleanup, but I'm not sure:

I changed all code related to OpenPGP that emits Content-Type lines, to achieve that we wrap earlier, and place the boundary= instruction on the next line.
This required to change the soft margin wrap line in jsmime.js to a smaller value, because jsmime is used to parse a content-type line, and it will then re-assemble all the parameters. Only by setting the smaller margin I was able to achieve that the line wraps earlier.

I added Content-Type 7bit to all places related to OpenPGP.

Because Topicbox is very picky, and requires an exact set of blank lines between MIME boundary, I had to tweak the code that adds line breaks in multiple places.

I had to increase the maximum column used by the MimeEncoder, to avoid that Topicbox rewraps HTML message parts.

I had to ensure that our HTML parts end with an additional blank line, to avoid that Topicbox adds a trailing "=" to the end of the HTML code.

I had to change how we encode PGP key attachments. If we use Quoted Printable encoding, then Topicbox adds a =0a= at the end of every line. (I couldn't find a justification why Topicbox would want to do that. My understanding of the QP encoding is that line ends \r\n should be left unmodified.

I couldn't switch to use base64 encoding for the pgp keys attachment, because Topicbox will re-wrap base64 contents at a different line length.

The only way that I found to make the pgp keys attachment go through unmodified: Attach it with 7bit encoding. (That's why I added an exclude rule for QP encoding for that MIME type.)

Assignee: nobody → kaie
Status: NEW → ASSIGNED
Attached file without-patch.eml —
Attached file with-armor-wrap.eml —
Attached file with-all-tweaks.eml —

I've attached 3 pairs of file to illustrate the changes of the patches, and to illustrate the behavior of topicbox.

without-patch shows today's code.

with-armor-wrap shows the effect of bug 1732909, and the topicbox behavior to add =0a=

with-all-tweaks is the combination of the patch from bug 1732909 with the patch I've attached here.

with-all-tweaks-topicbox.eml shows a valid signature.

Magnus, Patrick, what do you think?

Summary: tracker: OpenPGP signatures broken on Thunderbird mailing lists (topicbox bug, they rewrite the message body) → tracker: OpenPGP signatures broken on Thunderbird mailing lists (topicbox bug, they rewrite the multipart/signed part)

I made one more tweak. I changed the base64 encoder to wrap at line length 60. That's what topicbox sends out, regardless of input line lenghts.

With this change, even messages with binary attachments (image, pdf) pass through topicbox without signature corruption.

Should we consider to take these changes, a few tests will need changes, to check for wrapped results etc.

comm/mailnews/compose/test/unit/test_attachment.js
comm/mailnews/compose/test/unit/test_longLines.js
comm/mailnews/compose/test/unit/test_messageBody.js
comm/mailnews/compose/test/unit/test_messageHeaders.js
comm/mailnews/mime/test/unit/test_structured_headers.js

I want to split the patch.

I want to move the simpler set of changes about 7bit encoding and some header wrapping to bug 1731600, because in that bug we already have agreement on the 7bit changes.

Depends on: 1731600
See Also: → 1732909
No longer depends on: 1731600
See Also: → 1731600

Comment on attachment 9243217 [details]
Bug 1731109 - Experimental changes that allow signed messages to go through Topicbox without signature corruption. r=mkmelin

Revision D126783 was moved to bug 1731529. Setting attachment 9243217 [details] to obsolete.

Attachment #9243217 - Attachment is obsolete: true

Marking invalid.

It isn't reasonable to create exactly the format that Topicbox expects.

Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → INVALID

parts of the cleanup changes are moved to bug 1731529

We don't have a really robust "avoid rewrites within the signed bit" so we've now changed to just not rewrite at all when we get signed mail through Topicbox. This means that we're adding the List-* headers, but we're not rewriting the body (e.g. to add the footer) at all.

Thank you Bron, that's a great solution, I'm very happy about your fix!
Best Regards

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: