Closed Bug 1899110 Opened 2 years ago Closed 2 years ago

inline signed openpgp with utf-8 content gets garbled

Categories

(MailNews Core :: Security: OpenPGP, defect)

Product:
MailNews Core
Component:
Security: OpenPGP
Version:
Thunderbird 115
Type:
defect

Tracking

(thunderbird_esr115 wontfix, thunderbird_esr128 fixed, thunderbird128 affected)

Status:
RESOLVED FIXED
Milestone:
129 Branch
Tracking Flags:
Tracking Status
thunderbird_esr115 --- wontfix
thunderbird_esr128 --- fixed
thunderbird128 --- affected

People

(Reporter: vesely, Assigned: mkmelin)

Details

(Keywords: testcase)

Attachments

(3 files)

Source pane above, message window (partially covered) below
116.52 KB, image/png
Details
Full message resulting in the reported behavior
1.66 KB, message/rfc822
Details
Bug 1899110 - Fix charset issue for inline OpenPGP, when the key was not known. r=kaie
48 bytes, text/x-phabricator-request
wsmwk
: approval-comm-esr128+
Details | Review

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0

Steps to reproduce:

Send myself a PGP-signed message containing accented characters.

Actual results:

Accented characters are not recognized ().

In the source pane (ctrl-u), after View/Repair Text Encoding, they look good.

View source always shows as latin1 (there's a bug open for that).
Looks like what's sent wasn't properly encoded?

The signed text is utf-8. It wasn't signed by Thunderbird. I signed it off-line using gpg --clearsign.

Did you try to open the message with TB?

I tried it. Repair text encoding doesn't change it, still displays broken. Are you sure the .eml has the original encoding?

The content is text/plain; charset=UTF-8, as Content-Type: declares. Hex mapping brings:

00000000  c3 a0 c3 a8 c3 ac c3 b2  c3 b9                    |..........|

Where, for example, c3 a0 = 1100 0011 1010 0000 which translates to 1110 0000 = U+00e0 LATIN SMALL LETTER A WITH GRAVE.

It's strange that Repair text encoding doesn't fix it. The file should show five accented letters, àèìòù, when displayed in a text editor or in a terminal window.

I see. Must be the OpenPGP processing doing something wrong. If I edit the message not to have the openpgp delimiters it's displayed correctly.

Status: UNCONFIRMED → NEW
Component: Untriaged → Security: OpenPGP
Ever confirmed: true
Keywords: testcase
Product: Thunderbird → MailNews Core
Summary: Signed messages don't decode utf-8 → inline signed openpgp with utf-8 content gets garbled

Inline messages don't work well with non-ascii text. Our existing code to verify signatures (inherited from former Enigmail) makes several attempts in decoding an inline signed messages, trying out different encodings, to check if one interpretation results in a correctly verified message.

That code isn't great. But I don't know if a more reliable implementation is possible. Senders should rather use PGP/MIME whenever they must transport non-7bit characters.

Would you please shred some light on why does the encoding matter? I thought clear text just needs to normalize line endings, spaces and dashes.

Anyway, the Content-Type: header field does declare the encoding.

Now I've tested using your attached message.

I cannot fully reproduce your report.
Let me try to summarize what you report and what I see.

You have attached two different screens.

First, you show your screen of "view source".
That displays the correct rendering.
No bug here, correct?
But in any case, I would NOT expected "view source" to guarantee the correct rendering.

Second, you show a rendering of the message display.
This confuses me.

Your screenshot shows that Thunderbird shows the "BEGIN PGP SIGNED MESSAGE" starting line.

I don't understand why you see that line shown by Thunderbird.
If I open your attached message, I do NOT see that BEGIN PGP line.
In my test, it displays only the message text.

However, I notice a difference here, based on whether the signer's key is available or not.

If the signer's key is missing, then I see garbled text.
If the signer's key is available, then the correct text is shown.

So, when the signer's key is missing, and we cannot verify the signature, our code apparently does a second attempt, rendering the message with a different charset.

However, Alessandro, you said you have sent the message to youself.
I conclude in your scenario the signer's key was available when reading the message, because it's your own key.
You should have seen the correct text with a valid signature.

Can you please clarify which problem exactly you see and that you would like to see changed/fixed?

I may have a fix.

Assignee: nobody → mkmelin+mozilla

Magnus, can you please say what behavior you want to change?
I still don't clearly understand the bug here.

The message has inline openpgp that we can't verify. (Or at least I don't have the key setup.)
The message should not become corrupted but be shown plain text exactly like it was, like if we didn't run it through OpenPGP code.

(In reply to Magnus Melin [:mkmelin] from comment #13)

The message should not become corrupted but be shown plain text exactly like it was, like if we didn't run it through OpenPGP code.

Exactly! Since it was signed off-line, it is correct to see -----BEGIN PGP SIGNED MESSAGE----- and the signature, irrespective of its verification. I don't understand how come Kai doesn't see it. (Note that I could have written that at the beginning of a line. What would have happened to the mail message rendering then?)

(In reply to Magnus Melin [:mkmelin] from comment #13)

The message has inline openpgp that we can't verify. (Or at least I don't have the key setup.)

You can get the key. While the message is shown, click on the OpenPGP label, then click on the Discover button. it will find Alessandro's key. Then load the message again.

The message should not become corrupted but be shown plain text exactly like it was, like if we didn't run it through OpenPGP code.

That makes sense.

(In reply to Alessandro Vesely from comment #14)

Since it was signed off-line,

"off-line"? What do you mean with that?

it is correct to see -----BEGIN PGP SIGNED MESSAGE----- and the signature, irrespective of its verification.

But I never see this with Thunderbird.

Our implementation will render the message, and then scan for the BEGIN PGP... separator.

If the COMPLETE section consists of the PGP wrapper lines, then we automatically replace the display. We automatically remove the BEGIN line, end END line and the signature section, and only show the text between the BEGIN and END lines.

I don't understand how come Kai doesn't see it. (Note that I could have written that at the beginning of a line. What would have happened to the mail message rendering then?)

I don't understand your question.

Can you illustrate your theoretical scenario in more detail?

What is it that "you could have written at the beginning of the line"?

Wait.

More confusion.

I get different behavior based on the setting "view / message body as".

I am trying a few scenarios, and will report again soon.

Ok, sorry for the earlier confusion.
It isn't related to the "view" setting. It's related to the presence of the signer key.

If I do not have Alessandro's key imported,
then I see the BEGIN PGP and END PGP lines,
and the corrupted/garbles characters.

If the signer key is available,
those lines are no longer shown,
and I see the correct characters

So the bug is:

If we cannot verify the signature,
and Thunderbird continues to show the message source,
it should show the correct rendering, according to the message contents.

If Thunderbird's OpenPGP signature verification does any modification of the shown message body, for attempting to verify the message by interpreting it using alternative character encodings, then Thunderbird should revert to the original display after those experiments failed.

(In reply to Kai Engert (:KaiE:) from comment #17)

More confusion.

Me too. On May 16th I upgraded to thunderbird:amd64 (1:115.9.0-1deb12u1, 1:115.10.1-1deb12u1), and that was the last update in the log. Yet, now I opened that message again and only see the text, w/o BEGIN, correctly decoded. If I change View/Message Body As, I see the whole message, including PGP wrapper, for such a short time that I had to register it an play back frame by frame to realize that decoding was good even during that period.

I reopened the non-MIME message I had received before, and I still see the U+FFFD replacement characters (which don't show up in bugzilla preview). The difference must be having or not having the key. I imported my key after reporting the bug, and still don't have their one.

If the implementation replaces the wrapper, how does one tell a MIME message versus one signed off-line?

(my previous comment has been sleeping unsent during the lunch break)

I don't understand re-encoding attempts when the key is not present. Without key, TB should just forget about PGP.

IMHO, there should be an option to display the PGP wrapper anyway, for consistency with what was sent.

For UTF-8 we did not convert to bytestring to begin with, so upon conversion back we got the wrong result.

Status: NEW → ASSIGNED
status-thunderbird128: --- → affected
status-thunderbird_esr115: --- → wontfix
Target Milestone: --- → 129 Branch

Pushed by mkmelin@iki.fi:
https://hg.mozilla.org/comm-central/rev/95df1981616c
Fix charset issue for inline OpenPGP, when the key was not known. r=kaie

Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Keywords: checkin-needed-tb
Resolution: --- → FIXED
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Pushed by geoff@darktrojan.net:
https://hg.mozilla.org/comm-central/rev/7534d3608f2e
Fix charset issue for inline OpenPGP, when the key was not known. r=kaie

Status: REOPENED → RESOLVED
Closed: 2 years ago → 2 years ago
Keywords: checkin-needed-tb
Resolution: --- → FIXED

Comment on attachment 9407528 [details]
Bug 1899110 - Fix charset issue for inline OpenPGP, when the key was not known. r=kaie

[Approval Request Comment]
User impact if declined: garbled characters in certain situations
Testing completed (on c-c, etc.): c-c, beta
Risk to taking this patch (and alternatives if risky): may see garbled text

Attachment #9407528 - Flags: approval-comm-esr128?

If it is not a regression I'm thinking to wait til next week for esr128 to give another week on beta, unless someone posts a concern about that.

Comment on attachment 9407528 [details]
Bug 1899110 - Fix charset issue for inline OpenPGP, when the key was not known. r=kaie

[Triage Comment]
Approved for 128esr

Attachment #9407528 - Flags: approval-comm-esr128? → approval-comm-esr128+
See Also: 1911227 →
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: