1866453 - Thunderbird could not decode raw ISO-2022-JP encoded headers

Status: REOPENED

(MailNews Core :: MIME, defect)

Description

[Masatoshi Kimura [:emk]]

Some older Japanese emails have subject/from fields encoded in ISO-2022-JP without using MIME "B"-encoding for historical reasons. The latest Thunderbird could not decode such subject/from fields. I think this is a regression by JSMime. (Yes, this is not a recent regression.)

Comment 1

[Francesco]

Subjects need to be RFC-2047 encoded or raw UTF-8. Any other encoding is invalid, see RFC 6532: https://datatracker.ietf.org/doc/html/rfc6532. That subject fields are no longer repaired is bug 1739609.

Comment 2

[Masatoshi Kimura [:emk]]

So WONTFIX? My Mailbox has pre-RFC 6532 mails that Thunderbird can no longer show the subject correctly.

Comment 3

[Francesco]

I'm just commenting on new bugs, not taking any decisions. Did any TB version ever display raw ISO-2022-JP in the subject? JS Mime landed in TB 31 (from memory, in 2014 or so). I believe the real bug is that you can no longer repair such subjects.

Comment 4

[Masatoshi Kimura [:emk]]

Hm, my memory was incorrect. Even Tb 1.5 did not show the subject correctly. Apparently the message was imported from a Microsoft product. It had the following header:

X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1441

Comment 5

[Francesco]

MS products could also handle raw ISO-8859-1/windows-1252 in the subject which TB never did.

Comment 7

[Masatoshi Kimura [:emk]]

I'm reopening this bug because I found a STR and bisected. I had to import an mbox that was imported from Outlook Express and view the message list pane.
Last good: https://hg.mozilla.org/comm-central/rev/546a24b91d05
First bad: https://hg.mozilla.org/comm-central/rev/2608fae8b13b
Changeset: https://hg.mozilla.org/comm-central/pushloghtml?fromchange=546a24b91d05&tochange=2608fae8b13b
I guess bug 858337 or bug 790855 is the culprit.

Comment 8

[Masatoshi Kimura [:emk]]

Attachment: Bad example

Comment 9

[Masatoshi Kimura [:emk]]

Attachment: Good example

Comment 10

[Francesco]

As per my comment #1, it's likely a WONTFIX. Subjects need to be RFC-2047 encoded or raw UTF-8. There are a few bugs like bug 1829655 where people complain about raw ISO-8859-1/windows-1252 not being handled "correctly".

Comment 11

[Masatoshi Kimura [:emk]]

As I said in comment #2, some old users (like me) may have pre-RFC 6532 mailbox.

Comment 12

[Masatoshi Kimura [:emk]]

convert8BitHeader already supports fallback to non-UTF-8 charset, but it does not consider the case for the 7-bit non-ASCII charset (i.e. ISO-2022-JP). So I don't think this is the same pattern as the ISO-8859-1/windows-1252 case.