different encoding in message list and message header (list OK, header pane garbled)

NEW
Unassigned

Status

Thunderbird
Folder and Message Lists
--
minor
4 months ago
2 months ago

People

(Reporter: Hassan DRAGA, Unassigned)

Tracking

52 Branch
x86_64
Windows 7

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(5 attachments)

(Reporter)

Description

4 months ago
Created attachment 8901596 [details]
bug.png

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:51.0) Gecko/20100101 Firefox/51.0

Steps to reproduce:

Thunderbird 52.3.0 32Bit
windows 7 64Bit
---------

Email received with this headers : 
... ... ...
Subject: Test Accepté.
Content-Type: text/plain; charset="iso-8859-1"
... ... ...


Actual results:

In message/email list its show like this : 

[ Test Accept? ]

if i click on the concerned email in the list to show the full email body, the subject show correctly !


Expected results:

that mean thunderbird use different encoding in message list, and message body!, so the encoding in message list must be the same.
(Reporter)

Updated

4 months ago
Severity: normal → minor
OS: Unspecified → Windows 7
Hardware: Unspecified → x86_64
Summary: bad Unicode encoding in emails list (Subject) → different Unicode encoding in message list and message body

Comment 1

4 months ago
Please attach the message in question:
File > Save As (save as .eml file), remove any personal information with the editor, attach message here using "Attach File".

Alternatively, view the message source (View > Message Source) and copy the Subject: line here into a comment. Or if the message is short, paste the entire message here so I can also see the body which contains "Accepté" correctly decoded.

I know that at some stage the decoding in the thread pane (message list) and the header/message pane (preview area) were different, but I believe I fixed that, so I'm surprised this still happens in TB 52.

Do you use any add-ons?
(Reporter)

Comment 2

3 months ago
Created attachment 8903923 [details]
[iWeb] Paiement PA------7 Accept.eml
(Reporter)

Comment 3

3 months ago
Subject: Test Accepté.
Content-Type: text/plain; charset="iso-8859-1"
(Reporter)

Comment 4

3 months ago
Created attachment 8903924 [details]
bug2.png
(Reporter)

Comment 5

3 months ago
Subject: [iWeb] Paiement PA*******7 Accepté
X-PHP-Originating-Script: 1001:PhpMailWrapper.php
X-ASG-Orig-Subj: [iWeb] Paiement PA*******7 Accepté
From: "iWeb Technologies Inc." <compta@iweb.com>
Content-Type: text/plain; charset="iso-8859-1"

Updated

3 months ago
Attachment #8903923 - Attachment mime type: message/rfc822 → text/plain

Comment 6

3 months ago
That's for providing all the information. Like bug 1358463, this bug is invalid.

Header with non-ASCII characters must be RFC2047 encoded but raw UTF-8 is also permissible according to RFC6532.

The subject header "Accepté" is iso-8859-1 (windows-1252) encoded and if you play around with some setting, you can get it to display properly.

I'll attach the message with the header UTF-8 encoded and you'll see that it will work under all circumstances.
Status: UNCONFIRMED → RESOLVED
Last Resolved: 3 months ago
Resolution: --- → INVALID

Comment 7

3 months ago
Created attachment 8903951 [details]
[iWeb] Paiement PA------7 Accept.eml - with UTF-8 encoded header. Always working.

Comment 8

3 months ago
OK, I looked at the internals a little more. The treatment of the invalid header is a little inconsistent.

For showing the header in the thread pane (message list), the subject is decoded when storing the message in the database. At that stage, the message header
  Content-Type: text/plain; charset="iso-8859-1"
will be used as a fallback, so you get the correct decoding in the database, but only for single part messages with that header.

For the header pane (the area above the message preview/pane), we use another fallback, the one from the folder properties or the one you pointed out in the font settings.

We won't invest any work into making an invalid header more consistent.

Headers *must* be UTF-8 encoded or RFC 2047 encoded. Anything else will give trouble.

Comment 9

3 months ago
Comment #8 as only 90% correct. This should be 100% correct now:

For showing the header in the thread pane (message list), the subject is decoded using the charset from the message header
  Content-Type: text/plain; charset="iso-8859-1"
as a fallback, so you get the correct decoding. The only works in the particular case of that message, since it is a single part messages with that header.

For the header pane (the area above the message preview/pane), we use another fallback, the one from the folder properties or the one you pointed out in the font settings.

Comment 10

3 months ago
OK, changed my mind. We can fix this. I already invested time to investigate this.
Status: RESOLVED → UNCONFIRMED
Resolution: INVALID → ---

Updated

3 months ago
Assignee: nobody → jorgk
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true

Comment 11

3 months ago
OK, I looked at it a little more: For the header pane the charset comes from the folder override or some other override and not from the message header which seems to be used for the tread pane (message list):
https://dxr.mozilla.org/comm-central/rev/73875939393398711b85122ceaed07ccf5c709c2/mailnews/mime/src/nsStreamConverter.cpp#143

It might be quite hard to get the header charset at that point.
Summary: different Unicode encoding in message list and message body → different encoding in message list and message header (list OK, header pane garbled)

Comment 12

3 months ago
Created attachment 8904008 [details] [diff] [review]
1394244-iso-8859.patch - WIP

I tried to improve the charset selection, but no luck so far, turns out to be harder than I thought. The charset is taken from the folder or some other override and there is no easy way I can see so far to get to the header charset at that point.

So this is quite hard and we should spend time on improving something which is invalid to start with.

Comment 13

2 months ago
I won't be pursuing this further. I guess the previous comment meant to be:
So this is quite hard and we should NOT spend time on improving something which is invalid to start with.
Assignee: jorgk → nobody
Status: ASSIGNED → NEW
You need to log in before you can comment on or make changes to this bug.