513472 - Subject, which is not MIME encoded, but contains non-ASCII characters is not displayed as user wants in Message list (IMAP server doesn't return original/wrong header data to BODY.PEEK[HEADER.FIELDS. Please use BODY.PEEK[HEADER as Bug 468351 is fixed. )

Status: NEW

(MailNews Core :: Networking: IMAP, enhancement)

Description

[Vladimir]

User-Agent:       Mozilla/5.0 (X11; U; Linux x86_64; rv:1.9.1.1) Gecko/20090730 Iceweasel/3.5.1 (Debian-3.5.1-1)
Build Identifier: Mozilla/5.0 (X11; U; Linux i686 (x86_64); ru; rv:1.9.3a1pre) Gecko/20090829 Shredder/3.1a1pre

Subject, which is not MIME encoded, but contains non-ASCII characters is not displayed correctly in Message list: non-ASCII characters are replaced by question mark "?". In message part of window the topic line is displayed correctly.
It looks like application doesn't check "Content-type" header when is building message list, but, when message is opened, all headers are recognized correctly.
It doesn't match standards, but it exists.



Reproducible: Always

Steps to Reproduce:
Subject: some 8-bit encoded string
Content-type: text/html; charset=windows-1251

Actual Results:  
Question marks in topic in message list

Expected Results:  
Display correct subject

Comment 1

[WADA:World Anti-bad-Duping Agency]

Same problem as Bug 468351?

> Build Identifier: Mozilla/5.0 (X11; U; Linux i686 (x86_64); ru; rv:1.9.3a1pre)
> Gecko/20090829 Shredder/3.1a1pre

Bug 468351 is already fixed(fixed on 2009-07-28, so landed around 7/28).
Patch for Bug 468351 was landed on 1.9.1 branch only?

Comment 2

[Vladimir]

Attachment: Screenshot of window

Comment 3

[Vladimir]

Bug 468351 looks similar, but, as i understand there was no talk about "Content-type" header. When i tried to open attached to this bug example message Thunderbird crashed.
Anyway, as i said, the subject is displayed correctly in message part of window and when message is opened. When message is opened in another tab, application window and tab names contains "?"

Comment 4

[WADA:World Anti-bad-Duping Agency]

Checkin date of patch for Bug 468351...
> Mon Aug 31 00:19:26 2009 +0200 (at Mon Aug 31 00:19:26 2009 +0200)
See Bug 468351 Comment #43. 
Vladimir, can you produce problem with newest build the patch is landed on?

Comment 5

[Vladimir]

I've checked Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.3a1pre) Gecko/20090901 Shredder/3.1a1pre. The problem still exists, but i've found one big note:
Bug appears only with IMAP accounts, not with POP.

Comment 6

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #5)
> big note:
> Bug appears only with IMAP accounts, not with POP.

Is it for same mail data in local mail folder(POP3) and in IMAP folder?
If for different mail, charset may be relevant. Can you attach mail?
(save as .eml, selecting "mail file" as file type and putting ".eml" in file name explicitly at file picker dialog. See Bug 508597, Bug 509379.)

Comment 7

[Vladimir]

Attachment: Sample message

Sample message.
When it is opened from local file system it is displayed correctly, but when it is received from IMAP server it contains "?" in subject.

Comment 8

[Vladimir]

I've made additional tests with messages and found that the same problem appears with other headers in message list (fields from,to...), which are not mime-encoded. The subject of messages with non-mime headers, received using POP3 protocol is displayed correct, but "from" field is decoded using wrong codepage. In message part of window (or when message is opened in new tab) the headers are shown correct.
Note:
For fields other than "subject" it's not a big problem, because bad-written messages come from users of bad localized or very old software (it is very rare case), or from bots/notifiers/subscribes which is written with some simplification or for English-speaking users (when codepage problem don't appears at all). After the second source of "bad" messages is localized, usually only subject and content of message is translated, sender name is left in English or contains only e-mail address.

Comment 9

[Vladimir]

(In reply to comment #6)
> (In reply to comment #5)
> > big note:
> > Bug appears only with IMAP accounts, not with POP.
> 
> Is it for same mail data in local mail folder(POP3) and in IMAP folder?

It is actual for same message (and same account), received different way.
I've checked Gmail (pop3 and imap) account and account on local Zimbra server (also pop3 and imap).

Comment 10

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #7)
> Sample message

Checked with next build. Problem was reproduced.
> Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.4pre) Gecko/20090902 Shredder/3.0b4pre

(1) Save the mail in local mail folder file(add "From - ..." line), rebuild-index.
    => Subject is displayed as expected in Subject column.
(2) Copy the mail to a Gmail IMAP folder.
    => Subject is displayed as expected in Subject column.
    If copy from local to IMAP, subject of local mail looks to be copied
    to .msf of IMAP folder, instead of one in mail header returned by server.
(3) Rebuild-index of the Gmail IMAP folder (or delete xxx.msf and restart of Tb).
    => Subject is displayed as "?...? ????" in Subject column.
    Problem is reproduced by attached mail.

Confirming.

When IMAP, mail data is obtained by two steps. (non multipart mail case)
> 6 UID fetch 2 (UID RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type)]) 
> 7 UID fetch 2 (UID RFC822.SIZE BODY.PEEK[]) 
Subject is obtained at first "6 UID fetch 2 (... FLAGS BODY.PEEK[HEADER.FIELDS ...". Care for "raw binary without encoding" of IMAP should probably be done at different place from POP3 case.

Comment 11

[WADA:World Anti-bad-Duping Agency]

Attachment: IMAP log for FETCH of header fields and mail (Gmail IMAP)

Comment 12

[WADA:World Anti-bad-Duping Agency]

Who converted to "???????? ????" was Gmail IMAP.
What can IMAP client do?

(1) FETCH BODY[HEADER.FIELD ...
> 00000045 4.71495581 [1012] 2752[1f6ac40]: 28f2800:imap.gmail.com:S-NS:CreateNewLineFromSocket: * 1 FETCH (UID 74 RFC822.SIZE 1782 FLAGS (\Seen) BODY[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type received)] {1157} 	
>(snip)
> 28f2800:imap.gmail.com:S-NS:CreateNewLineFromSocket: Subject: ???????? ???? 	
> 00000097 4.71979713 [1012] 2752[1f6ac40]: 28f2800:imap.gmail.com:S-NS:CreateNewLineFromSocket: 18 OK Success

(2) FETCH BODY.PEEK[] (viewed as windows-1251)
> 00000107 4.77043772 [1012] 2752[1f6ac40]: 28f2800:imap.gmail.com:S-NS:SendData: 19 UID fetch 74 (UID RFC822.SIZE BODY.PEEK[])
>(snip)
> 00000160 4.96851540 [1012] 2752[1f6ac40]: 28f2800:imap.gmail.com:S-NS:CreateNewLineFromSocket: Subject: ЃEроткий теЃEЋт 	
> 00000181 4.99838066 [1012] 2752[1f6ac40]: 28f2800:imap.gmail.com:S-NS:CreateNewLineFromSocket: 19 OK Success

Comment 13

[Ivan]

(In reply to comment #12)
> Who converted to "???????? ????" was Gmail IMAP.
> What can IMAP client do?

But other clients deal with it somehow... At least Windows Live Mail and The Bat display these messages correctly.

Isn't it possible to use body's Content-Type option to select encoding for message list? According to my mailbox it looks that almost all messages which have non-standard encoded headers have same encoding of body and headers.

Comment 14

[jan3a1r]

same problem, think i'll try to take a closer look on it

Comment 15

[WADA:World Anti-bad-Duping Agency]

Bug of mailer which mail sender uses. Subject header should be correctly
encoded.
So, INVALID. 

Possible Tb change for torelance with such wrong header via IMAP:

As IMAP server usually returns as-is data(Gmail IMAP does do so) to
BODY.PEEK[HEADER(no FIELD qualifier), use of BODY.PEEK[HEADER instead of
BODY.PEEK[HEADER.FIELDS is probably a workaround in this case, because Tb
already has quirks(Bug 468351 for Tb3) for such malformed/wrong header of POP3 mail(raw binary of charset of mail body is written in header).
Because download of "Received: header"(which usually shares largets part of mal
headers) is required to support "Received column", change from
BODY.PEEK[HEADER.FIELDS to BODY.PEEK[HEADER is reasonable solution for both
issues, except in slow dial up connetion environment.

Following is request by Tb when custom filter for Content-Type, Received is
added.
> UID fetch 1:4 (UID RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type Received
I believe simpleest BODY.PEEK[HEADER is better if many fileds are required in
request.

Comment 18

[Altaveron]

Will this bug be fixed on the next release?

Comment 20

[Bastard]

Note that the bug is also present in Mozilla Firefox 7.0.1, going go your Gmail inbox -> Select problematic email -> droplist menu -> show original.

please check as well with your gmail account.

Comment 21

[Christian Reischl]

Maybe my comment an attachment on Bug 254519 are more related to this bug.

https://bugzilla.mozilla.org/show_bug.cgi?id=254519#c193
https://bugzilla.mozilla.org/attachment.cgi?id=589494

Comment 22

[Mark Banner (:standard8)]

(In reply to Christian Reischl from comment #21)
> Maybe my comment an attachment on Bug 254519 are more related to this bug.

This is about the subject field and not the from field. They aren't necessary related, especially as this is on more about the server.

Comment 23

[Serg Iv]

Please, guys, fix this bug. And why it's just "enhancement"? Bunch of question marks in the mail list instead of readable text is clearly "normal - some loss of functionality under specific circumstances".
It works in GMail webmail, it works in POP3, why can't it work in IMAP?

Comment 24

[Mike Kaganski]

(In reply to Serg Iv from comment #23)
It would qualify "normal" if the program failed to show the standards-complying data.

Suppose it's set in a community to communicate using English only. But some person comes and ignores this rule, and uses _broken_ mumbo-jumbo. You _may_ try to learn his tongue, and that would _enhance_ your ability to communicate with that selfish person. But completely ignoring such misbehaviour is just another acceptable option.

That said, I want to note that I am also affected by this, and I have voted for this issue. So please don't take it as a call to ignore this problem. But this is clearly just an enhancement (that would colaterally support these non-standard headers).

Comment 25

[Serg Iv]

Well said, Mike. But please do not forget we are talking about _importance_ of the bug. Not about "how much this sh*t broke the standards". And from the point of view of the average (russian for inst.) person this bug is quite important. Not just an enhancement that will not be implemented in a million years. For now I have to move all  messages with broken subject to a local folder and delete msf-files afterwards. Not very convenient.
BWT I do not understand why it's so hard to fix this bug. Does TB know how to decode the mail subject? Yes it does, 'cos I can see subj in the mail header without any problem. Next step - just show exactly the same line of text in the mail list. That's all.

Comment 26

[ap]

For me, it seems to "fix" the problem by the following way:

1. Drop the account from TB (I use Icedeove 10.0.11 from Debian).
2. Re-create the account with download of messages turned ON ("Синхронизация и хранение" -> "Хранить сообщения для этой учётной записи на этом компьютере" in localized version) in account's settings.

All and only new messages should be now shown correctly in the message list.

Comment 27

[Mike Kaganski]

(In reply to ap from comment #26)

Thank you for the useful tip.

However, it is not a fix, it's just a workaround.
As TB now stores all mails locally, it is able to see the real bytes of the message and guess its charset.

TB could do a better job without requiring to download everything just to be able to see the subject correctly. And if developers don't want to make these modification to save efficiency, they could create a context menu option to "download correct headers" so that users could decide if they need to fix the garbled strings. Or maybe this could be implemented as a plugin?

Comment 28

[LRN]

I've always had the "Keep messages for this account on this computer" option enabled in Daily 22.0a1, and i'm still experiencing this issue with e-mail messages that have subject field without MIME encoding. That is, daily doesn't grok the subject field encoding from the Content-Type header field.
What am i doing wrong?

Comment 30

[Ivan Kudryashov]

Attachment: DeepinScreenshot_20210613123007.png

A similar error in version 78.8.1 (64-bit) under Linux Mint 20.1:

When the Subject field is not Base64 or QM encoded, all non-ASCII characters are displayed as question marks in the mail list and in the tab header (if the mail is displayed in a separate tab).

At the same time, everything is in order in the source of the letter with the Subject field, it is also displayed normally in the 3rd panel (message viewing area).