550667 - 10% of letters in Russian are broken

Status: RESOLVED DUPLICATE of bug 513472

(Thunderbird :: Folder and Message Lists, defect)

Description

[Altaveron]

User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6
Build Identifier: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3

Hi! At least 10% of letters in Russian are broken. Subject of the letter below  in the messages grid is '?????????????? ???????? byfly'. But in the detail view below the grid the subject is correct! It is 'Предупреждение абоненту byfly' (koi8-r). Please fix this problem. That why I can't use Thunderbird. Thanks!

Subject: Предупреждение абоненту byfly
Content-Type: text/plain; charset=koi8-r

Предупреждение послано системой Абсолют.
Это автоматически сгенерированное письмо. Не надо направлять ответ отправителю.


Reproducible: Always

Steps to Reproduce:
1. Create testing letter in koi8-r codepage with headers described in Details.
2. Send this letter to your inbox (IMAP in my case).
3. Look to the letter in Thunderbird.
Actual Results:  
Letter is '?????????????? ???????? byfly' in the grid.

Expected Results:  
Letter is 'Предупреждение абоненту byfly' in the grid.

Please ask for more details. The bug presents in other codepages too. It may me any trash chars instead '?????'.

Comment 1

[Altaveron]

Attachment: Screenshot of the bug

Look to the subject of the message. The bug presents in the grid. In the details view below all is correct.

Comment 2

[Wayne Mery (:wsmwk)]

thread pane looks fine for me using Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.2pre) Gecko/20100224 Lanikai/3.1b1pre

I composed with encoding set to KOI8-R and subject Предупреждение абоненту byfly and received ...

Message-ID: <[e-mail]>
Date: Sat, 06 Mar 2010 08:16:17 -0500
From: "Wayne Mery (vn)" <>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.2pre) Gecko/20100224 Lanikai/3.1b1pre
MIME-Version: 1.0
To: vseerror <>
Subject: =?KOI8-R?Q?=F0=D2=C5=C4=D5=D0=D2=C5=D6=C4=C5=CE=C9=C5_=C1=C2=CF?=
 =?KOI8-R?Q?=CE=C5=CE=D4=D5_byfly?=
Content-Type: text/plain; charset=KOI8-R; format=flowed
Content-Transfer-Encoding: 7bit

Comment 3

[Nikolay Shopik]

Altaveron, your subject line isn't encoded property that's why it show up as question marks. Just select folder properties and change default charset encoding for such messages this helps.

Comment 4

[Altaveron]

> I composed with encoding set to KOI8-R and subject Предупреждение абоненту
> byfly and received ...

Wayne Mery, the subject of your message is wrong. Your subject is 'Subject: =?KOI8-R?Q?=F0=D2=C5=C4=D5=D0=D2=C5=D6=C4=C5=CE=C9=C5_=C1=C2=CF?=
>  =?KOI8-R?Q?=CE=C5=CE=D4=D5_byfly?=' instead: 'Subject: Предупреждение абоненту byfly'.

 
> Message-ID: <[e-mail]>
> Date: Sat, 06 Mar 2010 08:16:17 -0500
> From: "Wayne Mery (vn)" <>
> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.2pre)
> Gecko/20100224 Lanikai/3.1b1pre
> MIME-Version: 1.0
> To: vseerror <>
> Subject: =?KOI8-R?Q?=F0=D2=C5=C4=D5=D0=D2=C5=D6=C4=C5=CE=C9=C5_=C1=C2=CF?=
>  =?KOI8-R?Q?=CE=C5=CE=D4=D5_byfly?=
> Content-Type: text/plain; charset=KOI8-R; format=flowed
> Content-Transfer-Encoding: 7bit

Comment 5

[Altaveron]

> Altaveron, your subject line isn't encoded property that's why it show up as
> question marks.

It's encoded as 10% or more of other letters from different sources. That's a fact that many letters are the same. And all other mail-clients works fine.

> Just select folder properties and change default charset
> encoding for such messages this helps.

Changing of folder charset encoding do nothing.

Please look to the screenshot. The subject of the letter displays as ??? above and correctly below. The same letter on the same window of the same instance of Thunderbird at the same time. This is a prove of the bug.

Comment 6

[WADA:World Anti-bad-Duping Agency]

> Subject of the letter below in the messages grid is '?????????????? ???????? byfly'.
> Subject: Предупреждение абоненту byfly

Who converted the malformed "П...е а...у"(it should be encode properly) to "?...? ?..."? Who is probably your IMAP server.
Get IMAP IMAP log and check IMAP response.
> https://wiki.mozilla.org/MailNews:Logging

Does your server retrun '?...? ?...? byfly'? to next request?  
> UID fetch xxx (UID RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID ...) ...
If so, it's already known phenomenon.
If you use Gmail IMAP for the mail, it's dup. it's already known that Gmail IMAP retruns "???" to above request if such wrong Subject: header is sent by mailer. If ordinal IMAP, it's also dup of the already reported bug.

Trick at header pane:
  The wrong header is displayed as text by HTML in koi8-r like next internally.
    <pre>Subject: Предупреждение абоненту byfly</pre>
  As auto-detect(or quirks for no explicit charset) works in HTML parsing
  and rendering, the raw binary of koi8-r in Subject header is displayed
  as koi8-r.

Bug of mailer which mail sender uses. Subject header should be correctly encoded.
So, INVALID. 

Possible Tb change for torelance with such wrong header via IMAP:

As IMAP server usually returns as-is data(Gmail IMAP does do so) to BODY.PEEK[HEADER(no FIELD qualifier), use of BODY.PEEK[HEADER instead of BODY.PEEK[HEADER.FIELDS is probably a workaround of this case, because Tb already has quirks for such malformed header of POP3 mail(raw binary of charset of mail body is written in header).
Because download of "Received: header"(which usually shares largets part of mal headers) is required to support "Received column", change from BODY.PEEK[HEADER.FIELDS to BODY.PEEK[HEADER is reasonable solution for both issues, except in slow dial up connetion environment.

Following is request by Tb when custom filter for Content-Type, Received is added.
> UID fetch 1:4 (UID RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type Received
I believe simpleest BODY.PEEK[HEADER is better if many fileds are required in request.

Comment 9

[Altaveron]

PLEASE HELP TO DELETE MESSAGE:

 Altaveron      2010-03-08 09:05:05 PST 

AND TO DELETE E-MAILS IN MESSAGE:

 Altaveron      2010-03-08 09:06:15 PST 

THANKS A LOT!!!

Comment 10

[WADA:World Anti-bad-Duping Agency]

> Subject:
> =?windows-
> 1251?Q?=D3=E2=E5=E4=EE=EC=EB=E5=ED=E8=E5=20=EE=20н=EE=E2=EEй=20=F2=E5=EC=E5=20=E2=20ф=EEруме=20(=20Ф=EEрум=20Хакер=EE=E2=20Сн=EE=E2идений=20)?=
> From: "Форум Хакеров Сновидений" <[e-mail]>

Name part in From: is same as your comment #0.

For Subject: header.
Is the Subject: header in mail source displyed in three lines as you pasted?
Or in single line or two lines?
Who converted the Subject: header to string in thread pane? 
Have you read comment #6 and got and checked IMAP log?

As subject at thread pane is (decoded string before н=EE=E2=EEй=20...) + (=EE=E2=EE... after in correct н), and as message header pane display is like next, I guess Subject: is sen as single line or as two lines(Subject:[CRLF] + ?=windows-...[CRLF]).
> Subject: =?windows-
> 1251?Q?=D3=E2...
> (a. as wrong qoted-printable data, Tb displays as-is)
> (b. as long word, split in display occurs at "-")

Save mail data in .eml, remove mail data part by text eitor(header lines only is sufficient), get IMAP log for download of mail header and mail data of the mail, and attach(never paste long data any more) the edited .eml and log file(both with content-type=text/plain), please.

By the way, who sends mail of such wrong/corrupted header?
Does "IPB PHP Mailer" still has such big bug in these days? Does mail sender correctly setup "IPB PHP Mailer"? Or the mail is from spammer?

Comment 11

[Altaveron]

WADA, sorry, it's required a lot of lime to write answers for you. Is it possible to edit or delete comment, or entire all the bug?

Comment 12

[Ludovic Hirlimann [:Usul]]

(In reply to comment #11)
> WADA, sorry, it's required a lot of lime to write answers for you. Is it
> possible to edit or delete comment, or entire all the bug?

just close this one as INVALID. That won't delete it. Why do you want to do that ?

Comment 13

[Altaveron]

The bug is still valid. But I need to hide messages with private info. Please help to do this. Thanks!

Comment 14

[[:Aureliano Buendía]]

Dupe of bug #524287. Both bug are dupes of bug #425945? Reporter of bug #524287 in comment #9 disagree.
Altaveron, what you think about this?

Comment 15

[Altaveron]

(In reply to comment #14)
> Dupe of bug #524287. Both bug are dupes of bug #425945? Reporter of bug #524287
> in comment #9 disagree.
> Altaveron, what you think about this?

This bug is not dupe of the bugs above. In this bug you may see on the screenshot the correct (!) subject in the message window. This bug exists only in the message grid.

Aureliano Buendía, please, help to delete comments with private info! Thanks

Comment 16

[Wayne Mery (:wsmwk)]

Altaveron, I've marked comment 7 and 8 private, so that only a few privileged, security conscious bugzilla users can see it. If that is not sufficient please let me know.

Comment 17

[Altaveron]

Wayne Mery, thanks a lot! May I add the attachment with message's headers instead these two comments? Comment #9 now is not actual too. THANKS!

Comment 19

[WADA:World Anti-bad-Duping Agency]

Comment #0 case is dup of bug 513472.

Subject: of Comment #8 case is apparently INVALID. Who can decode such wrong(corrupted for any mail receipient)?
Altaveron, please open separete bug for Subject: of Comment #8 case without pasting, if you still think valid bug(flaw in Tb's code, or request for torelance of Tb with broken MUA or broken MTA), with next which I already requested you,
(1) Get IMAP log and check fetch response from IMAP server.
    Who converted the wrong(corrupted) encoded-word to string which was
    displayed at thread pane of Tb?
and with next additionally.
(2) Read RFC2047 for encoding of non-ascii text.
> http://tools.ietf.org/html/rfc2047

Any way, closing as dup of bug 513472.
Altaveron, you will be able to sleep well by duping, because very limited peoples only will access DUP'ed bugs(usually, QA peoples, Triage team members, ..., only).

Comment 20

[[:Aureliano Buendía]]

Cleanup *dupeme* whiteboard flag from bugs that are marked as Resolved Duplicate!