674247 - The subject field (in the list of emails) contains additional (to the original subject) string (When last Subject: header line is space only line=malformed mail, subsequent header data or garbage in buffer for previous mails is added to subject string)

Status: RESOLVED DUPLICATE of bug 1454257

(MailNews Core :: MIME, defect)

Description

[Marek Makowski]

Attachment: Screen dump illustrating the reported problem

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko/20100101 Firefox/5.0
Build ID: 20110615151330

Steps to reproduce:

Upgraded from ver 3.x (which was working correctly, i.e., in particular had not the bug reported below) to 5.0


Actual results:

The subject field (in the main window) contains (for some emails only) strings composed of the original (correct) subject and an "additional" string; the latter differs. I cannot figure out how the additional strings are composed.
The subject is displayed correctly in the header of each message; it is also correct in forwarded messages.




Expected results:

All emails from the IIASA Tennis Club (see the attached screen dump) should have (in the list of emails) the subject: "Court booking" (as correctly displayed with the msg displayed in the lower part of the screen), i.e. without the "additional" strings (seen in the attached dump).
BTW: the problem occurs only for emails from this address. I can of course forward you a sample of such an email, if this could help to solve the problem.

Comment 1

[Ludovic Hirlimann [:Usul]]

does rebuilding the index in which those emails are stored helps fixing the issue ?

Comment 2

[Marek Makowski]

(In reply to Ludovic Hirlimann [:Usul] from comment #1)
> does rebuilding the index in which those emails are stored helps fixing the
> issue ?

Rebuilding the index (I did it by using the "repair folder" from FolderProperties.GeneralInformation) did not help.

Additional observations/suggestions:
- today I migrated to version 6.0, the problem persists
- the "additions" to the subject field are different in the inbox and in a folder to which I move these emails
- the problem is ONLY for emails generated by an application; the application uses a very old C++ class for sending emails; therefore I can add e.g. trailing characters to the subject, if this can help to trace/fix the problem
- the problem did not occur in previous version of the Thunderbird; also other mail clients I use (the Mac standard mail, gmail, sun-solaris desktop mailer, standard unix mail) process all emails correctly; therefore I guess that the Thunderbird requires a termination of the subject string that other mailers (and previous versions of Thunderbird) do not require (e.g., I am not sure, if \n is required). The problem may be caused by a not properly (by \0) terminated subject string (generated by Thunderbird from the email header), and therefore the displayed string has appended "random" characters (which in the inbox appear to come from the header of a previous email).

Comment 3

[David :Bienvenu]

a sample message, saved as a .eml file, would be helpful.

Thunderbird requires some sort of line ending at the end of the subject line, but that's just standard rfc822 behavior, nothing particularly picky on Thunderbird's part.

Comment 4

[Marek Makowski]

Attachment: Here is the requested sample.eml

Comment 5

[Marek Makowski]

(In reply to David :Bienvenu from comment #3)
> a sample message, saved as a .eml file, would be helpful.
> 
> Thunderbird requires some sort of line ending at the end of the subject
> line, but that's just standard rfc822 behavior, nothing particularly picky
> on Thunderbird's part.

A comment, in addition to the uploaded sample:

The subject displayed in the "bottom window"(i.e. below buttons reply, reply all, etc) is correct (Court booking). The subject displayed in the upper window (i.e., in the list of emails) reads:
Court booking C?= xxx
where xx is the email address of the sender of email received just before the sample email.

The described situation occurs in the inbox.
I move all emails (from this particular address having subjects from a set of subjects) to a folder. In this folder some emails display (again, only in the list of email) the same subject as in the inbox; however, for other emails the "modification string"" is different than that in inbox. I can, of course, make a screen dump of both situations, if this could help.

Another comment: I've just checked the _very_ old code of the class that generates and sends emails. The generated string for the header ends with
"\n \n"; The puzzle I've just noticed": the od -c sample.eml shows (after the subject text) "\r\n\r\n; I have no idea which application adds \r before \n

Final comment on the Subject string (in sample.eml): there are three pairs of \r\n; the first one (with trailing space) is generated by the court booking application.

Hope that this comment will help to solve the puzzle/problem.
Thanks in advance.

Comment 6

[WADA:World Anti-bad-Duping Agency]

(In reply to Marek Makowski from comment #4)
> Here is the requested sample.eml

Subject header is as follows([HTAB]=0x09,[CRLF]=0x0D0A,[SP]=0x20)
> Subject: [HTAB]Court booking[CRLF]
> [SP][CRLF]
> [CRLF] <== delimiter of mail header lines and mail payload lines
> Dear Brian,[CRLF]
> (snip)

(Q1) No X-Mozilla-Status: header in attached .eml file.
     IMAP folder?
(Q2) Can you see your problem on local mail folder?
1. Copy several mails including the mail to folder "Test" under "Local Folders".
2. "Repair Folder" at folder property of the "Test" folder.
Note: I can't see your problem on the mail imported to local mail folder(drag&drop of the .eml file to a mail older) with Tb 6.0. 
(Q3) Tb 3.0 had problem of "Garbled char for HTAB in Subject: at thread pane".
     This problem looks to have disappeared after fix of Bug 460443.
     (Bug 460443 : Target Milestone: --- ➔ mozilla1.9.3a5)
     You didn't see such problem with Tb 3.1.x?

Comment 7

[Marek Makowski]

(Q1) Indeed, it is IMAP folder

(Q2) After "repair folder" the added strings changed, it almost all case it is composed of characters "SCI" (repeated several times, starting with any of these letters). Sometimes (rarely) such a string is prepended by a piece of the subject string of an earlier email'; after "repair" the pieces of email addresses disappeared from the displayed subject field. BTW: the subject string displayed with each email (in the subwindow with buttons "reply...") are always correct; only the subject field in the upper window (list of emails) is modified.

(Q3) I have not seen this problem in any previous Tb version (have been using since over four years; the application sending the emails that cause problems has been used since over 10 years, and such a problem has never been noticed (although we have over 50 tennis players each year using probably all popular mail clients, and receiving such each whenever they book a court).

Comment 8

[WADA:World Anti-bad-Duping Agency]

(In reply to Marek Makowski from comment #7)
> (Q2) After "repair folder" the added strings changed, (snip)

When local mail folder, is "subject at thread pane after repair folder" correct one(expected one)?
Or garbage other than "the pieces of email addresses" still exists even after "repair folder"?

Comment 9

[Marek Makowski]

The garbage (string added to the (correct) corresponding correct subject stings) are there also after the local folder repair; however, in the inbox the added garbage differs among the emails, and very often appears to be a part of headed of the previous (in the inbox) mail, almost always part of the "From"" field. In the local (repaired) folder the garbage part substantially differ from those in inbox, i.e., garbage parts are replaced by a string composed of a short sequence of characters (same for all garbage-parts, but starting with a different character).

Comment 10

[WADA:World Anti-bad-Duping Agency]

Can you attach two sets of <foldername> and <foldername>.msf file?
(1) Create folder TestA(IMAP account), folder TestB(Local Folders).
    At Folder Properties/Synchronization of TestA, enable offline use.
    Show "Order Received" column at TestA and TestB.
    If IMAP folder, value of "Order Received" column == UID of mail 
(2) Copy several mails to TestA, Repair Folder of TestA.
    Wait for "TestA is up to date" message at Activity Manager window.
(3) Copy mails in TestA to TestB, keeping order of mails:
      Copy mail of UID=1 in TestA to TestB,
      Copy mail of UID=2 in TestA to TestB, ...
      Copy mail of UID=X in TestA to TestB, ...
   and Repair Folder of TestB.
(4) Terminate Tb. (mandtory to get clean .msf backup)
(5) Keep backup of TestA, TestA.msf, TestB, TestB.msf.
(6) Attach above 4 files to this bug.

Comment 11

[Marek Makowski]

OK, I'll try my best to prepare these files tomorrow

Comment 12

[Marek Makowski]

Attachment: The files requested by Wasa yesterday

The zip file contains nine files. In addition to the four files with the content specified by Wasa, there are four screen dumps and the Readme (with a summary of the content of the files)

Comment 13

[WADA:World Anti-bad-Duping Agency]

(In reply to Marek Makowski from comment #12)
> The files requested by Wada yesterday

I could see same thread pane display as your screen shot, using your backup of TestB, with Tb 6.0 and next Tb trunk nightly.
> User Agent :  Mozilla/5.0 (Windows NT 5.1; rv:9.0a1) Gecko/20110828 Thunderbird/9.0a1
 1. Copy your backup of TestB to Tb's mail directory of "Local Folders" account.
 2. Restart Tb.
    No TestB.msf is copied, so internal rebuild-index is done by Tb.
 3. Same thread pane display as your screen shot was obtained.
    First set of 5 mails has no problem.
    Garbage is seen for second set of 5 mails only.
Confirming.

Quick observations:
(A) When second Subject: line([SP][CRLF]) is removed, no problem occurs.
(B) When single [TAB] is changed to multiple [TAB]s or multiple [SP]s, problem occurs, and length of garbage is changed.
(C) When single [SP] of second line is changed to multiple [SP]s, problem occurs, and length of garbage is changed.
(D) When single [SP] of second line is changed to [SP]X, no problem occurs.
So, I guess problem when "space only continued header line".
But why "second set of 5 mails only"?

Note: Because RFC prohibits "space only mail header line", the mail is malformed or broken. 

Question:
Line ending of all mail data lines is CRLF, and problem was reproduced with local mail folder. So, line ending is not relevant to problem, and IMAP or not is rrelevant to problem.
However, your backup of TestA(I requested IMAP folder) contains X-Mozilla-Status:/X-Mozilla-Status-2: header, and content of TestA is identical with TestB.
How did you get backup of TestA file?

Comment 14

[Marek Makowski]

I am sorry for the long delay in my response to your comment: I've been traveling a lot.

> How did you get backup of TestA file?
>
I run tb on Windows7 PC. After exiting tb I copied all requested files (TestA and TestB directories) to my unix (solaris), then zipped them, and uploaded.

From your summary I understand that the problem is caused by empty lines in the email header. Thus I can fix the source of the problem by modifying the application that generates emails. However, you may also want to modify tb to be "permissive", i.e., handle (e.g. ignore) such empty lines, like previous tb versions were doing, as well as all other mail clients I and my colleagues know.

Please let me know, if I can help anything else here, and what you plan to do with this issue.

Comment 15

[Thierry CHARLES]

I have the same issue.

I have it only when the sender uses "incredimail" email client.
As  what you said, the headers contains blank lines. In my study case, the line following the subject is followed by a line containing only 1 tab and then I can see the headers from my recipient server :

Return-Path: <[e-mail]>
Delivered-To: [e-mail]
Received: from b0.ovh.net (HELO queue) (213.186.33.50)
	by b0.ovh.net with SMTP; 23 Apr 2012 11:54:39 +0200
Received: from blu0-omc1-s5.blu0.hotmail.com (65.55.116.16)
  by mx3.ovh.net with SMTP; 23 Apr 2012 11:54:36 +0200
Received: from BLU0-SMTP143 ([65.55.116.8]) by blu0-omc1-s5.blu0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Mon, 23 Apr 2012 02:54:35 -0700
X-Originating-IP: [92.90.21.129]
X-Originating-Email: [[e-mail]]
Message-ID: <[e-mail]>
Return-Path: [e-mail]
Received: from user1-PC ([92.90.21.129]) by BLU0-SMTP143.phx.gbl over TLS secured channel with Microsoft SMTPSVC(6.0.3790.4675);
	 Mon, 23 Apr 2012 02:54:29 -0700
MIME-Version: 1.0
Date: Mon, 23 Apr 2012 11:49:37 +0200
Content-Type: multipart/related; charset="iso-8859-1";
	type="multipart/alternative";
	boundary="------------Boundary-00=_PAFXQL80000000000000"
X-Mailer: IncrediMail (6295175)
From: nelly <[e-mail]>
References: <[e-mail]>
X-FID: 108AAAA9-F41D-401B-BCC2-7F22B2F57BC7
X-Priority: 3
To: "Thierry CHARLES" <[e-mail]>
Subject: =?iso-8859-1?B?UulmLiA6IEJpc291cyBtYW1hbiBkJ2Ftb3Vy?=
	
X-OriginalArrivalTime: 23 Apr 2012 09:54:32.0150 (UTC) FILETIME=[14EB5F60:01CD2137]
X-Ovh-Tracer-Id: 2800113069244350000
X-Ovh-Remote: 65.55.116.16 (blu0-omc1-s5.blu0.hotmail.com)
X-Ovh-Local: 213.186.33.73 (mx3.ovh.net)
X-OVH-SPAMSTATE: OK
X-OVH-SPAMSCORE: -51
X-OVH-SPAMCAUSE: gggruggvucftvghtrhhoucdtuddrfeeggedrtddtucetufdoteggodetrfdofgetucfrrhhofhhilhgvmecuqfggjfenuceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmnegotedvleehqddtudculdegledmnecuhfhrohhmpehnvghllhihuceonhgvlhhlhigprhhoshhsihhgnhholheshhhothhmrghilhdrtghomheqnecuffhomhgrihhnpehsvghlohhgvghrrdgtohhmnecujfgurhepggfftgfohfhfrffvufesrhdtrehiihdtud
X-Spam-Check: DONE|U 0.5/N

Comment 16

[WADA:World Anti-bad-Duping Agency]

See also bug 544620. That bug is "[CRLF] for header folding is inserted at wrong position of Subject: header" case.

If header parsing is one like next, which does do "header unfolding"  at first of all, which is very easy if JavaScript code, this kind of problem is not usually exposed.
(1) Unfold Subject: header, by removing [CRLF] followed by a [WSP],
    and merge into single line.
(2) Parse the message header merged into single line.
Because spaces, Htabs, Null(not 0x00, string of length ZERO) is never prohibited at any place of single line message header, no problem occurs in message header parsing, as far ar header syntax is valid.
This code merely ignores a rule of "space only line is prohibited in message header" in RFC.
And, because there is no definition about "correct action when malformed mail" in mail rellated RFC, above code is never RFC violation by Tb.

Header processing for .msf data is unforunately not such logic.
As seen in bug 317263, Tb tries to merge multiple RFC2047 encoded atoms to single one one Subject RFC2047 encoded atom for message subject data held in .msf. In such process, Tb's C++ code　searchs each header line for words, space, Htab, line ending, null line, etc. So, Prohibited "space only line" in malformed mail, "[CRLF] for header folding is inserted at wrong position" in malformed mail, can produce different result in Tb from user's thoughts.

Comment 17

[Tony Mechelynck [:tonymec]]

See also the errata of RFC 5322, http://www.rfc-editor.org/errata_search.php?rfc=5322 , in particular about obsolete syntax (which must not be produced in outgoing mail but must be accepted in incoming mail IIUC) and the possibility of two consecutive folds.

Comment 18

[WADA:World Anti-bad-Duping Agency]

(In reply to Tony Mechelynck [:tonymec] from comment #17)
> See also the errata of RFC 5322,
> http://www.rfc-editor.org/errata_search.php?rfc=5322

Status of Errata ID: 1908 is Verified.
  Errata ID: 1908, 
  Status: Verified
  Date Verified: 2010-03-24

IIUC, header folding of RFC822 is;
  After a WhiteSpace, insert [CRLF][SP] and arbitary WhiteSpaces.
> definition in RFC822
> http://tools.ietf.org/html/rfc822#section-3.1.1
> The general rule is that wherever  there may  be
> linear-white-space (NOT  simply  LWSP-chars),
> a CRLF immediately followed by AT LEAST one LWSP-char
> may instead  be inserted.
>    LWSP-char   =  SPACE / HTAB                 ; semantics = SPACE
>    linear-white-space =  1*([CRLF] LWSP-char)  ; semantics = SPACE
>                                                ; CRLF => folding
> ("inserted at" in RFC 822 == after LWSP-char)
> In contrast to it,
>    RFC2822 and later: insert [CRLF] before a WhiteSpace.
> i.e. [SP][CRLF] is folding indicator in RFC822
       and any number of WhiteSpace can be added upon folding,
>  but [CRLF][SP] is folding indicator in RFC2822 and later
>      and anything MUST NOT be added.
If so, following may be permitted,([SPx]=added space)
    Subject: ...A...[SP]...B...[SP]...C...[SP][CRLF]
 => Subject: ...A...[SP]...B...[SP][CRLF]
    [SPx]...[SPx]...[SPx][CRLF]
    [SPx][CRLF]
    [SPx]...[SPx]...[SPx]...C...[SP][CRLF]
but followings can't occur by folding of any of RFC822 and RFC2822/RFC5322.
 => Subject: ...A...[SP]...B...[SP]...C...[SP][CRLF]
    [SPx]...[SPx][CRLF]
 => Subject: ...A...[SP]...B...[SP]...C...[SP][CRLF]
    [SPx][CRLF]
Even if "space only line" can occur in mid lines of folded header, "space only line at last line of folded header" can't occur.

Header of comment #15 is; 
> Subject: =?iso-8859-1?B?UulmLiA6IEJpc291cyBtYW1hbiBkJ2Ftb3Vy?=[CRLF]
> [HTAB][CRLF]
How can it be generated by header folding defined by RFC?
Should we be torelant with any bug of RFC and any bug of other casual mail applications?

Even though obs-FWS in "RFC5322 with the Errata ID: 1908" permits "space only line as last folded header line", I don't think "mailer MUST accept it", although I believe that simple rule of "Unfold first" which is perhaps clealy writtn in RFC ,or "Ignore space only line" which is perhaps implied in obs-FWS after the Errata, is best practice of any mailer.

I believe that this kind of problem should be reported to genetator of malformed mail first and fixing of bug should be asked to generator of malformed mail first instead of developer of Tb, even though I know that this kind of malformed mails are frequently sent from goverment whose mail applicatio was coded by casual application programmer(many of them may know VBS or Excel script only :-)).

Comment 19

[WADA:World Anti-bad-Duping Agency]

Phenomenon of comment #15 is;
(a) Actual header
> Subject: =?iso-8859-1?B?UulmLiA6IEJpc291cyBtYW1hbiBkJ2Ftb3Vy?=[CRLF]
> [HTAB][CRLF]
> X-OriginalArrivalTime: 23 Apr 2012 09:54:32.0150 (UTC) FILETIME=[14EB5F60:01CD2137][CRLF]
(b) Tb interprets it as; (fails to detect [CRLF] at end of second line)
> Subject: =?iso-8859-1?B?UulmLiA6IEJpc291cyBtYW1hbiBkJ2Ftb3Vy?=[CRLF]
> [HTAB]X-OriginalArrivalTime: 23 Apr 2012 09:54:32.0150 (UTC) FILETIME=[14EB5F60:01CD2137][CRLF]

Comment 20

[WADA:World Anti-bad-Duping Agency]

(In reply to Marek Makowski from comment #12)
> Created attachment 558480 [details]
> The files requested by Wasa yesterday

I couldn't observe problem with you test mails any more(checked with Tb 17.0.3).
  Last Subject line is single space only line.
  No other headers follows(Subject is last header).
Tb 17 looks to termiate searching at "End of message headers" correctly.

Marek Makowski(bug opener), do you still see problem?

Comment 24

[Raymond]

So what ? What is the next step ? Trying to cure it ?

Comment 25

[Ulf Zibis]

Attachment: Coruppted subject in messages list

Same problem here with TB 24.2

From message source I see:
Subject: =?iso-8859-1?B?QmVzY2h3ZXJkZSB3ZWdlbiBFdXJvYmV0cuRnZW4=?=[0A]
[09][0A]
X-Provags-ID: V02:K0:SSCHEYzJRhiq1Ap0xO6pS0+5SfXMjW9epYHtrCc4XeT

Comment 26

[Raymond]

I have to juste add that:
1. This happen only in the mails list of mails (not in the news list)
2. It looks like SM remove the CRLF instead of removing the extra space.

Comment 27

[Ludovic Hirlimann [:Usul]]

Removing myslef on all the bugs I'm cced on. Please NI me if you need something on MailNews Core bugs from me.

Comment 28

[Jorg K (CEST = GMT+2)]

This got fixed on bug 1454257.