240924 - Messages with TAB in subject header (folded/wrapped header) displayed incorrectly in thread pane

Status: NEW

(MailNews Core :: Backend, defect)

Description

[Dan Mellem]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7b) Gecko/20040316
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7b) Gecko/20040316

See bug 23635

A message with the header broken with CR + LF + HTAB shows blocks in the subject
in the message view and a TAB in the subject when opened in its own window. Seen
when using IMAP and POP.


Reproducible: Always
Steps to Reproduce:

Comment 1

[Mike Cowperthwaite]

I don't know about the timeframe of bug 23635, but Moz 1.4 and 1.6 exhibit this 
symptom.  At any rate, not a recent regression.

Actually, in 1.6 and 1.8a-0416, I see the 'unknown character' block in the title 
bar, and a tab-sized spacing in the envelope panel; but in the thread pane, I 
see a glyph that looks like an uppercase N, only not in the usual font (shorter, 
set slightly low within the line), followed by a space -- maybe this is one of 
those fonts which has a graphical representation of 'newline'?

Comment 2

[Dan Mellem]

Not that it really matters, but the character is the tab.

Example 1:
Subject: This
 is a test

(shows as "This is a test")

Example 2:
Subject: This
[tab]is a test

(show as "This[symbol]is a test")

The symbol is a solid square or a degree symbol (one computer shows one, another
the other). It should replace the whitespace/tab with a single space.

Comment 3

[Stanimir Stamenkov]

Attachment: Display of new line characters in Page Info: Forms field values

Could it be related?

Open "Page Info: Forms" and observe the values displayed for the form fields.

Notice, for the visible text fields values there is displayed only the first
line while for the TEXTAREA and the hidden fields there is shown all the text
(value) where the new lines are displayed with some strange characters, at
least on my system:

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7b) Gecko/20040421

Comment 4

[Mike Cowperthwaite]

I think that odd character between lines in the forms is similar to the 
character I see in the thread pane's subject -- but that's really not related to 
this bug, which is about headers in mail and how they do not remove the 
indentation that's been stuffed into the fold.

Bug 87653 and bug 226502 have similar problems, and there was a newsgroup 
posting just yesterday about a charset-tagged BASE64-encoded subject (like
this:  =?ISO=8859-1?B?....?=  ) that had been folded in the middle of the BASE64 
encoding; when unfolded, it could no longer be decoded.

Also, in bug 234547, there is mail from Yahoo that has a MIME boundary 
declaration which include a space, but the actual uses of the boundary do not.  
It's possible that the header was folded at punctuation and then when unfolded a 
space was left in, but you can't tell from the actual posted message source.

Without having studied the relevant RFC, I would recommend that any trailing 
space before the newline in a fold be left intact, but the newline and all 
whitespace following be deleted.  I also think that in many cases, it would be 
better to simply let the header go to 90 or 100 characters rather than folding 
it to fit within 80.

I think this bug might be better filed as a MIME bug, but I'm not sure; CC'ing 
ducarroz.

Comment 5

[John Williams]

*** Bug 266709 has been marked as a duplicate of this bug. ***

Comment 6

[Mike Cowperthwaite]

TB bug 271312 is about this same problem; I bet this is really a backend issue.
That bug pointed me to RFC2822, section 2.2.3:
  <http://rfc.net/rfc2822.html#s2.2.3.>
which states:
  Unfolding is accomplished by simply removing any CRLF that is immediately
  followed by WSP.

So, my recommendation in comment 4 is completely wrong.  :)   And Mozilla, as 
well as TB, is displaying the subject correctly in the envelope.

The little NL symbol in the thread pane, tho, is obviously incorrect.

Comment 7

[Dan Mellem]

In a way, this is partially an evangelism bug since some mail servers (e.g.,
Microsoft Exchange) improperly puts tabs in. This gets even more confused when
the headers are unfolded and refolded by another gateway. Messages can end up
with embedded tabs within the text.

Subject: This is a[tab]test
 of the emergency[tab]broadcast
 system...


Additionally, Mozilla should interpret HTABs instead of printing them as a
single character (which is the point of this bug). It doesn't matter too much
how they're interpreted, such as moving to the next tab stop, inserting a single
space, or inserting 8 spaces, but to be compatible with the (non-standard) mail
gateways out there, I'd propose that all TAB characters are replaced with a space.

Comment 8

[Jungshik Shin]

*** Bug 270012 has been marked as a duplicate of this bug. ***

Comment 9

[John Coonrod]

This is also true in long file names of attachments.

Comment 10

[Mike Cowperthwaite]

*** Bug 338902 has been marked as a duplicate of this bug. ***

Comment 11

[Stephen Warren]

I am seeing this problem using Thunderbird 1.5.0.5 on Windows XP.

In my case, mailman is what is indenting the wrapped subject lines with TAB characters. mailman is very popular software!

Here is a dump of an example message header:

0002020   C   c   :  sp  nl   S   u   b   j   e   c   t   :  sp   [   B
0002040   u   i   l   d   b   o   t   ]  sp   B   u   i   l   d   B   o
0002060   t  sp   S   U   C   C   E   S   S  nl  ht   (   B   l   d   _
0002100   g   s   s   d   k   _   g   s   s   d   k   _   t   e   s   t
0002120   _   2   0   0   6   _   0   9   _   1   2   _   0   0   _   0
0002140   1   _   0   0   _   I   n   c   r   e   m   e   n   t   a   l
0002160   _   2   )  nl   X   -   B   e   e   n   T   h   e   r   e   :

Comment 12

[Phillip Susi]

Wow this bug has gone unresolved for a very long time.  It is still here in thunderbird 3.0.3, though instead of showing the bad glyph trying to print the hard tab, the thread view just removes the tab, smashing the words before and after together.  It should replace the tab with a space, which is what happens when you open the message in its own tab.

Comment 13

[Phillip Susi]

Looks like this is a duplicate of #271312.

Comment 14

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #12)
> It is still here in thunderbird 3.0.3, though instead of showing the bad glyph trying to print the hard tab, the thread view just removes the tab, smashing the words before and after together.

(In reply to comment #32)
> It is still here in thunderbird 3.0.3, though instead of showing the bad glyph
> the thread view just removes the tab, smashing the words before and after together.

As I wrote in bug 271312, it's separately processed in Bug 553280(opened on same day as your report, I dupe'ed to Bug 593337)/Bug 593337 for the new phenomenon with Tb 3.0(and Tb 3.1). See Bug 593337 Comment #15 for check result with trunk builds(Tb 3.2pre/Sm 2.2pre), please.
Next can be said.
1. This bug at thread pane never occurs with Tb 3.0/Sm 2.0(this bug is fixed),
   because Tab is removed upon Subject display at thread pane.
   And, by Tb 3.0/Sm 2.0, Bug 553280/Bug 593337 is generated.
2. Bug 553280/Bug 593337 is fixed by Tb 3.2pre/Sm 2.2 pre. 

I think this bug and bug 271312 are better dup'ed to bug 64948.

Comment 15

[mark]

(In reply to comment #11)
> I am seeing this problem using Thunderbird 1.5.0.5 on Windows XP.
> 
> In my case, mailman is what is indenting the wrapped subject lines with TAB
> characters. mailman is very popular software!


I had this problem with mail sent from mailman with the 3.1.x releases of Thunderbird. However Thunderbird 5 seems to have resolved the issue and the subject line now displays correctly (i.e. with the tab character substituted by a space character) in the thread pane for me.