593337 - Wrapped spaces in subject line omitted from message list (Tab in Subject: for header folding is removed when shown at Thread pane)

Status: RESOLVED WORKSFORME

(MailNews Core :: Backend, defect)

Description

[William Blunn]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8 (.NET CLR 3.5.30729)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.1.11) Gecko/20100711 Thunderbird/3.0.6

Please do not mark this bug as a duplicate of 64948.

Let us say we have a message which contains the following (valid) header record:

Subject: [Dovecot] Documentation for "altpath" / "altmove" / ?"Alternate
	storage"

i.e. the Subject header record is over two lines; we have a wrap after the word "Alternate", and the second line of the header record begins with a TAB character.

In the message preview pane, the subject is correctly displayed as:

[Dovecot] Documentation for "altpath" / "altmove" / ?"Alternate storage"

whereas in the message list pane, the subject is incorrectly displayed as:

[Dovecot] Documentation for "altpath" / "altmove" / ?"Alternatestorage"

i.e. the space betwee the words "Alternate" and "storage" has gone missing.

Please do not mark this bug as a duplicate of 64948.

Reproducible: Always

Steps to Reproduce:
In order to reproduce the bug, you need to have a message on hand which has the provoking data in it, viz.

Subject: [Dovecot] Documentation for "altpath" / "altmove" / ?"Alternate
	storage"

The second line begins with a TAB.

Note that sending a message to yourself with a subject of

[Dovecot] Documentation for "altpath" / "altmove" / ?"Alternate storage"

does not do it because Thunderbird does not create a wrapped header record in this case.

The problem showed up when a message was sent through a mailing list, and the mailing list software generated the Subject header record.

Actual Results:  
Message preview pane shows subject as:

[Dovecot] Documentation for "altpath" / "altmove" / ?"Alternate storage"

Message list pane shows subject as:

[Dovecot] Documentation for "altpath" / "altmove" / ?"Alternatestorage"

(i.e. with the space between the words "Alternate" and "storage" having been omitted.)

Expected Results:  
Message list pane should show subject the same as the message preview pane, i.e.

[Dovecot] Documentation for "altpath" / "altmove" / ?"Alternate storage"

Comment 1

[William Blunn]

Attachment: Test message which should provoke the bug

This attachment is a small test case which should provoke the bug.

Drag the file into a folder in Thunderbird, then look at what appears for the Subject in the message list pane, and in the message preview pane.

Comment 2

[William Blunn]

TABs in Subject lines have little practical use.

Rightly or wrongly, common practice is to wrap long Subject line by replacing a space with CR, LF, TAB.

Removing the CRLF in this scenario leaves us with a TAB which seems to confuse the message list pane.

I would suggest the following treatment of TABs in the Subject field:

For the message preview pane, substitute the TAB with a space (0x20).

For the message list pane, substitute the TAB with a space (0x20).

This will then make Subject values appear in a way which is

(a) consistent, and
(b) sane

Comment 3

[William Blunn]

Think about it this way: If we display TABs as spaces (0x20), do you think that anyone is going to raise a bug about that?

Complainer: "I had a Subject line with a TAB character in it and Thunderbird displayed it as a space!"

Maintainer: "Yes? And what did you expect it to do?"

Complainer: "Er, don't know."

The point being that if you set up a behaviour that:

(a) no-one is likely to report as a bug, and

(b) no-one can practically provide a materially superior expected behaviour

then the problem kind of, er, goes away :-)

I don't think that my proposed solution is "bad" or "selling soul to the devil". It's just a pragmatism.

If Thunderbird behaves in an obviously inconsistent way, then you are forever going to get people popping up complaining about it.

If no-one can agree on the "right", "correct", "high fibre" answer, then you need a pragmatist to come along and say "OK we just need to make this go away. Do this <thing> and the problem goes away. Happy? Good."

Comment 4

[William Blunn]

I am a software engineer and have programmed significant projects in C C++ and other languages for many years. I dare say I could fix this in about 5 minutes flat if I had the Thunderbird build tree lying around.

As it happens I have better things to do, so I don't have the build tree lying around.

So, could someone with a build tree please fix it, commit the changes to the repository, and we can all get on with something more useful instead?

Comment 5

[Ludovic Hirlimann [:Usul]]

(In reply to comment #0)

> Please do not mark this bug as a duplicate of 64948.

Why ?

(In reply to comment #4)

> So, could someone with a build tree please fix it, commit the changes to the
> repository, and we can all get on with something more useful instead?

Sure look at https://developer.mozilla.org/en/Simple_Thunderbird_build, and in no time you'll have a tree and a build.

Comment 6

[William Blunn]

> > Please do not mark this bug as a duplicate of 64948.
> 
> Why ?

Because bug 64948 is very old, stale, and apparently deadlocked / stalemated.

I have found that if you report a bug, and someone marks as a duplicate of a bug with a number which is less than 90% of the new bug's number, you can forget about ever seeing a fix for it --- you are seeing a deadlocked / stalemated issue which no-one is interested in fixing, or people are more interested in arguing the ins and outs until hell freezes over rather than just fixing it and getting on.

I am pointing out a specific issue and suggesting a specific remedy.

I don't want the issue to get mired in people arguing about RFCs when it is an obvious problem with an obvious five-minute fix.

A few years ago when trying to persuade someone to do something, I realised I had spent 20 minutes arguing to get them to do a 30 second fix. It's like "why are we even arguing about this? It would take far less time to just *do* it than to stand around arguing about it."

The phrase I coined was "Easier done than said", a play on the old "Easier said than done".

I suppose I am just frustrated that it seems that the only issues I seem to have with Thunderbird are those ones which are deadlocked / stalemated.

As you get older, you grow tired of pussy-footing around and soft-soaping everyone. You just want to get things /done/. Which is probably why we have the phrase "grumpy old men". Ah well.

Comment 7

[William Blunn]

Stagnated -- that's the word I was looking for.

Comment 8

[WADA:World Anti-bad-Duping Agency]

Have you searched for "subject tab" instead of "subject tab char"? 

For thread pane display:
Dup of bug 240924(report of "tab is shown in U+FFFD"), with morphed external phenomenon by some intentional changes in other bugs or accidental changes, such as removal of "Tab char after CRLF for folding" upon thread pane display.
AFAIR, I saw a bug report for removal of "Space(Tab?) after CRLF for folding" at thread pane, but I'm not sure. 

For message pane display:
Result after some changes which fixed problem reported by bug 271312, with avoiding discussion on "what is consistent handling of Tab in mail header" in bug 64948.

Basically, all of bug 240924, bug 271312, bug I saw, and your bug, are dup of bug 64948, with slightly different external view/phenomenon, due to some intentional or non-intentional changes, due to some changes relevant or irrelevant to fixing of problem of bug 64948. "Old bug" is not reason of "not dup" in this case, because no one intentionally fixed problem of bug 64948, and code around bug 64948 had not been changed so much.   

Content of Subject: header lines is saved in .msf file, with some modifications.
  If Subject: consists of multiple RFC2047 encoded words, they are merged
  into single encoded word, with converting to charset of first encoded word.
For thread pane display, this data in .msf is used.
(In contrast to it, message pane display uses Subject: header lines in mail data stream itself. It's reason why "display at thread pane" and "display at message pane" should be analyzed/resolved separately, with caring for consistency.)

Is Tab character already removed in .msf file?
Or held in .msf and removed upon display at thread pane?

(Off-Topic for real "bug" of this bug report at B.M.O)
> Stagnated

It's simply because developers who can propose patch are very busy with much important/severe bugs, and who complaints on thread pane display of Tab did/does/will never propose patch.
Please note that "string in Subject header" is readable text for human being. And, "treatment of Tab in readable text for human being" depends on application and situation. Even though both Tab and Space can be used for header folding, Tab shouldn't be used in "readable text for human being" in message header, unless "treatment of Tab" is defined clearly.
  a) Text editor shows with indented when Tab exists.
     Note: Mozilla family still has next issue;
           Tab=4chars in mail editing, Tab=8char in mail display.
           It is probably applicable to Composer of Browser.
  b) HTML spec is similar to "replace by a white space" upon rendering.
     Tab in <pre> may be similar to a).
And, "Tab in Subject: header" is usually result of "inserting a Tab upon header folding"(can be called "bug of a mailer"), instead of "intentional Tab use for tabbing or spacing or some other purposes in subject text" nor "misuse of Tab for a space".   
Although I also think b) is better than a) for display at thread pane, any one can claim that a) should be applied at any place consistently. Phenomenon at message pane reported to bug 271312 was a result of application of rule of a).

Comment 9

[WADA:World Anti-bad-Duping Agency]

See Bug 317263 for Subject: data in .msf(merge of multiple rfc2047 encoded words is seen.)

Comment 10

[WADA:World Anti-bad-Duping Agency]

FYI.
See bug 587510 for difference between "thread pane display" and "message pane display" for wrongly encoded Subject: header.

Comment 11

[WADA:World Anti-bad-Duping Agency]

William Blunn(bug opener), no need to check .msf content by you.
Following is .msf content by Tb 3(3.0.5, 3.1.2). Tab(0x09) is kept in .msf.
(Checked with Subject: Alternate[CRLF][Tab]storage[CRLF]) 
> (86=Alternate$0D$0A$09storage)(87=4A4D5034.5030808@x.x.x.001)(89
Thread pane display by Tb 3.0.5/3.1.2 was 'Alternatestorage'(no quote), instead of 'Alternate storage'(our expectation) nor 'Alternate�storage'(by Tb 2 or former). 
It(0x09 is not tried to render by Tb 3) seems why Bug 524806(crash with CZTSans font and 0x09 in Subject:) occurs with Tb 2 but doesn't occur with Tb 3(3.0.x).

Comment 12

[WADA:World Anti-bad-Duping Agency]

Check result with SeaMonkey 2.0.2.
Thread pane  : 'Alternatestorage' (no quote, same as Tb 3.0/3.1, this bug)
Message pane : 'Alternate       storage' (no quote, same as bug 271312)
Changing to Product=MailNews Core.

Comment 13

[Nikolay Shopik]

This is WFM, correctly displayed in messages list with space.
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b6pre) Gecko/20100903 Thunderbird/3.2a1pre

Comment 14

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #13)
> This is WFM, (snip)

Me too, with next trunk builds of Tb & Sm.
> Mozilla/5.0 (Windows NT 5.1; rv:2.0b4pre) Gecko/20100816 Shredder/3.2a1pre
> Build identifier: Mozilla/5.0 (Windows NT 5.1; rv:2.0b4pre) Gecko/20100805 SeaMonkey/2.1a3pre

For Subject: Alternate[CRLF][Tab]storage[CRLF]
  Thread pane                                 : Alternate storage
  Message pane                                : Alternate storage
  Pre-set file name at save as dialog of mail : Alternate storage
=> WORKSFORME

By the way, interesting phenomeon was seen on (wrong) Tab for header folding in value of name parameter, with above two trunk builds. 
For Content-Type: text/plain; name="Alternate[CRLF][Tab]storage"[CRLF]
  Attachment name at attachment pane                : Alternatestorage
  File name at message pane(inline display)         : Alternate storage
  Pre-set file name at save as dialog of attachment : Alternate-storage
                                          (it was not Alternate%09storage)
William Blunn(bug opener), please open separate bug for such inconsistency, if you believe it's big problem as an mailer and Tb should be consistent in "(wrong) Tab in name parameter" handling.

Comment 15

[WADA:World Anti-bad-Duping Agency]

Additional check result.
> Subject: Alternate[Tab]stor1[CRLF][Tab]stor2[Tab]stor3[CRLF]
Thread pane          : Alternatestor1 stor2stor3
Message pane (Tb3.2) : Alternate stor1 stor2 stor3
Message pane (Sm2.2) : Alternate   stor1 stor2   stor3

Change history seems;
(A) Thread pane.
(A-0) Tab is shown in U+FFFD.
(A-1) Any Tab is removed upon Thread pane display by Tb3.0/3.1, Sm2.0.
(A-2) By report like this bug, [CRLF][Tab](Tab for folding) is shown as Space
      by Tb 3.2/Sm 2.2.
(B) Message pane
(B-0) Tabbing is done for Tab in Subject: upon header display.
(B-1) Tb only change : Tab is shown as a space.
(B-2) By change of (A-2), [CRLF][Tab] is replaced by a space upon display,
      then only [CRLF][Tab] part is shown as a space by Sm 2.2.

Because Tab in Subject: header is generated by "putting Tab upon Subject: header folding"(bug of mail generator), I believe "replace of [CRLF][Tab] by a space" only is sufficient. I don't think inconsistency between native [Tab] and [CRLF][Tab], inconsistency between Tb and Sm, are problem.

Comment 17

[WADA:World Anti-bad-Duping Agency]

"Bug I saw" was Bug 553280, which reported phenomenon on Tooltip also.
> Subject: Alternate[Tab]stor1[CRLF][Tab]stor2[Tab]stor3[CRLF]
Tooltip of Subject of thread pane : Alternate   stor1 stor2   stor3
[CRLF][Tab] in Subject is shown as a space by Tooltip too.