1130843 - Send format recognition does not recognize HTML e-mail (Link of Link Text==Link URL is considered Convertible by SeaMonkey, and Prefers=Unknown is currently similar to Text Domain, then HTML mail is sent as text/plain)

Status: NEW

(SeaMonkey :: MailNews: Composition, defect)

Description

[Rainer Bielefeld]

I found this problem during my research for "Bug 200128 - HTML Support in Emails" (comment 5)

Steps how to reproduce:
1. Send account settings to "Create HTML mails"
2. open new e-mail for send
3. from <http://en.wikipedia.org/wiki/Main_Page> copy / paste text line 
   "the free encyclopedia that anyone can edit." (below heading) to mail body
4. Save as draft and check view: looks ok, words with hyperlink in blue and underlined
5. Check menu 'Options  ► Format': Should show "Automatic"
6. Send mail and check result:
   » HTML will be broken, Hyperlinks shown as additional URLs
     Mail header shows "Content-Type: text/plain;"

10. Create new mail, try again, in step 5 now change to "only Rich text (HTML)"
   » Will work fine, received mail shows HTML

So it seems that auto recognition of HTML fails.
I don't use HTML e-mails, so I can't tell when this might have started, I see some very old Bugs what might have been related to this one, for example see <https://bugzilla.mozilla.org/show_bug.cgi?id=114477#c8>: "If the checkbox in the Options|Format menu in the HTML composer doesn't reflect your Prefs|Mailnews|Send Format pref, that's another bug as well."

Comment 1

[Thomas D. (:thomas8)]

(In reply to Rainer Bielefeld from comment #0)
> So it seems that auto recognition of HTML fails.

Indeed. For both TB and SM, Delivery Format > Auto-detect is a total misfit which, as WADA has put it, unexpectedly decides to send plaintext whenever it wants to send plaintext, radically ignoring the principle of ux-wysiwyg.

> I don't use HTML e-mails, so I can't tell when this might have started, I
> see some very old Bugs what might have been related to this one, for example
> see <https://bugzilla.mozilla.org/show_bug.cgi?id=114477#c8>: "If the
> checkbox in the Options|Format menu in the HTML composer doesn't reflect
> your Prefs|Mailnews|Send Format pref, that's another bug as well."

Excellent research. You're right there.
Bug 114477 Comment 8 by Ben Bucksch:

> The original bug reported here seems to be: You enable "Compose message in HTML
> format" in the Account Settings. But the message is sent in plaintext. [...]

> This is not a bug at all, it is a feature. By default, you *compose* the
> message
> in HTML, taking advantage of the rewrapping features in the HTML composer and
> giving you the option to style the message, but if the message can be
> represented fairly in plaintext, we use that for *sending*. (if there is
> fstyling being used, we ask the user what to do, unless we know that the
> recipient can recieve HTML.) Otherwise, we would send HTML all the time and
> that
> would pose a major interoperability problem. We try to figure it out for the
> user and do the right thing (because the user can't in most cases do it
> himself,
> because of lack of backgkround knowledge).
> 
> WONTFIX - works as specified.

So now you know the "logic" behind this amazing "feature" of downgrading HTML mails and randomly destroying user formatting in the process. And the culprit, too...
It's yourself the user, stupid! Who in their right mind would expect an explicitly HTML formatted message with links, styles and all to be sent as HTML when SM/TB can just "figure it out for the user and (because the user can't in most cases do it himself, because of lack of background knowledge)" and send plaintext instead!?

The year is 2015 A.D. The world is entirely occupied by HTML freaks. Well, not entirely... One small village of indomitable plain text defenders still holds out against the invadors!!!

For an overview of related bugs, see
Bug 889315 - [Meta] Tracker bug for delivery format UX issues (HTML vs. Plaintext; incl. interaction of {Delivery Format | Auto-Detect} with other settings and related UX-failures)

It's a deep and dirty pool. I've all but lost hope of drying that out. Perhaps in another life.
Thanks for calling.

Comment 2

[Arivald]

https://addons.mozilla.org/thunderbird/addon/always-html/ will help You

Comment 3

[Thomas D. (:thomas8)]

For examples how bad it can get, look at the screenshots I attached to Bug 414299.

Comment 4

[Thomas D. (:thomas8)]

(In reply to Arivald from comment #2)
> https://addons.mozilla.org/thunderbird/addon/always-html/ will help You

Hehe, thanks & YES... :)))

Arivald tends to turn up for some advertising whenever I speak about Auto-Detect misfeature...
Will I get provisions if I add the advertising myself?? :>>

Comment 5

[WADA:World Anti-bad-Duping Agency]

Unable to reproduce, with Link created by "Insert Link" of Composer of Thunderbird in HTML mode. 

(In reply to Rainer Bielefeld from comment #0)
> I found this problem during my research for "Bug 200128 - HTML Support in
> Emails" (comment 5)
> 
> Steps how to reproduce:
> 1. Send account settings to "Create HTML mails"
> 2. open new e-mail for send
> 3. from <http://en.wikipedia.org/wiki/Main_Page> copy / paste text line 
>    "the free encyclopedia that anyone can edit." (below heading) to mail body

This is merely a "text string pasted to Body Text(or PRE), unless you request Link in HTML == <a href="...">...</a>.

> 4. Save as draft and check view: looks ok, words with hyperlink in blue and underlined

When mail is saved as draft, draft is saved as text/html if compose in HTML mode, and is saved as text/plain if composed in Text mode or compsed in HTL mode with requesting Delivery Formaat=Plaa Text Only".
Thunderbird has feature of "Linkfy URL like string when showing Text mail". This is applied to text/html maail when View/Message Body As/Plain Text".

Because you composed in HTL mode and saved as draft, draft is saved as text/html.
With what "View/Messaage Body As" did you see your draft mail? 

> 5. Check menu 'Options  ► Format': Should show "Automatic"
> 6. Send mail and check result:
>    » HTML will be broken, Hyperlinks shown as additional URLs
>      Mail header shows "Content-Type: text/plain;"

This is well known phenomenon.
  Even when mail is composed in HTML mode,
  if "text only in Body Text without any formatting" and/or "text only in PRE without any formatting",
  and if color setting etc. is not touched with default color setting,
  and if all recipients are not explicitely defined as "HTML Domain recipients",
  Thunderbird currently silently sends it as text/html by UNKNOWN reason for Tb user,
  because of "display of the HTML is identical" as "display of text version of the HTML",
  in order to avoid "HTML mail vs. Text mail War".
This is mystery for Tb users, so peoples opened bug like this bug at bugzilla for such mystery.
Unfortunately, some of such reports was closed as dup of bug 414299 which is about "<tt> and friends".
See duped bugs of bug 414299, please. I believe that bug 414299 is absolutely irrelevnt to your case.
And, some peoples repeatedly said "Auto-Detect misfeature", "Format Loss!!!", "Data Loss!!!", "Never not lossless, too lossy!!" due to the mystery for user, as seen in bugs listed in dependency tree for meta bug 889315.

AS written in comment #2, simple/easy workarounds are already known.
To bug opener, if your case is "Text only HTML in Body Text and/or PRE" case, and if you are not merely complainting about Tb's Auto-Detect feature, please watch Bug 1210244.
We are tryng to fine one-liner patch for such mystery for Tb user.

Comment 6

[WADA:World Anti-bad-Duping Agency]

Woops. I missed "Hyperlinks shown as additional URLs" in next comment.
> 6. Send mail and check result:
>    » HTML will be broken, Hyperlinks shown as additional URLs

What phenomeno do you call by "Hyperlinks shown as additional URLs"?

This sounds similar phenomenon to one like next.
  When <a href="mailto:[e-mail]">XXX YYY</a> in HTML, following is generated by text conversion from HTML.
       XXX YYY <[e-mail]>
  note: if <a href="mailto:[e-mail]">[e-mail]</a>,
        "[e-mail] <[e-mail]> is generated by text conversion, so annoying in some case.
If you do "change to Delivery Format=HTML only" and "Send Later" with same procedure as your STR,
what HTML source is generated by Thunderbird?

Comment 7

[Thomas D. (:thomas8)]

WADA, this is reported against SeaMonkey. Did you try SeaMonkey before saying "unable to reproduce"?

I was also wrong because I didn't test this sufficiently, and I did not test in SeaMonkey either.
But as SeaMonkey is more conservative than TB, I think it's possible there might be slight variants in auto-degrade-and-detect.

On TB, I cannot reproduce this bug.
HTML-formatted links (<a href="...">foo</a>) are only removed for cases of <a href="www.foo">www.foo</a> where the link text is same as url.
But in comment 0, link text differs from URL, so HTML format is always preserved with default settings.

(In reply to WADA from comment #5)
> Unable to reproduce, with Link created by "Insert Link" of Composer of
> Thunderbird in HTML mode.

Why not use the actual links referred to in step 3?
 
> (In reply to Rainer Bielefeld from comment #0)
> > I found this problem during my research for "Bug 200128 - HTML Support in
> > Emails" (comment 5)
> > 
> > Steps how to reproduce:
> > 1. Send account settings to "Create HTML mails"
> > 2. open new e-mail for send
> > 3. from <http://en.wikipedia.org/wiki/Main_Page> copy / paste text line 
> >    "the free encyclopedia that anyone can edit." (below heading) to mail body
> 
> This is merely a "text string pasted to Body Text(or PRE), unless you
> request Link in HTML == <a href="...">...</a>.

No. The links provided in step 3 are fully formatted HTML links, and will remain so in TB.
But this is reported against SeaMonkey, so we'll have to check there.

> > 4. Save as draft and check view: looks ok, words with hyperlink in blue and underlined
> 
> When mail is saved as draft, draft is saved as text/html if compose in HTML
> mode, and is saved as text/plain if composed in Text mode or compsed in HTL
> mode with requesting Delivery Formaat=Plaa Text Only".
> Thunderbird has feature of "Linkfy URL like string when showing Text mail".
> This is applied to text/html maail when View/Message Body As/Plain Text".
> 
> Because you composed in HTL mode and saved as draft, draft is saved as
> text/html.
> With what "View/Messaage Body As" did you see your draft mail? 
> 
> > 5. Check menu 'Options  ► Format': Should show "Automatic"
> > 6. Send mail and check result:
> >    » HTML will be broken, Hyperlinks shown as additional URLs
> >      Mail header shows "Content-Type: text/plain;"
> 
> This is well known phenomenon.

For TB, yes. But in TB, not relevant for the particular sample of this bug.

>   Even when mail is composed in HTML mode,
>   if "text only in Body Text without any formatting" and/or "text only in
> PRE without any formatting",

That's not exactly true WADA, and you know it. Lots of styles get currently removed e.g. when inside the tags which are wrongly deemed inconsequential for formatting, and styles are formatting. So the condition "text only in Body Text without any formatting" does NOT adequately describe status quo.

>   and if color setting etc. is not touched with default color setting,
>   and if all recipients are not explicitely defined as "HTML Domain
> recipients",
>   Thunderbird currently silently sends it as text/html by UNKNOWN reason for
> Tb user,

Oh, I hope I haven't given "brain fried" symptom to WADA.
Correction: Thunderbird currently silently sends it as text/*plain*...

>   because of "display of the HTML is identical" as "display of text version
> of the HTML",

Not necessarily! E.g. Variable width vs. fixed width.

>   in order to avoid "HTML mail vs. Text mail War".
> This is mystery for Tb users, so peoples opened bug like this bug at
> bugzilla for such mystery.
> Unfortunately, some of such reports was closed as dup of bug 414299 which is
> about "<tt> and friends".
> See duped bugs of bug 414299, please. I believe that bug 414299 is
> absolutely irrelevnt to your case.

Huh? If the full HTML links really get removed by SM, then that looks very much like bug 414299 in its broader sense (not just <tt>, but <tt> and friends; links were even explicitly in the summary of that bugs before Ben Bucksch unilaterally removed that part).

> And, some peoples repeatedly said "Auto-Detect misfeature", "Format
> Loss!!!", "Data Loss!!!", "Never not lossless, too lossy!!" due to the
> mystery for user, as seen in bugs listed in dependency tree for meta bug
> 889315.

Some people includes me, and we had to repeat it because apparently nobody was listening nor understanding. I don't understand WADA why you're taking offence in that, it's a true user experience for users who correctly expect HTML formatting from HTML composition to be sent by default. Maybe it was not strategically ideal, but it's still true, and we have the right to voice our anger about all those failures in the current design. WADA you're repeatedly complaining about our repeated complaints about dataloss of formatting; so you're starting to be just as repetitive...

> AS written in comment #2, simple/easy workarounds are already known.

If having to find and install a particular addon is "simple" or not might be subject to discussion.
Many users will NEVER go into addons, they may not even know about them.
Same for the other workaround of including color, bold etc. It takes considerable effort and understanding on the side of the user to realize and implement that.

Please don't make it sound as if there's no problem. We have all those bug reports because it IS a real-life problem for many of our users.

> To bug opener, if your case is "Text only HTML in Body Text and/or PRE"
> case, and if you are not merely complainting about Tb's Auto-Detect feature,
> please watch Bug 1210244.

WADA, I understand your frustration about many bugs in this field which would be easy to avoid/close with some simple fixes, but please keep our users out of this. They have every right to complain about this as long as it is not fixed.

> We are tryng to fine one-liner patch for such mystery for Tb user.

"Mystery" is just another word for "surprising/unexpected change of delivery format" which is just another word for "dataloss of formatting". Of course if some users consider auto-degrade of HTML to plaintext a feature, for them it's never dataloss. For the rest of us (and most likely the majority), it IS dataloss. But yeah sure, we can call it "mystery" to avoid offending developers and triagers who for some strange reason don't like the term "formatting dataloss".

Otherwise yes, amazingly enough, some of these extremely annoying issues could be fixed with one-line patches if we reach a political consensus on the desired design/behaviour.

Comment 8

[WADA:World Anti-bad-Duping Agency]

(In reply to Thomas D. (currently busy elsewhere; needinfo?me) from comment #7)
> WADA, this is reported against SeaMonkey. 
> Did you try SeaMonkey before saying "unable to reproduce"?

Sorry, I missed Product=SeaMonkey.
However, I quickly tested with SeaMonkey before comment posting although "Insert Link" case only, because I'm SeaMonkey user, and I know that AutoDowngradeToText is well known phenmenon in SeaMonkey too.
And, I was aware of next today:
  In Address Book, Contact I used for test, which is z@z, had "Prefers HTML" in SeaMonkey.
I forgot "I intentionally changed Prefers of the z@z to HTML in the past for other testing"...
I thougt "Link only case" was downgraded to text in SeaMonkey, but I thought something was changed in SeaMonkry.
Sorry for my mistake.

"URL like string is pasted" or "Link is pasted" depends on "where Copy was done".
If copied at Text Editor, at SeaMonkey's URL bar etc., it's merely a text string,
(a) HTML source generated by SeaMonkey with simple Copy/Paste of text.
    http://en.wikipedia.org/wiki/Main_Page
If copied at Thunderbird's mail display of HTML mail with View/Message Body As/Plain Text or Text mail display, it's copied as Link, because Tb shows with Auto-Linkify and Copy generates Rich Text version and simple text version.
I checked again.
(b) HTML source generated by SeaMonkey's Insert Link:
    <a href="http://en.wikipedia.org/wiki/Main_Page">http://en.wikipedia.org/wiki/Main_Page</a>
(c) HTML source generated by SeaMonkey by this "Copy/Paste from Tb's mail":
    <pre wrap=""><a class="moz-txt-link-rfc2396E"
       href="http://en.wikipedia.org/wiki/Main_Page">&lt;http://en.wikipedia.org/wiki/Main_Page&gt;</a></pre>
There is no difference in HTML between SeaMonkey and Thunderbird.
In any case, if recipient is Unknown, above is sent as text/plain by SeaMonkey.
This is same result with SeaMonkey I saw in the past.
This is not observed in Thunderbird.

In SeaMonkey, change like next is made.
   Compose button has Drop Down menu, and HTML mode or Text mode is selectable.
Menu text in Option is still "Format" in SeaMonkey but it's "Delivery Format" in SeaMonkey.   
So, logic relevant to Auto-Deect may be different between SeaMonkey and Thunderbird.
Logic of gMsgCompose.bodyConvertible() is perhaps different between SeaMonkey and Thunderbird.

Anyway, if you talk about a specific case, please check HTML source which is actually generated by SeaMonkey or Thunderbird, instead of merely checking Content-Type:text/plain of sent mail where problem already occured.
If "text only in Body Text or PRE" case, gMsgCompose.bodyConvertible() or equivallent surely says "Convertible".
I think you perhaps don't think "Format Loss" in this case,.
So, there is no need to say "Format Loss" in this case.

What I want to say is that phenomenon of "auto downgrade to text" itself is already well known phenomenon as known by bugs listed in dependency tree for meta bug 889315. So, there is no need to shout "Format Loss!!!" in every bug relevant to Auto-Detect.
In bug 1210244, developer said "(B-2) in Table of User Story of bug 1210244" is a "format is lost" case. (B-2) is "if(allPlain && Not-Cnvertible), silently send as text/plain" part. A possble solution in this case is "Always ask instead of silently send as text/plain". If this kind of solution, if improved in "text only in Body Text or PRE" case, it's automatically applied to "<tt> only case", "style case" and so on.
Why "Shouting Format Loss!!!" is needed? "Finding solution" is far important than "Complain Format Loss in many bugs", isn't it?

Comment 9

[WADA:World Anti-bad-Duping Agency]

(In reply to Rainer Bielefeld from comment #0)
> Steps how to reproduce:
> 1. Send account settings to "Create HTML mails"
> 2. open new e-mail for send

  How did you invoked Tb's composer?
  If SEND-TO, different problem may occur, becuse "Save As Draft" is invlved in your STR.
  If Edit As New of mail created by other mailer such as MS application, different problem may be involved.

> 3. from <http://en.wikipedia.org/wiki/Main_Page> copy / paste text line 
>    "the free encyclopedia that anyone can edit." (below heading) to mail body

  In which aplication's at where, did you do Copy operation on what?

> 4. Save as draft and check view: looks ok, words with hyperlink in blue and underlined

  With which "View/Message Body As" option of SeaMonkey?

> 5. Check menu 'Options  ► Format': Should show "Automatic"
> 6. Send mail and check result:
>    » HTML will be broken, Hyperlinks shown as additional URLs
>      Mail header shows "Content-Type: text/plain;"

  If you do "set Delivery Format=HTML only" and do "Send Later" at this step,
  what HTML source is generated by SeaMonkey?

Comment 10

[WADA:World Anti-bad-Duping Agency]

Gotcha.
Differece between followng. (SeaMonkey, Delivery Format=Auto-Detect, Unknown Recipient)
(i)  <a href="http://abc.def.ghi/pqr.xyz">http://abc.def.ghi/pqr.xyz</a>   silently sent as text/plain 
(ii) <a href="http://abc.def.ghi/pqr.xyz">ZZZZZZZZZZZZZZZZZZZZZZZZZZ</a>   Send Options is used

If Copy of a link at some where and Paste at Tb, (i) is always generated by Paste.
In Link Location(double click of generated link), changeable part is URL only. Link Text can't be altered.
If string of the link in composing HTL is altered, both Link Text and URL is changed.
i.e. To create (ii), one of next is needed.
     different Link Text from URL upon Insert Link
     at Link Location, change URL.

I prefer folowing simple change to "request convertible check logic change".
  -  if (aConvertible == nsIMsgCompConvertible::Plain || allPlain)
  +  if (allPlain) OR if (aConvertible == nsIMsgCompConvertible::Plain && allPlain)
If user sets other than "Plain Text Only" at Send Options, no roblem occur.
If user sets "Plain Text Only", user can accept "sent as text/plain" because he requested by setting.

Comment 11

[WADA:World Anti-bad-Duping Agency]

Another solution : user's default choice for Unknown. (no need of send logic change) 
If Unknown is converted to HTML by setting, problem won't occur.
If Unknown is converted to Text by setting, user can accept.

Comment 12

[WADA:World Anti-bad-Duping Agency]

Bug summary of bug 414299 was "<tt> for "fixed width" sections or with simple links".
Thomas D. duping to bug 414299 is apropriate?

Comment 13

[WADA:World Anti-bad-Duping Agency]

There are 4 kinds of bug which can resolve problem you saw.
(1) Bug   56834 : If user can choose default for Prefers=Unknown,
                  Prefers=Unknown falls into one of HTML Domain or Text Domain.
                  So Unknown disappears. Then problem due to Unknown will never occur.
(2) Bug  136502 : if Auto-Detect feature is not used or killed by option,
                  or if other default action than Auto-Detect of Delivery Format menu can bet set by option,
                  Auto-Detect is not used, then your problem won't occur.
(3) Bug  414299 : If HTML you created is considered Not-Convertible-To-Text, your problem won't occur.
(4) Bug 1210244 : If send logic is modified,
                  and if Send options is referred before doing Auto-Downgrade-To-Text,
                  user can send HTML mail as text/html by setting HTML in Send Options.
                  (Surprizingly, one-liner patch was suffcient.)

To bug opener.
Which is good for porting to SeaMonkey?


  ,

Comment 14

[WADA:World Anti-bad-Duping Agency]

FYI.
Official solution in this case.
  1. Set Prefers=HTML of Contact for the recipient in Address Book.
  2. Request Delivery Format=HTML or Both explicitly.
Bug 1215791 is sucessor of Bug 1210244.
Seting dependency to that bug, instead of duping to Bug 414299 which is request of "Not-Convertible for simple link".

Comment 15

[WADA:World Anti-bad-Duping Agency]

To bug opener.
See "[3] Offcial methods, simple/easy workarounds, to avoid the mystery for user" section of attachment 8675255 [details] for known workarounds, please.