1202276 - Prevent silent, unpredictable, and potentially massive dataloss of formatting when sending HTML messages involving recipients of delivery format preference "Unknown" (even when mixed with "prefers-HTML")

Status: RESOLVED DUPLICATE of bug 584313

(MailNews Core :: Composition, defect)

Description

[Thomas D. (:thomas8)]

+++ This bug was initially created as a clone of Bug #414299 +++

Before even looking at the preferred delivery format of message recipients (plaintext|HTML), Auto-Detect runs DetermineConvertibility() to decide on behalf of the user which HTML formatting deserves to be preserved or dumped. If it thinks there's nothing worth keeping, it will silently downgrade HTML messages into plaintext messages before sending. Unfortunately, the algorithm is poor and outdated, so lots of clearly consequential HTML or css formatting goes down the drain, e.g. all style sheets on a range of tags which are killed, including plain links. Compare:

before auto-degrade (attachment 633913 [details]) vs.
after auto-degrade (attachment 633910 [details]).

That's a massive silent dataloss of formatting. Which is why many affected users hate that auto-*degrade* behaviour, and rightly so. More so as there's no way to switch off auto-degrade (bug 136502). The download statistics for Arivald's "Always HTML" addon which simply disables the Auto-Detect/Auto-Degrade HTML to plaintext "feature" speak for themselves:

12,027 Downloads (and counting, 233 in the last 30 days), 3 years after publication on 28th June 2012, perhaps only 1 year looking at the graph.
 1,742 Users per day on average (1,518 in the last 30 days)

Source: https://addons.mozilla.org/de/thunderbird/addon/always-html/statistics/

And that's just the tip of the iceberg, because the auto-degradation happens so cunningly that many users may never notice, and if they do, they might not know about the addon. And God only knows how many thousands/millions of users have switched Auto-Degrade off by using the known workarounds like HTML tags in their signature etc.

FOR MORE DETAILS why this violates the principles of ux-wysiwyg, ux-control, ux-efficiency and ux-consistency, pls refer to Bug 414299 Comment 131.

To prevent silent and unpredictable auto-*degrade* for recipients of type "Unknown", this bug will simply implement the addon code (2 lines).

Please note the difference between auto-degrade and auto-detect! This bug will switch off only auto-*degrade*, and only for the worst cases (involving recipients whose delivery format preference is *UNKNOWN*, which is true for all collected addresses and all new contacts unless explicitly specified otherwise).

Iow, the actual auto-*detect* of delivery format according to delivery format preferences of recipients is NOT (yet) changed by this bug per its current intention (although auto-*detect* has plenty of ux-problems, too).
Also, this bug (per current plan) will not even fix *all* cases of auto-*degrade*, but only those involving recipients of type UNKNOWN.

Theoretically, we could alternatively try to fix the algorithm which decides on the fate of tags and attributes, which in practice can never succeed because the formatting dataloss will just be less, but still remain. I agree with Arivald's Bug 414299 Comment 122:

> Fixing [bugs like bug 414299 in their] current form will be like fixing shape
> of car wheels, from square to pentagonal - it will be a bit better, but it
> will [still] not work properly anyway.

Besides, there's really no point of maintaining auto-*degrade* which has clearly outlived its original purpose of avoiding to send HTML as often as possible for MUA compatibility and message size reasons. The days of PINE and 56K modems are clearly gone. 100Mbit DSL, Terabyte data stores, XXL HD digital photo attachments and a global internet based on HTML and CSS, including mobile applications in the case of Firefox OS: All of these have made the historical auto-degrade approach redundant to say the least. Say goodbye to the dinosaur...

FTR: This bug in its current shape will leave all dinosaur auto-degrade aka nsIMsgCompConvertible algorithms intact and just stop using them for the subset of "UNKNOWN" recipients and combinations thereof.

Comment 1

[Thomas D. (:thomas8)]

I have talked about this problem with Blake (:bwinton) personally (at the MozSummit 2013) and he indicated his agreement with being more preservative about user formatting and more generally, to err in favor of HTML.

Comment 2

[Thomas D. (:thomas8)]

This also continues seamlesssly the development started by Bryan (:clarkbw) in Bug 466674, where we gave up the ever-nagging question about delivery formats (are you really sure that you want to send HTML???) in favor of just sending HTML+plaintext by default. Bryan (then UX-lead) even stated in Bug 466674 Comment 8 that
> now that I understand the workings of the Thunderbird code for sending it probably
> wouldn't matter if we switched to *HTML only* either.

I believe that Bug 466674 Comment 4 by Phil Ringnalda (:philor) still describes the current behaviour pretty well (hasn't changed much since 2008, except the better default which stops nagging about it...).

So this bug addresses the underestimated "not always perfectly correctly" part of the following scenario:
> If any recipient is unknown or text, or there's a newsgroup, and we believe
> (not always perfectly correctly) that we can losslessly convert to text, we
> send text only (even if it enrages Nelson ala bug 414299).
Nelson is definitely not the only one to be enraged, as seen from the addon statistics in comment 0...

Comment 3

[Thomas D. (:thomas8)]

For an overview of just how much users are suffering with delivery format issues, please refer to this bug and its dependencies:

Bug 889315 - [Meta] Tracker bug for delivery format UX issues (HTML vs. Plaintext; incl. interaction of {Delivery Format | Auto-Detect} with other settings and related UX-failures)

Comment 4

[Thomas D. (:thomas8)]

The Fix - 2 lines as in "Always HTML" addon, courtesy of Arivald (also author of Stationary addon, https://addons.mozilla.org/de/thunderbird/addon/stationery/)

(In reply to WADA from bug 414299 comment #131)
> (In reply to Thomas D. from comment #125)
> > FTR: Starting point in the code:

http://mxr.mozilla.org/comm-central/source/mail/components/compose/content/MsgComposeCommands.js#4009

> > 4111 function DetermineConvertibility()
> > 4112 {
> > 4113     if (!gMsgCompose.composeHTML)
> > 4114         return nsIMsgCompConvertible.Plain;
> > 4115 
> > 4116     try {
> > 4117         return gMsgCompose.bodyConvertible();
> > 4118     } catch(ex) {}
> > 4119     return nsIMsgCompConvertible.No;
> > 4120 }
> > 
*********************************************************************************************
> > Arivald's Always-HTML addon simply replaces this function with the following:
> > 
> > function DetermineConvertibility() {
> >   if (gMsgCompose.composeHTML) return nsIMsgCompConvertible.No;
> >   return nsIMsgCompConvertible.Plain;
> > }
*********************************************************************************************
>
> Auto Detect does do two different jobs.
>   (a) Auto Downgrade to Text from HTML
>   (b) Auto Delivery Format determination(or asking) only when mail is HTML,
> based on preference of HTML or Text in Address Book and Send Option.
> This bug is for (a), and complaints of this bug is for logic of
> gMsgCompose.bodyConvertible().
> AlwaysHTML addon looks to have worked around problem by "never call
> gMsgCompose.bodyConvertible()".
> 
> A problem in current code.
> (a) is done first, regardless of per recipient HTML/Text preference in
> address book, and once "Downgrade to Text" is recomended by 
> DetermineConvertibility() and gMsgCompose.bodyConvertible(), there is no way
> to send "mail composed in HTML by user all the way long" in HTML, and there is no
> chance of "(b) works" because state of the mail is already "Text mail".
> 
> [...]
> Is "making (a) optional" possible?
> Or it's already optionsal?  User can always choose (i) Don't install
> AlwaysHTML addon, and (ii) Install AlwaysHTML addon.
> 
> Is "change sequence of (a) and (b)" possible?
>   If all recipients prefers Text, try to down grade to Text by (a), 
>   then, if "convert to Text is rejected by gMsgCompose.bodyConvertible(),
> ask to user.
>  (New Send Option like "try to downgrade to Text if all recipients prefer
> Text" my be needed for this)

Just like the addon, my upcoming patch here will simply prevent "(a) Auto Downgrade to Text from HTML" for the most frequent scenario involving recipients with "UNKNOWN" delivery format preference. I bet nobody will notice nor shed a tear. After that, we can move on to fix other scenarios and reflect about the usefulness or otherwise of keeping the dinosaur under the hood.

If we'd succed to actually prevent the algorithms of gMsgCompose.bodyConvertible() from running, that should also come with performance gains: That whole exercise of scanning the whole body for "discardable" tags with lots of special casing is pretty expensive.

Comment 5

[Thomas D. (:thomas8)]

Attachment: Patch v. 1.0: Don't auto-degrade messages involving recipients with "Unknown" delivery format preference

The FIX (as explained in comment 4).

For review, this has been tried and tested in Arivald's "Always HTML" addon (whose name is a bit misleading because it just prevents the auto-degrade for messages involving recipients with "UNKNOWN" delivery format preference; otherwise both auto-degrade and auto-detect will still occur with this patch).

For UI-review, pls refer to comment 0 ff; this fixes several UX-violations as explained with more detail in Bug 414299 Comment 131. With general blessings from :bwinton (see commment 1), and continuing the good work started by Bryan (:clarkbw) in Bug 46667 (see comment 2). General overview in meta Bug 889315. This is meant as a first step towards improving the delivery format UX: Less dataloss(y), more predictable and consistent. Most of us don't want their HTML-formatting silently discarded upon sending.

Comment 6

[Thomas D. (:thomas8)]

(In reply to Thomas D. from comment #0)
> FOR MORE DETAILS why this violates the principles of ux-wysiwyg, ux-control,
> ux-efficiency and ux-consistency, pls refer to Bug 414299 Comment 131.

Typo: UX discussion in Bug 414299 Comment 133 (not 131).

(In reply to Thomas D. from comment #5)

> For UI-review, pls refer to comment 0 ff; this fixes several UX-violations
> as explained with more detail in Bug 414299 Comment 131.

Dito, that's Bug 414299 Comment 133 (not 131).

Comment 7

[Richard Marti (:Paenglab)]

Comment on attachment 8657637 [details] [diff] [review]
Patch v. 1.0: Don't auto-degrade messages involving recipients with "Unknown" delivery format preference

ui-r- because this makes the auto-detect a NOP. I'm more for a implementation of bug 136502. Maybe also memorizing the last user selected option in composer could help. Then he has not to change it every time.

Comment 8

[Joshua Cranmer [:jcranmer]]

(In reply to Thomas D. from comment #0)
> Besides, there's really no point of maintaining auto-*degrade* which has
> clearly outlived its original purpose of avoiding to send HTML as often as
> possible for MUA compatibility and message size reasons. The days of PINE
> and 56K modems are clearly gone. 100Mbit DSL, Terabyte data stores, XXL HD
> digital photo attachments and a global internet based on HTML and CSS,
> including mobile applications in the case of Firefox OS: All of these have
> made the historical auto-degrade approach redundant to say the least. Say
> goodbye to the dinosaur...

Alas, here you are somewhat mistaken. Yes, any decent mail client needs to support HTML email, and that's something that clients will live with. Unfortunately, there are ancillary parts of the email ecosystem that work far better with plain text than HTML (mailing list archives do come to mind here). There are also communities where HTML email is highly frowned upon (Usenet comes to mind here). I would suspect that, in practice, composing an email by typing the text in (or copy/pasting w/o formatting) and eschewing use of all formatting knobs is the most commonly-accepted way of acceding to send email in plain text.

Comment 9

[Joshua Cranmer [:jcranmer]]

Comment on attachment 8657637 [details] [diff] [review]
Patch v. 1.0: Don't auto-degrade messages involving recipients with "Unknown" delivery format preference

Alas, I cannot support this bug. It's not really fixing the issue, and it feels like a gross attempt to try to sidestep the actual crux of the debate.

The real problem is that the algorithm is flawed because it's trying to divine the intent of the user by looking at the generated document--but the intent of the user is most clearly indicated by the explicit actions a user takes (i.e., manually introducing formatting). Rather than saying "this is stupid, don't support this," you should look at actually making those actions inform the code that convertibility is not desired.

Comment 10

[Joshua Cranmer [:jcranmer]]

(Alternatively, Paenglab's section in comment 7 would probably be easier and suffice for most use cases).

Comment 11

[Arivald]

(In reply to Joshua Cranmer [:jcranmer] from comment #8)
> (In reply to Thomas D. from comment #0)
> > Besides, there's really no point of maintaining auto-*degrade* which has
> > clearly outlived its original purpose of avoiding to send HTML as often as
> > possible for MUA compatibility and message size reasons. The days of PINE
> > and 56K modems are clearly gone. 100Mbit DSL, Terabyte data stores, XXL HD
> > digital photo attachments and a global internet based on HTML and CSS,
> > including mobile applications in the case of Firefox OS: All of these have
> > made the historical auto-degrade approach redundant to say the least. Say
> > goodbye to the dinosaur...
> 
> Alas, here you are somewhat mistaken. Yes, any decent mail client needs to
> support HTML email, and that's something that clients will live with.
> Unfortunately, there are ancillary parts of the email ecosystem that work
> far better with plain text than HTML (mailing list archives do come to mind
> here). There are also communities where HTML email is highly frowned upon
> (Usenet comes to mind here). I would suspect that, in practice, composing an
> email by typing the text in (or copy/pasting w/o formatting) and eschewing
> use of all formatting knobs is the most commonly-accepted way of acceding to
> send email in plain text.

I agree, there are places where HTML is unwelcome. 
But IMO we should leave this decision to the user, and if he explicitly chosen HTML, let him send the HTML. If user chose plain text, TB should send as plain text. 

The point is that TB often send plain text when user wanted HTML. And by doing so, in many cases formatting, and sometimes the content is lost.

So, auto-detect should be trashed, but the preferences of HTML/plain-text format should be handled properly. In my opinion they currently works quite good.

Bug 136502 seems to be good alternative, especially if this preference would default to disabled auto-detection.

By the way, it may help to expose the UI to switch between HTML and plain-text, currently users often do not know how to do that. Another problem is that after switching from HTML to plain-text, message body is NOT converted to the plain-text right after switch, it still shows the HMTL version, it just hide the formatting controls. Also currently there is no possibility to switch to HTML if composer was started in plain-text mode, this should be fixed.

So, there is a lot do do. But sadly it all depends on the politics, on the minds of people who control TB evolution. And judging by lats ten years, there will be no change. I personally given up, I use my extensions to make TB work as I need, hoping that no one will remove support for HTML from TB.

Comment 12

[Charles]

(In reply to Joshua Cranmer [:jcranmer] from comment #8)
> Unfortunately, there are ancillary parts of the email ecosystem that work
> far better with plain text than HTML (mailing list archives do come to mind
> here). There are also communities where HTML email is highly frowned upon
> (Usenet comes to mind here).

Yes, we know that the 'old guard' prefers plain text, and would just as soon kill HTML if they could.

But then there is reality.

Both of those examples are corner cases...

a) many (or at least some) email archiving systems use HTML formatted emails, and

b) usenet? Really? You want to code to that extremely minor use case in preference to the vast majority of (potentially new) Thunderbird users who have never heard of usenet and want (properly functioning) HTML by default?

> I would suspect that, in practice, composing an email by typing the text in 
> (or copy/pasting w/o formatting) and eschewing use of all formatting knobs
> is the most commonly-accepted way of acceding to send email in plain text.

Sure - but even most of those people (I'm talking the younger/newer TB users, not the ancient ones) will still want HTML, even if only so they can bold and underline with a GUI button.

I agree with Thomas - the dinosaur must go.

Comment 13

[Dan Pernokis]

(In reply to Richard Marti (:Paenglab) from comment #7)
> Comment on attachment 8657637 [details] [diff] [review]
> Patch v. 1.0: Don't auto-degrade messages involving recipients with
> "Unknown" delivery format preference
> 
> ui-r- because this makes the auto-detect a NOP. I'm more for a
> implementation of bug 136502. Maybe also memorizing the last user selected
> option in composer could help. Then he has not to change it every time.

Specifically, memorizing the last user selected option:  This would introduce another randomness, because, gee, last time I sent something it worked such-and-such way, and so I might presume that must be normal now.  Then it changes again.


@ALL:

I agree very much with Thomas that there is too much guesswork in the system.  As a wobbly first step, let's immediately get rid of the auto-DEGRADE as proposed -- ie, for "Unknown" recipients.  But let's also include absolutely no downgrades if the recipient is explicitly HTML.

Then let's tackle the real problem that I and a million others have already complained about:  If I (we) want HTML -- either explicitly by choice or implicitly by use of formatting, etc -- then darn well send it that way -- no question!

This quickly gets us to the end point:  Send plain-text *ONLY* if and when explicitly requested, otherwise *ALWAYS* use HTML (or both, if that's what it takes to work).  Tagging the recipient for plain-text is one way to be explicit, adding a control button could allow the writer to specify text only (default always HTML, remember).

There is one big fly in the ointment, however:  Say I write a colourful formatted message to ten people, one of whom is explicitly tagged as a plain-text recipient.  TB has to know to send HTML to everyone else, then downgrade for the exception.  But the sent message must be saved in my SENT folder as pristine (original, not "recovered") HTML -- and likewise to my DRAFT folder if this is still only a draft message.  Downgrade to plain-text must happen only at the very end of the process, and only for those explicitly involved in text only.

Comment 14

[[e-mail]]

The design of the ask send format dialog assumes that the message has failed to convert to plain text, so we have to be careful when changing the format logic. Currently it works like this:
1. HTML, if all recipients are listed as wanting HTML
2. Plain, if all recipients are listed as wanting plain
3. Plain, if the message "is convertible"
4. As per the mail.default_html_action preference, if it's valid
5. Ask the user

A possible tweak would be to switch cases 3. and 4. This means that if the recipient preference is unknown then your default action takes precedence, unless it's set to always ask, in which case it will attempt to convert to plain, and only ask if that "fails". Additionally it might be desirable to attempt to convert to plain if at least one of the recipient prefers plain before checking the default HTML action.

Note that Thunderbird's default value for that preference is 3 which will mean it will start always sending messages as both HTML and Text rather than attempting to convert to plain, while SeaMonkey's default value is 0 so it won't be affected by default.

Comment 15

[Dan Pernokis]

@NEIL (re: your Comment 14)

In the common case of "all" being exactly one recipient, items 1 and 2 should be the exhaustive list.  Send both if possible, send plain to the recipient who wants plain.  The other items don't matter.  (Would a user ever want to send specifically plain text to a recipient who wants HTML?  Does it matter, especially if the user avoids use of any format controls and other implied HTML stuff?)

For multiple recipients, items 1 & 2 handle the "all recipients" cases.  But what of a mix?  The message format should not default to the lowest common denominator (plain) and spoil it for all.  Multiple versions of the message should go out, in which case I think the order should be 4, 5, 3.

My question is why would it "fail to convert to plain text"?  Why would it even try???  Send HTML (or both) whenever possible (ie, always), send plain text only as a last resort or if explicitly requested.

Comment 16

[Thomas D. (:thomas8)]

In reply to Richard Marti (:Paenglab) from comment #7)
> Comment on attachment 8657637 [details] [diff] [review]
> Patch v. 1.0: Don't auto-degrade messages involving recipients with
> "Unknown" delivery format preference
> 
> ui-r- because this makes the auto-detect a NOP. I'm more for a
> implementation of bug 136502. Maybe also memorizing the last user selected
> option in composer could help. Then he has not to change it every time.

Wow. That was quite quick to dismiss a well-thought-out proposal which 12,000 user downloads of "Always HTML" have already voted for. Please don't get confused by the name of the addon, it does NOT effect "always HTML", nor does it prevent auto-downgrading just because we set isConvertible to NO.

The addon, just like my patch here, only disables the wild and unpredictable guessing game of auto-*degrade* (isConvertible), which is decided already *before* the actual auto-*detect* continues (with more bugs, e.g. ignoring explicit pref mail.default_html_action when all recipients prefer plaintext - massive, silent dataloss of ALL HTML formatting, even inline images, in spite of the explicit pref set to ASK me in that exactly that case). Iow, the addon or patch do NOT even touch the later stages of auto-detect based on recipient preferences. So it's factually wrong to claim that this patch would make auto-detect a NOP (Null operation, disfunctional). Please test it: even with patch applied, that bug mentioned just now will still occur, of ignoring the pref and NOT asking when all recipients are prefers-plaintext and massive silent dataloss occurs, discarding inline images and all.

Bug 136502 might be a big improvement, but Auto-Detect will still be available as an option (perhaps default), so it will still misbehave and NOT do what it says it will do (like ask me before dataloss), and DO what it never said it would do (like serve dataloss to explicit HTML recipient if combined with "unknown" recipient).

Memorizing the last user selected per-message delivery format option also creates more problems than it solves, and defeats the very purpose of choosing the delivery-format per-message, but with sane and reliable defaults. If I usually write HTML and exceptionally choose "plaintext" for one particular message, does that mean that the next unrelated message should also become "plaintext"? Big red alarm: ux-error-prevention. Memorizing user choices is always a temptation, but for many cases it's much less helpful than it sounds. Which is what we found for other cases like vertical size of composition header. Expanding it one time to see those 20 recipients has no implications for my next message which might have only 2 recipients.

Bottom line: There are real bugs out there in Auto-degrade-and-detect which require fixing. Auto-*degrade* should probably be removed because we'll never get that right no matter where in the logical sequence we put it. Auto-*detect* must be fixed, and hopefully streamlined into something predictable and reliable with clearly understandable interaction of all involved UIs.

Neil's comment 14 sounds like a good start going in the right direction, and WADA has made a similar proposition of changing order of execution (Bug 414299 Comment 131). These things are tricky, it's an interaction of something like 6(!) different factors which inform/confuse the final decision which delivery format to use for a particular message, with some factors being virtually unpredictable from user's perspective. Neil's lineup is pretty good, but ignores all the current UX failures/bugs and also skips significant detail necessary to evaluate the user impact. For details, pls refer to Bug 1202227 Comment 7. As Dan's comment 15 points out, it also skips the main question, which is

> why would it "fail to convert to plain text"?  Why would it even try???  Send HTML (or both) whenever
> possible (ie, always), send plain text only as a last resort or if explicitly requested.

Something along those lines. I have tried 2 sketches for improved behaviour in Bug 1202227 Comment 6. I'm still hoping that something clear and simple will spring to mind. I'm glad we're talking. It even looks like we might be moving. Good for TB. I promise.

Comment 17

[Thomas D. (:thomas8)]

(In reply to [e-mail] from comment #14)
> The design of the ask send format dialog assumes that the message has failed
> to convert to plain text, so we have to be careful when changing the format
> logic. Currently it works like this:
> 1. HTML, if all recipients are listed as wanting HTML

That's the only thing which we can safely decide on behalf of the user unless told otherwise in once-only per-message delivery-format setting.

> 2. Plain, if all recipients are listed as wanting plain

That's where massive dataloss of full-fledged HTML message even inline images currently occurs because we don't bother to go to 4. and act as told by user's the mail.default_html_action preference.
We should even consider force-asking the user regardless of prefs because it's unlikely that he wants massive formatting like inline-images to be dumped. Also relates to failures of that "conversion" where inline images would have to be attached at least.

> 3. Plain, if the message "is convertible"

That's the biggest painpoint addressed in this bug. Here's where we force plaintext onto everyone. But we never told them (auto-detect != prefer plaintext at all costs), so we shouldn't do it.
A comment in the code claims we "guarantee lossless conversion".
Neil, do you think the following sample case, currently evaluated as isConvertible=true, should be considered "lossless conversion"?

before auto-degrade (attachment 633913 [details] [©] [details]) vs.
after auto-degrade (attachment 633910 [details] [©] [details]).

My take is, in an environment of HTML editor (! remember, we also have plaintext editor for those who love that!!!), lossless conversion of anything HTML is all but impossible unless there's REALLY NO FORMATTING whatsoever. <p> is formatting (although we might get away with that if we replace with blank lines). <p style=...> is definitely formatting. Leave them styles alone!!! <tt> is formatting because it's defined to render monospace, which can only be preserved by sending HTML (more so if interspersed with variable width). Think harder and realize that sending plaintext might cause the recipient to use fixed-width for display. But in my HTML-composition, I composed variable width, and that will be preserved if we send HTML. So, ultimately, even plain <p> is formatting because it implies and ensures variable width (deviating viewports and view settings notwithstanding).

> 4. As per the mail.default_html_action preference, if it's valid

That one gets bypassed by 3, unfortunately.

> 5. Ask the user

Dito, can be bypassed by 3, resulting in massive formatting dataloss.

> A possible tweak would be to switch cases 3. and 4. This means that if the
> recipient preference is unknown then your default action takes precedence,

That's going in the right direction. I am starting to wonder if we even need to disentangle this a bit by adding another pref. Isn't this about 2 things:

1) Should "unknown" be generally treated as "prefers plaintext" or "prefers html", or even "prefers both". That choice could become a new and exposed pref.

2) How to deal with cases where the delivery format preference is "plaintext", but the message is "formatted HTML" (any formatting counts!). Let the user choose:

- Try to downconvert "losslessly", ask only if that fails (I don't believe in lossless conversion, see my comment on 3.).
- ask me anyway (without trying to downconvert anything, because it's never lossless)
- just send plaintext (dump all formatting and image content without notice; or maybe we could attach images at least) - imo this dataloss option is so bad it should probably not even be offered - ux-error-prevention - makes no sense to compose HTML then dump it all, even for plaintext recipients, which I might not remember that they are.
- just send HTML - not ideal, but legitimate I guess
- just send HTML and plaintext - bigger messages, but safe

Or maybe we could get away with less options?

> unless it's set to always ask, in which case it will attempt to convert to
> plain, and only ask if that "fails".

I say NO to any non-approved conversion (downgrading of HTML to plaintext). It's never lossless.

> Additionally it might be desirable to
> attempt to convert to plain if at least one of the recipient prefers plain
> before checking the default HTML action.

I don't understand that. What if the other recipient prefers HTML?
Better let the user choose which degree of dataloss he wants to accept, if any.

> Note that Thunderbird's default value for that preference is 3 which will
> mean it will start always sending messages as both HTML and Text rather than
> attempting to convert to plain,

The preference is 3, but it's not honored, because auto-degrade decides for plaintext *before* we even get to using the pref. So no, it will NOT start always sending messages as both, and it DOES always try to convert to plain. That's exactly what this bug is about.

> while SeaMonkey's default value is 0 so it
> won't be affected by default.

Omg, SeaMonkey still nags each and every time...

Comment 18

[Thomas D. (:thomas8)]

(In reply to Thomas D. from comment #17)
> (In reply to [e-mail] from comment #14)
> > The design of the ask send format dialog assumes that the message has failed
> > to convert to plain text, so we have to be careful when changing the format
> > logic. Currently it works like this:
[snip]
> > 5. Ask the user
> 
> Dito, can be bypassed by 3, resulting in massive formatting dataloss.

So many bugs, I get confused and forgetful.
5. (Ask the user) is also bypassed in all cases of 2. (Plain, if all recipients prefer plain), resulting in massive silent dataloss of ALL html formatting, inline images and all. That's a bold bug.

Comment 19

[Thomas D. (:thomas8)]

An informed comment from a TB stalwart:

(In reply to Joe Sabash [:JoeS1] from bug 1202227 comment #32)
> Just a brief comment on this "Controversy"
> 
> The *ONLY* reason I have been involved with Thunderbird since 0.5a
> Is because of it's ability to render HTML/CSS
> My history is replete with workarounds, and tricks for the Editor
> to do what I want it to do.
> 
> Regarding rendering, TB is top of the line
> 
> Regarding the ability to compose HTML,
> and have it sent as intended, we fail miserably.
> 
> I imagine the whole HTML vs plaintext thing boils down to this
> 
> Do you want to convey information in your mail
> or something more. An emotion, a feeling, a thought
> 
> Poets may be able to do that, but I'm no poet
> I need HTML/Css to make my point
> 
> I fully understand the frustration involved here,
> This needs to be fixed,
> as well as a better easier HTML composer.

Comment 20

[WADA:World Anti-bad-Duping Agency]

(In reply to [e-mail] from comment #14)
> Currently it works like this:
> 1. HTML, if all recipients are listed as wanting HTML
> 2. Plain, if all recipients are listed as wanting plain
> 3. Plain, if the message "is convertible"
> 4. As per the mail.default_html_action preference, if it's valid
> 5. Ask the user

It's incorrect.
Current behavior is:
1. When mail is composed in HTML mode by user long the way,
   and when user didn't request Delivery Format=Plain Text Only nor Both HTML & Text nor HTML Only,
   (i.e. Delivery Format=Auto-Detect is used),
   If gMsgCompose.bodyConvertible() returns Yes,
   The "mail which was composed in HTML mode by user long the way"
   is always silently sent as plain/text by Thunderbird,
   Regardless of user's Send Options settings, 
   Regardless of Message preference setting of recipient in Address Book.
   i.e.
   If gMsgCompose.bodyConvertible() returns Yes,
   Send Options/"When mail is sent in HTML..." is always ignored by Thunderbird,
   and message preference in Address Book is always ignored by Thunderbird,
   because "When mail is sent in HTML..." in Send Options is always false
   when gMsgCompose.bodyConvertible() returns Yes.
2. When Delivery Format=Auto-detect is used,
   If gMsgCompose.bodyConvertible() fortunately returns No,
   Send Options/"When mail is sent in HTML" is fortunately applied by Thunderbird,
   and message preference in Address Book is fortunately referred by Thunderbird.

Please read bug 584363 comment #11 for Quick Check result of Tb's behavior. That was check result in 2012, but I think implementation is not so different in recent Thunderbird releases.
Known workarounds of problem of (1) is:
(a) Install "Always HTML" addon. The addon stops calling gMsgCompose.bodyConvertible().
(b) Force color=#000001 or background-color=#FFFFFFE, or use <B></B>(no string!) in HTML mail.
    gMsgCompose.bodyConvertible() fortunately returns No currently when one of them is used.
Many bugs in dependency tree for bug 889315 were produced by problem of (1).
Chaos, which is seen in some bugs in dependency tree for bug 889315, was produced by problem of (1).

Required action is : 
 Always apply Send Options when Delivery Format=Auto-Detect is used in HTML mode mail composition.
=== Because user composed mail in HTML mode long the way,
    Decision on "send as text/plain, no text/html part" should be made by user himself via one of next:
    - Explicit request of Delivery Format=Plain Text Only,
    - Message preference in AddressBook==Plain Text + Send Options + Reply to dialog if Ask 
    - Text Domain setting
=== Decision on "send as text/plain" should NOT be made by Thunderbird if the mail is composed in HTML mode by user.
=== Never call gMsgCompose.bodyConvertible() before application of Send Options/"When mail is sent in HTML..." when Delivery Format=Auto-Detect is used(== Same action as "Always HTML" addon). If calling gMsgCompose.bodyConvertible() is needed, it should be done in actions by Send Options setting.
I can not understand reason why calling gMsgCompose.bodyConvertible() is still mandatory even after implementation of Send Options/"When mail is sent in HTML..." and message preference in Address Book.

Comment 21

[Thomas D. (:thomas8)]

An informed summary comment from WADA:

(In reply to WADA from bug 584363 comment #11)

> As for text only HTML mail case, I think "preference=Unknown" and "not
> defined in Address Book" is beter interpreted as "preference=HTML" nowadays.
> I believe "silent downgrade by Auto-Detect to text/plain from text/html
> when no need of HTML formatting" should be applied to "preference=Plain
> Text" only nowadays.
> Problem due to "mail is sent in text/plain even though user composed mail as
> HTML" is far worse nowadays than problem due to "mail is sent in HTML even
> though formatting by HTML is not required".

Comment 22

[Thomas D. (:thomas8)]

(In reply to Thomas D. from bug 1202227 comment #6)
> Is there any other worthwhile MUA out there who degrades explicit HTML
> messages into plaintext before sending? Worse, without giving the user a
> choice??
> 
> Auto-*degrade* must go (because we can never get that right), but perhaps
> auto-*detect* can be improved to do the right thing (decide on the best
> delivery format for a given set of recipients, and take the lowest
> non-datalossy denominator.

************************************************************************
*** DRAFT for better auto-detect decision-making for delivery format ***
************************************************************************

a) If we could agree to just send HTML OR HTML+plaintext for "unknown" cases, it becomes very simple and straightforward:

recipients of type -> delivery format

all plaintext -> plaintext (and follow default_html_action pref, i.e. NO silent, hardcoded *full* dataloss downgrading like now, unless explicitly set or confirmed by user).
all html      -> html
all unknown         -> html(+plaintext)
unknown + plaintext -> html+plaintext
unknown + html      -> html(+plaintext)

b) Otherwise, if we want to give user more control and avoid sending "both" by default, we'd probably need a new pref to let the user decide if "unknown" should prefer html, plaintext, or both? (needs more thought)

all plaintext -> plaintext (and follow default_html_action pref, i.e. NO silent, hardcoded *full* dataloss downgrading like now, unless explicitly set or confirmed by user).
all html      -> html
all unknown   -> HTML OR Plaintext OR both OR ASK (per new pref, PLUS follow old default_html_action pref for cases of downgrading to plaintext)
unknown + plaintext -> if unknown==plaintext(new pref): plaintext (plus default_html_action pref for cases of downgrading to plaintext, which might add or substitute by html format, or ask);
                       if unknown==html|both(new pref): html+plaintext
unknown + html -> if unknown==plaintext|both(new pref): html+plaintext;
                  if unknown==html(new pref):           html

I wish we could have an XUL demo of all the combinations as they are now and how we could improve it.

Comment 23

[Thomas D. (:thomas8)]

(In reply to Thomas D. from comment #22)

For both a) and b), I forgot one case of mixed recipients, which however is obvious:

html + plaintext --> html+plaintext

Comment 24

[WADA:World Anti-bad-Duping Agency]

To bug opener:
Please never mix following different issues in one bug.
(i)  Problem when gMsgCompose.bodyConvertible() returned Yes.
     In this case, Send Options setting, message preference in Address Book,
     are absolutely irrelevant, because Tb won't apply such options.
(ii) Phenomena/issues, or improvements in Send Options + message preference in Address Book,
     when Tb applied such options.
     (gMsgCompose.bodyConvertible() returned No, or gMsgCompose.bodyConvertible() is not called)
If you talk about (i), there is no need to refer to Send Options nor Address Book setting.
And, if you talk about (ii), please surely rule out phenomenon due to (i) from your report or comment.
(i.e. Check always with "Always HTML" addon, or with color=#000001 in HTML mail composition)

Comment 25

[WADA:World Anti-bad-Duping Agency]

To bug opener:
Please stop repeated writing of complaints of "data loss".
Fault in Tb(and mismatch between user's normal/natural want/expectation and current Tb's code) is:
  Even though user composed mail in HTML mode long the way,
  when gMsgCompose.bodyConvertible() returned Yes,
  Tb always does do same thing as "Delivery Format=Plain Text Only"
  or same thing as "reply of Plain Text Only in Ask by Send options",
  regardless of Send Options settings and message preference setting in Address Book,
  then mail is sent as text/plain message.
  (text/plain part of multipart/alternative{text/plain + text/html} when Both HTML&Text is used)
Because text/plain part only is sent by Thunderbird, needless to say, information in text/html part is surely not sent by Thunderbird.

Comment 26

[WADA:World Anti-bad-Duping Agency]

To bug opener:
"Same code as Always HTML"(don't call gMsgCompose.bodyConvertible. Your proposal) is not "Only one solution of problem of (i) in my comment #24.
"msgcompose.background_color=#FFFFFE or msgcompose.text_color=#000001 in mailnews.js" is also an effective and sufficient solution for majority of Thunderbird users who uses HTML composition mode in his daily mail composition and who is not heavy user of "complicated formatting by HTML tags" and who wants/expects "mail is sent in text/html by mailer if user composed mail in HTML mode using the mailer, as far as user doesn't explicitly request to the mailer to send it as text/plain". 
Because gMsgCompose.bodyConvertible() is currently so coded that No is returned when msgcompose.background_color=#FFFFFE or msgcompose.text_color=#000001 is set,
"msgcompose.background_color=#FFFFFE or msgcompose.text_color=#000001" can be called officially provided way for Tb users to stop "Tb's automatic/silent downgrade to text/plain regardless of user's Send Options settings/Address Book setting".

Comment 27

[Thomas D. (:thomas8)]

A crisp summary comment from an annoyed user:

(In reply to Luke Plant from Bug 1202227 Comment 33)
> I have this in my preferences:
> 
> > Text format
> > 
> > When sending messages in HTML format and one or more
> > recipients are not listed as being able to receive HTML:
> >
> > Send the message in both plain text and HTML
> 
> This is absolutely explicit, yet TB does not obey my preference. It
> sometimes obeys it, according to a subtle and buggy algorithm that decides
> that you have 'really' composed in plain text, despite having chosen the
> HTML composer.
> 
> I do not understand how the TB developers cannot see this as a simple bug.
> However, I'm bored of arguing this, and bored with the noise in my inbox,
> since the "Always HTML" extension works well enough (for now). I won't be
> contributing any further.

That's why this bug is proposing to partially correct biased algorithm behaviour by making 2 lines of code of "Always HTML" the default. Please note that this does NOT effect "Always HTML", but it actually *enables* a proper auto-detection of the best delivery format in accordance with the users preferences.

Comment 28

[Thomas D. (:thomas8)]

(In reply to WADA from comment #24)
> To bug opener:
> Please never mix following different issues in one bug.

Thanks, I have a reasonable idea that they *can* be considered different issues.
However, they all contribute to delivery format determination, and the net result for the user is wrong for many cases (see e.g. comment 27). Moreover, even technically they are intertwined in illogical ways, it's a conglomerate of bugs and wrong assumptions which do not match what we promise in the UI. Given that an ui-reviewer has wrongly claimed that setting isConvertible.NO would annihilate the entire auto-detection mechanism, I believe we need to look at the entire delivery format detection algorithm to understand what's really going on and going wrong.

(In reply to WADA from comment #25)
> To bug opener:
> Please stop repeated writing of complaints of "data loss".

I will continue to complain about dataloss until that data loss is contained. I fully understand how the dataloss occurs, and how easily it can be avoided e.g. by the patch in this bug (for many, but not all cases). I'm entirely failing to see your problem with talking about dataloss. If developers would actually understand the massive amount of dataloss that is technically possible with the current buggy algorithm (randomly dumping many tags, loads of consequential CSS styling, inline images etc.), I believe they would land the patch in this bug immediately. The fact that we're still discussing shows that the nature and amount of dataloss UX for our users has not been fully understood and appreciated. I'm still waiting for a single explicit confirmation from any developer to acknowldegde that the difference between:

before auto-degrade (attachment 633913 [details] [©] [details]) vs.
after auto-degrade (attachment 633910 [details] [©] [details]).

is never "lossless conversion" (as we claim), but potentially massive dataloss of formatting which is totally unacceptable for default settings which all suggest HTML.

(In reply to WADA from comment #26)
> To bug opener:
> "Same code as Always HTML"(don't call gMsgCompose.bodyConvertible. Your
> proposal) is not "Only one solution of problem of (i) in my comment #24.
> "msgcompose.background_color=#FFFFFE or msgcompose.text_color=#000001 in
> mailnews.js" is also an effective and sufficient solution for majority of
> Thunderbird users who uses HTML composition mode

Again I'm not sure what you're trying to say. We all know the workaround, but it's wrong to force our users into workarounds because our delivery format algorithm fails to get it right. Are you suggesting that we should implement the workaround into TB default settings? I'd consider that a really bad idea, starting to implement tricks of color shading to work around our own failures. Probably you're trying to open up a road for compromise: By making the workaround the default, we can leave the current buggy algorithm untouched, so that the tiny minority of users who prefer unpredictable downgrading to plaintext in spite of composing HTML can have their way. That's a possibility, same effect as addon code, practically switching off auto-*degrade* (not auto-detect), but allowing plaintext lovers composing in HTML to undo it, but I think it's just consolidating and painting over the current madness which is in the code. That workaround also manifests the current waste of performance which comes with scanning every single node of HTML if it can be dumped or not by isConvertible(). Let's face the beast now, and fix it. I can already see a reasonable amount of agreement from various sides that status-quo is wrong and isConvertible checks are at least in the wrong place (before auto-detect, but if anything, they must come after auto-detect has decided for plaintext). My patch takes them out of the wrong place which immediately fixes the worst problems; I suggest that everyone runs with the patch and then reports what else needs to be done.

I'd rather compromise to return auto-detect to its original purpose and promise: When there's really NO TRACE OF HTML in the message, downgrade to plaintext. That's still dataloss of formatting (for <p> with padding, variable vs. fixed with, associated fonts in most viewers etc.), but much more acceptable than what we have now, removing consequential tags, styles, inline images etc. without warning.

I'd also be willing to introduce another pref (as attempted in comment 22 b) which allows the user to define how "Unknown" should be handled (prefers text/html/both), and then adapt auto-detect algorithms from there, with a bias for preserving any HTML formatting which is there by sending HTML+plain if in doubt, or cases of mixed recipients, or what not.

Changing black font to "not really black" does not look like a reasonable compromise to me, so I'd only consider that as a last resort if we are really failing to reach any other consensus.

Comment 29

[WADA:World Anti-bad-Duping Agency]

Correction on my "Regardless of Send Options setting/Address Book setting" in recent Thunderbird release when Yes==gMsgCompose.bodyConvertible case.
"Regardless of Send Options/Address Book setting" looks too strong in recent Thunderbird release when Yes==gMsgCompose.bodyConvertible() case.

When Yes==gMsgCompose.bodyConvertible(), if all recipients==prefers HTML in Address Book, it looks that "Send Options/Message preference in Address Book" is partially cared by recent Thunderbird.
  In Tb 38.2.0, When Yes==gMsgCompose.bodyConvertible(),
  If all recipients==prefers HTML in Address Book,
  "mail composed in HTML mode by user" looks to be sent as text/html(never multipart/alternative),
  and if "all recipients==prefers HTML" is false,
  "mail composed in HTML mode by user" looks to be sent as text/plain by Thunderbird.
Needless to say, when No==gMsgCompose.bodyConvertible() is forced by msgcompose.text_color=#000001,
if ("all recipients==prefers HTML" is false && "all recipients==prefers Text" is false),
i.e. if message format preference of recipients is mixed,
"Ask of Send Options" works as expected/as designed/as implemented in recent Tb too, as it worked in 2012.

I've now understood reason why you refer to "message format preference in Address Book" in your bugs.

Comment 30

[WADA:World Anti-bad-Duping Agency]

(In reply to Thomas D. from comment #28)
> I will continue to complain about dataloss until that data loss is contained.

I see.

> (In reply to WADA from comment #26)
> Again I'm not sure what you're trying to say.

I see.

I've understood that my comments in your bug is absolutely meaningless.
I'll never add any comment to bug at bugzilla.mozilla.org which was opened by you.
I'll never touch any bug at bugzilla.mozilla.org which was opened by you.
Thanks for your opening many many bug reports at bugzilla.mozilla.org.
I could know many issues in Tb's code by your opening of many many bug reports at bugzilla.mozilla.org, even if I can't agree with your claim and even if I can't understand your claim in many many cases.
Bye.

Comment 31

[:aceman]

Is it a problem if we send both format (HTML+plain) to recipients that only want plain?

Would it help if we sent both formats to recipients being in the Unknown group?
Thus if at least one recipient is Unknown, just send both formats to all recipients?
I don't think we have the luxury to produce different messages for different recipients from one composition (as was proposed in one of these bugs). We also want only one copy to reside in the Sent folder.

Comment 32

[Dan Pernokis]

In reply to Aceman comment 31:

>> Is it a problem if we send both format (HTML+plain) to recipients that only want plain?
>> Would it help if we sent both formats to recipients being in the Unknown group?
>> Thus if at least one recipient is Unknown, just send both formats to all recipients?
Clarity at last.  I agree fully that we should always send both formats, either always, or whenever possible (see below).

>> I don't think we have the luxury to produce different messages for different recipients 
>> from one composition (as was proposed in one of these bugs) ...ie, my comment 15 above.
Except that TB must never ever -- under any circumstances -- throw out any user formatting, so you *MUST* send HTML (or whatever new improved format might come along tomorrow).  Only if a user cannot accept non-text -- as signified by a recipient preference status of "plain-text only" -- then send plain-text to THIS user.  So, catch-22, TB may have to send out two versions in these cases.  It *must* not lose formatting for those who need it, and *should* not send to those who cannot handle formatting, so two versions may have to go out if necessary.  (Note the priority: SENDER's preference for intact formatting trumps the recipient.)

>> We also want only one copy to reside in the Sent folder.
Most definitely!  And it should always be the fully-formatted HTML version as written and intended by the sender, regardless of recipient preference.  That way, the sender always has the original on file for reference, and can always re-send or forward the message later (or cut-and-paste for another message), still with its format intact.

If it happens that the output goes to exactly one recipient AND that recipient is a text-only person, then downgrade to text.  But even so, the stored format for the SENT folder should always be the original  HTML -- so the sender has an intact original for possible re-use.


So just to be clear:  "HTML" or "formatted" means sending in [HTML+plain] as Aceman suggests.  (I don't think anyone really wants HTML-only messages.)

Chew on this one:  In TB's SEND OPTIONS, under Text Options, the prompt asks what to do:
>> When sending messages in HTML format and one or more recipients are not listed as 
>> being able to receive HTML:  + pulldown responses 
It says "NOT LISTED as being able" -- ie, unknown, not specified.
It does not say "is listed as NOT BEING ABLE to receive HTML" -- ie, explicitly known that it cannot handle formatting.
And one of the options is "Send HTML anyway" -- which does not say "HTML only".

Comment 33

[Charles]

(In reply to Dan Pernokis from comment #32)
> Clarity at last.  I agree fully that we should always send both formats,
> either always, or whenever possible (see below).

Just one question: are there any legitimate email clients out there that cannot handle an email that contains both HTML and plain text?

If not, then what is the problem? Just send both, all the time, unless ALL recipients are known to receive HTML, then just send HTML only.

Comment 34

[Dan Pernokis]

Reply to Charles Comment 33:

I agree we need to be sure there aren't any email clients that may choke on HTML (or any other formatted text for that matter -- just trying to future-proof this thing).

But some may argue that, for the foreseeable future anyway, caution should prevail:  Send BOTH formats all the time, regardless if all are "known" to be HTML recipients -- because maybe some aren't HTML users after all.

Comment 35

[Joshua Cranmer [:jcranmer]]

(In reply to Charles from comment #33)
> (In reply to Dan Pernokis from comment #32)
> > Clarity at last.  I agree fully that we should always send both formats,
> > either always, or whenever possible (see below).
> 
> Just one question: are there any legitimate email clients out there that
> cannot handle an email that contains both HTML and plain text?

Yes. Keep in mind that TB includes newsgroup support, and newsgroups are *much* more conservative about HTML than email.

Comment 36

[Dan Pernokis]

Good catch, Joshua.  In that case, I think the sender would likely know he's sending to a newsgroup and would have set up that recipient as plain-text -- or certainly will if he gets a few failures -- so TB would know to send in text mode.

So, if TB *knows* a recipient is plain-text-only, the send plain-text only.  Otherwise, in all other instances, my Comment 33 stands:  send both formats.

Comment 37

[Charles]

(In reply to Joshua Cranmer [:jcranmer] from comment #35)
> (In reply to Charles from comment #33)
>> Just one question: are there any legitimate email clients out there that
>> cannot handle an email that contains both HTML and plain text?

> Yes. Keep in mind that TB includes newsgroup support, and newsgroups are
> *much* more conservative about HTML than email.

That is just one mode of this particlar client.

It isn't the *client* that doesn't support it, it is just a *service* that the client happens to support that doesn't support it.

Big difference. The client itself supports it just fine.

That said, of course you are right that, so for newsgroup accounts, force all posts to plain text only...

Any other examples of when posting both html+plaintext doesn't make sense?

Comment 38

[WADA:World Anti-bad-Duping Agency]

Thomas D.(bug opener), is there any progress in this bug?

I believe following 4 kinds of job should be executed concurrently, instead of "try to do all jobs for all possible cases on all relevant problems in one bug or a few bugs".
(a) Remove wrong aConvertible=True for user, as many as possible.
(b) Find approprite workaround or required action by user.
(c) Find root cause of problem, try to resolve it.
(d) Try to resolve problems found during (a)/(b)/(c).
I believe there was no need to open bugs like this bug and bug 1210244, if bug like bug 766860, where you posted comment in 2012-06, was properly processed. It's not fault of developers. It's fault of QA peoples.
(a) is in progress(and continus forever)
(b) already completed.
A part of (c) is in progress in bug 1210244.
A part of (d) is being tried in bug 136502.

Comment 39

[WADA:World Anti-bad-Duping Agency]

Tohmaas D., finding "where problem is generated" was pretty hard job until change by bug 1204379 was done in 2014. Who can read and understand such complicated code easily before change by bug 1204379? 
Without that bug and without acemaan's patch proposal, I believe I couldn't know "where problem is generated".

Comment 40

[WADA:World Anti-bad-Duping Agency]

Oh, sorry, wrong copy/paste.
  "change by bug 970118" is correct.

Comment 41

[WADA:World Anti-bad-Duping Agency]

I looked into bug 414299 again after 3 years from trying to change bug summary of that bug in 2012, and checked History of that bug, and checked about "<tt> and simple link" in bug summary of that bug.
And, I found that that bug is currently requesting Convertible check logic change for "<tt> and simple link" case only(Style case should be processed in Bug 674184).

Phenomenon itself is prerry simple in any case.
  If mail is sent as text/plain, text/html part is not sent.
  If text/plain part only is sen and HTML tag in text/html part is not transfered by Tb upon mail send,
  mail recipient receives text/plain data only,
  so formatting of HTML by mailer of recipient is not done(impossible).
This is, needless to say, a matter of course. 

There are 4 kinds of bug which can resolve "sent as text/plain" issue in Auto-Detect/Auto-Downgrade,
(1) Bug   56834 : If user can choose default for Prefers=Unknown,
                  Prefers=Unknown falls into one of HTML Domain or Text Domain.
                  So Unknown disappears. Then problem due to Unknown will never occur.
(2) Bug  136502 : if Auto-Detect feature is not used or killed by option,
                  or if other default action than Auto-Detect of Delivery Format menu can be set by option,
                  Auto-Detect is not used, then your problem won't occur.
(3) If HTML you created is considered Not-Convertible-To-Text, your problem won't occur.
(3-1) Bug 414299 : For <tt> case and simole link case only.
(3-2) Bug 674184 : For style(style tag aand style attribute, link rel style sheet)
(4) Bug 1210244 : If send logic is modified,
                  and if Send options is referred before doing Auto-Downgrade-To-Text,
                  user can send HTML mail as text/html by setting HTML in Send Options.
                  (Surprizingly, one-liner patch waI recommend you. 

Because meta bug 889315 is too big, I think meta bug for solutions for "sent as text/plain" issue.
Please note that "P and PRE only" case can't be resolved by Bug 414299, because it's design.

I recommend you.
(1) Close this bug as WONTFIX, because this bug is request for "same code as Always HTML in Tb".
(2) Close your bug 1202227 as WONTFIX, because that bug is alreaady in sufficiently
    unable-to-read, unable-to-understand, unable-to-follow state.
(3) Open new meta bug for tracking soliuions for the issue only,
    with arranging bug summary and bug description of your two bugs.
    There is no need of description in the bug about phenomenon after mail is sent as text/plain,
    becuse such thing is a matter of course.
(4) Close Bug 414299 and open new bug for "<tt> and simple link" case only,
    with "<tt> and simple link only" in all of bug summary, comment #0, User Story. 
    <tt> case may be better removed, because it doesn't exist in HTML 5.

Comment 42

[WADA:World Anti-bad-Duping Agency]

Sorry, wrong paste.
  (Surprizingly, one-liner patch was sufficient)

Comment 43

[WADA:World Anti-bad-Duping Agency]

FYI.
It looks that Magnus still doesn't agree with aceman's one-liner patch.
If acceptable patch will be found, I'll close Bug 1210244 and open bug for landing the accepted patch.

Comment 44

[WADA:World Anti-bad-Duping Agency]

Thomas D. I prefer resolving by Bug 56834 to any other solution, because it never touches Auto-Detect feature itslf. It merely helps user to set Prefers=HTML for all Prefers=Unknown Contacts in all Address Books.

Comment 45

[WADA:World Anti-bad-Duping Agency]

Last comment in this bug from me.

FYI.

Thomas D., have you read and understood Table in "User Story" of Bug 1210244?

As I wrote in Bug 1210244 Comment #36, send logic is pretty simple now.

Pretty simple/clear logic after sorting out by bug 970118 implemented in 2014.
(line number is as of today)
> http://mxr.mozilla.org/comm-central/source/mailnews/compose/src/nsMsgCompose.cpp#4879
> 4879 nsMsgCompose::DetermineHTMLAction(int32_t aConvertible, int32_t *result)
> 
> 4969   // If everyone supports HTML, then return HTML.
> 4970   if (allHtml)
> 4971   {
> 4972     *result = nsIMsgCompSendFormat::HTML;
> 4973     return NS_OK;
> 4974   }
> 4975 
> 4976   // If we can guarantee that converting to plaintext is not lossy, send the
> 4977   // email as plaintext. Also send it if everyone prefers plaintext.
> 4978   if (aConvertible == nsIMsgCompConvertible::Plain || allPlain)
> 4979   {
> 4980     *result = nsIMsgCompSendFormat::PlainText;
> 4981     return NS_OK;
> 4982   }
> 4983 
> 4984   // Otherwise, check the preference to see what action we should default to.
> 4985   nsCOMPtr<nsIPrefBranch> prefService(do_GetService(NS_PREFSERVICE_CONTRACTID, &rv));

If allHtml==true, allPlain==false always, and if allPlain==true, allHtml==false always.
So, write if statement as follows for ease of understanding condition.
  write "if(allHtml)"  as "if(allHtml==true  && allPlain==false)".
  write "if(allPlain)" as "if(allHtml==false && allPlain==true)".
And, to make string short, use followin expression for aaConvertible value.
  aConvertible==nsIMsgCompConvertible::Plain  =>  IsConvertible
  aConvertible!=nsIMsgCompConvertible::Plain
  === aConvertible==nsIMsgCompConvertible::No =>  IsNotConvertible
And, to make string short, write statements in an condition as follows.
  *result = nsIMsgCompSendFormat::HTML;  ==>
  return NS_OK;                          ==> SendAsHTML;
  *result = nsIMsgCompSendFormat::PlainText;  ==>
  return NS_OK;                          ==> SendAsText;

After it, code is equivallent to following.
> if      (allHtml==true  && allPlain==false) { SendAsHTML; }
> else if (IsConvertible || (allHtml==false && allPlain==true) )  { SendAsText; }
> // Otherwise, check the preference to see what action we should default to.
> nsCOMPtr<nsIPrefBranch> prefService(do_GetService(NS_PREFSERVICE_CONTRACTID, &rv));
Because OR condition, 2nd if statement cn be re-written as follows.
> if      (allHtml==true  && allPlain==false) { SendAsHTML; }
> else if ( (allHtml==false && allPlain==true) || IsConvertible ) { SendAsText; }
> // Otherwise, check the preference to see what action we should default to.
> nsCOMPtr<nsIPrefBranch> prefService(do_GetService(NS_PREFSERVICE_CONTRACTID, &rv));

Call "// Otherwise and following codes" UseSendOptions.
This block can be expressed as follows. 
   else if (allHtml==false && allPlain==false) { UseSendOptions; } 
After it, code is equivallent to following.
> if      (allHtml==true  && allPlain==false) { SendAsHTML; }
> else if ( (allHtml==false && allPlain==true) || IsConvertible ) { SendAsText; }
> else if (allHtml==false && allPlain==false) { UseSendOptions; } 

"||" condition can be split to two "if statements" with same action.
> if      (allHtml==true  && allPlain==false)  { SendAsHTML; }
> else if ( allHtml==false && allPlain==true ) { SendAsText; }
> else if ( allHtml==false && allPlain==false && IsConvertible ) { SendAsText; }
> else if ( allHtml==false && allPlain==false ){ UseSendOptions; }

In each combintion of allHtml and allPlain,
both IsConvertible case and IsNotConvertible case exists.
So, above can be re-written as follows.
> (A-1)  if      ( allHtml==true  && allPlain==false && IsConvertible    ) { SendAsHTML; }
> (A-2)  else if ( allHtml==true  && allPlain==false && IsNotConvertible ) { SendAsHTML; }
> (B-1)  else if ( allHtml==false && allPlain==true  && IsConvertible    ) { SendAsText; }
> (B-2)  else if ( allHtml==false && allPlain==true  && IsNotConvertible ) { SendAsText; }
> (C-1)  else if ( allHtml==false && allPlain==false && IsConvertible    ) { SendAsText; }
> (C-2)  else if ( allHtml==false && allPlain==false && IsNotConvertible ) { UseSendOptions; }

This is "Current Logic" of Table in "User Story" of Bug 1210244, based on findings in Bug 1204379 where I requested to close as WONTFIX several times, and where I got WONTFIX at last by bug summary change.
(C-1) is where phenomenon of "sent as text/plain when Convertible" is generated by Auto-Detect/Auto-Downgrade.

As easily known by above re-written simple logic, problem in current (C-1) can be resolved by some solutions.

(i) Because of (allHtml==false && allPlain==false) case,
if Unknown falls into one of HTML or Text by "user's choice of default of Unknown",
(allHtml==false && allPlain==false) case is changed to "HTML Domain+Text Domain only".
Because of "HTML Domain + Text Domain only" && IsConvertible case, "silently send as text/plain" is a reasonble/acceptable action.

(ii) Because of IsConvertible case, if it's changed to IsNotConvertible case, falls into (A-2)/(B-2)/(C-2).
So, problem won't occur.

(iii) Because problem is generated by ction of SendAsText in (C-1), if action is changed to UseSendOptions, problem can be bypassed.

(iv) Because Auto-Detect is relevant, if Auto-Detectt is killed, problem can't occur.

(ii) is never only one solution.
And current request for <tt> may be invalid.
And current request for "Link of Link Text==Link URL" perhaps won't be accepted, because it looks current design.

Anyway, please stop posting many comments about anything which is relevant to "silently sent as text/plain by Auto-Detect", to a bug which is relevant to "silently sent as text/plain by Auto-Detect", merely by the bug is relevant to "silently sent as text/plain by Auto-Detect".

Comment 46

[:aceman]

(In reply to Richard Marti (:Paenglab) from comment #7)
> Comment on attachment 8657637 [details] [diff] [review]
> ui-r- because this makes the auto-detect a NOP. I'm more for a
> implementation of bug 136502. Maybe also memorizing the last user selected
> option in composer could help. Then he has not to change it every time.

Yes, the patch v.1.0 here achieves the same as my patch in bug 136502, just that the one here turns off any convertibility detection unconditionally. In bug 136502 it is a user's pref. Good that there is some agreement ;)

Comment 47

[Thomas D. (:thomas8)]

FTR: I admit that setting convertible::no is the wrong approach to tame Auto-Downgrade, whether unconditionally as in my 1st patch for this bug, attachment 8657637 [details] [diff] [review], or conditionally as in Aceman's attachment 8673195 [details] [diff] [review] on bug 136502. It's wrong because it sets a fake return value for isConvertible which can cause wrong behaviour for subsequent auto-detection, esp. ASK and newsgroups.

Instead, what we really want is a simple on/off switch for auto-downgrade-if-convertible-plain, without faking return values of isConvertible. That switch must be correctly placed in the main, "outer" auto-detect logic, as in my new patch, attachment 8682030 [details] [diff] [review] on bug 136502.

Compared to my patch here (attachment 8657637 [details] [diff] [review]), the simple Auto-Downgrade switch of my new patch (attachment 8682030 [details] [diff] [review] on bug 136502) also comes with the obvious advantage that users who actually appreciate the current auto-downgrade behaviour are free to keep it, and we can continue to default to that behaviour if that's politically wanted.

It's important to realize that Auto-Downgrade is inherently *message-centric* (triggered by evaluation of message body), and therefore currently bypasses 80% of auto-detection, which is inherently *recipient-centric*. 10% is allPlain scope where we currently force downgrading even for convertible!=plain, with massive, silent dataloss of full formatting including inline images etc. Another 10% is allHTML scope where surprisingly, we suddenly respect recipient preferences and do NOT auto-downgrade even though the message is convertible::plain. It's surprising and unpredictable because for all other scopes including 'unknown' and 'someHTML'(!) (when some, but not all recipients prefer HTML), we don't care about recipient preferences and just auto-downgrade based on *message* content.

My assumption is that users who agree to the general philosophy of auto-downgrade (when the *message* looks like plaintext, just send plaintext and ignore/bypass recipient preferences) will be fine with *always* applying auto-downgrade *whenever* the /message/ is convertible::plain (100% message-centric).
When auto-downgrade is *optional*, it's fine to auto-downgrade even for allHTML scope, which makes this behaviour much more consistent, straightforward, transparent, and predictable.

There's a big gap in recipient-centric auto-detection, namely that we don't let the user choose a preferred send format for recipients whose preferences is 'unknown' in AB, which is the biggest set of contacts because every new contact, or contacts who are not yet in AB, defaults to 'unknown'.
It would obviously be tremendously helpful for user-controlled auto-detection of best delivery-format if user could map 'prefers-unknown' to any of 'prefers-plain|html|both'. Then auto-detect can use that preference to arrive at the correct send format which user really wants to send; and user will even be free to set more radical downgrading for uncle Ben (because 'unknown' scope is no longer force-bundled with 'somePlain/allPlain' scope).

For other users of the 21st century who don't care about recipient-centric auto-detection at all (probably the majority), because the war is over, and because they don't have plaintext-loving uncle Ben, we need to offer *message-centric default delivery format*. Iow, allow user to freely set (in send options dialogue) one of the following delivery formats as a default for new compositions:
Auto-Detect | Plain(?) | HTML | Both
So if I want just want to send ALL of my messages as HTML-only, I can set that as a global default and rest assured that neither message-centric Auto-Downgrade nor recipient-centric Auto-Detect will ever get into my way again...

Wow. Imagine when TB will actually let you have YOUR WAY...

Comment 48

[Thomas D. (:thomas8)]

And as long as Auto-Downgrade is still the default, this also depends on Bug 584313 (Auto-Downgrades dumps styles, that's what I refer to as "potentially massive dataloss of formatting").

Comment 49

[Thomas D. (:thomas8)]

Let's put this to rest.
We've fixed a lot of bugs which mitigate this and which I campaigned for.

Notably, aceman fixed bug 584313 about silent downgrading of any style attributes to plain text, so the worst fallouts which motivated this bug and used to irate users are gone. There's a range of other bugs which have tamed the arrogant and predatory behaviour of Auto-Detect's original Auto-Downgrade algorithm with its blatant prejudice for plain text.

Another milestone in this respect was bug 136502 which I fixed myself, finally allowing users to skip Auto-Downgrade entirely, which allows for reliable "HTML-always" message format for those who so wish, in spite of all the unpredictable mess of recipient-centric Auto-Detect which is still active under the hood with countless bugs, false promises, and UX failures. In cooperation with Aceman and WADA, and after painstaking and painful analysis and demystification of the status quo, I proposed adding one pref and some UI changes in bug 1222176 to cure the current recipient-centric logic and make it much more transparent, controllable and predictable.

All said and done, I do sympathise with Magnus radical judgement in bug 1222176 comment 85 that the entire recipient-centric part of the Auto-Detect logic should be ripped out, as it's way too complex for dubious gain. Changing message formats in a (flawed) black box algorithm after sending really sounds like a genuinely bad idea which blatantly violates ux-error-prevention and ux-wysiwyg.

So that would leave us with a fully message-centric approach, where I have suggested to allow the user to set stable per-identity default message formats, optionally with message-centric Auto-Downgrade (which should be exposed in the primary UI rather than buried far too deep in Send Options). Details to be worked out. Btw dumping unpredictable post-sending format changes also requires something along the lines of Bug 470062 (immediate wysiwyg downgrade of composition when changing to plaintext), and Bug 140800 (allow switching between plain text and HTML format, (historically) one of the most-voted TB bugs - 100 to 150 votes, including the votes of duplicate bug 216132, currently 26 duplicates for both).

Bottom line: Much progress made, and there's some hope on the horizon to eventually dry out the rest of the cesspool of meta bug 889315 which has the full picture of delivery format related UX failures and shortcomings. Please note that a lot of related bugs are slumbering in our archives after being falsely invalidated or wontfixed due to an aggressive mantra of favoring plain text over HTML. I guess we'll get there one day...