1204379 - Please set msgcompose.background_color=#FEFEFE in mailnews.js ASAP, in order to protect developers and QA peoples from flood of bugs/comments like "Format Loss!!!", "Data Loss!!!"

Status: RESOLVED WONTFIX

(Thunderbird :: Preferences, defect)

Description

[WADA:World Anti-bad-Duping Agency]

"Send Options" lies on Tb's behavior when Yes==gMsgCompose.bodyConvertible().
  "Send Option" says:
    Text Format
    When sending message in HTML format
    and one or more recipients is not listed as being able to receive HTML:
    selectable action [ Ask me, Text Only, HTML Only, Both ]

However, it works as stated only when No==gMsgCompose.bodyConvertible().
If Yes==gMsgCompose.bodyConvertible(), current Tb's behavior is(Tb 38.2.0):
  Regardless of Send Options setting by user,
  - When all recipients are listed as being able to receive HTML,
      send the mail composed by user in HTML mode as text/html mail.
      (Note: not multipart/alternative mail)
  - When false == "all recipients are listed as being able to receive HTML",
      send the mail composed by user in HTML mode as text/plain.

Obviously, Send Options lies on Tb's behavior.
Util Send Options will stop to lie on Tb's behavior,
please set msgcompose.background_color=#FEFEFE in mailnews.js,
(or msgcompose.text_color!=#000000, e.g. #000001, in mailnews.js),
in order to avoid many bug opens at buzgilla.mozilla.org,
(please see dependency tree for bug 889315)
in order to avoid Thunderbird user's confusions due to hard-to-understand current Tb's behavior when Yes==gMsgCompose.bodyConvertible() unfortunately happened internally.

Note: This bug was intentionally opened as problem in "Send Options UI" in order to avoid referring to UX relevant complaints, such as bugs listed in dependency tree for bug 889315.

Comment 1

[WADA:World Anti-bad-Duping Agency]

IIUC, Auto-Detect is name for feature of gMsgCompose.bodyConvertible(), and purpose of the gMsgCompose.bodyConvertible() was to avoid "HTML vs. Text War" when HTML mode composition was set as default composition mode in Account Settings.
Because "Send Options" UI & UX is already implemented and sorted out well, I think Delivery Format=Auto-Detect is better renamed to Delivery Format=Chosen by Send Options. And I believe that call of gMsgCompose.bodyConvertible() should be done as an optional action in Send Options if it's required.

Comment 2

[WADA:World Anti-bad-Duping Agency]

I believe that "normal/natural/ordinal want/expectation of user of a mailer which supports HTML mail composition" is following.
  As far as user intentionally composed a mail in HTML mode,
  the "mail composed in HTML mode by him" is sent by the mailer
  as text/html or multipart/alternative{text/plain+text/html},
  unless user explicitly request to the mailer to send as text/plain
  by Text Domain Setting, by Delivery Format=Plain Text Only,
  or via. Send Options setting + message preference=Text in Address Book.
Current Tb's behavior is obviously far different from user's pretty normal expectation if gMsgCompose.bodyConvertible() unfortunately returns Yes for HTML mail composed by user.

Comment 3

[Kent James (:rkent)]

How is this a security problem? I don't see that sending HTML email is a security problem, even if the plan was to send text email.

Comment 4

[WADA:World Anti-bad-Duping Agency]

(In reply to Kent James (:rkent) from comment #3)
> How is this a security problem?
It's by mistake upon bug open. How can I clear it?

Comment 5

[Thomas D. (:thomas8)]

To bug opener:

Please file MORE such bugs which present the failure of delivery format UX very concisely and technically correct. Thank you. I appreciate.

(In reply to WADA from comment #0)
> in order to avoid Thunderbird user's confusions due to hard-to-understand
> current Tb's behavior when Yes==gMsgCompose.bodyConvertible() unfortunately
> happened internally.

Because gMsgCompose.bodyConvertible()==Yes/Plain causes so much user confusion and unpredictable outcome of auto-detect delivery format decision, plus ignoring user's explicit send options per this bug 1204379, I suggested in bug 1202276 as a first step of remedy to just disable this in its current location in the code workflow, by setting gMsgCompose.bodyConvertible()==NO (as in popular "Always HTML" addon which does NOT cause always HTML). With that, premature Auto-/Degrade/ is prevented, and Auto-/Detect/ and "Send Options UI" will work correctly for all cases except "all recipients prefer plaintext". That case needs separate fixing where we wrongly assume allPlain(recipients-prefer)==plain(delivery-format) in the code, which causes the other half of bypassing send options.
 
> Note: This bug was intentionally opened as problem in "Send Options UI" in
> order to avoid referring to UX relevant complaints, such as bugs listed in
> dependency tree for bug 889315.

"Send options lies on TB's behaviour", as a problem in "Send Options UI", looks very much like an UX problem to me, but I welcome WADA's initiative of making a clean start on this which is less wordy/emotional and more technical than mine, so it points out some of the real bugs here even more clearly. Afaics WADA and I agree in analysis of current behaviour and problems present wrt delivery format; largely but perhaps not fully on possible solutions; language, emphasis and strategy might differ. I fully appreciate and endorse WADA's painstaking and focused QA work, and in spite of recent misunderstandings, I believe we'll work best as a team. I'll try to learn from WADA and cut myself shorter, especially here on 'his' bug :) I hope that others will take the time to fully understand the UX implications of WADA's excellent shorthand description of some of the real problems at hand, which our users are complaining about in all those bugs referenced in comment 0 (meta bug 889315).

Comment 6

[Thomas D. (:thomas8)]

Technical/pragmatic difference between approaches:

(a) bug 1202276, attachment 8657637 [details] [diff] [review]
if (gMsgCompose.composeHTML) always return nsIMsgCompConvertible.No;

(b) this bug 1204379:
if (gMsgCompose.composeHTML) return gMsgCompose.bodyConvertible();
add subtle default HTML formatting via pref which is surely known to trigger
return nsIMsgCompConvertible.No;

So the net result is the same: we always return nsIMsgCompConvertible.No (never execute auto-degrade before auto-detect, use send options more correctly esp. for cases where recipient preference=="Unknown").

(a) is more resolute/clean. Majority of users who want HTML composition == HTML delivery format no longer need to find and apply "Always HTML" addon or delicate formatting workarounds. 5-line-addon is needed for minority of users who want to keep exactly the old inconsistent/unpredictable auto-degrade behaviour (with different results depending on the method of HTML formatting used).

(b) is more pragmatic/dirty. We'd make the workaround of subtle HTML color formatting a default feature, just to circumvent the bug. Maybe we'll get new bugs why our default background color is not technically plain white. Above minority who prefers random auto-degrade can remove the formatting easily to return to old buggy behaviour without using addon.

As a matter of principles, I think (a) is the better and more sustainable solution. I can even offer to write the Addon "Nostalgy Auto-Detect (Always prefer PlainText)" if that helps (seriously).
I concede (a) is a possible (workaround) solution which is definitely better than status-quo.

Comment 7

[Thomas D. (:thomas8)]

(In reply to Thomas D. from comment #6)
> Technical/pragmatic difference between approaches:
> (a) bug 1202276, attachment 8657637 [details] [diff] [review]
> if (gMsgCompose.composeHTML) always return nsIMsgCompConvertible.No;
> (b) this bug 1204379:
> if (gMsgCompose.composeHTML) return gMsgCompose.bodyConvertible();
> add subtle default HTML formatting via pref which is surely known to trigger
> return nsIMsgCompConvertible.No; [...]
> I concede (a) is a possible (workaround) solution which is definitely better
> than status-quo.

Damn. Correction:

I concede *(b)* is a possible (workaround) solution which is definitely better than status-quo.

Comment 8

[:aceman]

Jorg, is this class of composer bugs something you could look into?

Comment 9

[Jorg K (CEST = GMT+2)]

Yes, in principle. But so could you. I'm currently working on some M-C editor stuff that no one else dares to touch. That will keep me busy until Christmas.

Comment 10

[WADA:World Anti-bad-Duping Agency]

To Thomas D.:
Please don't make this bug hard-to-read and hard-to-understand by your comments.
As known by many long-lived/old bugs in dependency tree for meta bug 889315, phenomenon itself, why "mail composed in HTML is sent as text/plain" occurs, workarounds, are already known.
This bug's purpose is:
(a) To provide Quick Summary about phenomenon, why "sent as text/plain" occurs, etc.
(b) Propose pretty simple/easy one-liner-patch solution,
    which no one can call it "absolutely wrong/incorrect/invalid/improper action".
      "Default background color of slightly darker then PureWhite"  
      is never "absolutely wrong/incorrect/invalid/improper default".
      Rather, it'S better for old boys like me. PureWhite is too bright for old boys.
(c) By the pretty simple/easy one-liner-patch, close many many bugs in dependency tree for meta bug 889315 at once.
  - Many many recent complaints like "Format is Loss!", "Data Loss!" => WORKSFORME
  - Image of Image only HTML mail is not sent -> WORKSFORME
  - Requests around Delivery-Format=Auto-Detect menu, for example, other default than Auto-Detect
    => Can be closed as WONTFIX
  - Complaints on logic used in Auto-Detect, for example, <tt> should be kept
    => WONTFIX, WORKSFORME, FUTURE, ...

If proposal by this bug is not used by developer, and if other solution in other bug is used by developer, "Close this bug as WONTFIX" is sufficient. There is no need of further excess comments in this bug.

Comment 11

[WADA:World Anti-bad-Duping Agency]

To Wayne Mery:
Shall I change bug summary to one like next? :-)
  Please set msgcompose.background_color=#FEFEFE in mailnews.js ASAP,
  in order to protect "Send Options" feature from Tb user's misunderstanding like next,
     "Statement in Send Options UI is wrong because it's different from Tb's actual behavior"
  in order to avoid or reduce user's confusions,
  in order to protect QA peoples from flood of bugs like "Format Loss!!!", "Data Loss!!!",
  in order to close many many bugs in dependency tree for a bug at once, as soon as possible.

Comment 12

[WADA:World Anti-bad-Duping Agency]

FYI.
Quick Summary of known workaround or circumvention.

Cause of user's confusion is:
   If    (A) Yes==gMsgCompose.bodyConvertible()
      && (B) false == "all recipients are prefs=HTML"
   mystery of "mail is sent as text/plain even though he composed in HTML mode" occurs.
There are two simple methods to avoid the mystery.
(1) Make Condition (A) false.
(1-1) Force No==gMsgCompose.bodyConvertible() by msgcompose.background_color=#FEFEFE.
      By this, "Send Options" is fully respected always.
      This is a known workaround of the mystery and is proposal by this bug.
      Other workaround, such as color=#000001, using <BB></BB>, is for same purpose.
(1-2) Don't call gMsgCompose.bodyConvertible() and return "No" always.
      By this, "Send Options" is fully respected always.
      This is a known workaround of "install Always HTML addon".
      In a bug, "include Always HTML code in Tb as standard feature" is requested.
(2) Make Condition (B) false.
(2-1) If prefs=Unknown && prefs=Undefined is considered as prefs=HTML by Auto-Detect,
      "condition of mixed" is changed to following two types.
      (a) No prefs=Text recipient
          This is identical to "All is prefs=HTML",
          so mail is sent as text/html by Auto-Detect => mystery disappears.
      (b) At least one recipient is prefs=Text
          => sent as text/plain by Auto-Detect
          => Because user intentionally set prefs=Text,
             "sent as text/plain" is normal phenomenon and is never mystery.
      This is proposal of bug 584363.
(2-2) Change default of message preference in Address Book from Unknown to HTML.
      IIRC, this was requested in a bug, but I'm not sure.
(2-3) If you want to send mail as text/html or multipart/alternative{text/plain+text/html},
      set prefs=HTML in Address Book for all recipients.
      This is protection by Tb user.
(2-4) Upon each HTML mail send, choose Delivery Format=HTML only or Both HTML and Text.
      This is also protection by Tb user.

Comment 13

[WADA:World Anti-bad-Duping Agency]

Reason why I proposed msgcompose.background_color=#FEFEFE in mailnews.js :
1. I never want to join in discussion or complaints on Auto-Detect feature,
   and I never want to touch code relevant to Auto-Detect feature,
   and I never want to touch bug which shouts "Format Loss!!!" or "Data loss!!!".
2. Even though my only one objective of "msgcompose.background_color=#FEFEFE in mailnews.js"
   is forcing No==gMsgCompose.bodyConvertible(),
   it's merely a default background color change in HTML mail composition from PureWhite
   to "Grey which is very near to PureWhite, which is merely slightly darker than PureWhite".
   (Note: if text color change to other than #000000,)
   (      some one surely says "default of Non-PureBlack is never acceptable".)
3. Only required change is　one-liner change in mailnews.js.
   There is no need to touch any other code including "Send Options" code.
   And I can't imagine complaints from many Tb users on msgcompose.background_color=#FEFEFE.

Above is a method for "Gaining maximum fruit by minimum effort".

Comment 14

[Thomas D. (:thomas8)]

(In reply to WADA from comment #10)
> To Thomas D.:
> Please don't make this bug hard-to-read and hard-to-understand by your
> comments.

!?!?

To WADA: Please DON'T try to shut out the qualified comments of an experienced QA and UX contributor like me, just because you prefer another approach. This is never "your" bug. I'll leave it to the audience whose comments are harder to read or understand, yours or mine...

> As known by many long-lived/old bugs in dependency tree for meta bug 889315,
> phenomenon itself, why "mail composed in HTML is sent as text/plain" occurs,
> workarounds, are already known.
> This bug's purpose is:
> (a) To provide Quick Summary about phenomenon, why "sent as text/plain"
> occurs, etc.

Great!

> (b) Propose pretty simple/easy one-liner-patch solution,
>     which no one can call it "absolutely wrong/incorrect/invalid/improper
> action".
>       "Default background color of slightly darker then PureWhite"  
>       is never "absolutely wrong/incorrect/invalid/improper default".
>       Rather, it'S better for old boys like me. PureWhite is too bright for
> old boys.

Please file a new bug if you think the default background color should be changed. Working around other bugs is a poor excuse for that.

> (c) By the pretty simple/easy one-liner-patch, close many many bugs in
> dependency tree for meta bug 889315 at once.
>   - Many many recent complaints like "Format is Loss!", "Data Loss!" =>
> WORKSFORME
>   - Image of Image only HTML mail is not sent -> WORKSFORME
>   - Requests around Delivery-Format=Auto-Detect menu, for example, other
> default than Auto-Detect
>     => Can be closed as WONTFIX
>   - Complaints on logic used in Auto-Detect, for example, <tt> should be kept
>     => WONTFIX, WORKSFORME, FUTURE, ...

With all respect WADA, you can't be serious on that. Please correct me if I'm wrong:

- Per this bug 1204379, WADA proposes the following change for default background color:
current: msgcompose.background_color;#FFFFFF (plain white)
proposed: msgcompose.background_color;#FEFEFE (darker than plain white)
- I.e. he wants to make the current *workaround* for many problems of Auto-Degrade-and-Detect a default feature, which indeed improves the default behaviour out of the box a lot (while changing the default background color of the actual document, the consequences of which for viewing, printing etc. would have to be examined!).
- However, what happens if user prefers "plain white" (#FFFFFF) background color again?

==> All bugs will return immediately! (Please correct me if I'm wrong.)

So, unless we implement further code changes, just changing the default document background color might be a good workaround, but it's NOT a stable solution for those bugs and I'll personally ensure to keep those bugs open.

> If proposal by this bug is not used by developer, and if other solution in
> other bug is used by developer, "Close this bug as WONTFIX" is sufficient.
> There is no need of further excess comments in this bug.

Really WADA please control yourself. I'm certainly too wordy at times, but my comments on this bug so far haven't been anywhere near "excessive" but very focused on the way forward. I reserve the right to ensure that we understand what we're doing, because sometimes your technical shorthand can by quite cryptic, and as of lately you're also campaigning.

Also, by virtue of my massive QA and UX contributions, I am a member of TB Core Team; so contrary to your insinuations, I am also a developer, including analysis and contribution of code (although coding is not my priority, and others are much better/faster at that). I'm still failing to see your emotional problem with some of the bugs; but I'm sure they have been posted for a reason, and depending on perspective, they are not all the same but even technically different (albeit related). We don't even have an explicit bug for the dumping of styles on dumped tags yet (because Ben refused to take that into bug 414299 where I presented detailed evidence with test messages, screenshots and all).

Please also note the historical context of auto-degrade before auto-detect: It was invented when ASK ME was still the default for mail.default_html_action pref, as a WORKAROUND to prevent asking too frequently for what was (at the time) considered inconsequential formatting which can be "losslessly converted". You should know best from all those bugs that "lossless conversion" is just a *BOLD*, _plain_ /LIE/. CSS styles, classes, etc. were not even topical when the list of "discardable" tags was made, so nobody considered that dumping tags will also dump their styles, classes etc. UX-WYSIWYG has also become more relevant over time with HTML/CSS becoming dominant on the WWW. IMO, there's no true safety for HTML formatting as long as the auto-degrade dinosaur is still alive under the hood. After at least a decade of user complaints (right from the start when it was implemented), it's time to say goodbye to the dinosaur and not just build a fence around it.

As a matter of fact, this bug 1204379 is trying to fix the original Workaround (auto-degrade to avoid HTML question) with yet another Workaround (to avoid auto-degrade). What a nonsense. WONTFIX indeed comes to mind...

Comment 15

[WADA:World Anti-bad-Duping Agency]

(In reply to Thomas D. (currently busy elsewhere; needinfo?me) from comment #14)
> However, what happens if user prefers "plain white" (#FFFFFF) background color again?

It's biggest problem in my proposal of this bug report.
Any Tb user can set any value at any time.
And, if user changed background color to PureWhite, the Tb user surely can fully enjoy mystery again.
And, if you do it, many bugs will be opened again by you, or many bugs will be re-opened by you.

Comment 16

[Wayne Mery (:wsmwk)]

(In reply to WADA from comment #11)
> To Wayne Mery:
> Shall I change bug summary to one like next? :-)

Thanks for asking. But I really have no opinion - this isn't an area where I'm investing my time.


As to the relative merits of this bug vs the others, I can't help but feel the other bugs have reached a point of complexity and contentiousness that no one is going to want to touch them.  So please don't bring that mass of information also into this bug, i.e. I suggest there be no further comments in this bug except a *simple* *decision* to impliement it or not, preferably from someone who hasn't commented in the other bugs.  Forget the other bugs - let's have this bug move forward, or not, on the merits and objectives wada has outlined.

Comment 17

[WADA:World Anti-bad-Duping Agency]

(In reply to Thomas D. (currently busy elsewhere; needinfo?me) from comment #6)

For clean, dirty, correct and so on of an action or solution:
If one in next two is dirty or incorrect, another is also dirty or incorrect.
If one in next two is correct or clean, another is also correct or clean.
  (i)  Force return code=No by changing essentially irrelevant option's default,
       but by officially permitted/allowed user's action. 
  (ii) Return faked return code of No, with skipping "invoke a feature which has a purpose"
(i) is my proposal. (ii) is what is done by "Always HTML" addon.
If both are *CORRECT*, "Correctness of (i)" > "Correctness of (ii)".
- "Faked Return Code in a software in order to bypass hard-ro-resolve problem"
  is usually not proper action as proper solution of problem in software,
  even if it's valid and effective workaround of the problem.
- In contrast to it, action by (i) is officially permitted/allowed user's action.
Needless to say, Effectiveness of (i) << Effectiveness of (ii), because user can kill (i) any time.  

I believe proper solution is;
  Do all Delivery-Format related actions under control of "Send Options" feature.
because "Send Options" is already implemented and is already sufficiently sorted out well.

Comment 18

[:aceman]

This bug is getting hard to follow. Let's focus on the original comment. It is claimed that if any of the recipients does not support HTML email, the body is automatically converted to plain text, without asking (regardless of options set). Is that right WADA?

It seems to me the logic is implemented at:
https://hg.mozilla.org/comm-central/file/8b519841303f/mail/components/compose/content/MsgComposeCommands.js#l2654
https://hg.mozilla.org/comm-central/file/8b519841303f/mail/components/compose/content/MsgComposeCommands.js#l3990
http://mxr.mozilla.org/comm-central/source/mailnews/compose/src/nsMsgCompose.cpp#4879

Can anybody see any logic error at first glance?

Comment 19

[Thomas D. (:thomas8)]

(In reply to :aceman from comment #18)
> This bug is getting hard to follow. Let's focus on the original comment. It
> is claimed that if any of the recipients does not support HTML email, the
> body is automatically converted to plain text, without asking (regardless of
> options set). Is that right WADA?

Not quite, your wording isn't correct which touches exactly the main problem, namely that currently "prefers-Unknown" is often treated same as "prefers-Plain". So, correction:

"If any of the recipients is not *explicitly* listed by user to prefer HTML mail,..."

I. e. for mixed-preference recipients, if *any* one of the recipients either prefers-plain OR prefers-*Unknown*, then we always *try* to automatically "losslessly" downconvert to plaintext before even looking at the recipients' preferred format. Only IF body HTML formatting is wrongly considered "convertible:plain", we'll randomly/inconsistently downconvert to plaintext without asking and regardless of send options set; also ignoring any other recipients' preference for HTML. When formatting is deemed worthy to survive by isConvertible(), we honor send options for mixed recipients.

If *all* recipients are prefers-plain (also plain-domains), then we *always* ignore the send options, with massive dataloss potential for *all* types of formatting.

If *all* recipients are prefers-HTML (also HTML-domains), then we *always* send HTML-only (which looks correct).

> It seems to me the logic is implemented at:
> https://hg.mozilla.org/comm-central/file/8b519841303f/mail/components/
> compose/content/MsgComposeCommands.js#l2654
> https://hg.mozilla.org/comm-central/file/8b519841303f/mail/components/
> compose/content/MsgComposeCommands.js#l3990
> http://mxr.mozilla.org/comm-central/source/mailnews/compose/src/nsMsgCompose.
> cpp#4879
> 
> Can anybody see any logic error at first glance?

Yes, the main logic error (exactly as explained above in this comment) is here, and I have already offered a simple patch in bug 1202276 which got shot down prematurely without sufficient explanation because reviewers apparently were NOT familiar with the current behaviour (also note that Joshua rejected my patch because it is NOT RADICAL ENOUGH):

https://hg.mozilla.org/comm-central/file/8b519841303f/mail/components/compose/content/MsgComposeCommands.js#l3990

  4009 function DetermineConvertibility()
  4010 {
  4011     if (!gMsgCompose.composeHTML)
  4012         return nsIMsgCompConvertible.Plain;
  4013 
  4014     try {
  4015         return gMsgCompose.bodyConvertible();
  4016     } catch(ex) {}
  4017     return nsIMsgCompConvertible.No;

If you read this correctly, this is what happens for failure cases:

If (gMsgCompose.composeHTML) // whenever we use HTML composer
return gMsgCompose.bodyConvertible(); // try to downconvert first (*before* going into auto-detect)
So if HTML formatting is wrongly considered convertible:plain (dumping random tags, styles, classes, alignements etc.), then delivery format "plaintext" is already determined at this stage, and send options as well as other recipients preferences for HTML get ignored.

For the all-prefers-plaintext failure case:

4976   // If we can guarantee that converting to plaintext is not lossy, send the
4977   // email as plaintext. Also send it if everyone prefers plaintext.
4978   if (aConvertible == nsIMsgCompConvertible::Plain || allPlain)
4979   {
4980     *result = nsIMsgCompSendFormat::PlainText;
4981     return NS_OK;
4982   }

Note the wrong claim in the comment: IF WE CAN GUARANTEE that converting to plaintext is NOT LOSSY...
What a brazen lie...

At least the allPlain case needs nsIMsgCompSendFormat::AskUser;

Note the extra confusion that in my understanding nsIMsgCompSendFormat::AskUser; will NOT necessarily ASK the user but at this stage it just means "obey the send options pref and do whatever is set there" (maybe in former times the other non-ask send options did not exist).

Comment 20

[Thomas D. (:thomas8)]

(In reply to Wayne Mery (:wsmwk, use Needinfo for questions) from comment #16)

Not helpful at all, Wayne, especially given that you're not interested. The behaviour is so twistedly complex that most people except WADA and I do not fully understand it. We can't just take decisions without understanding the behaviour and the exact consequences of changing things.

Comment 21

[WADA:World Anti-bad-Duping Agency]

(In reply to Thomas D. (currently busy elsewhere; needinfo?me) from comment #19)

Do you understand "What occurs on following logic" by "Always HTML addon" and "msgcompose.background_color=#FEFEFE"?
>   4009 function DetermineConvertibility()
>   4010 {
>   4011     if (!gMsgCompose.composeHTML)
>   4012         return nsIMsgCompConvertible.Plain;
>   4013 
>   4014     try {
>   4015         return gMsgCompose.bodyConvertible();
>   4016     } catch(ex) {}
>   4017     return nsIMsgCompConvertible.No;

What is done by both "Always HTML addon" and "msgcompose.background_color=#FEFEFE" is:
  At "return gMsgCompose.bodyConvertible();" step in try block,
  force return of nsIMsgCompConvertible.No.
Difference is merely :
  Always HTML addon : If composed in HTML mode,
                      Don't call gMsgCompose.bodyConvertible(),
                      and return faked nsIMsgCompConvertible.No always.
  msgcompose.background_color=#FEFEFE :
    Fully utilize current logic of gMsgCompose.bodyConvertible().
      If msgcompose.background_color != #FFFFFF in HTML mail composition,
      gMsgCompose.bodyConvertible() currently returns No. 
    So, No==gMsgCompose.bodyConvertible() is always forced by background_color=#FEFEFE.
    gMsgCompose.bodyConvertible() is so coded. What's wrong in fully utilizing it?
Once "No" is returned by DetermineConvertibility(), there is no difference in Thunderbird behavior, then code of "Send Options" is normally invoked and "Send Options" is fully respected.

Biggest problem in "Always HTML addon" is:
  It skips "call of gMsgCompose.bodyConvertible() which is executed on purpose".
  And, it always returns *FAKED* nsIMsgCompConvertible.No.
Who can say that it's absolutely proper/valid/correct/mandatory action in Thunderbird?
I can say nothing about "Skipping call of gMsgCompose.bodyConvertible() is wrong/bad action or correct/good action".
If you can say "Skipping call of gMsgCompose.bodyConvertible() is absolutely proper/correct/valid action", what is your base of it and what is evidence of it?

If "install of Always HTML addon" is done by user as his workaround of a bug, there is no problem.
However, if it's done as standard Thunderbird feature, "bodyConvertible() is mandatory" vs bodyConvertible() is useless" War may happen. I dislike, rather hate, such useless War.
Please note that workaround is workaround, workaround is not always proper solution of a problem in software.

Comment 22

[WADA:World Anti-bad-Duping Agency]

(In reply to :aceman from comment #18)
> Let's focus on the original comment.

:aceman, if you focus on, please focus on my comment #11, and read/understand "User Story".
What I wanted to provide by this bug is : Steps to reproduce/check result in "User Story".

Comment 23

[WADA:World Anti-bad-Duping Agency]

:aceman, please read/understand my Comment #12 too.
It's another "what I wanted to provide by this bug".

Comment 24

[:aceman]

So if you compose a msg in HTML but it is convertible to plain text (no fancy tags) AND if any one of recipients is determined to not prefer HTML (or is unknown), THEN and email is automatically downgraded to plain and sent silently.  The Send options are not honored regardless what they say. Is that right?

You want it to still send as HTML if that is set in the Send options? Sounds logical to want that if the option is there.

Comment 25

[:aceman]

(In reply to Thomas D. (currently busy elsewhere; needinfo?me) from comment #19)
> If (gMsgCompose.composeHTML) // whenever we use HTML composer
> return gMsgCompose.bodyConvertible(); // try to downconvert first (*before*
> going into auto-detect)
> So if HTML formatting is wrongly considered convertible:plain (dumping
> random tags, styles, classes, alignements etc.), then delivery format
> "plaintext" is already determined at this stage, and send options as well as
> other recipients preferences for HTML get ignored.
This does not seem true to me by reading the code. Yes, it determines convertibility first, but still does go into parsing recipients and only then determines the final format, considering recipient preferences AND convertibility. If there is a bug in that consideration, yes, that is what we need to find here.

> For the all-prefers-plaintext failure case:
> 
> 4976   // If we can guarantee that converting to plaintext is not lossy,
> send the
> 4977   // email as plaintext. Also send it if everyone prefers plaintext.
> 4978   if (aConvertible == nsIMsgCompConvertible::Plain || allPlain)
> 4979   {
> 4980     *result = nsIMsgCompSendFormat::PlainText;
> 4981     return NS_OK;
> 4982   }
> 
> Note the wrong claim in the comment: IF WE CAN GUARANTEE that converting to
> plaintext is NOT LOSSY...
> What a brazen lie...
I've seen you have also claimed this in another bug. I assume the determination which tag is convertible and which is not is done at http://mxr.mozilla.org/comm-central/source/mailnews/compose/src/nsMsgCompose.cpp#5011 . Can yout please open a new bug proposing specific changes to be done in that function? Which tags (or which attributes to them should cause unconvertibility of the msg).

> At least the allPlain case needs nsIMsgCompSendFormat::AskUser;
> 
> Note the extra confusion that in my understanding
> nsIMsgCompSendFormat::AskUser; will NOT necessarily ASK the user but at this
> stage it just means "obey the send options pref and do whatever is set
> there" (maybe in former times the other non-ask send options did not exist).
This seems correct.

My next step would be to change that line 4978 to 
4978   if (aConvertible == nsIMsgCompConvertible::Plain && allPlain)
I.e. only if really all recipients are plaintext and msg is mostly convertible, send plain text. Or do we really want to ask even in this case?
This would also cause any one Unknown recipient to skip this block (as allPlain == false) and actually look at Send Options, as this bug requests.

If anybody gets around to testing that faster, feel free to report the findings ;)

Comment 26

[:aceman]

(In reply to :aceman from comment #25)
> (In reply to Thomas D. (currently busy elsewhere; needinfo?me) from comment

> > Note the wrong claim in the comment: IF WE CAN GUARANTEE that converting to
> > plaintext is NOT LOSSY...
> > What a brazen lie...
> I've seen you have also claimed this in another bug. I assume the
> determination which tag is convertible and which is not is done at
> http://mxr.mozilla.org/comm-central/source/mailnews/compose/src/nsMsgCompose.
> cpp#5011 . Can yout please open a new bug proposing specific changes to be
> done in that function? Which tags (or which attributes to them should cause
> unconvertibility of the msg).
Seems that can partly be bug 414299, but that one is again drowned in comments about how to force the wished send format.

Comment 27

[:aceman]

Attachment: patch

This patch makes the change as described in my comment 25.

Aryx, please push to try server together with https://bugzilla.mozilla.org/attachment.cgi?id=8661999 so Thomas and WADA can play with the try builds.
Thanks.

Comment 28

[WADA:World Anti-bad-Duping Agency]

(In reply to :aceman from comment #27)
> patch
> -  if (aConvertible == nsIMsgCompConvertible::Plain || allPlain)
> +  if (allPlain && (aConvertible == nsIMsgCompConvertible::Plain))
>    { *result = nsIMsgCompSendFormat::PlainText; return NS_OK; }
>    // Otherwise, check the preference to see what action we should default to.

After process allHtml==true case(SendFormat::HTML is returned regardless of Send Options),
  aConvertible ==      allPlain  Current                   After your patch
   Convertible::Plain
1. true                true      SendFormat::PlainText     SendFormat::PlainText
2. true                false     SendFormat::PlainText     Go next(Use Send Options)
3. false               true      SendFormat::PlainText     Go next(Use Send Options)
4. false               false     Go next(Use Send Options) Go next(Use Send Options)
Next step(Use Send Options step) currently invoked only when allPlain==false(&&allHtml==false).
i.e. Next step(Use Send Options step) may assume allHtml==false&&allPlain==false(i.e. mixed case).
Is case 3, allHtml==false&&allPlain==true&&aConvertible!=Convertible::Plain case, processed normally, as expected, as designed?
Does Next step(Use Send Options step) process all cases again?

I think following is sufficient, because Send Optionsis already available.
  After process allHtml==true case(SendFormat::HTML is returned regardless of Send Options),
  // Because allPlain==true, send in text/plain regardless of Send Options setting.
  if (allPlain) { *result = nsIMsgCompSendFormat::PlainText; return NS_OK; }
  // Otherwise, check the preference to see what action we should do...
  // and Process allHtml!=true&&allPlain!=true case(===mixed case).
     => By Send Options, Asked, or sent as format user sets in Send Options,
        regardless of aConvertible value.

Comment 29

[WADA:World Anti-bad-Duping Agency]

(In reply to :aceman from comment #18)
> Let's focus on the original comment.
(In reply to :aceman from comment #27)
> patch

Please propose proper solution in separate bug.
- This bug was opened with incorrect bug summary intentionally.
  "Tb's actual behavior when Delivery Format=Auto-Detect
    !== What stated in Send Options panel" is true,
  but it's normal state, because "What stated in Send Options panel" is never
  "What is done by Tb when Delivery Format=Auto-Detect".
  "What stated in Send Options panel" is "What is done by Send Options code".
  Therefore, "Send Options UI panel lies" is surely wrong.
  Bug started from wrong assumption should be closed as WONTFIX, INVALID etc.
  although my want is "close this bug as WONTFIX after implementing proper solution".
- As you know, this bug is already Hard-to-read/Hard-to-understand/Hard-to-follow state,
  because I failed to keep this bug clean.
  Opening as "Bug in Send Options panel" was insufficient to protect this bug.

Comment 30

[Magnus Melin [:mkmelin]]

Comment on attachment 8662001 [details] [diff] [review]
patch

Review of attachment 8662001 [details] [diff] [review]:
-----------------------------------------------------------------

Didn't look in detail, but I don't think this is correct. If you really haven't added any formatting, we shouldn't send as HTML. Frankly I doubt many people have html plaintext prefs set for their contacts.

Comment 31

[:aceman]

(In reply to Magnus Melin from comment #30)
> Didn't look in detail, but I don't think this is correct. If you really
> haven't added any formatting, we shouldn't send as HTML. Frankly I doubt
> many people have html plaintext prefs set for their contacts.

That is the question this bug is about. How do we know there is no formatting? If the determination of convertible::plain is buggy then that can be helped in bug 414299.

But the user started in HTML formatting mode. TB silently decides there is no formatting and downgrades to plain text and then the Send Options are not needed, as there is no HTML message sending. Is that what the user expected?

Comment 32

[Thomas D. (:thomas8)]

(In reply to Magnus Melin from comment #30)
> Comment on attachment 8662001 [details] [diff] [review]
> patch
> 
> Review of attachment 8662001 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Didn't look in detail, but I don't think this is correct. If you really
> haven't added any formatting, we shouldn't send as HTML.

Magnus, it's about time that we DO look at this in DETAIL. Without details, we'll not get anywhere.

1) If you really don't want any formatting to be sent, then don't compose in HTML. Why not just use plaintext editor?

2) Where in the UI do we promise/notify to downgrade messages from HTML format to plaintext, especially for contacts whose preference is "Unknown"? You seem to be pretty much the only one who likes uninvited degradation. Our users have complained about this in many, many different bugs and support threads.

3) A message composed in HTML is ALWAYS formatted, and "lossless conversion" is a mystery. Even if there's no tag except <html>, downgrading to plaintext comes with big chances of violating UX-wysiwyg:
Composition shows my messages in a variable-width font. When sent as HTML, it will be rendered in variable-width for most recipients. When sent as plaintext, the rendering result is much more random, with a much higher chance of being rendered as fixed-width (which is NOT what I sent, and much harder to read). Similarly, using <p> (which we don't, another bug) is formatting, and we have no right to remove it without giving user a choice at least. But we must default to preservative behaviour. <p> in composition describes the logical entity of a paragraph, including vertical padding. Even if we would fix that bug where we don't translate vertical padding into blank lines, blank lines are NOT the same as a clean-cut logical and formatted <p>, with high chances of being rendered very much the same for recipient with standards-conform MUA.

4) WHY should we prefer sending plaintext when there's no formatting, but all indications of composition are HTML?? No formatting == no tags. So even the reduction in message size will be totally insignificant. So what's the higher goal that would make us prefer plaintext?

5) What we really SHOULD do is return ux-control to our users. What we SHOULD NEVER do is take delivery format decisions ABOVE THE USER'S HEAD. That's not just violating UX-principles, that's violating some of the highest standing principles of MOZILLA (e.g. Mozilla Manifesto, Principle 5: "Individuals must have the ability to shape the Internet and their own experiences on it.").

> Frankly I doubt many people have html plaintext prefs set for their contacts.

That's why we need to default to something which does not violate ux-wysiwyg, ux-control, ux-consistency, ux-efficiency, ux-trust. Really Magnus, please look into the details of this. Currently User has no way of reliably ensuring HTML format out of the box. We can't continue forcing users who just want reliable HTML format into extra steps for each message, workarounds, or addons.

For the minority who really care about sending plaintext messages:
- Please use plaintext editor
- Please set your recipients to "prefers-plaintext"
- If you insist on composing in HTML (why???) then please use per-message delivery-format setting "plaintext".

We must get it right for the majority of users who rightly expect the HTML they compose to be sent as HTML, thus preserving any formatting which is there while composing (ux-wysiwyg), not more, not less.

Comment 33

[Thomas D. (:thomas8)]

(In reply to Thomas D. (currently busy elsewhere; needinfo?me) from comment #32)
> 3) A message composed in HTML is ALWAYS formatted, and "lossless conversion"
> is a mystery.

Correction: "Lossless conversion" is a *myth* (and currently a mystery, too...)

Comment 34

[WADA:World Anti-bad-Duping Agency]

(In reply to Magnus Melin from comment #30)
> patch
> -----------------------------------------------------------------
> Didn't look in detail, but I don't think this is correct.
> If you really haven't added any formatting, we shouldn't send as HTML. 

It looks that I should have added "<img> tag only case", "<tt> tag only case", "<img> tag and <tt> tag only case" and so on, to my "User Story" of this bug, in addition to simplest/easiest-to-understnd "no HTML tag, text in Body Text only case", for you...

I believe;
- When Tb's default mail composition mode was changed from Text to HTML,
  Gain by "silently send as text/plain" >> Gain by "send as text/html even when not-so-heavy-formatting-by-Tag"
  Loss by "silently send as text/plain" << Loss by "send as text/html even when not-so-heavy-formatting-by-Tag"
    Where Gain by "silently send as text/plain" = avoiding HTML vs Text War
          Loss by "send as text/html even when not-so-heavy-formatting-by-Tag" = confusion by HTML vs Text War
- However, because HTML mail composition is pretty popular nowadays and "Send Options" is already availble,
  Gain by "silently send as text/plain" << Gain by "send as text/html even when not-so-heavy-formatting-by-Tag"
  Loss by "silently send as text/plain" >> Loss by "send as text/html even when not-so-heavy-formatting-by-Tag"
    Where Gain by "send as text/html even when not-so-heavy-formatting-by-Tag" = match with user's expectation
          Loss by "silently send as text/plain" = many user's confusion

> Frankly I doubt many people have html plaintext prefs set for their contacts.

Me too.
It's reason why "Send Options", which has action of Ask, exists, isn't it?

Comment 35

[:aceman]

This is the resulting message if I compose in HTML mode, but use no formatting options, just type the text (Enter pressed at some places, those are the <br> tags):
<html>
  <head>
    <meta http-equiv="content-type" content="text/html; charset=utf-8">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    I have used no tags here. sdkfjsk dfj skdfjak sjdf kasdjf kjsd kfjsk
    fj sdk<br>
    askdfj skdjf kasdjf ksjdf ksjdfk jk kajs dkfjas kdfj aksdfjaksdjf<br>
    aksj dfksj dfkjas dfkajsk dfjask dfjk sdjfk asdjfk sjd fksj dkfas
    dkfjasdkf<br>
    k asjkdfj skdfj askdfj ksdjfksj hjf hjsdf<br>
    as dfsjdfk asjdkf jskd fjksd fjksd fjkas dfksd fjksdj fks dk jksjdkf
    askdfj aksd<br>
    k asjdfk sdjf kasdjfk asjd fksjd fkasjd fksdj fks dkasfkasdfasd<br>
     skdjfk asjdfk asdjf kasjdf kj kasdjfkasdfksjd fkasjd k  s sldfksldf
    lsdkf <br>
    l sdfklsk dflksd flks      asjdfksjdfkjsdkf jskdfj<br>
    skdfj skfj askdfjk asdjfksdfjksd fjsdk<br>
  </body>
</html>

Magnus, do you want to optimize for exactly this case? User unknowingly leaving HTML composition on by default, but does not actually use any of the options.

Maybe we could settle on this, and nothing more. All other tags than these and message is no longer convertible::plain.
See https://bug414299.bmoattachments.org/attachment.cgi?id=8661999. So in bug 414299 we need to make the checking function even more strict, e.g. if even <br> or <p> contains ANY attributes (not just style), bail out and make it above convertible::plain (::Yes, ::Altering or ::No). But let us discuss that there or in a new bug.

And what happens if you have this degradable message but all the recipients want HTML? We send this simple message as HTML so no optimization here.

Comment 36

[Magnus Melin [:mkmelin]]

(In reply to :aceman from comment #31)
> That is the question this bug is about. How do we know there is no
> formatting? If the determination of convertible::plain is buggy then that
> can be helped in bug 414299.

Indeed, such bugs should be fixed.

> But the user started in HTML formatting mode. TB silently decides there is
> no formatting and downgrades to plain text and then the Send Options are not
> needed, as there is no HTML message sending. Is that what the user expected?

Yes, I think that's perfectly reasonable.

(In reply to Thomas D. (currently busy elsewhere; needinfo?me) from comment #32)
> 1) If you really don't want any formatting to be sent, then don't compose in
> HTML. Why not just use plaintext editor?

Because when you start to compose you still want to reserve the possibility that you would add something formatted. Most people probably don't even know the plaintext editor exits. The plaintext editor also has it's various quirks :/

> 2) Where in the UI do we promise/notify to downgrade messages from HTML
> format to plaintext, especially for contacts whose preference is "Unknown"?
> You seem to be pretty much the only one who likes uninvited degradation. Our

The vast majority of messages are not hand carved to be some kind of masterpieces for the recipient to enjoy. They are information, and you want to get that information through as nicely as possible. If there's no real formatting, the recipient’s mailer will likely render the plain text the best way. 
This is also about the data - e.g. I don't think our own (non-gloda) search finds stuff accurately in html.

I also don't think this is really such a common complaint (what's complained about is bugs in the algorithm). You've probably written more complaints about it yourself than all the other users combined, over the years.
That said, the algorithm to determine what can be sent as plaintext could be a bit stricter.

> 3) A message composed in HTML is ALWAYS formatted, and "lossless conversion"
> is a mystery. Even if there's no tag except <html>, downgrading to plaintext
> comes with big chances of violating UX-wysiwyg:

HTML is not meant to be wysiwyg either. And it's not, especially now when many people read their mails on the phone where screen real-estate is limited.
 
> formatted <p>, with high chances of being rendered very much the same for
> recipient with standards-conform MUA.

Many people use Outlook, so you're really far from anything called standards conformant rendering engine.

> 4) WHY should we prefer sending plaintext when there's no formatting, but
> all indications of composition are HTML?? No formatting == no tags. So even
> the reduction in message size will be totally insignificant. So what's the
> higher goal that would make us prefer plaintext?

If you want a higher goal: essentially you preserve the data as basic as it can be.

> We must get it right for the majority of users who rightly expect the HTML
> they compose to be sent as HTML, thus preserving any formatting which is
> there while composing (ux-wysiwyg), not more, not less.

Yes, bugs in the detection should be fixed.

Comment 37

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Try push: https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=dea5a2d67bb8

Comment 38

[Thomas D. (:thomas8)]

(In reply to Magnus Melin from comment #36)

I give up, just like many of our users complaining about this have given up before. I can't argue against a sea of mythology, distorted descriptions, lack of detail and illogical conclusions without any principles. Sorry I can't handle wobbly ideosyncracies. I prefer systematic, detailed analysis of facts and principled UX design.

I've made my point before why those thousands of users who have downloaded the "Always HTML" addon, and the unknown number of users who are using those other workarounds, maybe thousands, or hundred of thousands, might still just be the tip of the iceberg because users might never notice that TB degrades their messages *after* sending.

This whole saga looks to me like an unfortunate confirmation of the "bad PR" thread on tb-planning: Thunderbird is stuck in the past. Sadly, such deficiencies will drive away exactly those advanced users and administrators who are vital to promote Thunderbird. Really, the way we handle such problems is entirely hopeless, and can kill every inspiration to contribute to this project.

I think Aceman has a very sober and practical understanding now of many of the issues, so I'm hoping his patches will go through, so that more sanity may prevail.

Comment 39

[:aceman]

(In reply to Magnus Melin from comment #30)
> Didn't look in detail, but I don't think this is correct. If you really
> haven't added any formatting, we shouldn't send as HTML. Frankly I doubt
> many people have html plaintext prefs set for their contacts.

So the patch does fix one big error: If all recipients want plain text, we silently convert ANY HTML (not just convertible) without asking, disregarding the Send options. That should be fixed now.

The patch does this:
If a message is convertible to plain text, then do:
1. if all recipients want HTML
- send HTML
2. if all recipients are unknown
- send from Send Options, default Both (without patch sends Plain)
3. if all recipients want plain
- send Plain
4. if recipients are mixed
- send from Send Options, default Both (without patch sends Plain)

If a message is NOT convertible to plain, then do:
1. if all recipients want HTML
- send HTML
2. if all recipients are unknown
- send from Send Options, default Both
3. if all recipients want plain
- send from Send Options, default Both (without patch sends Plain)
4. if recipients are mixed
- send from Send Options, default Both

So which case do you propose to behave differently?

Comment 40

[WADA:World Anti-bad-Duping Agency]

(In reply to :aceman from comment #39)

Thanks for providing idea for "improvements in great Send Options feature".
It's similar to my expectation.
  Do all Delivery Format=Auto-Detect related actions under control of Send Options.
  Example of my idea.
  - Delivery Format=Auto-Detect => Delivery Format=Chosen by Send Options
  - When sent as HTML, option to choose text/html or multipart/alternative, even when allHtml==true
  - Utilize HTML analysis result of Convertible==Convertible::Plain in action of Send Options,
    in order to reduce "Ask is needed case for user"
  - If asking, show reason why asked("who is not HTML domain recipient" etc.) 
  - In Send Options panel, "When sent in HTML ..." => "When composed in HTML mode..."
  and so on.

> So the patch does fix one big error:
>  If all recipients want plain text, we silently convert ANY HTML (not just convertible) without asking,
>  disregarding the Send options. That should be fixed now.

Why it's wrong?
"Text Domain" and "message preference=Text in Address Book" is for "mail recipient who can't correctly understand HTML"(e.g. old mobile phone), "who rejects HTML mail, accepts Text mail only"(e.g. goverment etc.) and so on, isn't it?
As Magnus says as follows,
  Most people probably don't even know the plaintext editor exits.
  The plaintext editor also has it's various quirks :/
majority of users use HTML composition mode which is defuled in Tb for lo--ng time.
"Text Domain" and "message preference=Text" is service for such peoples, isn't it?
I think following is very kind for users and all is selectable by it, but I don't think it's mandatory.
  If all recipients want plain
     If Convertible == Convertible::Plain
        Ask, or Send Both, or Force HTML only, or Send Text Only, or Don't send 
     If Convertible != Convertible::Plain
        Ask, or Send Both, or Force HTML only, or Send Text Only, or Don't send

Comment 41

[WADA:World Anti-bad-Duping Agency]

(In reply to Thomas D. (currently busy elsewhere; needinfo?me) from comment #38)
> I give up,

Thanks for giving up. I'm very appriciated. By it, I believe that at least several developers and QA peoples don't think "never touch this bug any more" in this bug.

Comment 42

[WADA:World Anti-bad-Duping Agency]

(In reply to Magnus Melin from comment #36)
> especially now when many people read their mails on the phone where screen real-estate is limited.

I also user of small mobile phone in Japan. Mobile phone which I used 10 years ago didn't have caapability to show HTML mail. But now, it'a no-so-new type but it has capabilty to show HTML mail. 
And I use "mail is HTML" as "condition to ignore the mail", because HTML mail which I receive by mobile phone is AD mail which can be called SPAM even if I requsted/allowed to send such mail to companies.

(Q1) I thought majority of mobile mail user in the world is smart phone user.
     Many many peoples stil use small mobile phone which can not show HTML mail correctly?

Magnus, please note that this bug is for issue of "(2-2) of User Story" only since initial, even though this bug was filled by comments which came from other bug reports. "What is same as other bug reports" is "start point is (2-2) of User Story of this bug" only. I never said "lossless conversion" nor "not lossy conversion" like thing. I never requested "skip calling gMsgCompose.bodyConvertible()". All of them came from other bug reports.

(Q-2) What do you think about Tb's current behaaviou of (2-2) in User Story of this bug?
      If it's not absolutely proper/valid/correct/... behavior as mailer for Tb users,
      what do you think about proper solution of issue of (2-2) in User Story?

Please note that if change like next is made,
> -  if (aConvertible == nsIMsgCompConvertible::Plain || allPlain) Send as text/plain
> +  if (allPlain ) Send as text/plain
any allHtml!=true&&allPlain!=true case(===any mixed case) is processed by "Send Options" regrdless of aConvertible value which is set by Auto-Detect, and user is asked by Tb if user sets Ask in opti0n, or mail is sent in format which user requested.

(Q3) What's wrong in the change?
If aConvertible==nsIMsgCompConvertible::Plain is set by Auto-Detect, Thunderbird should always send the mail as text/plain silently, without permission or request by user?

Comment 43

[:aceman]

Thomas, WADA, please try the build at http://ftp.mozilla.org/pub/mozilla.org/thunderbird/try-builds/[e-mail]-dea5a2d67bb8/ . Only use on test profiles as there are some crashes in it when managing accounts.

Comment 44

[WADA:World Anti-bad-Duping Agency]

(In reply to :aceman from comment #43)
> please try the build

Thanks for try-build.
What change is made by the try-build?
Please answer to my question of comment #28. (following aConvertible!=Convertible::Plain && allPlain==true case)
     aConvertible ==      allPlain    Current                 After your patch
     Convertible::Plain
  3. false                true        SendFormat::PlainText   Go next(Use Send Options)

Comment 45

[WADA:World Anti-bad-Duping Agency]

(In reply to :aceman from comment #43)
> please try the build

Quick check result.
  aConvertible ==     allPlain Current                   After your patch
   Convertible::Plain
1. true               true     SendFormat::PlainText     SendFormat::PlainText     => silently sent as text/plain
2. true               false    SendFormat::PlainText     Go next(Use Send Options) => Asked. problem is fixed
3. false              true     SendFormat::PlainText     Go next(Use Send Options) => Asked. improved
4. false              false    Go next(Use Send Options) Go next(Use Send Options) => Asked

Case-3 is allHtml==false&&allPlain==true case.
In Send Options logic, it looks that "!allHTML => Ask" works as stated in Send Options panel.

What a powerful one-liner patch!
I think "your patch only" is sufficient to close many bugs which complaints due to problem in Case-2, although I think change like "Delivery Format=Auto-Detect => Determined by Send Options", "When HTML is sent... => When mail is composed in HTML mode...", are better done at same time.

Comment 46

[WADA:World Anti-bad-Duping Agency]

:aceman, please open separate bug with following bug summary(statement in your patch)
> if the recipient does not preferer HTML, honor Send options to choose format of outgoing messages
and propose final patch in it and land the patch, please.
Many bugs can be closed at once as "FIXED by the separated bug" or "WORKSFORME because problem won't occur any more", and this bug is automatically closed as "WONTFIX because far better proper solution is implemented".

Comment 47

[:aceman]

(In reply to WADA from comment #44)
> Thanks for try-build.
> What change is made by the try-build?
It contains this patch and the one from bug 414299 (making stricter checks for convertivility)

> Please answer to my question of comment #28. (following
> aConvertible!=Convertible::Plain && allPlain==true case)
>      aConvertible ==      allPlain    Current                 After your
> patch
>      Convertible::Plain
>   3. false                true        SendFormat::PlainText   Go next(Use
> Send Options)

I tried to answer that in my comment 39.

(In reply to WADA from comment #46)
> :aceman, please open separate bug with following bug summary(statement in
> your patch)
> > if the recipient does not preferer HTML, honor Send options to choose format of outgoing messages
> and propose final patch in it and land the patch, please.
> Many bugs can be closed at once as "FIXED by the separated bug" or
> "WORKSFORME because problem won't occur any more", and this bug is
> automatically closed as "WONTFIX because far better proper solution is
> implemented".

Why should I make a new bug? This one has all the history and comments. This one is about "Send Options" lying. Does the patch not fix that problem? I think we can skip the "chaning of color" part :)

Comment 48

[WADA:World Anti-bad-Duping Agency]

(In reply to :aceman from comment #47)
> Why should I make a new bug?

Read comment #29, please. And, comment #11 is written on assumption of "this bug will be surely WONTFIXed".

> This one is about "Send Options" lying. Does the patch not fix that problem?

"Send Options lies" is intentional my lie.

Comment 49

[Magnus Melin [:mkmelin]]

(In reply to WADA from comment #42)
> (Q-2) What do you think about Tb's current behaaviou of (2-2) in User Story
> of this bug?
>       If it's not absolutely proper/valid/correct/... behavior as mailer for
> Tb users,
>       what do you think about proper solution of issue of (2-2) in User
> Story?

If it's not convertible then we want to send as html.

I think we have too many preferences. In the end all the prefs would appear slightly irrelevant to me. If it's convertible we convert to plaintext, if not we don't.

Comment 50

[WADA:World Anti-bad-Duping Agency]

(In reply to Magnus Melin from comment #49)
> If it's not convertible then we want to send as html.

Tb's behavior : before patch and after patch

            aConvertible                                         (SF::HTML      = SendFormat::HTML     )
            ==                                                   (SF::PlainText = SendFormat::PlainText)
    allHtml Convertible allPlain  Before patch   After patch
            ::Plain                             (* = changed by patch)
 0. true     N/A        false     SF::HTML        SF::HTML       =>  sent as text/html
 1. false    true       true      SF::PlainText   SF::PlainText  =>  sent as text/plain 
 2. false    true       false     SF::PlainText  *UseSendOptions => *Asked. Problem is resolved
 3. false    false      true      SF::PlainText  *UseSendOptions => *Asked. Improved for users, bad for me
 4. false    false      false     UseSendOptions  UseSendOptions =>  Asked

(Q-a) Is it acceptable for you? Is it correct action for you?
Case-2 should be SendFormat::PlainText because aConvertible==Convertible::Plain?
(Auto-Detect requested to send in Text? Or Auto-Detect merely says "Coversion to Text" is OK?).

(Q-b) What's wrong in my proposal of "In Case-3, SF::PlainText" == "Because Text Domain recipients only, send as text/plain regardless of aConvertible value"?
Because Auto-Detect says "Don't do downgrade", "send as text/plain" should not be executed?
IIUC, Text Domain setting means "mail should be sent in Text because recipient can't handle HTML well", so if user composed mail in HTML mode even though Text Domain recipient, Tb kindly sends the mail as text/plain for user who forgot his "Text Domain setting"...

(Q-c) When allHtml==false&&allPlain==false case(mixed case), is optional action better provided, which fully respects/utilizes aConvertible==Convertible::Plain, because Auto-Detect feature says "Do downgrade to Text"?

Anyway, "cause or start point of many bugs relevant to Auto-Detect" is current Tb's behavior of Case-2 in above table. So, I believe that many complaint only comment posting in many bugs will stop by patch.

Comment 51

[Thomas D. (:thomas8)]

(In reply to WADA from comment #50)
> (In reply to Magnus Melin from comment #49)
> > If it's not convertible then we want to send as html.
> 
> Tb's behavior : before patch and after patch
> 
>             aConvertible                                         (SF::HTML  
> = SendFormat::HTML     )
>             ==                                                  
> (SF::PlainText = SendFormat::PlainText)
>     allHtml Convertible allPlain  Before patch   After patch
>             ::Plain                             (* = changed by patch)
>  0. true     N/A        false     SF::HTML        SF::HTML       =>  sent as
> text/html
>  1. false    true       true      SF::PlainText   SF::PlainText  =>  sent as
> text/plain 
>  2. false    true       false     SF::PlainText  *UseSendOptions => *Asked.
> Problem is resolved
>  3. false    false      true      SF::PlainText  *UseSendOptions => *Asked.
> Improved for users, bad for me
>  4. false    false      false     UseSendOptions  UseSendOptions =>  Asked

For the uninitiated and the superficial readers amongst us, please note that "Asked" in WADA's table does NOT necessarily mean we'll actually ASK. It actually means we'll default to NOT asking and sending BOTH formats. Otherwise, we'll do whatever user prefers in that case, i.e. we'll respect Send Options, which can be either of Ask/Plain/HTML/Both.

> (Q-b) What's wrong in my proposal of "In Case-3, SF::PlainText" == "Because
> Text Domain recipients only, send as text/plain regardless of aConvertible
> value"?
> Because Auto-Detect says "Don't do downgrade", "send as text/plain" should
> not be executed?

No, when Auto-Detect-Convertible says "Not convertible", it means there's formatting which can't be sent in text/plain. Per ux-error-prevention, to avoid accidental/unexpected dataloss of formatting, we should send multipart/alternative by default. (We also need to improve conversion to include inline images as parts when downgraded to plain). Recipient who prefers text can read the text/plain part of multipart/alternative. Is there any serious email client, even when generally text-oriented, which can't handle multipart/alternative? But when same prefers-plaintext recipient uses another device for receiving, or upgrades to a recent/decent MUA like TB, he can still see text/html part, or save that part and look at in browser. I'm still assuming that every formatting is there for a reason, and deserves respect, never silent force-dumping. For those who don't like their own formatting, please use plaintext editor, or just force delivery-format plain manually from menu. Otherwise, user is free to set "ask", or "downgrade without asking" in Send options. However, that would currently also downgrade for "unknown", which is unwanted. I propose that "prefers-Unknown" should be disentangled from "prefers-plain" because they are not the same type of recipients. We can either decide that "prefers-unknown" should translate to SF::BOTH, or we introduce a new pref where user can define prefered format for "unknown" recipient type. In both cases, Auto-Detect can work very reliably, and for almost every case except all-plain perhaps, we don't even need isConvertible (so to improve performance, we shouldn't run it when it's not needed, but that's for other bugs).

> IIUC, Text Domain setting means "mail should be sent in Text because
> recipient can't handle HTML well", so if user composed mail in HTML mode
> even though Text Domain recipient, Tb kindly sends the mail as text/plain
> for user who forgot his "Text Domain setting"...

As explained above, "prefers-text" imo does NOT mean "suffocates when receiving 'multipart/alternative'". For any serious email client, in the worst case it will mean "can't display text/html part in proper/full formatting". Even PINE has basic HTML support!!!
And for exceptional tiny minority cases like age-old newsgroups, where perhaps text/plain is technically the only accepted format, and they might refuse multipart/alternative, well I guess that's where you should just use plaintext editor to ensure wysiwyg for your own sake.

> (Q-c) When allHtml==false&&allPlain==false case(mixed case), is optional
> action better provided, which fully respects/utilizes
> aConvertible==Convertible::Plain, because Auto-Detect feature says "Do
> downgrade to Text"?

Auto-Detect must NEVER decide above the user's head. FULL STOP.

> Anyway, "cause or start point of many bugs relevant to Auto-Detect" is
> current Tb's behavior of Case-2 in above table. So, I believe that many
> complaint only comment posting in many bugs will stop by patch.

Indeed. FTR, I'm widely in agreement with WADA and Aceman's proposals, and anything which takes out Auto-Degrade for "unknown" even when mixed with "prefers-HTML" recipients will be a massive improvement. So if for the time being we can just limit the dataloss to "all-plain" case, I won't be in the way of that although I think it's wrong.

Comment 52

[Thomas D. (:thomas8)]

(In reply to Magnus Melin from comment #49)
> (In reply to WADA from comment #42)
> > (Q-2) What do you think about Tb's current behaaviou of (2-2) in User Story
> > of this bug?
> >       If it's not absolutely proper/valid/correct/... behavior as mailer for
> > Tb users,
> >       what do you think about proper solution of issue of (2-2) in User
> > Story?
> 
> If it's not convertible then we want to send as html.

Where in the UI do we inform the user that "if TB thinks your formatting is "convertible" (not important), we'll just dump it"!?? Lossless conversion is a myth, as I've shown in countless comments, testcases etc. Besides we have a lot of evidence from user feedback that users expect their formatting to be preserved to the jota *for sending*. What the recipient will ultimately be able to see is a different story and totally irrelevant for send format, given that we are in HTML editor which is intended to compose HTML. So what on earth makes anybody think we have any rights to FORCE lossy conversion onto every user without ways of opting out??

> I think we have too many preferences. In the end all the prefs would appear
> slightly irrelevant to me. If it's convertible we convert to plaintext, if
> not we don't.

Oh come on please!!! (That effectively translates to: What all of you have said so far is slightly irrelevant to me, let's just ignore it and pretend the world is flat!)

The main problem is not that we have too many preferences, but that they interact in unpredictable ways for the case of delivery-format::auto-detect:
a) because there are hidden algorithms like auto-degrade which distort the results
b) because we don't do what we promise to do wrt send options
c) because in that conglomerate algorithm we ultimately make wrong assumptions like "prefers-unknown"=="prefers-plaintext", which as an absolute is clearly wrong in these HTML-everywhere times.

If auto-degrade-and-detect would stop forcing everyone into dataloss of formatting (auto-degrade) regardless of their send options, then loads of bugs which are hated by many users can be solved.

<sarcasm>
But following comment 8 logic of "higher goal: essentially you preserve the data as basic as it can be", perhaps we should just remove the HTML editor and return to sending text in ASCII, in the spirit of "back to the basics". Then we don't even need to worry about maintaining TB any more and making it a competitive email client. Plain old PINE will be sufficient to "preserve the data as basic as it can be"...
</sarcasm>

Comment 53

[Thomas D. (:thomas8)]

(In reply to Thomas D. (currently busy elsewhere; needinfo?me) from comment #52)
> <sarcasm>
> But following comment 8 logic of "higher goal: essentially you preserve the
> data as basic as it can be", perhaps we should just remove the HTML editor
> and return to sending text in ASCII, in the spirit of "back to the basics".
> Then we don't even need to worry about maintaining TB any more and making it
> a competitive email client. Plain old PINE will be sufficient to "preserve
> the data as basic as it can be"...
> </sarcasm>

Ooops, sorry, that refers to Magnus' comment 36, NOT Aceman's comment 8. But who cares, as long as TB thinks it can just force-send "B" when I composed "A"!?

Comment 54

[WADA:World Anti-bad-Duping Agency]

(Revised) Check result of Tb's behavior : before patch and after patch, *** with Send Options=Ask ***

            aConvertible                                         (SF::HTML      = SendFormat::HTML     )
            ==                                                   (SF::PlainText = SendFormat::PlainText)
    allHtml Convertible allPlain  Before patch   After patch
            ::Plain                             (* = changed by patch)
 0. true     N/A        false     SF::HTML        SF::HTML       =>  sent as text/html
 1. false    true       true      SF::PlainText   SF::PlainText  =>  sent as text/plain 
 2. false    true       false     SF::PlainText  *UseSendOptions => *Asked. Problem is resolved
 3. false    false      true      SF::PlainText  *UseSendOptions => *Asked. Improved for users, bad for me
 4. false    false      false     UseSendOptions  UseSendOptions =>  Asked

Comment 55

[:aceman]

Please, if there is a bug in determining if a message is downconvertible, defend that in bug 414299 and add examples. Nobody yet provided an example that does not work with my patch there.
If you find serious problems, maybe Magnus will agree that we can't automatically downgrade in any case.

Comment 56

[Magnus Melin [:mkmelin]]

(In reply to Thomas D. (currently busy elsewhere; needinfo?me) from comment #52)
> Oh come on please!!! (That effectively translates to: What all of you have
> said so far is slightly irrelevant to me, let's just ignore it and pretend
> the world is flat!)

Not really, I just don't think those prefs are really used by anyone, so what you're discussing are extreme edge cases. 
I bet for way over 99% of mails it's just all-unknown. For that 1% how much would have the slightest formatting, maybe 5%? When does it do anything wrong? Not that often. You get the picture.

You can claim "Lossless conversion is a myth" all you want, but it doesn't make it any more true. If there's no real formatting there's no loss.

Comment 57

[WADA:World Anti-bad-Duping Agency]

To Magnus and Tohmas D., this bug is for phenomenon/issue of "(2-2) in User Story of this bug" and patch(proper solution for the problem) proposed by :aceman.
Please stop adding comment which is irrelevant to phenomenon/issue of "(2-2) in User Story of this bug" and patch(proper solution for the problem) proposed by :aceman.
If you want to continue, please continue at appropriate place. You can pretty easily find many appropriate places in dependency tree for meta bug 889315, where is already sufficiently hard-to-read, hard-to-understand, hard-to-follow bug report at bugzilla.mozilla.org, or where is for discussing(many are not for resolving) such issue or theme.

Comment 58

[WADA:World Anti-bad-Duping Agency]

To Magnus, why I asked you some questions is your next comment #30 on proposed patch.
> Didn't look in detail, but I don't think this is correct.
> If you really haven't added any formatting, we shouldn't send as HTML.

You said the patch is not correct, so, I asked about "you stil think incorrect or not", and asked about meaning of aConvertible==Convertible:Plain.
  aConvertible==Convertible:Plain === mail should be sent as text/plain, or not.
  aConvertible!=Convertible:Plain === mail should not be sent as text/plain, or not.
  A value of aConvertible forces Tb to send as text/html or text/plain, or not.
  A value of aConvertible is merely a recommendation by Auto-Detect, or not.
  "Mail send logic" should subject to Convertible value always, in any case, or not.
  aConvertible can be ignored by "Mail send logic", or not.

> If you really haven't added any formatting, we shouldn't send as HTML.

Who is "you" and "we"? you=Tb user? or Email user? we=Thunderbird? or Tb/Email user?
Because comment on patch, I thought you=Tb or Email user and we=Thunderbird(or developer)...

Comment 59

[:aceman]

(In reply to WADA from comment #54)
>             aConvertible                                         (SF::HTML  
> = SendFormat::HTML     )
>             ==                                                  
> (SF::PlainText = SendFormat::PlainText)
>     allHtml Convertible allPlain  Before patch   After patch
>             ::Plain                             (* = changed by patch)
>  0. true     N/A        false     SF::HTML        SF::HTML       =>  sent as
> text/html
>  1. false    true       true      SF::PlainText   SF::PlainText  =>  sent as
> text/plain 
>  2. false    true       false     SF::PlainText  *UseSendOptions => *Asked.
> Problem is resolved
>  3. false    false      true      SF::PlainText  *UseSendOptions => *Asked.
> Improved for users, bad for me

Why is this case bad for you?
I think this bug was about this. The text in Send options says "if any of the recipients can't receive HTML, do this:". allPlain case fullfils this condition so SendOptions are used.

So how would you like it to behave?

Comment 60

[Thomas D. (:thomas8)]

(In reply to :aceman from comment #59)
> Why is this case bad for you?
> I think this bug was about this. The text in Send options says "if any of
> the recipients can't receive HTML, do this:"

No, it says "not listed as being able to receive HTML" which means (translated per current behaviour):
If any recipient is listed prefers-plaintext AND/OR
If any recipient is listed prefers-unknown
That's exactly what I've come to see as a design problem, that we put "prefers-plain" and "prefers-unknown" into the same category (which they are clearly NOT), AND apply an uncontrollable bias for plaintext (auto-degrade before auto-detect). Even if we fix the uncontrolled bias, the problem remains that we force user to set the same format decision for the two very different groups, plain and unknown. Therefore, I'd really wish someone could comment on my tentative proposal to have a *separate*, explicit user-definable format-preference setting for "prefers-unkown", so user decides:
prefers-unknown should be handled as:
-> prefers plain | html | both
If we have that explicit information, I think we could derive a more predictable and precise auto-detect mechanism (which should be preservative and send multipart/alternative where in doubt). I'm not 100% sure though.

The other design problem of that send option is that it doesn not reflect how to handle the MIXED cases (prefers-plain & prefers-html; prefers-unknown & prefers-html).
Practically, for all the auto-degrade cases, AND all the mixed cases (I think), we IGNORE prefers-HTML by either just sending plain (auto-degrade), or following the send options (which however should only apply to prefers-plain, and per current misdesign, to prefers-unknown, but certainly not where prefers-plain are involved). I'm trying to say that maybe instead of letting the user define fixed delivery format end results which will always get it wrong for one recipient or another, for the sake of auto-detect we should perhaps just let user set preferences ("TRY/PREFER to send as...") and then use a sane auto-detect decision to create mostly multipart/alternative or sometimes other formats as needed. Auto-Detect must become a data-preservative default; if you want to force degradations, please use per-message delivery format setting from menu, or write your own addon which downgrades your HTML formatting to your likings.

Also note that the option UI does not say that the whole send options setting will only apply when Auto-Detect is used.

> allPlain case fullfils this
> condition so SendOptions are used.

+1. That's how I would read the current wording, too. But it's bad/insufficient options design as I explained above.

> So how would you like it to behave?

Definitely HTML-preservative whereever in doubt/mixed etc.

Comment 61

[:aceman]

(In reply to Thomas D. (currently busy elsewhere; needinfo?me) from comment #60)
> No, it says "not listed as being able to receive HTML" which means
> (translated per current behaviour):
> If any recipient is listed prefers-plaintext AND/OR
> If any recipient is listed prefers-unknown
> That's exactly what I've come to see as a design problem, that we put
> "prefers-plain" and "prefers-unknown" into the same category (which they are
> clearly NOT), AND apply an uncontrollable bias for plaintext (auto-degrade
> before auto-detect). Even if we fix the uncontrolled bias, the problem
> remains that we force user to set the same format decision for the two very
> different groups, plain and unknown. Therefore, I'd really wish someone
> could comment on my tentative proposal to have a *separate*, explicit
> user-definable format-preference setting for "prefers-unkown", so user
> decides:

I see. Maybe that is also what Magnus says (force downconvert if all plain text recipients).

But then that goes against the description of this bug as filed and as the current state of the options dialog is.

I asked in another bug if it is bad to send Both to plaintext recipients. Theoretically it should be fine, but if there are clients that really fail on multipart messages, then we do not support them currently. Because in the AB contact, the field says "User prefers: Plain", not "the only thing the user accepts: plain".

I'd be fine if you file this request as a new bug as we would need to change the wording in the Options and the AB contact dialog. Or introduce new fields/options to be able to support the scenario. As I think we should not force that scenario (downconvert) when the current wording does not imply anything like that (thus so many bugs when it happens).

Comment 62

[WADA:World Anti-bad-Duping Agency]

(In reply to :aceman from comment #59)
> (In reply to WADA from comment #54)
> >             aConvertible                                         (SF::HTML  
> > = SendFormat::HTML     )
> >             ==                                                  
> > (SF::PlainText = SendFormat::PlainText)
> >     allHtml Convertible allPlain  Before patch   After patch
> >             ::Plain                             (* = changed by patch)
> >  0. true     N/A        false     SF::HTML        SF::HTML       =>  sent as
> > text/html
> >  1. false    true       true      SF::PlainText   SF::PlainText  =>  sent as
> > text/plain 
> >  2. false    true       false     SF::PlainText  *UseSendOptions => *Asked.
> > Problem is resolved
> >  3. false    false      true      SF::PlainText  *UseSendOptions => *Asked.
> > Improved for users, bad for me
> 
> Why is this case bad for you?
> I think this bug was about this. The text in Send options says "if any of
> the recipients can't receive HTML, do this:". allPlain case fullfils this
> condition so SendOptions are used.
> 
> So how would you like it to behave?

This bug is for Case-2 in the table, and I say "problem is resolved" for the Case-2.

I think that Tohmas D. never repeatedly complainted on "All is Text Domain recipiets" case, because he wrote Unknown(==not all is Text Plain) in bug summary of many his bugs.
And, I believe that "Test Domain setting" is for "send mail as text/plain even when Tb user composed mail in HTML mode, in order to protect mail recipents who can't understand HTML, for example, smaiil (and old) mobile phone which doesn't support HTML mail.
So, I think that "send as text/plain always if allPlain==true" is reasonable.
i.e. If allPlain==true, it's Casr-1 && Case-3 in table, "send as text/plain" can be done regardless of aConvertible value. Please note that Case-2 is allPlain==false case.
And, because aConvertible value is not used in Next step which uses Send Options, my claim is identicaal to "there is no need to refer to aConvertible value any more" in Send logic for allHTML=true and allPlain==true case.
And, if there is no need to refer to aConvertible value in Case-3, and if refering to aConvertible value in Case-2 will not be done, my claim is identical to "there is no need to call gMsgCompose.bodyConvertible() any more".
So, I asked to Magnus about "aConvertible can be ignored or not".

Comment 63

[:aceman]

1. OK, it seems WADA and Thomas want this interpretation ("plain text" in AB contact means "always plain, not just prefers if possible"), I can look at implementing that here and fix the wording accordingly.

2. But Magnus had a different issue. If message is convertible to plain text and there is at least one non-HTML recipient (Unknown or plain text), send ONLY plain text to all of them. Not both formats.

Magnus, can you confirm if you agree with 1.?

WADA, Thomas, can you please comment on problem 2.?

Comment 64

[WADA:World Anti-bad-Duping Agency]

(In reply to :aceman from comment #63)
> 2. But Magnus had a different issue.
> If message is convertible to plain text and there is at least one non-HTML recipient (Unknown or plain text),
> send ONLY plain text to all of them. Not both formats.

It's reason why I asked to Magnus about "aConvertible==Convertible::Plain means that an HTML mail should be sent as text/plain always regardless of Text Domain/HTML Domain seting, or not", and about "aConvertible!=Convertible::Plain means that an HTML mail should be sent as HTML always regardless of Text Domain /HTML Domin setting, or not".

Comment 65

[Thomas D. (:thomas8)]

(In reply to :aceman from comment #63)
> 1. OK, it seems WADA and Thomas want this interpretation ("plain text" in AB
> contact means "always plain, not just prefers if possible"), I can look at
> implementing that here and fix the wording accordingly.

*NO*, I never asked for that. I think "prefers-plain" means exactly that: Prefers plaintext (so we know the preference, but we should only send plain-only if "lossless" conversion is possible AND ALL recipients prefer plain; in most cases, we should just include a plaintext version as a part of multipart/alternative). Which means that as soon as there are other types of recipients (unknown, html) it's no longer possible to send plaintext for default settings. For the case of "all recipients prefer plaintext", I differ from WADA and probably Magnus because I believe even in that case, if there's any HTML formatting, we should *default* to sending multipart/alternative with plaintext+html.

So I'm tending to think we could try the following
*****************************************************
Tentative Proposal for new Send Options UI:
*****************************************************

Settings for Delivery Format Auto-Detect

Consider recipients whose preferred delivery format is unknown to prefer:
- Plaintext
- HTML
- Both plaintext and HTML

When composing messages in HTML format and ALL recipients prefer plaintext:
- Ask me what to do                                  --> This is a new option, skip isConvertible (i.e. always ask when message was composed in HTML but all recipients prefer plaintext).
- Ask me only when lossless conversion is impossible --> Try isConvertible:Plain, otherwise ask
- Convert the message to plaintext (removes all HTML formatting!)
- Send the message in HTML anyway
- Send the message in both plaintext and HTML

I've made a tentative outline how this options UI would translate to safe and predictable auto-detect decisions, in bug 1202276 comment 22, which for easier reference I'll repost below.

> 2. But Magnus had a different issue. If message is convertible to plain text
> and there is at least one non-HTML recipient (Unknown or plain text), send
> ONLY plain text to all of them. Not both formats.

I'd consider that a bug if there's no way to opt out of auto-degrade by default, because we are in HTML composition. An Unknown recipient is usually NOT a non-HTML recipient these days, please mind your wording. "Potential non-HTML recipient" perhaps.
It's also a big bug because any recipients' explicit preference for HTML will be ignored without need or benefit. I really think we need to give the user a choice and never force plaintext onto them when they composed HTML, no matter how little the formatting effect may be. But if we continue to force unwarranted downgrade, it will be a little bit better if we actually fix the auto-degrade (isConvertible) algorithm to be less datalossy about styles and other attributes.

Comment 66

[Thomas D. (:thomas8)]

(In reply to Thomas D. (currently busy elsewhere; needinfo?me) from bug 1202276 comment #22)
> (In reply to Thomas D. from bug 1202227 comment #6)
> > Is there any other worthwhile MUA out there who degrades explicit HTML
> > messages into plaintext before sending? Worse, without giving the user a
> > choice??
> > 
> > Auto-*degrade* must go (because we can never get that right), but perhaps
> > auto-*detect* can be improved to do the right thing (decide on the best
> > delivery format for a given set of recipients, and take the lowest
> > non-datalossy denominator.
> 
> ************************************************************************
> *** DRAFT for better auto-detect decision-making for delivery format ***
> ************************************************************************
> 
> a) If we could agree to just send HTML OR HTML+plaintext for "unknown"
> cases, it becomes very simple and straightforward:
> 
> recipients of type -> delivery format
> 
> all plaintext -> plaintext (and follow default_html_action pref, i.e. NO
> silent, hardcoded *full* dataloss downgrading like now, unless explicitly
> set or confirmed by user).
> all html      -> html
> all unknown         -> html(+plaintext)
> unknown + plaintext -> html+plaintext
> unknown + html      -> html(+plaintext)
> 
> b) Otherwise, if we want to give user more control and avoid sending "both"
> by default, we'd probably need a new pref to let the user decide if
> "unknown" should prefer html, plaintext, or both? (needs more thought)
> 
> all plaintext -> plaintext (and follow default_html_action pref, i.e. NO
> silent, hardcoded *full* dataloss downgrading like now, unless explicitly
> set or confirmed by user).
> all html      -> html
> all unknown   -> HTML OR Plaintext OR both OR ASK (per new pref, PLUS follow
> old default_html_action pref for cases of downgrading to plaintext)
> unknown + plaintext -> if unknown==plaintext(new pref): plaintext (plus
> default_html_action pref for cases of downgrading to plaintext, which might
> add or substitute by html format, or ask);
>                        if unknown==html|both(new pref): html+plaintext
> unknown + html -> if unknown==plaintext|both(new pref): html+plaintext;
>                   if unknown==html(new pref):           html
> 
> I wish we could have an XUL demo of all the combinations as they are now and
> how we could improve it.

Comment 67

[Thomas D. (:thomas8)]

(In reply to Magnus Melin from comment #56)
> (In reply to Thomas D. (currently busy elsewhere; needinfo?me) from comment
> #52)
> > Oh come on please!!! (That effectively translates to: What all of you have
> > said so far is slightly irrelevant to me, let's just ignore it and pretend
> > the world is flat!)
> 
> Not really, I just don't think those prefs are really used by anyone, so
> what you're discussing are extreme edge cases. 
> I bet for way over 99% of mails it's just all-unknown. For that 1% how much
> would have the slightest formatting, maybe 5%? When does it do anything
> wrong? Not that often. You get the picture.

I agree on the picture wrt 99% unknown.
Which means that 99% are affected by forced auto-degrade without prior notice in the UI nor consent.
I'm failing to understand how you can't see the problem. The integrity of the message exactly as seen on screen is sacrosanct, unless user has explicitly agreed otherwise.

> You can claim "Lossless conversion is a myth" all you want, but it doesn't
> make it any more true. If there's no real formatting there's no loss.

So who entitles us to define "real formatting" without user consent? That's just plain arrogance of the application to decide above the user's head.

Sure, if we actually fix the auto-degrade algorithm so that it stops downgrading tags with style attributes and any other attributes in fact, it will be a LOT better than before. But unless we check for styles in the head (which we don't afasics), there can still be massive dataloss of formatting when the styles are designed to apply to all tags of one sort without using in-tag style and class attributes.

Otherwise, even for the less datalossy cases like <p> (logical entity with vertical spacing, usually variable font) etc., I've already commented why dropping those is still dataloss of formatting so we must at least allow the user to opt out of downgrading completely.

Comment 68

[Thomas D. (:thomas8)]

(In reply to WADA from comment #62)
> I think that Tohmas D. never repeatedly complainted on "All is Text Domain
> recipiets" case, because he wrote Unknown(==not all is Text Plain) in bug
> summary of many his bugs.

Well, the unknown case is much more disturbing than downgrading for explicit plaintext recipients only.

> And, I believe that "Test Domain setting" is for "send mail as text/plain
> even when Tb user composed mail in HTML mode, in order to protect mail
> recipents who can't understand HTML, for example, smaiil (and old) mobile
> phone which doesn't support HTML mail.

Right, but who protects the users composition as intended and seen on screen?

WADA, as I explained in my comment 60, it's a big design problem that we lump "prefers-plain" and "prefers-unknown" together in terms of send options choice for user. But you and I degree that prefers-unkown is now probably prefers-HTML more often than not and hence very different from (never same as) prefers-plain.

If we disentangle "prefers-plain" send options from "prefers-unknown" (like I suggest in comment 65 and comment 66), everything becomes much easier and we'll give more control to the user while allowing to send as few and appropriate data as possible. 

> So, I think that "send as text/plain always if allPlain==true" is reasonable.
> i.e. If allPlain==true, it's Casr-1 && Case-3 in table, "send as text/plain"
> can be done regardless of aConvertible value.

I understand you are saying we should continue to hardcode (without opt-out) the behaviour where a fully-formatted HTML message with styles, inline images and all will silently be downgraded to plaintext when sending (without warning, without any prior opt-out from settings), just because all recipient(s) happen(s) to be "prefers-plaintext" (and since we don't mark recipients, user might not even remember that his old uncle John is still hooked to his old plaintext mobile). Sorry, such behaviour with massive dataloss potential must never be hardcoded, because it violates ux-error-prevention.

If we change the scope of the current send options to cover only the case of "allPlain"...
> When composing messages in HTML format and ALL recipients prefer plaintext:
...the user can decide himself which degree of control over downgrading to plaintext he needs.

What's wrong with applying/limiting those send options to the allPlain scenario for which they were originally intended anyway (under the outdated assumption that "unknown"=="prefers-plain")?

Alternatively/additionally as suggested on another bug:
- We should inform the user *before* sending (i.e. during composition) about the actual delivery format in which the message will be sent (especially important for format==auto-detect). I could even imagine placing a copy of the format switch into the status bar for that purpose, and if auto-detect, just add the current outcome format if message gets sent now as-is.
- If really all recipients are prefers-plaintext, so auto-detect might want to send as plain-text IF the user has consented/been informed to that, we should probably switch into plaintext composition mode.
I repeat: There's not much point of composing in HTML if what gets sent will be plaintext at the end.

Comment 69

[WADA:World Anti-bad-Duping Agency]

For lossless, lossy, ..., war.

I can't understand reason why "<body background-color=#FFFFFF>ABC</body>" is lossless and "<body background-color=#FFFFFE>ABC<B></B></body> is lossy. What is difference between #FFFFFF and #FFFFFE even though this is Email? Why can <B></B> be so important even though no string in it?
And, I'm never interested in logic to determine lossless, lossy, ....
I'm never interested in discussion on lossless, lossy, ..., or lossless vs lossy war.
I believe current logic is for "If not so lossy, send as text/plain even when composed in HTML mode, in order to avoid useless HTML vs Text war", which was implemented while Text Domain/HTML Domain setting only was available/implemented, which was implemented when default mail composition mode in Mozilla MailNews was changed from Text to HTML.

aceman, my claim is:
- If majority of Tb users(not Tb developers or QA peoples) can accept next,
     If all recipients is Text Domain recipients, send as text/plain regardless of his mail composition mode.
  there is no need of "HTML content check by Tb for lossless or lossy determination",
  unless aConvertible value(determined by gMsgCompose.bodyConvertible) will be utilized in Send Options action.
- If majority of Tb users(not Tb developers or QA peoples) can not accept next,
     If all recipients is Text Domain recipients, send as text/plain regardless of his mail composition mode.
  and if majority of Tb users want to send his mail as HTML mail when he composed his mail in HTML mode,
  and if majority of Tb users can accept current Send Options or enhanced Send Options,
  there is no need of "HTML content check by Tb for lossless or lossy determination".
  unless aConvertible value(determined by gMsgCompose.bodyConvertible) will be utilized in Send Options action.
I'm never opposite to improvement or change of logic in gMsgCompose.bodyConvertible().
I'm never opposite to utilizing aConvertible value(determined by gMsgCompose.bodyConvertible).
Any code in Tb can utilize(or respect) aConvertible value which is determined by gMsgCompose.bodyConvertible().
Simply, I'm never interested in such things.

As I wrote "problem is resolved" for Case-2 and "Improved for users" for Case-3, I believe that any Tb user will never suffer from following funny phenomenon(=mystery) any more, as far as your patch proposed in this bug will be landed,
  Thunderbird somehow sent his HTML mail as text/plain by unknown reason,
  even though he actually composed his mail in HTML mode using Thunderbird.
even if some Tb users may complain "his HTML mail was not sent as multipart/alternative",
even if some developers may complain "even though no formatting by HTML tag, sent as text/html".
And, I believe "Send Options" should be designed for majority of Thunderbird users, instead of for developers, instead of for some Tb users who repeatedly shout in loud voice at bugzill.mozilla.org.

Comment 70

[WADA:World Anti-bad-Duping Agency]

To Tohmas D.:

As I repeatedly wrote, this bug is for "Issue of (2-2) in User Story of this bug".
Please keep adding comments related to "the issue of this bug and patch for the issue of this bug" only.
If you want to discuss on imporovement or change of Send Options relevaant UX/UI, menu etc., please do it in appropriate other bug.
Please dont change this bug from "hard-to-read, hard-to-understand, haard-to-follow" to "unable-to-read, unble-to-understand, unable-to-follow".

Comment 71

[:aceman]

WADA, it seems your 2 options could be done by my proposal in bug 136502. Then no majority needs to accept only one of the options. Everybody can choose.

Otherwise, how do we determine what the majority thinks? We have no mechanisms for that.

Comment 72

[Magnus Melin [:mkmelin]]

(In reply to :aceman from comment #63)
> 1. OK, it seems WADA and Thomas want this interpretation ("plain text" in AB
> contact means "always plain, not just prefers if possible"), I can look at
> implementing that here and fix the wording accordingly.
 
> Magnus, can you confirm if you agree with 1.?

I don't think that's reasonable. If you composed a mail with say lot of inline pictures we shouldn't convert that to plain text just because of a pref. Like I wrote before, "If it's convertible we convert to plaintext, if not we don't.". But this bug is so wordy so it's really hard to make any sense of it.

Comment 73

[:aceman]

Yes, so can you accept the patch as is (it should be on the safe side to only convert when really all are plain)? And then we should make new bugs to finetune it for other cases?

Comment 74

[Magnus Melin [:mkmelin]]

No because you have "allPlain = false;" for unknown, so you'd ignore convertibility in practice.

Comment 75

[:aceman]

Yes for unknown or a mix we send both. That seems safe to me.

So then why do we force HTML for pure HTML recipients even if message is convertible?

Comment 76

[WADA:World Anti-bad-Duping Agency]

(In reply to :aceman from comment #71)
> Everybody can choose.

It's good for me too, as far as (a) "Send Options panel size" is "hight/width < physical window size of my PC", and (b) "used font size at panel is not too small(text is visible/readable for me. please don't force 1pt or 1px), and (c) statements in panel is understandable for me, and (d) structure of options in the panel is not too complicated and never too deep(please avoid "advanced button", "multiple windows/tabs", "horizontal scroll bar/virtical scroll bar is needed always even when large physical window size", in Send Options).
Please keep "simple is best" at as many places as possible.

I think "If preference=Unknown or not defined in Address Book : consider it [ Text , HTML ]" like one can reduce required variations of option in Send Options panel(this is an answer to proposal in bug 584363. note: I couldn't reproduce bug 584363 yet). I think this kind one will immediately make some bugs WONTFIX or WOKSFORME.

Comment 77

[WADA:World Anti-bad-Duping Agency]

(In reply to :aceman from comment #71)
> Everybody can choose.

If so, how about next? Tb user can select any action on any condition freely.
Therefore, this bug will be resolved without your patch proposed to this bug.

What's wrong in this proposal?
Panel size shortage? Both Magnus and Tohmas D. rejects this? :-) 

                 
[1. Send Logic of Tb]
> 1. Apply default of message preference of recipient.
> 2. Set allHTML and allPlain.
> 3. Do actions defined in following table.
>    Pretty simple. Allways Go to "Use Send Options step!.  
> ------  ------------------     ---------------------------------------  
> CaseID  Taken Action by Tb     <------ Corresponding Condition ------>  
>                                                   aConvertible
>                                allHtml  allPlain  ==Convertible::Plain
> ------  ------------------     -------  --------  --------------------  
> (A-1)   UseSendOptions         true     false     true                  
> (A-2)   UseSendOptions         true     false     false                 
> ------  ------------------     -------  --------  --------------------  
> (B-1)   UseSendOptions         false    true      true                  
> (B-2)   UseSendOptions         false    true      false                 
> ------  ------------------     -------  --------  --------------------  
> (C-1)   UseSendOptions         false    false     true                  
> (C-2)   UseSendOptions         false    false     false                 
> ------  ------------------     -------  --------  --------------------  
 
[2. Send Options panel] 
> (Default of message preference in Address Book) 
>  If preference=Unknown or not defined in Address Book, consider it :
>     [ x ] HTML Domain recipient
>     [   ] Text Domain recipient
> 
> (User option setting table)
>  Because of table, it can be put in current Send Options panel. 
> ------  ---------------------    ---------------------------------------
> CaseID  Check an option          <------ Corresponding Condition ------>
>                                                     aConvertible
>         Ask  Both  HTML  Text    allHtml  allPlain  ==Convertible::Plain
> ------  ---  ----  ----  ----    -------  --------  --------------------
> (A-1)    x                       true     false     true                
> (A-2)        x                   true     false     false               
> ------  ---  ----  ----  ----    -------  --------  --------------------
> (B-1)              x             false    true      true                
> (B-2)                    x       false    true      false               
> ------  ---  ----  ----  ----    -------  --------  --------------------
> (C-1)        x                   false    false     true                
> (C-2)    x                       false    false     false               
> ------  ---------------------    -------  --------  --------------------
> Note: Because Unknown is converted to one of HTML or Text by default choice,
>       ( allHtml==false && allPlain==false ) === ( Mix of HTML and Text ) 
> 
> (HTML Domain/Text Domain setting Table)

Comment 78

[WADA:World Anti-bad-Duping Agency]

(In reply to Magnus Melin from comment #72)
> But this bug is so wordy so it's really hard to make any sense of it.

It's not by me==Bug opener.
Please look following table in comment #54.
> (Revised) Check result of Tb's behavior : before patch and after patch, *** with Send Options=Ask ***
> 
>             aConvertible                                         (SF::HTML      = SendFormat::HTML     )
>             ==                                                   (SF::PlainText = SendFormat::PlainText)
>     allHtml Convertible allPlain  Before patch   After patch
>             ::Plain                             (* = changed by patch)
>  0. true     N/A        false     SF::HTML        SF::HTML       =>  sent as text/html
>  1. false    true       true      SF::PlainText   SF::PlainText  =>  sent as text/plain 
>  2. false    true       false     SF::PlainText  *UseSendOptions => *Asked. Problem is resolved
>  3. false    false      true      SF::PlainText  *UseSendOptions => *Asked. Improved for users, bad for me
>  4. false    false      false     UseSendOptions  UseSendOptions =>  Asked

(In reply to Magnus Melin from comment #74)
> (In reply to :aceman from comment #73)
> > Yes, so can you accept the patch as is (it should be on the safe side to
> > only convert when really all are plain)? And then we should make new bugs to
> > finetune it for other cases?
> 
> No because you have "allPlain = false;" for unknown, so you'd ignore convertibility in practice.

Magnus, which case of above table are you talking about?
Changed case by aceman's current patch is Case-2 and Case-3 only of above table. Nothing is changed in other cases of above table by aceman's patch.
allPlain==false(&&allHtml==false) case is Case-2. This is mixed case. When Case-2, Send Options is always applied after aceman's patch.
Do you say as follows?
  Because Unknown is involved, mail composed in HTML mode should not be sent as text/plain merely by
  aConvertible==Convertible::Plain, merely by user's Text Only setting in Send Options.
  Because Unknown is involved, it should be sent as HTML mail always, regardless of aConvertible value,
  regardless of user's Text Only setting in Send Options setting.
If so, what should be done if user sets Ask option and replied "Text Only"?
How about Delivery Format=Text Only case?

Please note that "message preference in Address Book" == "per mail address HTML Domain or Text Domain setting which is held/saved/kept in Address Book".

Comment 79

[Magnus Melin [:mkmelin]]

(In reply to :aceman from comment #75)
> Yes for unknown or a mix we send both. That seems safe to me.

I don't agree, we have to be able to trust the isConvertible. If it's convertible there's little point in sending anything but plaintext.

> So then why do we force HTML for pure HTML recipients even if message is
> convertible?

Because they actually opted into it?

Comment 80

[Magnus Melin [:mkmelin]]

(In reply to WADA from comment #78)
> Do you say as follows?

I think the opposite. If uknown, send as plaintext if possible (and if not possible, use html).

Comment 81

[Thomas D. (:thomas8)]

(In reply to Magnus Melin from comment #80)
> (In reply to WADA from comment #78)
> > Do you say as follows?
> 
> I think the opposite. If uknown, send as plaintext if possible (and if not
> possible, use html).

Magnus, it's fair if you want that behaviour for yourself, but what makes you think you can *force* that down-conversion onto *all* users without offering opt-in or opt-out? Also, you're pretty much the only one...

Comment 82

[WADA:World Anti-bad-Duping Agency]

Even if aceman's patch which is relevant to Auto-Detect is attached to this bug by aceman, this bug is request for "msgcompose.background_color=#FEFEFE in mailnews.js" since initial.
So, many UX bugs is irrelevant to this bug. Removing Blocks: 889315.

Comment 83

[WADA:World Anti-bad-Duping Agency]

(In reply to Magnus Melin from comment #80)
> (In reply to WADA from comment #78)
> > Do you say as follows?
> 
> I think the opposite.

I'm still slightly unclear about what you say on which case/which condition. 
  If unknown,     : == allHtml==false && allPlain==false case == mixed case
                  : For Unknown only case?
                  : or For pure mixed case with Unknown? (HTML+Unknown, Text+Unknown, HTML+Text+Unknown)
                  :   Proposal of Default preference in Send Options is to rule out ambiguos this mixed case.
                  : How about HTML+Text case?
  (A) send as plaintext if possible, 
        : Case-2 allHtml==false && allPlain==false && aConvertible==Convertible::Plain
        : Right?
  (B) and if not possible, use html.
        : Case-4 allHtml==false && allPlain==false && aConvertible!=Convertible::Plain
        : Right?

Does "html" in "use html" mean text/html? or multipart/alternative? or any of them?(user can request via Send Options)
Is "send" / "use" in "send as plaintext" / "use html" whose action?
Tb user? Code of Thunderbird? Developer of Thunderbird?
Does it implies "SHOULD send/SHOULDuse", "MUST send/MUST use"?

Behavior of your (A) is same as current Tb's code in Case-2.
This is the cause of following phenomenon:
  (C) Thunderbird somehow sent his HTML mail as text/plain by unknown reason,
      even though he actually composed his mail in HTML mode using Thunderbird.
And (C) is start point of many bugs relevant to Auto-Detect.

What do you think about phenomenon of (C)?
Is "Silently send mail composed in HTML mode by user as text/plain, without any user's request or any user's permission" proper behavior as mailer which supports HTML mail composition?
Tb user has to set Text or HTML in all contacts in all Address Books?

I believe that "Number of Tb users who confused due to (C)" is already far larger than "Number of Tb users who could avoid HTML vs Text War by (C)".

Comment 84

[:aceman]

(In reply to Magnus Melin from comment #79)
> (In reply to :aceman from comment #75)
> > Yes for unknown or a mix we send both. That seems safe to me.
> 
> I don't agree, we have to be able to trust the isConvertible. If it's
> convertible there's little point in sending anything but plaintext.
We can't trust it yet and Thomas even claims we can't trust it ever (as it is theoretically not possible).
It is probably the question of how much we can afford to loose in the conversion.
I've got no replies yet in bug 414299 so we can't move this trust issue forward.

> > So then why do we force HTML for pure HTML recipients even if message is
> > convertible?
> 
> Because they actually opted into it?
No, they just prefer it (per current wording). The plain text recipients also "opted into" plain text and we sometimes send them Both.

Somehow we do not cover the case where plain text recipients only want plain or their client will choke on both (multipart). That was covered before the patch.

Comment 85

[:aceman]

(In reply to WADA from comment #83)
> I'm still slightly unclear about what you say on which case/which condition. 
>   If unknown,     : == allHtml==false && allPlain==false case == mixed case
>                   : For Unknown only case?
>                   : or For pure mixed case with Unknown? (HTML+Unknown,
> Text+Unknown, HTML+Text+Unknown)
>                   :   Proposal of Default preference in Send Options is to
> rule out ambiguos this mixed case.
>                   : How about HTML+Text case?
>   (A) send as plaintext if possible, 
>         : Case-2 allHtml==false && allPlain==false &&
> aConvertible==Convertible::Plain
>         : Right?
I could agree with this.

>   (B) and if not possible, use html.
>         : Case-4 allHtml==false && allPlain==false &&
> aConvertible!=Convertible::Plain
>         : Right?
I could agree with this, but send Both (not only html)

> Behavior of your (A) is same as current Tb's code in Case-2.
> This is the cause of following phenomenon:
>   (C) Thunderbird somehow sent his HTML mail as text/plain by unknown reason,
>       even though he actually composed his mail in HTML mode using
> Thunderbird.
> And (C) is start point of many bugs relevant to Auto-Detect.
I think many of these cases in (C) are due to the bad detection of what is convertible and what is not. But that is sorted out in bug 414299.

Comment 86

[:aceman]

(In reply to Thomas D. (currently busy elsewhere; needinfo?me) from comment #81)
> Magnus, it's fair if you want that behaviour for yourself, but what makes
> you think you can *force* that down-conversion onto *all* users without
> offering opt-in or opt-out? Also, you're pretty much the only one...

I offered exactly this option in the UI to choose between Magnus' behaviour and your behaviour (ignore convertibility) in bug 136502 (and that option would stick for the user permanently). But you personally were against it in that bug.

I'm starting to be lost in what you guys actually want. All three of you have 3 different ideas of what should happen.

Comment 87

[Thomas D. (:thomas8)]

WADA, STOP. Please don't start the Ben Bucksch way of removing links to other bugs with valuable information just because you personally don't like it; I'm really allergic to that. If there's any other pragmatic/strategic reasons for wanting to remove the link, let me know via private mail, I might accept that but personally I'm less concerned about it these days.

This bug is about send options; in fact, even though the UI says nothing about it (another bug), these send options ONLY apply when using delivery-format auto-detect. The current wording of these send options does NOT match the actual behaviour at all; which gives confusion to our users (hence "send options lies" in the summary of this bug). Send options directly influence delivery format. Bug 889315 is meta bug where all delivery format UX issues are tracked:

Bug 889315 - [Meta] Tracker bug for delivery format UX issues (HTML vs. Plaintext; incl. interaction of {Delivery Format | Auto-Detect} with other settings and related UX-failures)

So this bug is definitely a delivery format UX issue, and therefore should be correctly included in the "depends on" list of meta bug 889315.

Comment 88

[WADA:World Anti-bad-Duping Agency]

(In reply to Thomas D. (currently busy elsewhere; needinfo?me) from comment #87)

OK.
I stop to touch this bug any more since now, as I had stopped to touch Bug 1202276, because this bug is already sufficiently hard-to-read, hard-to-understand, hard-to-follow for me(rather, near to impossible-to-read, impossible-to-understand, impossible-to-follow for me).
Please change any field of this bug including bug summary, User Story etc. freely.

I hope aceman will open crisp, never-hard-to-read/never-hard-to-understaand/never-hard-to-follow bug, and will propose patch for proper solution in the crisp bug. 
And I hope aceman's patch will be accepted by developers such as Magnus and will be landed.

By the way, please note that there is no difference between Bug 1202276(opened by you) and this bug(opened by me) except requested action in bug by bug opener.
  Bug 1202276 : same code as Alwys HTML addn in Tb
  This bug    : msgcompose.background_color=#FEFEFE in mailnews.js
As I wrote in comment #21, these are essentially same workaround of exactly/absolutely same phenomenon in Thunderbird which was produced by absolutely same code in Thunderbird. So, start point is absolutely same and requested change is essentially same.

And, please note that;
  Why I opened this bug was :
    I saw some bugs in dependency tree for meta bug 889315, which were opened by you recently.
In two such bugs, I stopped to touch any more.
Have you read and understood my comment #48 and my comments pointed in it? (comment #11 and comment #29)

Anyway, Bye.

Comment 89

[:aceman]

Guys, can you please try the build in bug 414299 and comment on its function there?

Bug 1202276 wants to fix the incorrect determination of convertibility by disabling it (I want to do that same in bug 136502, but based on a pref).

The bug here specifies that Send options are not obeyed when they should (at least depending on their wording in the Prefs). That happens e.g. in the allPlain=true, regardless of convertibility.

So I don't think they are the same.

I want to fix the bug here, as it is part of the problem. Then we should fix bug 414299. Then we will see if anything else actually remains. If anybody wants the "don't check convertibility and force HTML if composed in HTML" then I offer a toggle for that in bug 136502. I don't see any more issues.

Comment 90

[Thomas D. (:thomas8)]

(In reply to Magnus Melin from comment #80)
> (In reply to WADA from comment #78)
> > Do you say as follows?
> 
> I think the opposite. If uknown, send as plaintext if possible (and if not
> possible, use html).

Magnus, there are several problems with this approach:

1) You still assume (for the limited case of isConvertible==yes) "prefers-unknown"=="prefers-plain". But LOTS of users have complained exactly about this, even for scenarios "without" formatting. They just expect auto-detect to send HTML for unknown cases, or both plain+html, according to personal preference. All 5 duplicates and 17 votes on bug 136502 cover exactly this case of "no formatting". So there are real users out there who want prefers-unknown==prefers-html. Is there anything wrong with giving them a simple choice as both WADA and I have suggested? (single dropdown field in options:
Consider recipients whose preference is unknown to prefer: plain | html | both?)

2) You are understimating what our users consider "formatting". Again, there's plenty of evidence across many bugs that many users want even the slightest hint of formatting to be preserved when sending. For example, please look at Bug 314213, 11 duplicates, 9 votes, where users are complaining that when they reply in HTML to a plaintext message, during composition we show some vertical spacing between the quotes and the reply text, and after sending, that spacing is gone because we decide on behalf of the user that the message should be sent as plaintext even though composed in HTML. Yes, we can try to fix it by inserting empty lines into plaintext, but the net effect of converted plaintext will never be the same as clean HTML. Compare:

<p>my first paragraph</p>
<p>my second paragraph</p>

(This will result in nicely formatted default paragraphs, variable font, moderate vertical spacing in most clients)

vs.

my first paragraph\n
\n
my second paragraph

(The logical information about paragraph entities is all but lost; plaintext can easily result in poorly formatted paragraphs with fixed font, too much vertical spacing from extra blank lines)

3) Auto-Degrade was introduced in a historical context which no longer exists.
- When HTML mail was still new, it was hated by some users. It was also causing problems for many old clients who were not yet able to cope with HTML or at least multipart/alternative. HTML mail was bigger which caused speed problems on 56K Modems, and capacity problems on Email accounts with capacities of 50MB etc. So TB took a design decision to prefer plaintext whenever possible, and offer opt-in downconversion where not possible. All of these problems are practically a thing of the past for the vast majority of users. Only TB's enforced bias for plaintext has still survived.
- Every downconversion comes with dataloss of formatting (no matter how little, even logical dataloss as demonstrated above in this comment), so TB introduced send options to control the dataloss (unfortunately not for the dataloss of isConvertible==true, bad dataloss now, less dataloss if we fix the algorithm). To effect delivery-format plaintext as often as possible (i.e. to enforce the plaintext bias), we used to ASK the user if downgrading is OK.
- Now unfortunately users complained that we were asking TOO often, and even when there was "no significant formatting". So to avoid the question which results from our own bias for plaintext, Auto-Degrade before Auto-Detect was introduced (always send plaintext for IsConvertible::Plain, ignore any other user preferences). Interestingly, we have stopped asking entirely and instead moved towards preferring html+plain; but unfortunately the move was only half-hearted (deliberate plaintext-bias? or maybe even because developers did not understand the net results of all settings involved?).
- Auto-Degrade (isConvertible==plain) has a bias for "unknown"=="prefers-plain". As WADA and I and many others have pointed out, times have changed and for the majority of users, "unknown"=="prefers-html" or "prefers-both" is perfectly reasonable these days.
- Auto-Degrade (isConvertible) algorithm was never adapted to reflect new ways of formatting like inline styles, styles in header for unmarked tags, other formatting attributes, etc. So over time, especially for advanced users, the dataloss potential of the buggy isConvertible algorithm became worse.

So in a nutshell, Auto-Degrade was just a WORKAROUND to avoid ASKING too frequently, and we only asked to enforce our own bias for plaintext (and protect users at the time). We're no longer asking by default, and a single <b> will trigger sending HTML+plain per default settings (so for many, but not all cases, we already gave up the plaintext bias), so what's stopping us from fully accepting the new realities of HTML as an equally valid message format and dumping the Auto-Degrade workaround, at least for default cases where we don't ask?

Comment 91

[Thomas D. (:thomas8)]

(In reply to WADA from comment #88)
> (In reply to Thomas D. (currently busy elsewhere; needinfo?me) from comment
> #87)
> 
> OK.
> I stop to touch this bug any more since now, 

I never asked you to stop touching the bug.

> I hope aceman will open crisp,
> never-hard-to-read/never-hard-to-understaand/never-hard-to-follow bug, and
> will propose patch for proper solution in the crisp bug. 
> And I hope aceman's patch will be accepted by developers such as Magnus and
> will be landed.

I think we are very close to significant improvements.

> By the way, please note that there is no difference between Bug
> 1202276(opened by you) and this bug(opened by me) except requested action in
> bug by bug opener.
>   Bug 1202276 : same code as Alwys HTML addn in Tb
>   This bug    : msgcompose.background_color=#FEFEFE in mailnews.js
> As I wrote in comment #21, these are essentially same workaround of
> exactly/absolutely same phenomenon in Thunderbird which was produced by
> absolutely same code in Thunderbird. So, start point is absolutely same and
> requested change is essentially same.

Maybe it would help if you stated more clearly that the design concept of Auto-Degrade before Auto-Detect is just nonsense, and we both agree on it.

> And, please note that;
>   Why I opened this bug was :
>     I saw some bugs in dependency tree for meta bug 889315, which were
> opened by you recently.
> In two such bugs, I stopped to touch any more.
> Have you read and understood my comment #48 and my comments pointed in it?
> (comment #11 and comment #29)

No, to be honest, I did not understand comment 48. I guess it's something strategic because one or two developers are stubbornly ignoring the results of our UX analysis based on many, many bugs filed by users, and ignoring UX-principles at random, and failing to understand our proposals in the jungle of delivery-format settings.

Please stop blaming me for making bug hard-to-read etc. You've added just as many long and repetitive comments which are clearer than mine in analysis, but harder in language style. And the reason why both of us keep adding such comments is that some other involved parties don't read our initial comments carefully enough, and don't understand the problem, nor the suggested solutions, nor the ux-principles involved; AND our very specific questions and proposals are never sufficiently, honestly, and completely answered.

I think Aceman has a very good understanding so hopefully he'll navigate some of this.

Comment 92

[:aceman]

Let's add in Ian into the mix here (Seamonkey will also be affected by this), maybe he gets some great idea how to solve the disagreement here :)

Comment 93

[:aceman]

(In reply to Thomas D. (currently busy elsewhere; needinfo?me) from comment #90)
> 1) You still assume (for the limited case of isConvertible==yes)
> "prefers-unknown"=="prefers-plain". But LOTS of users have complained
> exactly about this, even for scenarios "without" formatting. They just
> expect auto-detect to send HTML for unknown cases, or both plain+html,
> according to personal preference. All 5 duplicates and 17 votes on bug
> 136502 cover exactly this case of "no formatting". So there are real users
> out there who want prefers-unknown==prefers-html. Is there anything wrong
> with giving them a simple choice as both WADA and I have suggested? (single
> dropdown field in options:
Yeah, I've read the 5 dupes and yes, all the users want to preserve the tiniest formatting (e.g. proportional font, which I think may be wrong expectation as the recipient can override that).
So if Magnus wants to keep the autodetection in, the only option I see is forcing the HTML e.g. by the setting I propose in 136502.

> Consider recipients whose preference is unknown to prefer: plain | html |
> both?)
This could also work.


> <p>my first paragraph</p>
> <p>my second paragraph</p>
> 
> (This will result in nicely formatted default paragraphs, variable font,
> moderate vertical spacing in most clients)
> 
> vs.
> 
> my first paragraph\n
> \n
> my second paragraph
> 
> (The logical information about paragraph entities is all but lost; plaintext
> can easily result in poorly formatted paragraphs with fixed font, too much
> vertical spacing from extra blank lines)
Yes, I understand this. Some users may want to preserve even this bit of formatting. I proposed the option for them to get this behaviour (just force HTML if composed in HTML). But you don't like it because you want to remove autodetect (at least as long as it contains downgrade in some case). So what if autodetect was not the default?
Magnus?

Comment 94

[Thomas D. (:thomas8)]

(In reply to :aceman from comment #93)
> > Consider recipients whose preference is unknown to prefer: plain | html |
> > both?)
> This could also work.

I think it's reasonably easy and makes everything much more user-controllable/predictable than now.

> > <p>my first paragraph</p>
> > <p>my second paragraph</p>
> > 
> > (This will result in nicely formatted default paragraphs, variable font,
> > moderate vertical spacing in most clients)
> > 
> > vs.
> > 
> > my first paragraph\n
> > \n
> > my second paragraph
> > 
> > (The logical information about paragraph entities is all but lost; plaintext
> > can easily result in poorly formatted paragraphs with fixed font, too much
> > vertical spacing from extra blank lines)
> Yes, I understand this. Some users may want to preserve even this bit of
> formatting. I proposed the option for them to get this behaviour (just force
> HTML if composed in HTML). But you don't like it because you want to remove
> autodetect (at least as long as it contains downgrade in some case).

No. I actually want to preserve auto-detect and make it just auto-detect the best data-preservative delivery format, based on recipient preferences and user preferences, but without prior interference/plaintext bias by auto-degrade (or maybe keep auto-degrade only for allPlain case).

In case it has not been noticed, if we implement that simple dropdown where user can pre-define the interpretation of "prefers-unknown", then users like Magnus can just set it to "prefers-plaintext". Then, if we change the scope of send options to only cover "allPlain" case, and still keep auto-degrade for allPlain case (or even make it optional by adding one more option to send options), and the default setting of "html+plain" for the options, then Magnus usecase "when 'no' formatting, and all unknown or plain or mixed without prefers-html, send plaintext" can still work exactly as it does now. Only the rest of us will have a choice.

So Magnus, if there's a way of restoring your usecase with a slightly modified options layout, would that suffice?

> So what
> if autodetect was not the default?

Allowing users to default to another per-message format (account-based pref) helps to work around the failures of auto-degrade-and-detect, but it doesn't change those failures of auto-detect, including the problem of unexpected auto-degrade before auto-detect which is described nowhere in the UI nor documentation afasics.

And the question remains, WHY is plaintext deemed so superior that we would need to *force* users of HTML composition into it, or - if we allow changing the per-message-default - all of them will just switch off auto-detect, the net result of which is probably that plaintext will NEVER get sent even where it would be possible and reasonable.

Comment 95

[Thomas D. (:thomas8)]

(In reply to :aceman from comment #89)

I widely agree with Acemans general strategy.

> I want to fix the bug here, as it is part of the problem. Then we should fix
> bug 414299. Then we will see if anything else actually remains. If anybody
> wants the "don't check convertibility and force HTML if composed in HTML"
> then I offer a toggle for that in bug 136502. I don't see any more issues.

Aceman, to add "delivery format: as composed" which was suggested in that bug is unfortunately useless-ui.
I understand using that format effects: "Always HTML" for all compositions in HTML editor, and "Always Plain" for all compositions in plaintext editor. The latter is meaningless because we already do that now and plaintext editor can only produce plaintext. "Always HTML" for HTML can be realized by simply allowing user to default the per-message deliveryformat to something other than auto-detect, which includes "HTML only", so that's covered in the original/simpler intention/proposal of the bug.

Comment 96

[:aceman]

(In reply to Thomas D. (currently busy elsewhere; needinfo?me) from comment #94)
> Allowing users to default to another per-message format (account-based pref)
> helps to work around the failures of auto-degrade-and-detect, but it doesn't
> change those failures of auto-detect, including the problem of unexpected
> auto-degrade before auto-detect which is described nowhere in the UI nor
> documentation afasics.

Because the auto-degrade is a feature, not a failure (after bug 414299 improves the detection), for some users (e.g. Magnus, heavy NNTP users, even me (I do not case if the reply header gets down-converted, I actually never compose in HTML in the first place :) So I am not hit by this bug, I just can understand there is some perceived loss for some users.))

So I'm fine with preserving that option, we just need to decide where it should reside, if it is the default and what we can automate so that not half of our users need to set 5 options at TB install.

Comment 97

[:aceman]

Actually I do not understand why plain and unknown recipients would be a different category if the message is convertible. Is it just because the downgrading loses a tiny bit of formatting and you want to preserve that for the unknown users?

Comment 98

[Magnus Melin [:mkmelin]]

I think that's what he says yes. In any case I think the magnitude of the problem is very much exaggerated. Like the cited "5 duplicates and 17 votes on bug 136502". That's NOT a lot for a 13 year old bug!

There's not a lot to do here I think. It's fine to offer a choice to disable conversion to plaintext. Sending both I think has a lot of unknown risk + would still break for cases like mailing lists that require plain text only. It's also fine to improve the convertible detection, but all of this already have their own bugs. So let's fix bugs that actually matter, and don't try to fix things that aren't really a problem.

Comment 99

[Dan Pernokis]

I've only read some of the comments here (although I've read many of the related bugs too), so forgive me if this comment is a little late or out pf place.

The issue of downgrading to plaintext when not absolutely necessary has this downside:
The recipient *might* want to REPLY or FORWARD with highlighted parts of the original text (bold, colour, etc), or insert his own formatted text.  I'm not sure all HTML recipients across all email clients would be able to do this if the message is now plaintext.

Comment 100

[:aceman]

(In reply to Dan Pernokis from comment #99)
> The recipient *might* want to REPLY or FORWARD with highlighted parts of the
> original text (bold, colour, etc), or insert his own formatted text.  I'm
> not sure all HTML recipients across all email clients would be able to do
> this if the message is now plaintext.

We haven't changed anything here, the patch is not accepted.
But I am sure the recipient can enable HTML mode in his mail client even if the message is plaintext. You can also do that in TB via Shift+click on Reply (if your default compose mode is plain text).

Comment 101

[WADA:World Anti-bad-Duping Agency]

(In reply to Magnus Melin from comment #98)
> It's fine to offer a choice to disable conversion to plaintext.

Magnus, thanks for this answer.
"disable conversion to plaintext" means "ignore aConvertible which is set by gMsgCompose.bodyConvertible()".
So I asked whether "aConvertible which is set by gMsgCompose.bodyConvertible()" can be ignored or not under some conditions, for example, by user option of "he wants to send in HTML" as done by explicit "Delivery Format=HTML".

For some peoples, following (X) is equal to following (Y).
  (X) Bugs in dependency tree for meta bug 889315 is kept as Open for long time,
      and no bug is changed to FIXED for long time.
  (Y) Developers rejects fixing of, or ignores, or doesn't understand,
      many bugs in dependency tree for meta bug 889315.
Proposition　(X) is absolutely Truth.
By your answer, "proposition　(Y) is absolutely False, is never Truth" has been proven.
Therefore, frustration in some peoples due to (X) will disappear :-) 

If "disable conversion to plaintext by user" will be officially possible, "HTML post vs Text post War in News" may arise, so protection from it should be considered.
I think following can be an answer/solution for both "HTML mail vs Text mail War in mail" and "HTML post vs Text post War in News".
  Send Options panel
    If recipient is Unknown or not defined in Address Book, consider it :
       Email : [ HTML , Text ] 
       News  : [ HTML , Text ]
By this, Unknown disappears, then mixed case == "HTML+Text" case only.

Anyway, thanks for closing this bug(==request of msgcompose.background_color=#FEFEFE in mailnews.js ASAP) as WONTFIX by developer.
By this action, I believe that at least one bug for following can be closed as WONTFIX.
  Essentially same request for workaround in Tb as this bug
  for exactly/absolutely same phenomenon in Tb as this bug
  which is produced by absolutely same code in Tb as this bug.

Comment 102

[WADA:World Anti-bad-Duping Agency]

(In reply to Magnus Melin from comment #98)
> It's fine to offer a choice to disable conversion to plaintext.

FYI.

A problem in bug processing for report of "when I composed mail in HTML mode, sent as text/plain" :
  Such bug was usually closed as dup of bug 414299, immediately, quickly, or sooner or later.
  Then, "what is problem", "what is solution", discussion, always goes to logic of gMsgCompose.bodyConvertible().
    Problem == HTML format loss, instead of Problem == sent as text/plain even though composed in HTML mode.
> bug 414299 Format | Auto-Detect: Messages composed as HTML with <tt> for "fixed width" sections or with simple links get sent as plain text
As written in bug 674184 comment #4, bug 414299 is surely about "<tt> and friends".
> This is not a DUP. The other bug is about <tt> and friends, not about styles.
> This is a different case. I have no objections in principle here.
Bug 603474 is a dup of bug 414299, and it looks report for same phenomenon as STR of this bug - No format case.
Reporter never shouts "Format Loss!!!". Simply says "sent as text/plain".
> Bug 603474 HTML-composed email gets sent as plain text

If considered as "problem==sent as text/plain even though user composed in HTML mode" instead of "Format Loss!!!", I believe that solution like "user's option to send as HTML mail", "default of messaage preference in Addres Book" etc. was proposed sooner or later at somewhere, even if it's not quickly fixed/closed because Severity/Priority is never high and number of victims is not so large.

Comment 103

[WADA:World Anti-bad-Duping Agency]

Anyway, by acemn's patch atached to this bug, as seen in Table in User Story of this bug, we could know about at where of Thunderbird "phenomenon of (2-2) in User Story of this bug" is generated.

Comment 104

[WADA:World Anti-bad-Duping Agency]

What is needed next is "fill 6 cases in following table", without lengthy words.

  It sounds for me that Mugnus says as follows.
    In (B-2), SF::PlainText should be executed as default action of Tb, as currently done,
    because News Post case should be always considered in mailer.
    However, it can be optional.
    In both (A-1)/(A-2), SF::HTML is sufficient and best, because all recipients are HTML Domain.
    There is no need of options in (A-1)/(A-2).
    In (A-1)/(A-2), if send format is not optional,
    multipart/alternative should be avoided, because unwanted/unexpected problem may occur.

> Note: Current Action in (B-2) is cause of user's confusion,
>       cause of following funny phenomenon == mystery for user.
>          Even though user actually composed mail in HTML mode,
>          the mail was sent as plain/text by Tb silently, with unknown reason.
> 
> ------  ----------------  --------------------------------------  ----------------------------------------
> CaseID  Action by Tb      <------ Corresponding Condition ----->
>           Should be                        aConvertible                             (* = chnged by patch)
>             user option?  allHtml allPlain ==Convertible::Plain   Current Action  Action proposed by patch
> ------  ----------------  ------- -------- --------------------   --------------  ------------------------
> (A-1)    ???              true    false    true                   SF::HTML         SF::HTML           
> (A-2)    ???              true    false    false                  SF::HTML         SF::HTML           
> ------  ----------------  ------- -------- --------------------   --------------  ------------------------
> (B-1)    ???              false   true     true                   SF::PlainText    SF::PlainText      
> (B-2)    ???              false   true     false                  SF::PlainText   *UseSendOptions     
> ------  ----------------  ------- -------- --------------------   --------------  ------------------------
> (C-1)    ???              false   false    true                   SF::PlainText   *UseSendOptions     
> (C-2)    ???              false   false    false                  UseSendOptions   UseSendOptions     
> ------  ----------------  ------- -------- --------------------   --------------  ------------------------
> 
> Note: If Unknown is converted to one of HTML or Text by user's default choice,
>       Unknown disappears, so useless discussion on Unknown disappears.
>         (allHtml==false&&allPlain==false) === (HTML + Text)

Comment 105

[WADA:World Anti-bad-Duping Agency]

Last question to Magnus in this bug.

(In reply to Magnus Melin from comment #98)
> Resolution: --- → WONTFIX

My expectation/want was:
(A) Closed as WONTFIX.
(B) Get proper solution for issue of (2-2) in User Story of this bug.
If current bug summary since initial, which I actually wanted to request by this bug, I can get (A) only, so I intentionally started with partially-inncorrect initial bug summary and partially-incorrect comment #0.
Sorry for such bad behavior at bugzilla.mozilla.org.

Is your following comment "your answer to (B)"?
> It's fine to offer a choice to disable conversion to plaintext.

Comment 106

[WADA:World Anti-bad-Duping Agency]

Sorry, type in comment #104.
(Wrong)
   In (B-2), SF::PlainText should be executed as default action of Tb, as currently done,
   Note: Current Action in (B-2) is cause of user's confusion,
(Correct)
   In (C-1), SF::PlainText should be executed as default action of Tb, as currently done,
   Note: Current Action in (C-1) is cause of user's confusion,

Comment 107

[WADA:World Anti-bad-Duping Agency]

To :aceman.

I think following from Magnus is applicable to your patch for (C-1) of table in comment #104 which is for (2-2) in User Story of this bug.
> It's fine to offer a choice to disable conversion to plaintext.
I believe it's never "applicable only to Disable Delivery Format=Auto-Detect" nor "applicable onlt to other default than Auto-Detect un Delivery Format".

What is proposed by your patch for "(2-2) in User Story of this bug" == "(C-1) of table in comment #104" is user oprion of "Ask, Both, HTML Only, Text Only" in current "Send Options" panel.
I believe that "User's option other than Text Only in Send Options panel" === An "offering a choice to disable conversion to plaintext" in "(2-2) in User Story of this bug".  
So, I believe that your patch has been accepted by Magnus by his following comment.
> It's fine to offer a choice to disable conversion to plaintext.

aceman, can you open crisp bug for "landing your patch" with correct bug summary/correct comment #0?
Oe we are better to open bug like following?
- Please give us option to send mail which I composed in HTML mode as HTML mail, 
  even when Delivery Format=Auto-Detect is used or forced,
  even when gMsgCompose.bodyConvertible() says Convertible,
  even when we forgot to change Unknown to HTML for all contacts in all Address Book,
  even when I mixed HTML recipients and Text recipients and Unknown recipients.
- Please never force text/plain when I intentionally composed my mail in HTML mode, 
  even when Delivery Format=Auto-Detect is used or forced,
  even when gMsgCompose.bodyConvertible() says Convertible,
  even when we forgot to change Unknown to HTML for all contacts in all Address Book,
  even when I mixed HTML recipients and Text recipients and Unknown recipients.

Comment 108

[WADA:World Anti-bad-Duping Agency]

Or following is big problem after your patch?
(a) If user sets HTML in Send Options, mail composed in HTML mode is;
    (a-1) sent as HTML in mail sending, and (a-2) sent as HTML in news posting,
    even when aConvertible==Convertible::Plain.
    If (a-2) occurs, HTML post vs. Text post War may arise.
(b) If user sets Text in Send Options, mail composed in HTML mode is sent as text/plain,
    so problem of (2-2) in User Story of this bug is not resolved.
(c) There is no option for different action betwenn (a-1) and (a-2).
(d) If Ask is set, frequent asking by Tb is annoying for user.

Comment 109

[WADA:World Anti-bad-Duping Agency]

FYI.
I've opened bug 1210244 for enhancement request of comment #107.

Comment 110

[Magnus Melin [:mkmelin]]

(In reply to WADA from comment #105)
Yes I think so.

Comment 111

[WADA:World Anti-bad-Duping Agency]

(In reply to Magnus Melin from comment #110)
> (In reply to WADA from comment #105)
> Yes I think so.

Thanks.

I'll sort out issue and solution in bug 1210244.
My current idea is "aceman's patch" + "User's default choice of Unknown for each Email and News".
If Unknown for Email is interpreted as HTML, and if Unknown for News is interpreted as Text, user can avoid mystery in Email, and can avoid HTML vs. Text War in News.
And, because user's choice for Unknown is shown in Send Options panel, "mystery by UNKNOWN reason" is changed to "mystery by Understandable reason".

This is similar request to existent bug for "Options to kill Auto-Detect" or "Other default than Delivery Format=Auto-Detect" etc. However, there is big difference : No complaint on Auto-Detect, no complaint on Auto-Degrade-to-Text in my request :-)
I hope that bug open for complaint and comment posting for complaint will be reduced.

Comment 112

[Dan Pernokis]

Please forgive that I don't know the functions and code logic being used here to be able to comment fully.  But from a user perspective (someone who has filed bugs on this matter), I do have some concerns -- and I don't understand the discussion well enough to know if the solutions being proposed are OK or not.  [I give thanks & appreciation to Aceman & WADA and others who do!]


If the user creates a message in HTML (formatting mode), either explicitly by an option setting or use of a control, or implicitly by using/invoking any formatting (even if withdrawn/cancelled/undone), then:

(1) Does the message go through as HTML -- that is, in formatted text as the user would expect -- to everyone *except* those explicitly listed as plain-text?

(2) Is the message saved in SENT BOX as HTML for future use & reference, regardless of destination (even if only one plain-text recipient)?

(3) If an in-progress message is saved to DRAFT, closed, and later re-opened, is HTML (formatting) preserved?  (Even if not used yet, user may now want to add formatting.)

(4) If the email is initiated by the Windows SEND-TO feature, or the Internet's MAILTO: feature, is the edit mode considered to be HTML as per my introductory sentence?  (Launch method should not change the format type from normal editing.)

(5) Does formatting disappear (ie - the message is downgraded to plain text) only as a last resort in the process?  (HTML should be attempted in all cases *unless* explicitly told to do otherwise.)

(6) In the case of mixed recipient types, does HTML go out to everyone *except* those explicitly listed as plain-text?  

If the answer is "No" to any of these questions, then I would need to be convinced this is the right approach.  For example, I would want to see HTML in the SENT BOX even in my Case #2 and Case #5 -- because formatting should never be lost to the user regardless of the recipient.


As for News etc, operationally is there any way to know, other than by user configuration of such a recipient, that the recipient is News etc, (ie - that *this* recipient should be plain-text)?  Is a recipient magically flagged as "News etc", or just that he/it should be plain-text?

Comment 113

[WADA:World Anti-bad-Duping Agency]

(In reply to Dan Pernokis from comment #112)
> Please forgive that I don't know the functions and code logic being used here to be able to comment fully.

Read "User Story" of this bug well before posting further comment, please.

> As for News etc, operationally is there any way to know, (snip)

???
Regardless of news poting or mail sending, mail/news post is composed using Identity which is defined for news account or mail account.
News posting is requested by "Newsgroup: NewsgroupName" or "Followup-To: NewsgroupName" at Composer window.
I don't know whether HTML Domain/Text Domain in Send Options is applied to Newsgroup or not, and I don't know whether HTML or Text in Address Book is applied to Newsgroup or not.
But used term is "message" in Domain setting and "Email address" in Address Book.
So I think Send Options/Address Book is not used for News posting.
My idea of "user's default choice of Unknown for News" is:
   Even when user started Composition in HTML mode via. Identity of Email account,
   If Newsgroup is involved in recipients via Newsgroup: or Followup-Tp:,
   Force "send as text/plain" via "Unknown for News==Plain Text Only" in Send Options.

> (4) If the email is initiated by the Windows SEND-TO feature,
> or the Internet's MAILTO: feature,
> is the edit mode considered to be HTML as per my introductory sentence?
> (Launch method should not change the format type from normal editing.)

This bug is for phenomenon after Composer is invoked in HTML mode.
"How to in voke with HTML mode", "When composer is invoked with HTML mode" etc. is absolutely irrelevant to this bug.
If you want to ask for such thing, please ask at appropriate place such as support forum.

As for "SEND-To under MS Win where MAPI is used", following is already known.
  When used identity by "SEND^TO via MAPI of MS Win" has attribute of default composition mode = HTML,
  and when Composer is invoked with HTML mode by it,
  "Save As Draft" saves draft mail as text/plain.
  So, Composer is always invoked in Text mode by "Edit Draft".
This is absolutely different phenomenon from Auto-Detect relevant phenomenon, and is absolutely independent issue from Auto-Detect relevant issues.
Please search bugzilla.mozillaaoeg by yourself well.

Comment 114

[WADA:World Anti-bad-Duping Agency]

(In addition to comment #113)

Auto-complete at To:, Newsgroup: etc. is currently as follows.
(a) When Composer is invoked using Identity for Email account.
    Show contact in Address Book.
(b) When Composer is invoked using Identity for News account.
    Show newsgroup under news account where the used Identity is defined.
I believe Tb user almost always invokes Composer via Identity for News account, and user usually set "default composition mode=Text" in news account setting if user wants Text posting at Newsgroup of the news server, and user usually set "default composition mode=HTML" in news account setting if user wants HTML posting at Newsgroup of the news server.
So I think "deafult choice of Unknown for News" like one is not needed.
Such idea is for edge case, and for protecting from repeated complaints or discussion like "Format loss", "lossless or lossy" and so on.

Comment 115

[Dan Pernokis]

WADA, I don't know specifically what the functions do, but I do know from a user perspective what seems to be working and what doesn't.  I DID read the user story -- or tried to, anyway -- makes sense if one knows the internals, but someone can easily get lost in the weeds and I obviously did.  I often ask "dumb questions" to stimulate thoughts & ideas -- sometimes new & improved thoughts & ideas -- or to create a change in perspective.  Please don't write me off for trying to help.

As for MAILTO, I DID post Bug 1095629 about it -- and you were involved there too -- but this issue has not been resolved AFAIK.  Again, your logic statements may make sense to those who know the internals, but I can't tell if your fixes here are helping or hurting this issue.  In hindsight + your comments, I agree the discussion of MAILTO doesn't belong here.  I'm just trying to make sure fixing or redesigning one aspect of the overall HTML debate doesn't create yet another quirk somewhere else.

As for saving to DRAFT, what sane person would want an HTML-formatted message to come back in plain-text when re-opened?  I'm sorry but imho this is crazy.  Again, not for discussion here, I agree, but it's still part of the overall issue of losing deliberate/explicit formatting. 

I just want TB to work well, and do things predictably, consistently, and correctly.  I don't want to go through rework & tedium (my Bug 1095629) because formatting unexpectedly goes away when it shouldn't, regardless of how I invoked the editor and regardless of what the logic indicates "should" happen.   Fixing individual pieces of the HTML problem is not always the best approach to fixing the whole.

Thanks for your comments & understanding.

Comment 116

[WADA:World Anti-bad-Duping Agency]

(In reply to Dan Pernokis from comment #115)
> WADA, I don't know specifically what the functions do, (snip)

Me too.
Please note following.
Here is bugzilla.mozilla.org for developers to fix bugs, implementing enhancements. Here is not support forum, even if any Tb user can open any bug and add any comment to bugs at bugzilla.mozilla.oeg.  

> I DID read the user story -- or tried to, (snip)

Thanks.
Please ask me about "what of where in the User Story you can't understand" in this bug, if you have a question on the User Story.

Comment 117

[WADA:World Anti-bad-Duping Agency]

(In reply to Dan Pernokis from comment #115)
> As for MAILTO, I DID post Bug 1095629 about it (snip)
> As for saving to DRAFT, (snip)

I think main phenomenon of that bug is.
1. Because "SEND-TO via MAPI in MS Win" is for sending file by attaching files to an simple email,
   MAPI code tries to force Text mode by setting A_Value=true.
   MAPI code looks to have code for "try to invoke composer in HTML mode, if body contains <html>".
2. "Draft Save" used the A_Value.
3. After it, imlementation change was done.
   Mail composition mode is always determined by "composition mode which is set in Identity's setting".
4. Draft Save still uses the A_Value, so draft is saved as text/plain becuse of A_Value=true.
5. If a draft mail is text/plain, there is no way to open the draft mail in HTML mode.
   If a mail is text/plain, there is no way to open composer in HTML mode by Edit As New.
   This is current design/implementation.
Problem is "mismatch among multiple features which are relevant to mail composing/sending".

Needless to say, there is no "formatting related data by HTML tag" in text/plain. So, there is no need to say "format loss" like one repeatedly. "Sent or saved as text/plain when mail is composed in HTML mode" is sufficient.

Comment 118

[Dan Pernokis]

Reply to WADA / Comment 117:  I believe what you say, because it perfectly describes what happens from the user's point of view -- my pov -- that's exactly what I saw.  However...

>> So, there is no need to say "format loss" like one repeatedly. "Sent or saved as 
>> text/plain when mail is composed in HTML mode" is sufficient.

This is just semantics.  You're talking steps and procedures from a programming pov: this happened, that happened, and why.  I'm talking observations from the end-user perspective: I added formatting of some kind, and it disappeared/vanished/went away/was removed/ceased to exist -- in other words, the formatting was "lost".  We're both right, just two sides of the same coin.

The difference is this:  Out of the blue (ie- with no particular reason to look), a sharp programmer or QA person *might* spot a fault in the code, and a systems analyst might see a fault in the logic stream if he/she happened to look.  But a large (millions) user base WILL eventually trip over a problem -- and the bigger the impact, the sooner it gets reported.  So then the pgmrs and QAs and SAs have a reason to look, find the cause, and fix it.

The user can't be expected to see the magic that makes a process happen, and the programmer can't possibly know every single way a user might break something.  I've spent a lifetime writing & debugging code and fixing procedures -- I just don't know this language and don't have the time or talent to delve that far into it.  I can only give a user's perspective (in as clear & technical a way as I can) in the hope that it helps.  So thank you (and the others like you) that can pick up the ball and run with it.

Comment 119

[WADA:World Anti-bad-Duping Agency]

(In reply to Dan Pernokis from comment #118)

Again, here is bugzilla.mozilla.org for developers, not support forum.

"Where format is lost" is step 0 in below.
  0. Tb has both HTML data and Text data which is converted from HTML,
     because there is need to send as multipart/alternative.
  1. Tb decides to send as text/plain, by various reasons, for example, Text recipients only.
  2. Tb sends as text/plain.
  3. Because of text/plain, "formating made by you using Tag, Style" is not transfered to recipients.
  4. This is essentially irrelevant to recipients,
     because recipients merely received text/plain mail.
     No one can know about your hard job for formting HTML mail utilizing Tags or Style,
     *WITHOUT USING ANY OF HX, P, INSERT LINK/IMAGE/TABLE, COLOR/FONT/BOLD/ITLIC ETC. OF THUNDERBIRD*.
  5. After it, you know that "formating made by your pretty hard job" is lost at mail recipients. 

"Where formatting is lost" is "When conversion from HTML to Text is done". This is step 0.
"When HTML data is not transfered to recipients" is "When sent as text/plain". This is step 5.
"Repeated saying of Format Loss!!!" is identical to "repeated saying of that HTML data is lo--st by Tb's conversion from HTML to Text" from perspective of programming or software development. This same as nonsense.
 
What haapens is:
(A) You composed mail in HTML mode.
(B) Tb decided to send as text/plain by various reasons.
(C) So Tb transfered only Text data of "HTML data + Text data converted from HTML data".
(D) All phenomena you saw, such as "HTML Tag/Style was not transfered to recipients",
    "you were aware of that your hard job had done nothing...", 
    is phenomenon after (C).
    This is similar to phenomenon by problem after problem after .... after problem.
"What is needed to be referred" is (A) and (B) and (C) only.
Any phenomenon observed after (A) and (B) and (C) is useless for resolving problem.

Comment 120

[WADA:World Anti-bad-Duping Agency]

By the way, Dan Pernokis, have you viewed message source of HTML mail generated by mail senders using MS's software?
Such mail looks for me same as "Send 'merely several text lines' as huge Word file".
Characteristics of such mails.
  In deeply nested tables with many <TR>s and <TD>s,
  with long style attribute in many tags, with long <style> in <head>,
  with useless HTML comments generated by application,
  text is contained in a few or several <TD>s,
This is Email for commuicatioon between human beings, isn't it?

Comment 121

[Ben Bucksch (:BenB)]

WADA, I would instead suggest to change the foreground color:
msgcompose.background_color = #FFFFFF
msgcompose.text_color = #FFFFFF

Comment 122

[WADA:World Anti-bad-Duping Agency]

If background_color/text_color=#FFFFFF is set in mailnews.js as official option to skip "auto-downgrade", user shouts "text is lo---st!". Then user is exhausted by the shouting, Therefore, user doesn't shout when he see "silently sent as text/html". :-)