414299 - Format | Auto-Detect: Messages composed as HTML with <tt> for "Fixed Width Font use" sections or with <a> of "link text==link url" get sent as plain text

Status: RESOLVED WONTFIX

(MailNews Core :: Composition, defect)

Description

[Nelson Bolyard (seldom reads bugmail)]

Gecko/2008012501 SeaMonkey/2.0a1pre on WinXP home SP2.

With this version of SeaMonkey, I cannot send any html emails.

I bring up the html composer (using shift-click on the compose button).
I select "Body Text" and "Fixed Width" font, then paste in the body of 
a plain text message that includes some URLs.  These URLs do not appear
as links initially.  Then I highlight each of the URLs, one at  a time, 
and for each one, I type Ctrl-L, which is the "Insert Link" shortcut, 
and then enter.  That turns the URL into an anchor tag.  
When I'm done, I click send.  I am not prompted with a question about 
whether to send plain text or html.  It just sends.  

The result: the message sent is a single part message, encoded in text/plain.
It is not a multipart-alternative message with an html part and a plain part.
All traces of HTML have been stripped out.  

I have checked that my addresses are marked in the address book as either 
preferring to receive html email or "unknown", but not preferring plain text.
I have checked that the preference when sending an html message to a user 
who is not known to prefer html is set to ask me what do do, not set to 
automatically send only html or only send plain text, or automatically send
both.

Comment 1

[Magnus Melin [:mkmelin]]

Seems WFM on Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9b3pre) Gecko/2008012803 Lightning/0.6a1 Thunderbird/3.0a1pre ID:2008012803 - I get the send format dialog.

Comment 2

[Nelson Bolyard (seldom reads bugmail)]

I tried setting the preference so that, when sending mail to someone who is 
not known to be able to receive html, it sends both.  Still no joy.

Is there some rule built-in to the composer that, when the email contains 
only text in "body text" style with "fixed width" formatting, and only 
other tags are anchor tags (links), then it will be sent as plain text only?

Is it perhaps enabled by some preference that I can change?  

I really need to send an html email with some links in it, and it's beginning
to look like I will need to use something other than SeaMonkey to do it.

Comment 3

[Nelson Bolyard (seldom reads bugmail)]

This does not appear to be a recent regression.  I tried composing and sending 
an html email with a 6-month old trunk build, and a different profile.  
It sent only plain text.  

Something about the particular email I'm composing, such as the links, or the 
text or the recipient(s) is causing the composer to send only plain text, 
rather than html.  I think I've set all prefs to allow sending of both html 
and plain text, and still it sends only plain text.   

Comment 4

[Mike Schroepfer]

This should block TB3 for sure - and we'd take a core fix in the 1.9 timeframe

Comment 5

[Phil Ringnalda (:philor)]

Certainly seems to not be recent - doesn't it date back to bug 28420 comment 14, in May 2000?

Comment 6

[David Ascher (:davida)]

I may be missing something, but I can send HTML email w/ Tb 3.0apre2.

Nelson, are you still seeing this?  Have you tried w/ Tb?

Comment 7

[Nelson Bolyard (seldom reads bugmail)]

David, 
How fortunate you are to not experience these bugs that I experience daily.  
Do you test in Windows?  

Comment 8

[Phil Ringnalda (:philor)]

Yep, you probably are missing something: you have to do the right things with only the right addresses and the right HTML. Here's some detailed steps, which you can follow in any version of Tb or SM or the Suite (with the default mail.default_html_action of 0, always ask), going back to May 2000.

1. Put one of your own addresses in the addressbook, and say that you prefer to receive HTML.
2. Compose an HTML message (shift-click the Write button if you don't default to HTML).
3. Select your HTML-lovin' address from autocomplete as the only To:
4. In the body, type http://www.mozilla.org/, then ctrl+a to select all, then ctrl+l to insert a link, then enter to create a link with both the href and the link text being the same, http://www.mozilla.org/.
5. Send the message (you'll get no format prompt), then when you receive it view source, you'll see that it was HTML.

Part deux:
6. Change your addressbook card to say that your HTML preference is unknown
7. Repeat steps 2 through 5, except that when you hit step 5 you'll find that the received message is plain text only, just the URL rather than a link.

For good or ill, what Nelson is describing in comment 0 (so long as he wasn't ever sending to only addresses which are marked as preferring HTML, but was always including at least one marked "unknown") is exactly what was intended by bug 28420, and what mailnews has done for eight years. Depending on that "so long as", this bug might be about one of three things that I can think of:

* some way that multiple addresses that all prefer HTML failed to trigger the "send it in HTML even though we think we can convert without information loss" code (or, the rough equivalent, failing to match an address with the card that says it prefers HTML)

* a request to revisit various can of worms about text-only vs. text+html

* a request to revisit which things in an HTML message can be losslessly converted to plain text

(The fourth possible thing, drop the lossless conversion entirely, isn't an option: it was actually added because without it, the html/head/body/p/br of truly plain text written in the HTML composer was prompting every time.)

Comment 9

[Nelson Bolyard (seldom reads bugmail)]

Phil, I think your comment 8 can be summarized by saying that:

1) In html messages, links (anchors) whose string value exactly matches the 
URI in the href attribute can be converted to plain text by dropping the 
anchor tags, and this conversion is considered to be "lossless", and

2) When the user has composed an html message, whose only html features are
those that can be "losslessly" converted from html to text, and the list
of recipients includes at least one whose preference for receiving html mail
is "unknown", that message can and will be *SILENTLY* converted to plain text 
and sent as plain text, without prompting the user to make a conversion 
choice, even when the user's preference clearly demands that he be prompted 
to make that choice.  

Giving the user a preference about this matter, and then choosing to ignore
it, certainly violates the principle of least astonishment.  

Comment 10

[David Ascher (:davida)]

ah, i see.  Am I right that the subject of this bug could then be changed to:

"html emails with nothing but links in them get sent as plaintext when in doubt over recipient preferences"?

This is not a problem if the receiver's email client auto-linkifies URLs, right?

Comment 12

[Phil Ringnalda (:philor)]

Grr. Why can't it ever be simple?

No, but it could be resummarized as "Fixed Width can't be losslessly converted to plain text (except when it can)." Unless he's got a compelling reason we haven't heard yet, the link part of Nelson's problem doesn't excite me much, but the Body Text - Fixed Width part (which was also the issue in the dupe) does. The link thing was easier to turn into STR, but if instead you set Body Text - Fixed Width, then do some ASCII art, then send it to your preference-unknown Gmail address and read it through the web interface, you'll see that our belief that we can losslessly convert <tt> is wrong.

So, that's easy enough, I just moved it down in http://bonsai.mozilla.org/cvsblame.cgi?file=/mozilla/mailnews/compose/src/nsMsgCompose.cpp&rev=1.570&mark=4797,4819#4797 and was trying to decide whether it's really ::Altering or ::No, when davida reminded me that we have UI to set Fixed Width as your default, and so we would be throwing anyone who has back into the pre-28420 "I typed 'George: sure, see you at 5' and it wants me to decide whether to lose information by sending in plain text!" state.

Almost enough to tempt me into reopening the "screw it, HTML won, send everything as HTML-plus-text" can of worms.

Comment 13

[Nelson Bolyard (seldom reads bugmail)]

Phil, I agree, why can't Mail/news simply do what the user's preferences say
it should do?

These are the preference choices available to me (text from the dialog):

> When sending formatted (HTML) messages and one or more recipients are not
> listed as being able to receive HTML:
> 
>  (.) Ask me what to do (Mail prompts you to choose a format)
> 
>  (.) Convert the message to plain text (some formatting may be lost)
> 
>  (.) Send the message as formatted text (HTML) anyway (some mail programs
>      may not be able to display it)
> 
>  (.) Send the message in both plain text and HTML (larger message size)

Now, what you've revealed in this bug is that the first of those choices
actually means "Ask me what to do except when a conversion to plain text 
would result in an acceptably small level of loss".  

IMO, if you want to offer that choice, then there should be a 5th choice
in the list above, that says just what I wrote above.

So, now I think there are two issues here:

Issue 1: When mail/news offers users a choice that says "You're in control", 
then that's what it should actually implement.

Issue 2: The criteria for determining "lossless" conversion obviously need
more work.  

The first of these should DEFINITELY be addressed, IMO.  The second probably
should be addressed.  

I might even consciously choose that "except for lossless conversion choice" 
if I agreed with the definition of lossless.  I consider that links cannot
be losslessly converted.  The whole reason I filed this bug in the first 
place was that the links were lost in the message that was sent.  That's 
not "lossless" by any definition I would accept.  

I normally send only plain text emails.  One of the very few reasons that 
I ever send HTML emails is when I want to put clickable links into those 
emails.  To have those clickable links removed defeats the purpose ENTIRELY!

People who read plain text emails with Thunderbird find that any URLs in 
the plain text get automatically converted to links when they read the mail,
so when they see a URL in the plain text email, they can just click on it.
Perhaps that is why someone thought that removal of links was "lossless".

But Microsoft mail clients don't do that, and none of the webmail services
that I occasionally use does that, either, so the removal of a link means
that the recipients of the "losslessly" converted emails won't have a link
to click on.  

My experience shows that my correspondents don't get a clickable link in 
their email, they won't attempt to copy-n-paste the URL from the mail body.  
I've actually had correspondents write back to me saying "I don't see any 
links in your email", when I carefully created an html email just to have
links.  It was one such reply that triggered this bug report.  

Comment 14

[Nelson Bolyard (seldom reads bugmail)]

As long as we're discussing undocumented behaviors for sending messages 
composed as HTML, let me ask about another pref.

If I had chosen the pref that says:
>  (.) Send the message in both plain text and HTML (larger message size)

what would have happened?  Would it send both? or would it decide to do a
(not) "lossless" conversion and send only plain text?

Comment 15

[Phil Ringnalda (:philor)]

Found your workaround in the twisted bowels of composition: the one and only time we'll really listen to you is if you express a per-message choice in the Compose window's Options - Format (well, a choice other than Auto-Detect, that means Do What We Feel Like Doing).

Comment 16

[Nelson Bolyard (seldom reads bugmail)]

Options - Format ?  
No such animal in SM trunk :(

BTW, in answer to my question in comment 14, 
I tried it, and it seems that if the pref is "send both", it will.

Comment 17

[Nelson Bolyard (seldom reads bugmail)]

(In reply to comment #16)
> Options - Format ?  
> No such animal in SM trunk :(

My mistake.  It's there in the html composer only.  
Thanks!  It's good to know that this workaround exists.

Comment 18

[Magnus Melin [:mkmelin]]

(In reply to comment #13)
> People who read plain text emails with Thunderbird find that any URLs in 
> the plain text get automatically converted to links when they read the mail,
> so when they see a URL in the plain text email, they can just click on it.
> Perhaps that is why someone thought that removal of links was "lossless".
> 
> But Microsoft mail clients don't do that, and none of the webmail services
> that I occasionally use does that, either, so the removal of a link means
> that the recipients of the "losslessly" converted emails won't have a link
> to click on.  

Even MS (e.g. Outlook) does autolinking of plain text links. And I can't remember any webmails I've had access where naive autolinking of full http urls weren't used - even if the web interfaces were real crappy in other aspects.

Comment 19

[Nelson Bolyard (seldom reads bugmail)]

Outlook (corporate edition) might.  
I'm pretty sure that Outlook Express does not.  
mail.com and comcast.net web mail do not. 

Comment 20

[Phil Ringnalda (:philor)]

Outlook Express does, on XP and 2K. I don't have a copy of Vista to test whatever the new name is, though I might be able to find one. mail.com does, though I wouldn't be shocked to find that their algo misses some links. comcast.net I'm unable to test - everyone I trolled up by hostmask on IRC claimed not to know how to log in, or just to be borrowing the connection.

Comment 21

[Nelson Bolyard (seldom reads bugmail)]

Well, I'm not joking that I send html messages with embedded links to 
various correspondents, and they ROUTINELY write me back saying "there's
no link in the email you sent me".  The emails they send back claim to 
have been sent from OE or comcast.net's webmail or mail.com's webmail.  

So, the upshot is: what SM mail does doesn't work for my correspondents.
If it didn't molest the html, and sent the html as is, I'm sure they'd 
see things differently.

Comment 22

[Magnus Melin [:mkmelin]]

I'm sure there are problems at times, but the fact that there have been no other complaints about this since ever indicates it's a very minor problem globally. 

I don't imagine there are many people writing mails and inserting links manually with the url as link text. (And if the text and link isn't the same, you don't get the problem - which I think isn't a bad workaround.)

Comment 23

[Phil Ringnalda (:philor)]

If the very last bug left in Tb3 was that it does the same thing it did in Tb 0.1 and Mozilla 1.0 with HTML email only HTMLified by having fixed-width text and links with the URL as the link text, no, we wouldn't block the release on changing (even if we were sure we wanted to change).

Comment 24

[rsx11m]

I ran into the issue of Fixed/Variable Width and Preformatted sections not triggering HTML auto-detect as well for the http://gsfn.us/t/16j03 thread.
Some discussion with BenB in bug 136502 comment #48. Technically this would
be bug 385162, but it has been duped against this one here.

Personally I think there shouldn't be any assumptions on what defaults the users have defined for composing or displaying mails. If I compose in Variable Width and intentionally switch to Fixed width for some passages, it's an explicit change which gets lost when sending as plain text only. Also, it cannot be assumed that the recipient views plain-text messages in fixed width, thus preformatting information - even if the text is not subjected to line wrapping during conversion - may get lost on the recipients end and requires switching to fixed width.

Comment 25

[Ben Bucksch (:BenB)]

We needed to limit the amount of times when we pop up the dialog, to not annoy people for every single email, even when it's unnecessary.

We send plaintext without asking, if you have no formatting whatsoever.

Preformat <pre> == plaintext. (Our Plaintext editor is basically the HTML editor with a <pre>, and our plaintext mail display is basically HTML with a <pre>.)
Obviously, if you change some part explicitly to plaintext, that part is not a reason to send HTML :).

Similarly, a link that is equal to the link text is not considered formatting either, otherwise we'd trigger too often (because we also detect and mark links, and so does other software that we quote etc.).

WONTFIX - works as intended

Comment 26

[Ben Bucksch (:BenB)]

(Please also note that the plaintext vs. HTML question is very delicate and *the* flamebait in mail area. Everybody has a certain opinion and is convinced it's right, just that everybody has a slightly different one. This is part of the compromise.)

Comment 27

[rsx11m]

Ok, I'll shut up now, but I still think it's wrong... ;-)

Comment 28

[Nelson Bolyard (seldom reads bugmail)]

Ben, since when did you become a module owner of peer of the relevant code?

Comment 29

[Ben Bucksch (:BenB)]

Since I wrote it.

Comment 30

[Ben Bucksch (:BenB)]

(And one doesn't have to be a module owner to close a bug.)

Comment 31

[M]

Please reconsider the <pre> = plain text assumption.

I often have this problem. When sending code, commands or log file excerpts, or very simple tabular data, I mark it "Fixed Width". But if I don't also use colors, bold or something, the whole message is sent as plain text.

The result is an unreadable mess, because almost nobody uses a fixed width font to display emails. As far as I known, variable width is the default in all email clients, and also what all normal users prefer. Most users aren't even aware they can change this.

Preserving "Fixed Width" when specifically marked as such is important.

It is different for links. If a link text is identical to the link's URL, it's indeed unnecessary to send HTML since all current email clients recognize links and make them clickable.

Comment 32

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #25)
> We needed to limit the amount of times when we pop up the dialog, to not annoy
> people for every single email, even when it's unnecessary.

When multiple mail recipients, and some are HTML capable but some are plain text only?
If so, can current behaviour be optional?
I believe that "option of HTML mode==text/html body or part is ALWAYS sent" and "option of plain text mode==text/plain body or part is always sent" is basic and mandatory behaviour of a mail client. Different behaviour from it should be optional extension for user's convenience.

Comment 33

[WADA:World Anti-bad-Duping Agency]

Current behaviour was OPTIONAL, but the optional behaviour was defaulted for user's convenience(and probably to avoid html vs. text war).
If Option/Format/HTML only or Plain text&HTML is explicitely selected instead of Auto-Detect, HTML body/part was generated as expected even when recipient's preference=plain text.
I've probably understood reason why WONTFIX by Ben Bucksch.

Ben Bucksch (:BenB), is there any way to set default of Option/Format to other than Auto-Detect?

Comment 34

[rsx11m]

No, that's bug 136502 - current behavior is to always default to Auto Detect when composing a message in HTML mode, then to figure out what to do.

Comment 35

[WADA:World Anti-bad-Duping Agency]

A simple workaround by bug 136502 comment #32.
  Use HTML signature with <b>&nbsp;</b>.

Comment 36

[rsx11m]

There are various known workarounds, but that's all what they are.
The question here (and in bug 136502) is if the current behavior is "correct"
(and if it should be customizable by preference setting).

Comment 37

[Ben Bucksch (:BenB)]

I think comment 31 makes a good argument. It seems backwards to use HTML to force old-style fixed width plaintext, though.

Comment 39

[Ian Scott]

I agree with comment 31.

As for comment 37, It doesn't seem backwards to use HTML to force a fixed-width presentation, though, especially when I want to mix variable-width (for prose) and fixed-width (for code) in a single email.

If I explicitly specify that I want both fixed and variable width text in an email, I fully expect the recipient to see my mail as I composed it.

Comment 40

[Ben Bucksch (:BenB)]

> I fully expect the recipient to see my mail as I composed it.

Forget that. Will never be true anyway. Not for plaintext, not for HTML.

Comment 41

[Ian Scott]

(In reply to comment #40)
> Forget that. Will never be true anyway. Not for plaintext, not for HTML.

I certainly understand that client limitations and settings will mean that the recipient won't see exactly what is created in the compose window. But if the user changes styles with the rich text editor, the Auto-detect should allow them be sent out in the HTML. Once some parts of the mail are fixed-width and others are variable-width, it ceases to be just plaintext.

Comment 45

[Thomas D. (:thomas8)]

(In reply to Ian Scott from comment #41)
> Once some parts of the mail are
> fixed-width and others are variable-width, it ceases to be just plaintext.

+1 (me)
+1 from duplicate bug 763855

Ben, the problem is this:
If deliberate alternations of "Variable width" and "Fixed width" are used, what users see during composition is *very* different from what others will receive, because we erase <tt> tag which would force monospace/fixed width font. Instead, we sent whole text with same style, text/plain + format=flowed. Two problems:

1) differences between "variable width" and "fixed width" are wiped

2) Depending on receiver's mail rendering,
- either whole text is displayed as "Fixed width", as in TB (which is odd for sections originally marked as "Variable width" and appearing with "nice"/readable fonts in composition)
- or whole text is displayed as "Variable width", as in some other mailers (which can completely break sections originally marked as "Fixed width" like code etc.)

I wonder if it can be very difficult to treat <tt> tag in same way we treat <b> tag, namely triggering text/HTML, to ensure wysiwyg and respect user's style choices.

Similarly if user deliberately applies links:
To assume lossless conversion of
> <a href="http://www.abc.com">http://www.abc.com</a>
into plaintext
> http://www.abc.com
might still be acceptable because most mail readers will still parse that correctly as a clickable link (what about web mailers? any other known exceptions?).

But it's definitely dataloss when we also convert (as we do! TB13/WinXP)
> <a title="Click me" href="http://www.abc.com">http://www.abc.com</a>
into plaintext
> http://www.abc.com
because we are loosing relevant content on the way (here: the title attribute).

Worse, we can mess up the whole format of the msg if advanced user uses styles:
> <a style="float:right;background-color:red" title="Click me"
> href="http://www.abc.com">http://www.abc.com</a>
is also shreddered into plain text
> http://www.abc.com
which is really unacceptable.

Comment 46

[Thomas D. (:thomas8)]

Do we apply conversion to plain text for any other tags apart from these?
<tt>
<a href...> where link text = src attribute

Comment 47

[Ben Bucksch (:BenB)]

If you use <pre>, your linebreaks will be preserved, even in format=flowed.

Whether the recipient uses fixed width or variable width fonts is his choice and you can't force something on him.

You can always force sending HTML, if you want to define exact formatting (but even then he's free to ignore your formatting).

Comment 48

[Ian Scott]

(In reply to Ben Bucksch (:BenB) from comment #47)
> Whether the recipient uses fixed width or variable width fonts is his choice
> and you can't force something on him.

That's the recipient's prerogative, not Thunderbird's. If the originator specifies markup in his/her email, it should be sent, and the recipient is free to honor it or not.

Comment 49

[WADA:World Anti-bad-Duping Agency]

Removing "format=flowed" from bug summary, because "automatic format=flowed" is internally disabled for some character encodings such as iso-2022-jp etc. due to problems by "format=flowed without DelSp=Yes parameter, and because format=flowed is disabled by mailnews.send_plaintext_flowed=false in any character encoding.

Comment 50

[Ben Bucksch (:BenB)]

It *is* possible to manually say that a message should be sent as HTML or plaintext:
Composer | menu | Options | Format | As Rich Text (HTML) only
(default is auto-detect)
And that's the workaround for this bug, if you really do care about this. The autodetection should only get it right in the majority of cases, not in all cases.

Comment 51

[rsx11m]

As for the "defaulted Options->Format->Auto-Detect is always applied regardless of user's want/expectation, and sadly, there is no way to change the default action" part which WADA added and BenB removed, that's bug 136502 and would indeed cover (and avoid) a lot of discussions like this.

In general, I agree with the notion that if the user applies /any/ formatting on purpose, his or her intention is obviously to retain such formatting to the message, and the auto-detect mechanism should respect that.

Comment 52

[Ben Bucksch (:BenB)]

rsx, are you saying that my workaround in comment 50 does not work currently? I just tried it: mark part of the text as menu | Format | Text Style | Fixed With, then do menu Options | Format | HTML only, send, and it does arrive as HTML only, and it works as you expect.

Comment 53

[rsx11m]

Correct, if you just mark some part and change variable to fixed width it will nevertheless go out as plain text despite explicit formatting. Options | Format | HTML only works of cause, but that's bug 136502 and not really related to the decision auto-detect makes in this specific case.

Comment 54

[Ben Bucksch (:BenB)]

> that's bug 136502

That's what confused me. Bug 136502 is not necessary to use the workaround. That bug is about *always* setting the format. Here, you apply very specific and delicate formatting to one message. If you do that, then you can also set Format | HTML or whatever you want, and it works as you expect.

Comment 55

[rsx11m]

Point taken, the question remains though why auto-detect doesn't consider <tt> as "sufficient formatting" for the purpose of auto-detect (e.g., to highlight function names or code snippets in the message). Changing the font color on the other hand does trigger auto-detect to propose/send an HTML part, thus it remains a bit try and error to figure out which markup causes an HTML part to be generated and which isn't sufficient on its own.

I've been using the workaround in bug 136502 comment #33 for years now to make sending in HTML explicit when I compose in HTML (with plain-text being my default), thus making the choice for each message as needed rather than guessing how it might be sent if left for auto-detect to decide. Anyway, bottom line is the apparent difficulty in discovering what auto-detect is basing its choice on, and ignoring an explicit fixed/variable-width change is rather unexpected not to justify an HTML part.

Comment 56

[Ben Bucksch (:BenB)]

> the question remains though why auto-detect doesn't consider <tt> as "sufficient formatting"
> for the purpose of auto-detect

Because plaintext mails are traditionally rendered in fixed width font. Some people display them as variable width (including myself), but that's not the assumption.

The bar for sending HTML is intentionally high. It must be something that really alters the meaning significantly, to an extend that makes the reader misunderstand the message, before we suggest HTML. Color is significant, e.g. some people separate quote and response only by color. <tt> isn't the same category.

Comment 57

[Ben Bucksch (:BenB)]

(FWIW, yes, there might be *some* cases where <tt> carries significant meaning. But I don't think it's very uncommon, and have to draw a line somewhere.)

> making the choice for each message as needed rather than guessing how it might be sent
> if left for auto-detect to decide.

This is exactly the purpose of the menu Options | Format option: If you know what you want, and you've carefully crafted the message, then by all means please use this menu item, that's what it's for.

Comment 58

[Thomas D. (:thomas8)]

(In reply to Ben Bucksch (:BenB) from comment #54)
> > that's bug 136502
> 
> That's what confused me. Bug 136502 is not necessary to use the workaround.
> That bug is about *always* setting the format. Here, you apply very specific
> and delicate formatting to one message.

I don't see what's so specific and delicate about alternating "fixed width" and "variable width", but nevertheless, the whole point of having "Auto-detect" is that if there's *any* explicit and consequential formatting, we have to use HTML format. We can only use plain text if a) no formatting applied or b) inconsequential formatting where lossless conversion is possible. To level off explicit user formatting of "fixed width" vs. "variable width" isn't lossless. And that's more than just line breaks, we are talking about e.g. code snippets that become unreadable in "variable width" (because TB dropped <tt> tag which would force other readers into "fixed width"), and flowing explanatory text which looks odd as fixed width, and becomes undistinguishable from code snippets.

> and delicate formatting to one message. If you do that, then you can also
> set Format | HTML or whatever you want, and it works as you expect.

That argument is absurd for a number of reasons:
1) *Every* manual formatting is applied to just *one* message - so why not ignore <b> tag, or any tag, and just convert to plaintext always when auto-detect=true? According to the logic of comment 54, users should "just set Format | HTML... and it works as you expect"...?
Only... what's the purpose of Auto-detect then?

2) So on the contrary: The more "specific" and "delicate" the formatting, the *more* reason for TB to preserve it and send as HTML when auto-detect=true!

3) clear violation of ux-efficiency!!! why force users into extra steps while it's obvious that consequential formatting has been applied and should be preserved? Ben, you think there is any user who applies "fixed vs. variable width" who will then be happy when we remove that formatting?

4) violation of ux-consistency, ux-control, ux-error-prevention, you name it.

Furthermore, I'd still maintain that it's *dataloss* if I compose "specific and delicate" formatting like 
> <a style="float:right;background-color:red" title="Click me"
> href="http://www.abc.com">http://www.abc.com</a>
and Thunderbird just wipes out all that formatting and sends
> http://www.abc.com,
thus messing up my entire layout while pretending to "do the right thing automatically" (auto-detect=true).
Ben, can you explain to me why this is not dataloss?

I appreciate that the general question of plain text vs. HTML might be a question of opinion, as Ben suggests, but it ceases to be such when we start to violate several ux-principles. And, compared to these violations, what's the *benefit* of enforcing plaintext for these cases (of *consequential* formatting)? To save a few bytes for an extra <HTML> tag and a <TT> tag? Really, this discussion doesn't make sense. If we can pls focus on what needs to be done to get this right and predictable, and I think that's pretty straightforward and not very complex to fix.

Comment 59

[Thomas D. (:thomas8)]

(In reply to Ben Bucksch (:BenB) from comment #56)
> > the question remains though why auto-detect doesn't consider <tt> as "sufficient formatting"
> > for the purpose of auto-detect
> 
> Because plaintext mails are traditionally rendered in fixed width font. Some
> people display them as variable width (including myself), but that's not the
> assumption.
> 
> The bar for sending HTML is intentionally high. It must be something that
> really alters the meaning significantly, to an extend that makes the reader
> misunderstand the message, before we suggest HTML. Color is significant,
> e.g. some people separate quote and response only by color. <tt> isn't the
> same category.

Before we get more such metaphysical explanations, can someone please add some testcases including screenshots before and after sending, and including "style" case of comment 45? I'm starting to feel unwell if programmers start to decide on my behalf which of my formatting is significant, and which isn't. Ben, really, your argument works just the other way round: If you are the type of person that applies a lot of formatting to your message which you then want to have wiped out upon sending - then *you*, by all means, should please set format:plain-text manually. Otherwise, pls let format:auto-detect work efficiently as expected for the rest of us without violating obvious ux-principles and unexpectedly nullifying the specific consequential formatting of our compositions without any substantial benefit.

Comment 60

[Ben Bucksch (:BenB)]

> inconsequential formatting where lossless conversion is possible

I guess we'll just disagree about whether <tt> in plaintext (which is assumed to be fixed width anyway) is inconsequential or not.

Given that we at the level of "absurd" and accusations, I think it's best to stop the discussion at this point.

Comment 61

[rsx11m]

(In reply to Ben Bucksch (:BenB) from comment #56)
> Because plaintext mails are traditionally rendered in fixed width font. Some
> people display them as variable width (including myself), but that's not the
> assumption.

Well, so if I compose a message in variable-width default and try to highlight something in fixed-width font, it's sent out as plain text anyway and everything becomes fixed or variable width depending on the preference of the recipient.

Thus, the main issue appears to be able to distinguish between "intentional" and "unintentional" formatting (as also differentiated in bug 28420 if I read this correctly).

(In reply to Ben Bucksch (:BenB) from comment #57)
> This is exactly the purpose of the menu Options | Format option: If you know
> what you want, and you've carefully crafted the message, then by all means
> please use this menu item, that's what it's for.

It would appear that the user applying intended formatting would imply the users intention to use that option, wouldn't it? Thus, a trade-off between interests and solution to some of these issues might be to set a boolean flag when the user applies some formatting from the format bar or menus and let auto-detect treat it as a "sufficient" criteria. If there are still objections, make such flagging optional but persistently user-selectable.

Comment 62

[Thomas D. (:thomas8)]

(In reply to rsx11m from comment #61)
> Thus, the main issue appears to be able to distinguish between "intentional"
> and "unintentional" formatting (as also differentiated in bug 28420 if I
> read this correctly).

I appreciate the idea, but I doubt we have any way of reading the users mind of "intentional" vs. "unintentional" formatting. What I see in Bug 28420 Comment 14 (12 years ago) is an attempt to remove *inconsequential* html tags that don't have logical nor layout consequences for the content (or, as in blockquote, programmers think they can do more or less lossless conversion of logic and layout). Interestingly, even for simple and frequent cases like <b>, TB now accepts these as formatting to trigger HTML, and more interestingly, some of the same people were having exactly the same discussions as here (e.g. Bug 28420 Comment 9, which puts it very clearly and correctly).

> It would appear that the user applying intended formatting would imply the
> users intention to use that option, wouldn't it? Thus, a trade-off between
> interests and solution to some of these issues might be to set a boolean
> flag when the user applies some formatting from the format bar or menus and
> let auto-detect treat it as a "sufficient" criteria. If there are still
> objections, make such flagging optional but persistently user-selectable.

We cannot make prescriptions to the user how to apply "intended formatting". If user copies and pastes HTML into his composition (injecting code or using editor UI), we have to assume that it's intended formatting unless user explicitly instructs us otherwise by setting format:plain-text. So I don't see a need for new flags that further complicate the code and ux, while the "benefit" is entirely unclear.

I also think the age of Bug 28420 where these conversions were first tried is also telling: 12 years ago, in 2000, email was certainly a different animal from what it is now. For a vast majority of users, plain text vs. html debate is completely irrelevant because mail storage capacity or modem speed or html-incapable mail readers are more or less issues of the past. Also at that time of browser wars, using sophisticated style attributes (css) and respect for correct wysiwyg rendering was probably much less topical than it is now.

Comment 63

[Thomas D. (:thomas8)]

In a nutshell (from Bug 28420 Comment 9):
> What's really important is that we don't do magic over the user's head when the
> magic isn't "perfect".

Comment 64

[Charles]

(In reply to Ben Bucksch (:BenB) from comment #60)
> > inconsequential formatting where lossless conversion is possible
> 
> I guess we'll just disagree about whether <tt> in plaintext (which is
> assumed to be fixed width anyway) is inconsequential or not.

???

If, as Thomas says, the possibility of the use of <tt> (personally I don't even know what it does) has even a minor chance of meaning that the user intends the message to be sent as HTML, then obviously it may not be 'inconsequential', and auto-detect should err on the side of *not* destroying user data...

I also agree with Thomas that, at this point in time, it would make more sense for auto-detect to simply default to HTML if *any* direct formatting has been applied - why even *try* to 'determine if such formatting is 'consequential' or not? The bottom line is, since the 'Option: Format' choice is only available when the user is composing an email *in HTML format*, this alone should make it clear that the user generally intends to use some direct formatting, and consequently, if *anything* is found in the content that could even *remotely* require HTML formatting to render correctly, auto-detect should *always* err on the side of HTML - meaning, don't even *try* to 'determine if such formatting is 'consequential' or not, if it finds *something*, select HTML.

> Given that we at the level of "absurd" and accusations, I think it's best to
> stop the discussion at this point.

Ben - all Thomas said was that your argument was absurd. Yes, maybe a bad choice of words (been guilty of that many times myself when arguing passionately for some feature I was interested in), but maybe he has a point? Sadly, it appears that you must have just ignored the rest of his very articulate and on point arguments, because you didn't even try to refute them.

This almost sounds to me like an HTML email hater making subtle arguments in favor of plain text, but trying not to look like an HTML email hater.

Comment 65

[Ben Bucksch (:BenB)]

For the record: I personally have no problem with HTML email (structured and well-formatted), and I'm on record with this. I don't argue for myself here at all. Please stop the personal attacks, thanks.

Comment 66

[Thomas D. (:thomas8)]

Most people on this bug see one or more ux-problems with the current behaviour (notwithstanding the general goodness of Thunderbird and its programmers!), so I have clarified the summary with some details which imho are necessary to better understand the nature of this problem.

FTR: I fully appreciate all TB programmers inluding Ben both as a person and for the gift of Thunderbird which they present to us. I'm sorry if any of my comments can be misunderstood as a personal attack, but they *never* have that intention (and while I obviously need to mention persons to reject their arguments or ask questions for clarification of the matter, I don't see where I ever *attacked* *persons*). I'm very aware of and thankful for Ben's great achievements as a volunteer TB programmer, including but not limited to his work in txt-html conversion and the immeasurable gift of sanitizing the Automatic Account Configuration, a long and tedious process in which Ben surpassed himself for the common good of Thunderbird.

I *do* reserve the right to point out problems with existing behaviour and UI, but I *always* strive to give reasons. So when I consider a particular *argument* (not a person) "absurd" (which might be an unfortunate impression, but that's what I think), you'll find as many as 4 different non-personal reasons to support that statement. I agree with Charles' comment 64 that it would be helpful to discuss (prove right or wrong) those arguments instead of going into personal blame games. And I'm not arguing for myself either, I have never used "fixed width" so far nor do I typically insert css-styles into my links. So I am arguing on behalf of other users who do (including those from 5 duplicate bugs), and to improve the quality of our product along recognized design and ux-principles.

I'm trying to do this in line with bmo netiquette:
> Constant and intense critique is one of the reasons we build great products.
> It's harder to fall into group-think if there is always a healthy amount of
> dissent. We want to encourage vibrant debate inside of the Mozilla community,
> we want you to disagree with us, and we want you to effectively argue your
> case. However, we require that in the process, you attack things, not people.
> Examples of things include: interfaces, algorithms, and schedules. Examples of
> people include: developers, designers and users.

And while my comment 59 is certainly a bit worked up ("genervt"), and I do *mention* people (which is not forbidden), I don't see any personal *attack* in that comment, and I am just using Ben as an example of any other user who *might* wish to have explicit and consequential HTML formatting wiped out upon sending. I do doubt there are many users who expect such behaviour. If comment 60 suggests it's inconsequential, then we need to verify that. In fact, in that same comment 59 I suggested that we come together and look at some testcases so that we can all better understand the effects of this bug, and then get it out of the way, for which we will depend on the qualified help of Ben.

Comment 67

[rsx11m]

(In reply to Thomas D. from comment #62)
> I appreciate the idea, but I doubt we have any way of reading the users mind
> of "intentional" vs. "unintentional" formatting.  [...]
> If user copies and pastes HTML into his composition (injecting code or using
> editor UI), we have to assume that it's intended formatting unless user
> explicitly instructs us otherwise by setting format:plain-text.

It's certainly impossible to /exactly/ read the intentions of the user, and copy-pasting of already formatted text is a special class to be considered.
For new messages, I figured that any user-invoked format would indicate such intention, thus forgoing the need to explicitly choosing the format with three additional mouse clicks per comment #50, thus it still may be a viable approach.

Looking at nsMsgCompose::TagConvertible() referred to in comment #12, there are various classes of nsIMsgCompConvertible::* return values depending on node type and tag name. Those are hard wired; interestingly, both <tt> and <pre> are in the nsIMsgCompConvertible::Plain class whereas <code> is ::Altering and indeed causes auto-detect to propose HTML to be sent.

Thus, given that some people have different ideas what formatting is consequential than others (and it appears to be unlikely that common ground for a fixed list will ever be found, judging from the discussions here and elsewhere), another option might be to introduce a preference to allow additional ::Altering tags to be defined, thus triggering auto-detect earlier than by default. In this way, <tt> can be added to be at the same level as <code> (for which no button exists), and if you don't want to rely on the recipient's e-mail client to be smart enough to linkify web addresses on its own, add <a> to ensure that those messages are sent with an HTML part (as per initial bug description).

Comment 68

[Thomas D. (:thomas8)]

Attachment: TC1-Source1.eml: Composition (test HTML message with explicit <tt> for 'fixed width' vs. 'variable width' formatting by user)

OK, let's focus on facts.

Here's Testcase 1.00: Source code of saved draft message where user explicitly alternates between "variable width" and "fixed width" fonts formatting.

To test:
- adjust your settings as described below
- load this .eml in TB and Ctrl+E to edit as new

Relevant Initial Settings of Testcase 1.00 (you might have to change some of these during testing):

1) per-identity: "Compose Messages in HTML format"
mail.identity.default.compose_html;true
mail.identity.id#.compose_html;true
Tools > Account Settings > YourAccount > Composition and Adressing >
[x] Compose Messages in HTML format (checked)

That's TB default setting.

2) per AB-card: recipient's preferred format
AB > Select recipient's card > Contact tab > Prefers to receive messages formatted as:
[Unknown] (select from dropdown)

That's TB's default for new addresses; so I assume most recipients will have that status.

3) general setting: Default format for sending in cases of doubt
mail.default_html_action;2
Tools > Options > Composition > Send Options > Text format > When sending messages in HTML format and one or more recipients are not listed as being able to receive HTML:
[Send the message in HTML anyway]

From the UI, it's unclear if this setting refers to recipients of type "Unknown", or "Plain text", or both.

TB's default setting is:
mail.default_html_action;3 - "Send the message in both Plain text and HTML"

4) general setting: Font control for displaying plain text messages
mailnews.display.prefer_plaintext;false
Tools > Options > Display > Advanced > Font Control >
[ ] Use fixed width font for plain text messages (unchecked)

That's TB's default setting ('Variable width' for displaying plaintext messages - makes sense if we want to encourage plain text for unformatted messages; breaks bugmail); comments in this bug say other mailers have same default setting, e.g. Google.

Comment 69

[rsx11m]

(In reply to Thomas D. from comment #68)
> Here's Testcase 1.00: Source code of saved draft message where user explicitly
> alternates between "variable width" and "fixed width" fonts formatting.

That's probably a case where you'd use <pre> rather than <tt> by switching from "Body Text" to "Preformat", but the result is the same given that the <pre> tag resolves to nsIMsgCompConvertible::Plain instead of ::Altering as well.

Comment 70

[Thomas D. (:thomas8)]

Attachment: TC1-Screenshot1: Composition Window (showing intended visual difference between variable-width and fixed-width sections)

Screenshot1: This is how Testcase1.00.eml of attachment 633824 [details] looks while composing, with settings as explained in comment 68.

User explicitly alternates between sections of "variable width" font, and other sections of "fixed width" font formatting. Sample use cases for fixed width are Code or ASCII art (or any other reason why user wants fixed width). Both of them rely on fixed-width, and will look distorted if variable-width. Variable-width is default used for anything else the user normally writes.

Pls note: While composing in HTML editor (with Format:Auto-detect), there is a clear visual difference between fixed-width and variable-width formatting.
(At least) for recipients with preferred format "Unknown", this difference is removed by Format:Auto-detect, and recipient has no way of recovering the difference (as intended by sender), regardless of his display settings (this bug).

Comment 71

[Magnus Melin [:mkmelin]]

(In reply to Thomas D. from comment #58)
> 3) clear violation of ux-efficiency!!! why force users into extra steps
> while it's obvious that consequential formatting has been applied and should
> be preserved? Ben, you think there is any user who applies "fixed vs.
> variable width" who will then be happy when we remove that formatting?

Definitely! Or, at least recipients will certainly be more happy.
If i get to choose reading arbitrarily formatted mails from someone vs reading them with standard settings i'd alwas choose the latter. It's in the same category as people composing mails with font size 25 to see what they are writing, but nobody wants to receive that - they want to have the normal sized message which they can apply their standard zooming to if needed.

Comment 72

[Thomas D. (:thomas8)]

Attachment: TC1-Source2.eml: Actual Result-Received (downgraded to text-plain, format-flowed, formatting lost).eml

This the source of what eventually gets sent and is received by recipient:
- format:Auto-detect downgrades everything to text/plain, format=flowed
- received msg source no longer shows any trace of difference between sections originally formatted differently (fixed vs. variable)

Comment 73

[Thomas D. (:thomas8)]

Attachment: TC1-Screenshot2: Actual Result-Received, view:variable (recipient using TB, default 'variable width' to display plaintext)

This screenshot depicts what recipient will see when receiving message of TC1-Source1, converted to TC1-Source2 upon sending, viewed with TB default display settings for plaintext (variable width): Garbage (this bug).

This looks *very* different from what sender saw (and intended!) when composing in TB (see TC1-Screenshot1). Differences between fixed-width and variable-width are gone, the whole message is now single part of text/plain format=flowed.

This is how most recipients will see the message, if they have their display options for plaintext messages set to 'variable width'.

Comment 74

[Thomas D. (:thomas8)]

Attachment: TC1-Screenshot3: Actual Result-Received, view:fixed

This screenshot shows what recipients will see when they receive TC1 and have their plaintext display options set to 'fixed-width'. Not as bad and broken as 'variable width', but still not showing the formatting differences which sender saw and intended - all sections are now fixed-width.

Comment 75

[Thomas D. (:thomas8)]

Attachment: TC1-Screenshot4: Expected Result-Received, view:HTML (if sent as HTML, will always display correctly as HTML, regardless of recipient settings)

Expected Result for <tt> and <pre>:

Finally, if TB's format:Auto-detect would do the right thing and just send TC1-Source1 as text/HTML (see attachment 633824 [details]), as requested by this bug for cases of <tt> formatting (among others), then *every* recipient (regardless of their display settings for plaintext) will normally just see the message correctly, because it will be rendered as HTML. Of course, if recipient insists on viewing our text/HTML message as unformatted plaintext, he is still free to do so. This might be helpful for users who prefer ASCII garbage over ASCII art.

Comment 76

[Thomas D. (:thomas8)]

(In reply to Magnus Melin from comment #71)
> (In reply to Thomas D. from comment #58)
> > 3) clear violation of ux-efficiency
> > ...you think there is any user who applies "fixed vs.
> > variable width" who will then be happy when we remove that formatting?
> 
> Definitely! Or, at least recipients will certainly be more happy.

That's a very ambiguous and twisted answer. You don't honestly believe that erasing deliberate formatting without confirmation will make the *sender* happy.

> If i get to choose reading arbitrarily formatted mails from someone vs
> reading them with standard settings i'd alwas choose the latter. It's in the
> same category as people composing mails with font size 25 to see what they
> are writing, but nobody wants to receive that - they want to have the normal
> sized message which they can apply their standard zooming to if needed.

- What's "arbitrarily" formatted about a simple ASCII art or code section with <tt> or <pre>? Why do we offer "Fixed width" in the font menu then?
- I am completely failing to see how simple <tt> and <pre> is in the same category of screamingly bad formatting like font size 25? What other options from the UI do I have to mark a section 'fixed width' if that's what I need and want? What's so special about having code or ASCII art in 'fixed width'?

Magnus, most importantly, your argument is besides the point because *as recipient*, you are *always* free to display text/html part as plain text, please use this command:

View > Message Body as > Plain text.

That's not what this bug is about. This is about datalossy behaviour when *sending*, because after TB unexpectedly erases <tt> and <pre> from user's composition (when format:auto-detect and recipient's preferred format:unknwon), that difference in format is gone without trace and cannot be recovered at the receiving end.

Comment 77

[rsx11m]

(In reply to Magnus Melin from comment #71)
> people composing mails with font size 25 to see what they are writing,

Ironically, if you change "Size: [normal]" to "[large]" in the Composition prefs, the message goes out as HTML with <font size="+1">, and also the <big> tag will be considered "sufficient" to make auto-detect happy...  ;-)

Comment 78

[Magnus Melin [:mkmelin]]

(In reply to Thomas D. from comment #76)
> That's a very ambiguous and twisted answer. You don't honestly believe that
> erasing deliberate formatting without confirmation will make the *sender*
> happy.

I think *deliberate* is the key here. Of course if you mark code as monospaced it's not what you want - but then again that should be marked <code> and not <tt>. I imagine the average user ends up with <tt> just because he  wanted to see the font differently but could care less for monospaced or not.

> - What's "arbitrarily" formatted about a simple ASCII art or code section
> with <tt> or <pre>? Why do we offer "Fixed width" in the font menu then?
> - I am completely failing to see how simple <tt> and <pre> is in the same
> category of screamingly bad formatting like font size 25? What other options
> from the UI do I have to mark a section 'fixed width' if that's what I need
> and want? What's so special about having code or ASCII art in 'fixed width'?

To mark code as <code> go to Options | Text style | Code

> Magnus, most importantly, your argument is besides the point because *as
> recipient*, you are *always* free to display text/html part as plain text,
> please use this command:
> 
> View > Message Body as > Plain text.

I prefer not to, as people do the silliest things like color their responses instead of quoting etc.

> That's not what this bug is about. This is about datalossy behaviour when
> *sending*, because after TB unexpectedly erases <tt> and <pre> from user's
> composition (when format:auto-detect and recipient's preferred
> format:unknwon), that difference in format is gone without trace and cannot
> be recovered at the receiving end.

I can agree <pre> should probably be nsIMsgCompConvertible::Altering because you don't end up with it (easily) by accident.

Comment 79

[Ben Bucksch (:BenB)]

> I can agree <pre> should probably be nsIMsgCompConvertible::Altering

<pre> we absolutely cannot make trigger HTML, for 2 reasons:
1. The whole definition of <pre> is that it should be considered plaintext. We send as plaintext,
  so we 100% fulfill it. Again, what the recipient does is none of our concern.
2. <pre> is used extensively internally both by us and other mailers to render plaintext in HTML.
We are in the HTML composer, so all replies to plaintext mails go into a <pre>.
If <pre> was to trigger HTML, then we would almost never send plaintext.

You can argue that we should skip quotes in the detection, but then somebody else like
you comes along and argues that we are not sending quotes correctly.

Also, there are other cases where <pre> is used, I think in copy&paste. I think there are many other places, too.

You only see the "Preformatted" menu item, but <pre> is used for a lot lot lot more than that. THe composer internally has no concept of differentiating between tags inserted manually by the user using the UI, and tags inserted by other means or for other reasons. There are literally countless reasons for a <pre> to enter the message, and most of them are fulfilled just fine by sending in plaintext.

This alone means that <pre> must never trigger HTML. But even for a manually inserted "Preformatted", I am firmly of the believe that sending as plaintext not just fulfills <pre>, but overfulfills it, because <pre> is just an emulation of plaintext in rich text. Sending a <pre> as plaintext is the ideal.

And this is why I am unmoving on the topic of <tt>, because I know <pre> will be next, and we cannot allow that, because it would break many many cases.

Comment 80

[Ben Bucksch (:BenB)]

<pre> definition: <http://www.w3.org/TR/REC-html40/struct/text.html#h-9.3.4> (note "may")

Comment 81

[Thomas D. (:thomas8)]

(In reply to Ben Bucksch (:BenB) from comment #80)
> <pre> definition: <http://www.w3.org/TR/REC-html40/struct/text.html#h-9.3.4>
> (note "may")

(also note: "how this is typically rendered", which is true for most, if not all current browsers/mailers, and that's exactly where this bug fails us by dumping <pre> in favor of text/plain, which most mailers will then display as 'variable-width', thus badly distorting the original content, as shown in TC1-Screenshot2 of attachment 633835 [details])

Comment 82

[Thomas D. (:thomas8)]

Attachment: TC2-Source1.eml: HTML Composition with styled simple links <a style="..." href="xyz">xyz</a> and <pre style="...">

Enters Testcase 2:

<a style="..." href="xyz">xyz</a>

Demonstration of "Format:Auto-Detect" dumping so-called "simple" link tags (where href=linktext) and losing consequential style attributes on the way...

Result: major distortions of layout and broken links (where broken links might be another bug, but it would be completely avoided if we did the right thing for this bug).

Comment 83

[Thomas D. (:thomas8)]

Attachment: TC2-Screenshot1: Composition window

In composition window: Applied styles still look good (some css is not rendered correctly in editor, which is another bug: wrong link-color and ignored text-decoration; however, source code is kept intact, as can be seen from saved draft in message reader).

<a style="float:..."> moves things around for layout purposes, as rendered correctly in compose window.

Comment 84

[Thomas D. (:thomas8)]

Attachment: TC2-Source2.eml: Actual Result-Received (downgraded by Format:Auto-detect to text-plain, format-flowed)

... and these are the bare bones which will get sent after Format:Auto-Detect has dealt with the HTML, "converting" it into text/plain, format=flowed.

Actual Result:
- <a>, <pre> tags: dumped
- link styles on those tags: dumped along
- quite a mess in source text, two links concocted into one -> both links broken

Comment 85

[Thomas D. (:thomas8)]

Attachment: TC2-Screenshot2: Actual Result-Received-view-variable (layout & links completely broken)

This is how most recipients will see our message (display text/plain as variable width), after Format:Auto-Detect has "converted" it to plaintext

> _http://www.getthunderbird.comhttp://www.getfirefox.comLorem_ ipsum dolor sit
> amet, consectetur adipiscing elit. Vestibulum 

- Layout completely broken
- color: gone
- 2 links concocted into one, and even concocted with neighbouring plaintext!
- Both links broken, the concoction points to the following "location" (That's TB's message reader failing to convert bad plaintext into link):
> http://www.getthunderbird.comhttp//www.getfirefox.comLorem
- ASCII art: distorted (because no longer protected by <pre>, which would trigger correct fixed-width HTML display regardless of recipient's settings)
- ASCII art no longer centered

Comment 86

[Thomas D. (:thomas8)]

Attachment: TC2-Screenshot3: Actual Result-Received-view-fixed (layout & links broken)

For other recipients with non-default 'fixed-width' display, layout and links still badly broken (see previous comment about TC2-Screenshot2), only ASCII art looks a bit more tidy, albeit still in wrong colors and wrong position.

Original font formatting differences between 'variable width' and 'fixed width' have disappeared (same as testcase 1), both here (fixed) and in TC2-Screenshot2 (variable).

Comment 87

[Thomas D. (:thomas8)]

Attachment: TC2-Screenshot4: Expected Result-Received-view-HTML (same as saved draft, correct layout, links & titles)

And this is how beautiful it would look if Format:Auto-detect did the right thing: Never dump HTML tags if they have user styles applied.
Works for every recipient regardless of their plaintext display settings, because text/HTML with <pre> will ensure correct rendering everywhere! :)

BTW, distortions list of Comment 85 still isn't complete:
Of course, Format:Auto-detect also dumps any other attributes that happen to be on the doomed tags, like the title attributes on the links.

<tt>, <pre>, styles, title attributes, you name it: We have no right whatsoever to wipe out user's HTML without notice, and without recovery. Mind you, after sending (crushed-to-plaintext), all your formatting is gone for good. Which is what I call "dataloss", but I tried adding that keyword and it was removed.

Even small things like title attributes are user data which might contain important information for recipient, like this one on the "getthunderbird" link:

<a title="Get the best mailer now! (Caveat: neglected bird with lots
      of bugs)" style="float: right; background-color: blue; font-size:
      18px; text-decoration: none; color: white; padding-top: 90px;
      padding-bottom: 90px; padding-right: 50px; padding-left: 50px;"
      href="http://www.getthunderbird.com">http://www.getthunderbird.com</a>

And, as a sidenote, TB even manages to distort the title attribute with extra whitespace in the tooltip, as seen on screenshot, when trying to squeeze user's HTML, which originally looked like this:

<a title="Get the best mailer now! (Caveat: neglected bird with lots of bugs)"
  style="float: right;
  background-color: blue;
  font-size: 18px;
  text-decoration: none;
  color: white;
  padding-top: 90px;
  padding-bottom: 90px;
  padding-right: 50px;
  padding-left: 50px;"
  href="http://www.getthunderbird.com">http://www.getthunderbird.com</a>

Food for thought (re: auto-"converting" to plaintext): consequential styles (with layout and design impact) can happen on almost any HTML tag, not only on those few where auto-detect currently respects them (e.g. divs and spans).
I'm happy to find that it was BEN who fine-tuned auto-detection to preserve divs and spans if they have styles. We need more such quality fine-tuned code.

And FTR: I like the bird, I like the programmers, and this isn't personal in any way. The only thing I don't understand is why Mozilla doesn't allocate more staff and resources for Thunderbird.

Comment 88

[Thomas D. (:thomas8)]

(In reply to Ben Bucksch (:BenB) from comment #79)
> <pre> we absolutely cannot make trigger HTML, for 2 reasons...
> If <pre> was to trigger HTML, then we would almost never send plaintext...
> just an emulation of plaintext in rich text. Sending a <pre> as plaintext is
> the ideal...
> And this is why I am unmoving on the topic of <tt>, because I know <pre>
> will be next, and we cannot allow that, because it would break many many
> cases.

There would be a lot to answer to this comment, too much for this bug. I sent a detailed response to Ben via PM.

So per comment 79, this bug is also a problem in terms of ux-implementation-level:

> User experience principle: interfaces should not be organized around the
> underlying implementation and technology in ways that are illogical, or
> require the user to have access to additional information that is not found in
> the interface itself.

And I agree with Ben, and M's Comment 31, and rsx11m's Comment 69, and Magnus' Comment 78, that <pre> is the other part of this bug, as seen in Testcase 2.

Comment 89

[Ben Bucksch (:BenB)]

Thomas, please stop campaigning, and stop messing with summary and the other fields. Thanks.

Comment 90

[Ben Bucksch (:BenB)]

> I agree with Ben

Don't twist my words. I have said clearly that <pre> will *never* trigger HTML. Doing so would be both highly ironic and damaging. See my last comment. 

What's next? Our plaintext composer sending HTML in a <pre>, because somebody might view the message in variable width font? This is email, it's highly heterogeneous. We have to find a balance between all interests. If we automatically send HTML as you wish, then yet other people will be upset.

Please. Just use menu Options | Format | HTML. Be pragmatic, please.

Comment 91

[Ben Bucksch (:BenB)]

> See my last comment. 

Sorry, I meant comment 79. Please also see comment 25 and esp. comment 26.

Comment 92

[Thomas D. (:thomas8)]

(In reply to Ben Bucksch (:BenB) from comment #91)
> > See my last comment. 
> 
> Sorry, I meant comment 79. Please also see comment 25 and esp. comment 26.

Comment 79 (I paraphrase):
> <pre> can never trigger HTML because...

<pre> already gets sent as HTML if there's any HTML-triggering tag like <b>

> - we would almost never send plaintext then

So what? According to Ben's own comment 65, there's no problem with sending HTML.

> - we are using <pre> internally for other plaintext things (which will break
>   if <pre> triggers HTML)

Irrelevant for *users*, and invisible (hence ux-implementation-level, see comment 88). If users using <pre> can break code logic, that's a bug which needs to be fixed. And technically, we already have code to distinguish between user-<pre> and TB-<pre> for quotes, as Ben knows.

(In reply to Ben Bucksch (:BenB) from comment #25)
> We needed to limit the amount of times when we pop up the dialog

This bug is not requesting to re-introduce the dialogue.
Showing the dialogue is not necessary to fix this bug.
So that's off-topic.

> We send plaintext without asking, if you have no formatting whatsoever.

Not true: Pls see Testcase 2 (attachment 633907 [details]) which proves beyond doubt that Auto-detect *does* purge correct and consequential formatting, sending completely broken messages as a result (attachment 633910 [details]).

> Preformat <pre> == plaintext.

Not really. <pre> == plaintext *within HTML*.
And <pre style="..."> == HTML (for sure!).

> Obviously, if you change some part explicitly to plaintext, that part is not
> a reason to send HTML :)

Well, to reporter of this bug, and 5 duplicates, and other commenters, it seems *not obvious*. Personally, I stop smiling when looking at broken result of TC2-Screenshot2 attachment 633910 [details] (compared to what I composed, which is attachment 633913 [details]).

> Similarly, a link that is equal to the link text is not considered
> formatting either, otherwise we'd trigger too often (because we also detect
> and mark links, and so does other software that we quote etc.).

"...trigger HTML too often" - is that a problem? For whom?

<a href="URL">URL</a> => plaintext URL (I suppose I can live with that)
<a title="User-defined title" href="URL">URL</a> => plaintext URL? (datalossy!)
<a style="..." title="User-defined title" href="URL">URL</a> => plaintext URL!? (no way! dataloss, this is clearly HTML which needs to be preserved, as we preserve any other HTML for much more simple cases like <b>).

(In reply to Ben Bucksch (:BenB) from comment #26)
> (Please also note that the plaintext vs. HTML question is very delicate and
> *the* flamebait in mail area. Everybody has a certain opinion and is
> convinced it's right, just that everybody has a slightly different one. This
> is part of the compromise.)

Ben, this bug is not a question of opinion, whether people *generally* prefer plaintext or HTML. This bug is about users who compose correct HTML messages with the *default settings of TB*, and *recipients with default settings* of TB or other mailers, and *TB losing data* on the way, making such messages unreadable, without warning, and without sufficient need or substantial benefit.

I fully appreciate the good intentions of your code, which were certainly useful at the time, but I think times have moved on, and the user experience of this is no longer good enough, and, as Charles says in Comment 64,
> auto-detect should err on the side of *not* destroying user data...

The argument of this bug runs like this:
1) Format:Auto-detect is default setting for TB.
2) Display plaintext as "variable width" is default setting (for TB and others)
3) Default settings should be "lossless" (vs. datalossy).
4) Currently, because of this bug, *TB's default behaviour* is *not* lossless, it
   is datalossy (or even significant *dataloss*, as proven in Testcase 2).

Ben, you are saying that users who want "lossless" should force HTML with extra steps, because TB's auto-detect prefers plaintext over HTML for cases of this bug. Iow, you are saying: "TB reserves the right to remove consequential formatting from normal users' composition without warning, for the higher goal of sending plaintext." But *for cases of this bug* (not in general), *what* is the extraordinary *benefit* of sending plaintext, while we are already *always* sending HTML if we find a single little <b> in user's composition?
And would that "benefit" justify *violating ux-consistency, ux-efficiency, ux-control, ux-error-prevention, and ux-implementation-level*? (btw, removing those keywords does not change the facts...)

Again: I strongly believe it should be the other way round: /Those users who want their own *relevant* formatting removed upon sending/ can be expected to take the extra step of forcing plain text. I strongly suspect those users are a minority, if they exist at all.

And for the technical side:

Suppose we would remove <tt> from the list of things that trigger plaintext (as requested by this bug). *Please provide testcase or scenario* with steps where this causes any problem (I doubt it will cause any problem whatsoever).

Same for <pre>: *Please provide testcase or scenario* with steps to explain how this will break so many code logics so badly as you claim.

This is a *fact* (for Format:Auto-detect and "recipient prefers:unknown"):
<b>bold</b><pre>Preformat</pre> *does* trigger HTML *without breaking anything*.

If just <pre>Preformat</pre> breaks anything, I suspect there is something wrong with TB's code, or the assumptions made in that code. IMHO, instead of ignoring/denying that there is a ux-problem, we should *work together* to find out about the nature of the problem, and try to fix it.

This is another *fact*:
We already *do* have *code to distinguish between user-defined tags and tags used by TB for internal purposes*, namely quotes. (So I don't see why the same or similar type of code, in principle, cannot work to fix this bug).

I might accept if Ben says "given our current code logics, it would be too much work to fix this", but I will not accept if people completely *ignore/deny* that there is any problem while it is so obvious from the testcases and my explanations given. I have provided reasons for every step of my argument: If any of these reasons are wrong, please refute them, or if you need more epxlanation to understand this ux-problem, please ask.

Comment 93

[Santiago Romero]

 Hi!

 As an "end-user", I created a "bug request" last week that has been marked as a duplicate of this one.

For me, the behaviour that you can see in the attachment "bug_images.tar.gz" in https://bugzilla.mozilla.org/show_bug.cgi?id=763855 (opened by me) it's a **really annoying** bug that has forced me to put HTML code inside my signature to "fix" it.

Please, download the tar.gz file and take a look at the 2 examples.

Thanks.

Comment 94

[Charles]

(In reply to Ben Bucksch (:BenB) from comment #89)
> Thomas, please stop campaigning, and stop messing with summary and the other
> fields. Thanks.

Campaigning? Ben, I have a better suggestion - why don't you address the actual and numerous examples that Thomas has taken such painstaking time to put together to document *why* your argument is incorrect, to show how/why *he* is wrong?

Comment 95

[rsx11m]

(In reply to Ben Bucksch (:BenB) from comment #79)
> 2. <pre> is used extensively internally both by us and other mailers to
> render plaintext in HTML.
> We are in the HTML composer, so all replies to plaintext mails go into a
> <pre>.
> If <pre> was to trigger HTML, then we would almost never send plaintext.

It is correct that all <pre> cannot be just taken on face value, otherwise a simple plain-text signature would trigger auto-detect to send a text/html part for that reason alone. This brings me back to comment #51 to use some measurable indication of the user's intention rather than the current approach of trying to define subjectively whether or not a specific markup is consequential.

> THe composer internally has no concept of differentiating between tags inserted
> manually by the user using the UI, and tags inserted by other means or for
> other reasons.

Actually, there is a way of differentiating such which is already applied for the signature case. A plain-text signature is wrapped in <pre class="moz-signature"> and as such uniquely identified as being the signature. Similarly, one could come up with <pre class="moz-manual"> to mark <pre> nodes which were caused by manual action and thus should be considered intentional changes to be respected.

Sure, there are other questions then, like should such a marker be used internally only or sent out and thus being persistent at least within the Mozilla world, but those should be discussable and solvable in a truly pragmatic way to remove the element of surprise and unexpected behavior for the user.

Comment 96

[Thomas D. (:thomas8)]

Attachment: TC3-Screenshot1: Composition (with fixed-width vs. variable-width and long lines)

(In reply to Santiago Romero from comment #93)
> As an "end-user", I created a "bug request" last week that has been marked
> as a duplicate of this one.
> 
> For me, the behaviour that you can see in the attachment "bug_images.tar.gz"
> in https://bugzilla.mozilla.org/show_bug.cgi?id=763855 (opened by me) it's a
> **really annoying** bug that has forced me to put HTML code inside my
> signature to "fix" it.
> Please, download the tar.gz file and take a look at the 2 examples.
> Thanks.

Santiago, thanks for contributing another testcase to prove this bug.
For ease of access and reference, I will include the contents of double-zipped bug_images.tar.gz (attachment 632169 [details]) as attachments on this bug, and rename them according to the logic of Testcase1 and Testcase2 for easy comparison.

Enters Testcase 3 (unzipped from duplicate Bug 763855, attachment 632169 [details]).
The attached screenshot shows composition window, with alterations of 'variable-width' and 'fixed-width', and long lines of both types.

This testcase is interesting and relevant because it observes an additional problem not covered by Testcase 1 and Testcase 2:

Testcase 3 screenshots show that Format:Auto-Detect "converts" long lines (formatted both fixed and variable width) into short lines with line breaks, which per Bug 763855 comment 0 is unexpected (-> violating ux-consistency). I couldn't reproduce that so far, so we need to look into that and find out (discussion of that particular problem preferably on Bug 763855, not here, let's come back here with results).

Comment 97

[Thomas D. (:thomas8)]

Attachment: TC3-Screenshot2: Actual Result-received, view:variable (long lines now short with hard breaks)

After Format:Auto-Detect has handled the message:
- HTML with <tt> downgraded to text/plain
- format differences between variable vs. fixed width lost
- *long lines lost* (received msg has short lines with extra line breaks)

Comment 98

[Thomas D. (:thomas8)]

Attachment: TC3-Screenshot4a: Compose-Workaround (add bold tag)

This screenshot shows how user resorts to workaround of adding <b> tag in composition (and user isn't happy being forced into those workarounds, hence reported Bug 763855 *after* trying this workaround).

User explains in Bug 763855 Comment 5 why he was expecting TB to preserve HTML formatting when sending, after having set mail.identity.id#.compose_html;true.
See Comment 68 for a more complete collection of relevant settings.

Comment 99

[rsx11m]

(In reply to Thomas D. from comment #98)
> This screenshot shows how user resorts to workaround of adding <b> tag in
> composition (and user isn't happy being forced into those workarounds, hence
> reported Bug 763855 *after* trying this workaround).

Yes, there are plenty of signature-based workarounds for what's bug 136502.
That's another motivation to make auto-detect meet the user's expectation in general, rather than tricking the algorithm into effectively being disabled.

Comment 100

[Thomas D. (:thomas8)]

Attachment: TC3-Screenshot4b: Expected Result-received-view-HTML (with workaround bold)

This screenshot proves that <pre> can be sent and received as HTML by TB without problems (triggered by workaround <b>). So *in principle*, there is no technical problem of letting simple <pre> (without other HTML tags) trigger HTML. There *might* still be a problem due to current internal code logics used for plaintext messages (as claimed by Ben); if that's the case, we need to fix those internal logics (-> ux-implementation-level).

Comment 101

[Thomas D. (:thomas8)]

Attachment: TC3-ScreenshotX1: About (reported against TB13)

Comment 102

[Santiago Romero]

Glad to see that my screenshots are useful :-)

I've been for YEARS, at work, having to put something bold or italic in my messages to get the formatting preserved.

I'm an UNIX system administrator and I send tons of emails with bash/perl/python source code, output of commands and log-file entries (always as 'Fixed width text' for that lines, to improve readability) , and I had to finally put an HTML signature with a FONT COLOR to ensure that my receiver  get a readable message...

Defaulting to plain text sucks, but what really 'broke' my messages was line wrapping... and of course, source code, command output and log lines were less readable...

Comment 103

[Thomas D. (:thomas8)]

Attachment: TC3-ScreenshotX2: AccountSettings for Account "localhost": Composition&Adressing

This screenshot shows where user tried (in vain) to make TB send HTML compositions as HTML:

Tools > Account Settings > User-Account > Composition & Adressing >
[x] Compose messages in HTML format

For other involved settings, see comment 68.
I have some doubts that these settings interact correctly, transparently, and predictably in all cases so that users actually feel in control, which might be part of the bad UX described in this bug (-> ux-control, ux-error-prevention).

FTR: This bug is *not* requesting that we should *always* send HTML for Format:Auto-Detect, but we need to send HTML *when necessary* to preserve correct formatting of the message as composed by user, or to avoid dataloss and broken layouts, as shown in Testcase 1, Testcase 2, and Testcase 3. qed.

Comment 104

[Thomas D. (:thomas8)]

(In reply to rsx11m from comment #99)
> (In reply to Thomas D. from comment #98)
> > This screenshot shows how user resorts to workaround of adding <b> tag in
> > composition (and user isn't happy being forced into those workarounds, hence
> > reported Bug 763855 *after* trying this workaround).
> 
> Yes, there are plenty of signature-based workarounds for what's bug 136502.
> That's another motivation to make auto-detect meet the user's expectation in
> general, rather than tricking the algorithm into effectively being disabled.

+1.
Ben, believe it or not, this bug is very much *in favor* of the good intentions of your code (Format:Auto-Detect), namely to *send plaintext as often as possible* without causing dataloss(y) results. This bug has the intention of */improving/ Format:Auto-Detect* so that users can *continue using that setting*.

Alternatively, if we refuse to improve the default behaviour of our default settings, well then, as rsx11m says, users will just completely disable Format:Auto-Detect either using the current workarounds, or, after fix for bug 136502, just switch it off and always send HTML.

So *absurdly*, your refusal to look into this bug in a cooperative manner, because of your obvious, well-documented, and *good* intention of sending plaintext more often, will *effect the exact opposite*: Affected users will just sideline Format:Auto-detect, so they will *never send plaintext*, and *always send HTML*. *That*'s irony, and *that*'s damage. And yes, I might be campaigning, but I campaign with solid arguments and testcases of bad UX that are open for everyone to discuss, because I believe such campaigning is for the common good unless proven otherwise with solid arguments and testcases or plausible scenarios.

(In reply to Ben Bucksch (:BenB) from comment #90)
> > I agree with Ben 
> Don't twist my words.

I apologise for deliberately twisting your words in that particular sentence, and I shouldn't have done that given that I don't appreciate when other people twist my words, and given the history of this bug. Sorry for that.
 
> What's next? Our plaintext composer sending HTML in a <pre>, because
> somebody might view the message in variable width font?

No, that's not part of this bug. If users want to ensure that their plaintext is viewed as fixed width, they will add <tt> or <pre> to their compositions, which makes these compositions HTML compositions. But that's broken for Format:Auto-Detect (this bug). So we are requesting that <tt> (and <pre>) should be preserved when composing HTML.

> highly heterogeneous. We have to find a balance between all interests. If we
> automatically send HTML as you wish, then yet other people will be upset.

I still want to meet those people who use <tt> or <pre> in their compositions and will be upset when it gets sent exactly as they see it in their composition, following the ux-principle of wysiwyg.

Comment 105

[Thomas D. (:thomas8)]

(In reply to rsx11m from comment #95)
> (In reply to Ben Bucksch (:BenB) from comment #79)
> > 2. <pre> is used extensively internally both by us and other mailers to
> > render plaintext in HTML...
> Actually, there is a way of differentiating such which is already applied
> for the signature case. A plain-text signature is wrapped in <pre
> class="moz-signature"> and as such uniquely identified as being the
> signature. Similarly, one could come up with <pre class="moz-manual"> to
> mark <pre> nodes which were caused by manual action and thus should be
> considered intentional changes to be respected.

Or perhaps the other way round: In the same way we mark <pre class="moz-signature">, also mark any other stuff for internal purposes. Do not mark user's <pre>. That might be easier and more compatible with the current approach.

If <pre> gives such a headache, personally I don't mind just adressing <tt> first, to have some progress with less discussion on this bug. Which might be as easy as removing a single line of code.

Comment 106

[Ian Scott]

(In reply to Thomas D. from comment #105)
> If <pre> gives such a headache, personally I don't mind just adressing <tt>
> first, to have some progress with less discussion on this bug. Which might
> be as easy as removing a single line of code.

I agree - there's really no easy way for a user to enter <pre> in the composer anyway, so just leave that alone. But <tt> is what happens when the user hits Ctrl-T or picks Fixed Width from the font list. If it's a common-enough style to warrant a shortcut in the composer, auto-detect should consider it enough to trigger sending as HTML. I think that's what's caused most people to comment or vote for this bug.

Comment 107

[Thomas D. (:thomas8)]

(In reply to Thomas D. from comment #105)
> If <pre> gives such a headache, personally I don't mind just adressing <tt>
> first, to have some progress with less discussion on this bug. Which might
> be as easy as removing a single line of code.

Removing this single line 5108 will fix a major concern of this bug for <tt>:
http://mxr.mozilla.org/comm-central/source/mailnews/compose/src/nsMsgCompose.cpp?mark=5108,5114#5108

Formerly found at
http://bonsai.mozilla.org/cvsblame.cgi?file=/mozilla/mailnews/compose/src/nsMsgCompose.cpp&rev=1.570&mark=4797,4803#4797

Comment 108

[Thomas D. (:thomas8)]

(In reply to Thomas D. from comment #87)
> Food for thought (re: auto-"converting" to plaintext): consequential styles
> (with layout and design impact) can happen on almost any HTML tag, not only
> on those few where auto-detect currently respects them (e.g. divs and spans).
> I'm happy to find that it was BEN who fine-tuned auto-detection to preserve
> divs and spans if they have styles. We need more such quality fine-tuned
> code.

Fwiw, dataloss of all css style attributes will occur for all of the following HTML tags (with default settings: Format:Auto-Detect and Recipient's preferred format:unknown):

5105               element.LowerCaseEqualsLiteral("br") ||
5106               element.LowerCaseEqualsLiteral("p") ||
5107               element.LowerCaseEqualsLiteral("pre") ||
5108               element.LowerCaseEqualsLiteral("tt") ||
5109               element.LowerCaseEqualsLiteral("html") ||
5110               element.LowerCaseEqualsLiteral("head") ||
5111               element.LowerCaseEqualsLiteral("title")
5153     else if (element.LowerCaseEqualsLiteral("body"))
5246       if (element.LowerCaseEqualsLiteral("a"))

Considering how many style properties might be added to any of these, that's quite a big potential for dataloss and broken layouts, all with TB's default settings.

And it can happen to all sorts of users:
- Joe User (when copying HTML with styles from his browser into the composition)
- Advanced User (when [select text and]Insert > HTML, or double-click on link > Link properties > Advanced Edit > Inline Style)

Ben, do you prefer separate bug for dataloss of styles?

Comment 109

[Thomas D. (:thomas8)]

Format: Auto-Detect is a necessary precondition for this bug, so it's a necessary part of the bug's summary. Without this addition, the summary is insufficient to understand the problem of this bug. Don't remove this part of the summary unless you provide very good reasons why. Short summaries are not a goal in itself, if the information contained is too little.

Comment 110

[Thomas D. (:thomas8)]

(In reply to Thomas D. from comment #103)
> For other involved settings, see comment 68.
> I have some doubts that these settings interact correctly, transparently,
> and predictably in all cases so that users actually feel in control, which
> might be part of the bad UX described in this bug (-> ux-control,
> ux-error-prevention).

For another example of the amount of confusion created by the unpredictable interaction of relevant settings, even for advanced users, see Bug 766860.
Obviously, such users no longer feel in control of their UX (-> ux-control).

Comment 111

[Thomas D. (:thomas8)]

Format:Auto-Detect also prevents users from saving their HTML compositions in HTML format *directly from composition window*, for all cases of this bug where auto-detect does dataloss conversion to text/plain (<tt>, <pre>, and many other tags listed in comment 108, including their style attributes).

STR:
1 Compose msg with TB default settings
2 add <tt>, <pre>, or any tags of comment 108 with or without style attributes
3 File > Save (from composition window)
4 File > Save as > File (from composition window)
5 select saved draft msg in main 3pane mail win, then File > Save as > File

Actual result

after 4: Dialogue: "Save TEXT as..."
available file types for saving: Text files | All files
File type "HTML files" is missing - but there is a lot of HTML in user's composition...

after 5: Dialogue: "Save Message as..."
file types for saving: Mail files | HTML files | Text files | All files

Expected result

after 4: Dialogue: "Save message as..."
file types for saving: Mail files | HTML files | Text files | All files
default type: Mail files

The difference between 4 and 5 is clearly violating ux-consistency:
There is no good reason why saving saved draft behaves so different from saving current composition. If there's a single HTML tag in composition, we need to offer save as HTML, too.

I discovered that problem when testing for Bug 743939, which I think is this case (but there's a language problem, so it's not clear).

Comment 115

[Arivald]

(In reply to Ian Scott from comment #106)
> (In reply to Thomas D. from comment #105)
> > If <pre> gives such a headache, personally I don't mind just adressing <tt>
> > first, to have some progress with less discussion on this bug. Which might
> > be as easy as removing a single line of code.
> 
> I agree - there's really no easy way for a user to enter <pre> in the
> composer anyway, so just leave that alone. 

You are wrong, there is really simple way: Just select "preformatted" paragraph style. In same way You can use <p> tags.

Comment 116

[Arivald]

(In reply to Ben Bucksch (:BenB) from comment #50)
> It *is* possible to manually say that a message should be sent as HTML or
> plaintext:
> Composer | menu | Options | Format | As Rich Text (HTML) only
> (default is auto-detect)
> And that's the workaround for this bug, if you really do care about this.

This is not helping. You expect me to click this damn menu every time I send HTML mail? Every time I send my business mail, using one of my CSS styled templates? Are You nut?


(In reply to Ben Bucksch (:BenB) from comment #56)
> > the question remains though why auto-detect doesn't consider <tt> as "sufficient formatting"
> > for the purpose of auto-detect
> 
> Because plaintext mails are traditionally rendered in fixed width font. Some
> people display them as variable width (including myself), but that's not the
> assumption.

Your "fixed width font" assumption is very stupid. Even You view plain-text in variable fonts...


> The bar for sending HTML is intentionally high. It must be something that
> really alters the meaning significantly, to an extend that makes the reader
> misunderstand the message, before we suggest HTML. Color is significant,
> e.g. some people separate quote and response only by color. <tt> isn't the
> same category.

You say "Color is significant"... Thewn why this damn code drop all colors and text styles I set using CSS?
I have very good looking templates for my business mails, most of them styled in one <style> block in <head>.
And this damn "auto-detect" mis-feature broke it so often and send it all as ugly and barerly readable plain-text...


(In reply to Ben Bucksch (:BenB) from comment #79)
> <pre> we absolutely cannot make trigger HTML, for 2 reasons:
> 1. The whole definition of <pre> is that it should be considered plaintext.

****! When I compose using variable font, and I put one preformatted paragraph (implemented on <pre>), i REALLY want to have all main text to be variable width, with only this one part preformatted!


(In reply to Ben Bucksch (:BenB) from comment #79)
> <pre> we absolutely cannot make trigger HTML, for 2 reasons:
> 2. <pre> is used extensively internally both by us and other mailers to
> render plaintext in HTML.
> We are in the HTML composer, so all replies to plaintext mails go into a
> <pre>.
> If <pre> was to trigger HTML, then we would almost never send plaintext.

Exactly! If we got HTML (with <pre>), we should forward HTML, without any change! Dropping essential information is unacceptable.



And it is nothing personal. I hate ALL people who force his own twisted idea on me, or claim to have know better what I want. This is reason why I give up on Outlook, and almost give up on Thunderbird. 

Luckily for me, I finally find way to disable this "clever" code, and now I'm able to send my mail always as HTML.

Comment 117

[Arivald]

(In reply to Thomas D. from comment #104)
> (In reply to rsx11m from comment #99)
> > (In reply to Thomas D. from comment #98)
> > > This screenshot shows how user resorts to workaround of adding <b> tag in
> > > composition (and user isn't happy being forced into those workarounds, hence
> > > reported Bug 763855 *after* trying this workaround).
> > 
> > Yes, there are plenty of signature-based workarounds for what's bug 136502.
> > That's another motivation to make auto-detect meet the user's expectation in
> > general, rather than tricking the algorithm into effectively being disabled.
> 
> +1.
> Ben, believe it or not, this bug is very much *in favor* of the good
> intentions of your code (Format:Auto-Detect), namely to *send plaintext as
> often as possible* without causing dataloss(y) results. This bug has the
> intention of */improving/ Format:Auto-Detect* so that users can *continue
> using that setting*.
> 
> Alternatively, if we refuse to improve the default behaviour of our default
> settings, well then, as rsx11m says, users will just completely disable
> Format:Auto-Detect either using the current workarounds, or, after fix for
> bug 136502, just switch it off and always send HTML.

Exactly! In my Stationery extension I already add code to disable this whole auto-detection mess... I have lot of complaints from users, lot of questions why theit HTML templates ends as ugly plaintext.

Comment 118

[Arivald]

(In reply to Thomas D. from comment #108)
> (In reply to Thomas D. from comment #87)
> > Food for thought (re: auto-"converting" to plaintext): consequential styles
> > (with layout and design impact) can happen on almost any HTML tag, not only
> > on those few where auto-detect currently respects them (e.g. divs and spans).
> > I'm happy to find that it was BEN who fine-tuned auto-detection to preserve
> > divs and spans if they have styles. We need more such quality fine-tuned
> > code.
>
> Fwiw, dataloss of all css style attributes will occur for all of the
> following HTML tags (with default settings: Format:Auto-Detect and
> Recipient's preferred format:unknown):
>
> 5105               element.LowerCaseEqualsLiteral("br") ||
> 5106               element.LowerCaseEqualsLiteral("p") ||
> 5107               element.LowerCaseEqualsLiteral("pre") ||
> 5108               element.LowerCaseEqualsLiteral("tt") ||
> 5109               element.LowerCaseEqualsLiteral("html") ||
> 5110               element.LowerCaseEqualsLiteral("head") ||
> 5111               element.LowerCaseEqualsLiteral("title")
> 5153     else if (element.LowerCaseEqualsLiteral("body"))
> 5246       if (element.LowerCaseEqualsLiteral("a"))
>
> Considering how many style properties might be added to any of these, that's
> quite a big potential for dataloss and broken layouts, all with TB's default
> settings.

There is also other problem: in my templates I have all CSS in <style> tag inside <head>. So no HTML tag have explicit "style" attribute. Tags are styled, but You can't know about it by just analysing tags.

Yet another problem if re reply to mail with <style> block.

So finally I think it is better to do not degrade mail if there is ant HMTL tag.

And case when we use HTML composer to edit text in plain-text mode is very simple. We do not need to bother about internal use of <pre> or other background magic. All we need is to check composition mode (gMsgCompose.composeHTML), and send plain or HTML accordingly.

Comment 119

[Thomas D. (:thomas8)]

Getsatisfaction has more than enough evidence that the current interaction of settings to send HTML isn't transparent and predictable enough, and users who are confused that their HTML compositions do not get sent as HTML, but as plaintext.

http://getsatisfaction.com/mozilla_messaging/tags/bug_414299

And many users have realized that Format: Auto-Detect doesn't act according to their expectations, resulting in the understandable request to just switch it off.

http://getsatisfaction.com/mozilla_messaging/tags/bug_136502
(among these, http://gsfn.us/t/16j03)

Comment 120

[Thomas D. (:thomas8)]

(In reply to Thomas D. from comment #119)
> Getsatisfaction has more than enough evidence that the current interaction
> of settings to send HTML isn't transparent and predictable enough, and users
> who are confused that their HTML compositions do not get sent as HTML, but
> as plaintext.

...and while the obvious need for *documentation* of the current confusing behaviour was recognized more than 2 years ago (bug 560633, triggered by GS report (1)), afasics all we have in terms of documentation is an empty stub (2) and a traditional mozillaZine article (3), not linked from SuMoMo, and misleadingly claiming that

> If your message does not use any HTML features (bold, underline, italics,
> colors, etc.), and if Thunderbird has no information about the recipients'
> preferences, then it sends plain text by default. 

<tt>, <pre> and <a> are HTML features just as well as <b>, <u>, and <i>...
The article also hastens to add some of the known workarounds for disabling Format:Auto-Detect.

(1) https://getsatisfaction.com/mozilla_messaging/topics/send_in_html_ends_up_in_plain_text
(2) https://support.mozillamessaging.com/en-US/forums/kb-articles-needs-draft/50
(3) http://kb.mozillazine.org/Mail_content_types#Sending_messages

Comment 121

[Ben Bucksch (:BenB)]

> This is not helping. You expect me to click this damn menu every time I send HTML mail?
> Every time I send my business mail, using one of my CSS styled templates? Are You nut?

No to both. CSS styles are *not* part of this bug. This bug is specifically only about fixed width vs. variable width.

And, no, I am not nuts, and comments like that are not helpful.

Comment 122

[Arivald]

(In reply to Ben Bucksch (:BenB) from comment #121)
> > This is not helping. You expect me to click this damn menu every time I send HTML mail?
> > Every time I send my business mail, using one of my CSS styled templates? Are You nut?
>
> No to both. CSS styles are *not* part of this bug. This bug is specifically
> only about fixed width vs. variable width.

Yes, this bug is not about CSS. But functionality mentioned in this bug, "auto detection", broke HTML mails in so many ways, including CSS styled mails. IMHO it should be removed, because it broke too much, for too small gain. 

Fixing this bug in current form will be like fixing shape of car wheels, from square to pentagonal - it will be a bit better, but it not work properly anyway.

Guess why nobody else in the world use such "auto-detection"?

 
> And, no, I am not nuts, and comments like that are not helpful.

Sorry, it just frustration. It was very unpleasant discovery to find why TB does not send my mail as HTML. Very unpleasant.

Comment 123

[Arivald]

(In reply to Arivald from comment #117)
> (In reply to Thomas D. from comment #104)
> > (In reply to rsx11m from comment #99)
> > > (In reply to Thomas D. from comment #98)
> > > > This screenshot shows how user resorts to workaround of adding <b> tag in
> > > > composition (and user isn't happy being forced into those workarounds, hence
> > > > reported Bug 763855 *after* trying this workaround).
> > > 
> > > Yes, there are plenty of signature-based workarounds for what's bug 136502.
> > > That's another motivation to make auto-detect meet the user's expectation in
> > > general, rather than tricking the algorithm into effectively being disabled.
> > 
> > +1.
> > Ben, believe it or not, this bug is very much *in favor* of the good
> > intentions of your code (Format:Auto-Detect), namely to *send plaintext as
> > often as possible* without causing dataloss(y) results. This bug has the
> > intention of */improving/ Format:Auto-Detect* so that users can *continue
> > using that setting*.
> > 
> > Alternatively, if we refuse to improve the default behaviour of our default
> > settings, well then, as rsx11m says, users will just completely disable
> > Format:Auto-Detect either using the current workarounds, or, after fix for
> > bug 136502, just switch it off and always send HTML.
> 
> Exactly! In my Stationery extension I already add code to disable this whole
> auto-detection mess... I have lot of complaints from users, lot of questions
> why theit HTML templates ends as ugly plaintext.

I got user requests to make simple extension just to disable this mis-feature... I made it and posted to AMO. https://addons.mozilla.org/thunderbird/addon/always-html/ Hope it will help other poor souls troubled with this bug.

Comment 124

[Thomas D. (:thomas8)]

Arivald's two addons provide workarounds for this bug:

1. Plain vanilla: This addon effectively bypasses {Delivery Format: Auto-Detect} and sends HTML always for HTML compositions (amazing, ain't it?), so it's also a workaround for Bug 136502. As a disclaimer, I must add that this defeats the purpose of Auto-Detect, but I guess we don't have much of a choice until the Auto-Detect algorithm will be improved to cause less formatting dataloss (and probably to stop downgrading for recipients with unknown preference):

https://addons.mozilla.org/thunderbird/addon/always-html/

2. If you want to use HTML files as templates/letter paper for your messages, try this (and obviously, those require and ensure sending as HTML, too):

https://addons.mozilla.org/de/thunderbird/addon/stationery/

Comment 125

[Thomas D. (:thomas8)]

FTR: Starting point in the code:

http://mxr.mozilla.org/comm-central/source/mail/components/compose/content/MsgComposeCommands.js#4111

4111 function DetermineConvertibility()
4112 {
4113     if (!gMsgCompose.composeHTML)
4114         return nsIMsgCompConvertible.Plain;
4115 
4116     try {
4117         return gMsgCompose.bodyConvertible();
4118     } catch(ex) {}
4119     return nsIMsgCompConvertible.No;
4120 }

Arivald's Always-HTML addon simply replaces this function with the following:

function DetermineConvertibility() {
  if (gMsgCompose.composeHTML) return nsIMsgCompConvertible.No;
  return nsIMsgCompConvertible.Plain;
}

Comment 126

[Luke Plant]

The Format:Auto-detect feature is clearly totally flawed, and needs to be removed entirely as far as I can gather, because of the following considerations:

1. Everything that has already been said a hundred times about software doing as it is told. [rest of rant snipped]

2. Beyond what has already been written: Many, many people are now viewing email on devices with small screens, and the trend is accelerating massively (see http://www.smartinsights.com/digital-marketing-strategy/internet-trends-2014-mary-meeker/ for example).

The TB behaviour of converting 'unformatted' emails (however that is detected) to plain text means that it hard wraps the email. The result is that emails sent from TB come out looking mangled on small screens. Please see attachment for example on my own phone (in landscape orientation). This is, quite simply, terrible default behaviour. Having to choose some option for every email, or install some extension, is not an adequate solution.

Because of this hard wrapping, there is no way to safely downgrade to plain text, no matter what formatting or lack of formatting is applied. It is a misfeature that needs to be removed.

Comment 127

[Luke Plant]

Attachment: Effect of hard wrapping when viewing on mobile phone

Comment 128

[Ben Bucksch (:BenB)]

> plain text means that it hard wraps the email

It doesn't. We use "format=flowed", and Internet standard to use soft (!) line wraps in plaintext emails. Many email programs support it, including Apple Mail and Hotmail, and even console clients like Pine, from what I read. Apparently yours doesn't. You might want to ask them to support it, if you care about this, or use another program.

http://joeclark.org/ffaq.html

Comment 129

[Arivald]

Funny. How line breaks in plain text (CR|LF) could be "soft"? 
Yes, I know what the "format=flowed" means, but is is just a bit hacky way to tell which line breaks should be removed. And yes, a good MUA should handle this... Not all of them are good enough, though.

On the other hand, the HTML by its nature require explicit line breaks, the text is flowed automatically by any MUA. Overhead of the HTML over the plain-text is small enough, while ways to properly format the messages are very rich. The HTML is just better.

In this context, it is a shame that TB forcefully degrade the HTML to plain text, loosing sometimes a lot of information and formatting... It was infuriating for me, until I disabled it for myself.

I do not think that "auto-detect" mis-feature will be removed from TB soon... Too many plain-text worshipers controls development of TB.

Comment 130

[Luke Plant]

(In reply to Ben Bucksch (:BenB) from comment #128)
> > plain text means that it hard wraps the email
> 
> It doesn't.

You appear to be using unique definitions of what "soft wrap" and "hard wrap" mean. That really doesn't help communication…

> We use "format=flowed"

At first I couldn't reproduce this. On all the emails that have been auto-converted to Plain Text, there is no "format=flowed" added. In Thunderbird itself (e.g. in outbox, as well as inbox if I send to myself), they display hard wrapped. (Using v 31.4, which appears to be the most recent I can download).

However, it looks like this might be the consequence of using the Enigmail add-on. Still, that is a pretty common and very important extension.

Comment 131

[WADA:World Anti-bad-Duping Agency]

(In reply to Thomas D. from comment #125)
> FTR: Starting point in the code:
> 
> http://mxr.mozilla.org/comm-central/source/mail/components/compose/content/
> MsgComposeCommands.js#4111
> 
> 4111 function DetermineConvertibility()
> 4112 {
> 4113     if (!gMsgCompose.composeHTML)
> 4114         return nsIMsgCompConvertible.Plain;
> 4115 
> 4116     try {
> 4117         return gMsgCompose.bodyConvertible();
> 4118     } catch(ex) {}
> 4119     return nsIMsgCompConvertible.No;
> 4120 }
> 
> Arivald's Always-HTML addon simply replaces this function with the following:
> 
> function DetermineConvertibility() {
>   if (gMsgCompose.composeHTML) return nsIMsgCompConvertible.No;
>   return nsIMsgCompConvertible.Plain;
> }

Auto Detect does do two different jobs.
  (a) Auto Downgrade to Text from HTML
  (b) Auto Delivery Format determination(or asking) only when mail is HTML, based on preference of HTML or Text in Address Book and Send Option.
This bug is for (a), and complaints of this bug is for logic of gMsgCompose.bodyConvertible().
AlwaysHTML addon looks to have worked around problem by "never call gMsgCompose.bodyConvertible()".

A problem in current code.
(a) is done first, regardless of per recipient HTML/Text preference in address book, and once "Downgrade to Text" is recomended by  DetermineConvertibility() and gMsgCompose.bodyConvertible(), there is no way to send "mail composed in HTML by user long way" in HTML, and there is no chance of "(b) works" because state of the mail is already "Text mail".

To Ben Bucksch (:BenB).
Is "making (a) optional" possible?
Or it's already optionsal?  User can always choose (i) Don't install AlwaysHTML addon, and (ii) Install AlwaysHTML addon.

Is "change sequence of (a) and (b)" possible?
  If all recipients prefers Text, try to down grade to Text by (a), 
  then, if "conver tto Text is rejected by gMsgCompose.bodyConvertible(), ask to user.
 (New Send Option like "try to downgrade to Text if all recipients prefer Text" my be needed for this)

Comment 132

[Ant]

Others and I just discovered this issue too. All I did was centered a section/paragraph. I did not bold, color, etc. SeaMonkey v2.33.1 thought I was sending a plain text without asking. :(

Comment 133

[Thomas D. (:thomas8)]

I have provided extensive evidence in many comments above why the current behaviour of and around {Delivery Format: Auto-Detect} violates ux-principles. While this bug's summary was deliberately and repeatedly mutilated by Ben to play down the real issues affecting our users, this bug does have substantial discussion of the issue including detailed testcases and user feedback. Also, the download statistics for Arivald's "Always HTML" addon which simply disables the Auto-Detect/Auto-Degrade HTML to plaintext "feature" speak for themselves:

12,027 Downloads (and counting, 233 in the last 30 days), 3 years after publication on 28th June 2012
 1,742 Users per day on average (1,518 in the last 30 days)

And that's just the tip of the iceberg, because the auto-degradation happens so cunningly that many users may never notice, and if they do, they might know about the addon. And God only knows how many thousands/millions of users have switched Auto-Degrade off by using the known workarounds like HTML tags in their signature etc.

The UX problems at hand are (even for the mutilated version of this bug which tries to create the wrong impression that this is just about <tt>):

ux-wysywig: Users rightly expect that their carefully composed HTML-formatted message will get *sent* exactly as seen on screen, and in the HTML source. We can't just drop formats silently and post-facto. Users never asked for auto-degradation; we're forcing it on them per default settings. This is about the integrity of the *message (source)* which gets sent. HTML is a widely used and well-defined standard with very high chances of being rendered on the recipient side exactly as sent. Recipient's actual ability or willingness to render the message correctly in HTML-format is not relevant for the integrity of the message sent; it's true that I can't stop any recipient from crumpling up, shredding, or burning my letter if he so pleases.

ux-control: No matter how much users try to set everything to HTML (per-account, per-recipient, send-HTML-anyway), auto-degrade will still kill their formatting by default, which makes them feel not in control of their software.

ux-error-prevention: Auto-degrade dataloss occurs unexpectedly (ux-wysiwyg), and cunningly/silently *after* sending, in spite of users maximum efforts to compose in HTML to HTML recipients. User might never notice if he doesn't check sent message or gets feedback about garbled message from recipient. Even recipient can't know for sure if perhaps sender was lazy or inexperienced and actually sent poorly formatted garbage, most of us don't usually write back to say "what's wrong with your formatting, please do better next time!". Everything in the UI suggests nicely-formatted HTML all the way long, and then we auto-degrade it in the last minute. Clearly violating ux-error-prevention.

ux-efficiency: Users who generally prefer composing plaintext messages can use plaintext composition mode. We offer HTML as default composition mode and send Plain+HTML by default when in doubt with mixed recipients. It's absurd to force users who have done everything in their power to compose in HTML into confirming that choice for every single message, to ensure their HTML-formatting will actually get sent. And then you never know which formatting is "worthy" enough to survive auto-degrade. As Comment 132 correctly reports, even consequential basic formatting like centering text is silently dumped when sending - wtf!? To force users of HTML composition, even with explicit HTML recipients, to confirm their choice for every message is a clear violation of ux-efficiency. For the minority of those who prefer to format their messages in HTML and then have those formats dumped, please use plaintext editor, or make your recipients plaintext-only, or YOU can change that per-message format setting to get the non-standard behaviour. A more practical solution for both types of users is something along bug 136502, i.e. to allow users to set their own per-account default for delivery format between (Auto|Plaintext|HTML|Both). Nevertheless, Auto-Detect, especially as long as it is the default format, must be tamed to avoid dataloss of formatting.

ux-consistency: As I have shown in my testcases attached to this bug, and plenty of user testimony, there's absolutely no way to predict *which* formatting will be auto-degraded or not. <b> survives the purge, <div align="center"> doesn't. Styles on <div> survive, styles on other elements like <a href> get ripped out. There's no consistency whatsoever. Mostly because consistency would require to preserve every HTML formatting now matter how little. Even plain <p> has relevant formatting because it structures text into paragraphs with vertical padding. The whole idea of auto-degrading has outlived its purpose. We no longer need to fight for every byte on slow modem connections, and every reasonable mail client can handle HTML. For those who can't, please compose in plaintext mode. Why compose in HTML mode and expect that to be auto-degraded into plaintext just before sending, with unpredictable results which you've never seen on your screen? It just doesn't make sense.

As a high-profile member of TB QA/UX, I'm explicitly setting these ux-flags on this bug because they are relevant for understanding the discussion and making informed decisions about the general problem at hand (not just the <tt> to which this bug has been misleadingly curbed).

*** Evolutionary perspectives - beyond auto-degrade by auto-detect ***

I'm starting to feel that really Auto-Detect should no longer be the default (or even ripped out entirely to become addon fodder), because we'll never get it right as the whole idea is flawed in principle, like Arivald says in comment 122:

> Fixing this bug in current form will be like fixing shape of car wheels,
> from square to pentagonal - it will be a bit better, but it will not work properly anyway.

I'm also more and more failing to see the point of Auto-Detect: As we can't, for the default HTML case, strip *any* HTML formatting which might have the slightest formatting implications (ux-wysiwyg), practically (even now), we can only downgrade to plaintext when there's really NO formatting. However, when there's virtually no formatting, the reduction of message size will be totally insignificant, as auto-degrade will just rip out one or two lines in the header and one or two html tags, perhaps only the <html> tags around the body. That's a lot of very complex, performance-expensive and error-prone application code for little gain. And, as mentioned before, in the age of 100Mbit DSL and Terabytes hard disk capacity, HD-photo attachments etc., finetuning the message size in the range of (kilo)bytes is really ridiculous, more so compared to the big annoyance of unexpected, silent formatting dataloss for the majority of our users (and yes, it's often silent because the question doesn't come up in all cases even if you chose the question, see WADA's spot-on analysis of comment 131).

For taming auto-degrade, WADA's comment 131 sounds like a good starting point. Also bug 136502.
And, it's about time that we stop misreading "Unknown" per-recipient delivery format preference as a preference for plaintext. The whole world runs HTML. Firefox OS apps are just HTML, javascript, and CSS. HTML is not the enemy. Proprietary languages and devices are. Welcome to the real world.

Comment 134

[Thomas D. (:thomas8)]

(In reply to Thomas D. from comment #133)
> Also, the download statistics for Arivald's "Always HTML"
> addon which simply disables the Auto-Detect/Auto-Degrade HTML to plaintext
> "feature" speak for themselves:
> 
> 12,027 Downloads (and counting, 233 in the last 30 days), 3 years after
> publication on 28th June 2012
>  1,742 Users per day on average (1,518 in the last 30 days)
> 
> And that's just the tip of the iceberg, because the auto-degradation happens
> so cunningly that many users may never notice, and if they do, they might
> know about the addon.

Typo: ...even if they DO notice silent post-facto degradation, they might NOT know about the addon. Iow, many users affected by auto-degrade are not included in the total download or user statistics of the addon.

> And God only knows how many thousands/millions of
> users have switched Auto-Degrade off by using the known workarounds like
> HTML tags in their signature etc.

Comment 135

[Thomas D. (:thomas8)]

(In reply to Thomas D. from comment #133)
> addon which simply disables the Auto-Detect/Auto-Degrade HTML to plaintext
> "feature" speak for themselves:
> 
> 12,027 Downloads (and counting, 233 in the last 30 days), 3 years after
> publication on 28th June 2012
>  1,742 Users per day on average (1,518 in the last 30 days)

Source: https://addons.mozilla.org/de/thunderbird/addon/always-html/statistics/

Comment 136

[Thomas D. (:thomas8)]

(In reply to Ben Bucksch (:BenB) from comment #128)
> > plain text means that it hard wraps the email
> 
> It doesn't. We use "format=flowed", and Internet standard to use soft (!)
> line wraps in plaintext emails. Many email programs support it, including
> Apple Mail and Hotmail, and even console clients like Pine, from what I
> read. Apparently yours doesn't. You might want to ask them to support it, if
> you care about this, or use another program.
> 
> http://joeclark.org/ffaq.html

Yes, only that the user never asked for plaintext + f=f, but we're forcing him into it, even with all settings/prefs set to HTML, except Auto-Detect. I would be interested to see in-product or other easily accessible documentation which explains to our users that Auto-Detect is NOT, as the label suggests, just trying to decide on the best delivery format given the delivery format preferences of each recipient; rather, what we call Auto-*Detect* involves massive, unpredictable, and silent Auto-*degrade* of consequential HTML formatting into plaintext, even for recipients whose explicit preference is "HTML" (if combined with "unknown" recipients).

Please compare:
before auto-degrade (attachment 633913 [details] [©] [details]) vs.
after auto-degrade (attachment 633910 [details] [©] [details]).
Is that "lossless conversion"!!!???

So the correct label of Auto-Detect would be "Prefer plaintext at all costs". I'll forever fail to understand what entitles Ben Bucksch to decide for me and all users of HTML composition mode out there which HTML is consequential formatting and which isn't. Apart from the inconsistencies that depending on HOW you apply it, one and the same formatting will be discarded OR preserved (e.g. <div align=center> used by TB vs. <div style=text-align:center>). Looking at my testcases at the top of this bug, the promise to "guarantee lossless conversion" (as a comment in the code suggests) is clearly NOT honored, the potential formatting dataloss (not only) in styles is massive (even entire inline images get wiped).

Given that the user never asked for "Prefer plaintext at all costs" (after all, we are in HTML editor, those who prefer can always use Plaintext editor), I find it arrogant to tell our users either of:
- Why don't you just confirm for every message that you want HTML and you actually MEAN IT?
- Why don't you just ask your email client to support sophisticated plaintext standards like f=f?
While the largest part of the world wide web, and even entire mobile operating systems in the open ecosystem that Mozillians are part of and campaign for, are using which open standard for formatted content again? Is it HTML, or is it plaintext+f=f!???

Rather than forcing our users to adapt to the ideosyncracies of some age-old auto-degrade-and-then-detect algorithms, TB should seek to adapt to changing times and offer the best possible UX which is based on tried and tested UX principles. Please refer to my Bug 414299 Comment 133 for details about all the UX-principles which auto-degrade-and-detect is violating. If we continue to ignore the legitimate concerns of our users, and unpredictably downgrade formatted messages into plaintext+f=f garbage, the ultimate reaction of affected users will be plaintext+wtf!!?? or even just plain f*** TB. Which is clearly NOT what we want, if by way of listening, openness, honest, and argument-driven discussion we can find better solutions.

For starters, can somebody (Ben, perhaps?) explain to me why we auto-detect does NOT honor the following pref:

When sending messages in HTML format and one or more recipients are not listed as being able to receive HTML" (i.e. prefers "unknown" or "plaintext" in AB card): e.g. ASK me!

Right, so I'm composing a full-fledged HTML-message including inline images, loads of substantial formatting like bold, colors, styles etc. you name it.
Then I pick uncle Ben as my recipient who happens to be marked in my AB as "prefers plaintext" but alas, TB is giving me so much headaches that I forgot what he prefers, and just click SEND (with auto-detect).
SO, we're talking about exactly that pref case of "sending messages in HTML format and one recipient is not listed blabla", SO, I should be ASKED because that's what I set, isn't it?

Guess what REALLY happens when you press SEND: TB detects "allPlain", and just DUMPS your entire full-fledged HTML formatted message, images and all, WITHOUT any warning whatsoever (AGAINST your explicit pref which says ASK me in that case).

That auto-degrade-and-detect "feature" is a BUG more often than not. I could go on with "Unknown" cases which fail in similar ways and also affect/ignore any "prefers-HTML" recipients which happen to be in the same message with somebody "unknown" or "prefers-plaintext"...

Comment 137

[:aceman]

So is anybody AGAINST moving "tt" from the convertible::Plain to convertible::Altering block at http://mxr.mozilla.org/comm-central/source/mailnews/compose/src/nsMsgCompose.cpp#5027 ? Similarly to e.g. <b>. Or do you want it to be full convertible::No as e.g. <font> seems to be?
Both of the options should cause the message no longer be convertible to plain text and at it will be sent as HTML or the user asked about downgrading. At least as long as some of the other related bugs like bug 1202227 are fixed :)

Comment 138

[:aceman]

Attachment: patch

So in the attached examples I could only see a use of <tt> and 'style' attributes on pre and <a> and other elements.
So I propose to declare a message unconvertible if any 'style' is encountered.

Comment 139

[:aceman]

Attachment: patch v2

Sorry, incorrect patch.

Comment 140

[:aceman]

Attachment: WIP patch v3

This version makes it even more strict, e.g. if <div> and <span> and <a> have any other attributes than our internal ones, it causes inconvertability.

This still makes a message composed in HTML mode than does not use any formatting options to be considered convertible to plain text. However, any replies in HTML mode aren't convertible, as the reply header is already some formatting.

Comment 141

[rsx11m]

(In reply to :aceman from comment #140)
> any replies in HTML mode aren't convertible, as the reply header is already some formatting.

Hmm, so you are saying that if I reply to a plain-text message without using any formatting whatsoever in my reply, but happen to be in HTML composition mode, that alone will be sufficient already to trigger sending a text/html part? That's rather unexpected...

Comment 142

[:aceman]

That is what we need to discuss here. That case could be special cased as there is only a limited set of tags/attributes used in the reply header. But surely some people would say that yes, it is already HTML formatting and should be preserved. As converting that header will not be 100% visually the same. See also last comments in bug 1204379.

Comment 143

[rsx11m]

Specifically that's the header with <div class="moz-cite-prefix"></div> followed by the quote, <blockquote cite="..." type="cite"></blockquote>

A reply to a plain-text message contains <pre> tags within the <blockquote type="cite"> which would probably trigger sending a text/html part. Maybe ignoring everything in that <blockquote> if the original message didn't contain a text/html part is the way to go, but may be a bit more complex (and, when replying to an HTML message I'd expect my reply to be sent as HTML as well rather than downconverted to plain text, even if it's feasible without formatting in that msg).

Comment 144

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Try push: https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=dea5a2d67bb8

Comment 145

[:aceman]

(In reply to rsx11m from comment #143)
> Specifically that's the header with <div class="moz-cite-prefix"></div>
> followed by the quote, <blockquote cite="..." type="cite"></blockquote>

Yes, the blockquote type="cite" is already special-cased as being convertible.
If I add exception for <div class="moz-cite-prefix"> and <pre wrap="">, plaintext quotes/replies can be convertible again.
If that is what we want.

> A reply to a plain-text message contains <pre> tags within the <blockquote
> type="cite"> which would probably trigger sending a text/html part. Maybe
> ignoring everything in that <blockquote> if the original message didn't
> contain a text/html part is the way to go, but may be a bit more complex
> (and, when replying to an HTML message I'd expect my reply to be sent as
> HTML as well rather than downconverted to plain text, even if it's feasible
> without formatting in that msg).

If the quoted message or your new text contains any other tag (other than <br> and <p>) then the message is not convertible to plain text again.

Also note all this downconverting happens only if send format is "autodetect" and your recipients prefer plain text. If all of them want HTML (per you addressbook contact properties), HTML is forced on any message (unless you compose directly in plain text). If there is a mix or some recipients with unknown preferences, bug 1204379 aims to ensure the Send Options in prefs is honored. And the default there is "send both formats".

Comment 146

[Arivald]

I appreciate Your effort, but I think You missing the point here. The point is to completely remove this "is this message a plain text" guessing. Why? Because it is unreliable. And I strongly believe that it isn't fixable anyway.

For example, do You check for <style> blocks? The CSS can change a lot of rendering, even for tags/attributes You consider internal only. In my mail template I have styles for the <blockquote cite="..." type="cite"> and other, just to make TB and other MUA render them in my way.

Instead of this guess work, there should be simple rule: You edit in HTML - send in HTML, You edit in plain, send plain. Simple, no guessing, no confusion, no silent loss. There is a lot of evil in automatic conversion, when user is not able to check and fix the conversion result.

Then if You add a way to easily switch between HTML and plain, user will be always able to send plain or HTML, as he needs. Plus some check when a recipient is added, and warn user that some recipients prefers plain messages, so user could choose to switch to plain, or send the HTML anyway. Let the user decide. Power to the people ;-)

Comment 147

[:aceman]

(In reply to Arivald from comment #146)
> For example, do You check for <style> blocks? The CSS can change a lot of
> rendering, even for tags/attributes You consider internal only. In my mail
> template I have styles for the <blockquote cite="..." type="cite"> and
> other, just to make TB and other MUA render them in my way.

Yes, we detect <style> (that it is not listed in the function I linked to, means the tag will cause inconvertibility of the message. I also chceck for 'style' attribute now.
However, how do you include <style> into the message in Thunderbird?

> Instead of this guess work, there should be simple rule: You edit in HTML -
> send in HTML, You edit in plain, send plain. Simple, no guessing, no
> confusion, no silent loss. There is a lot of evil in automatic conversion,
> when user is not able to check and fix the conversion result.

I am sure we can add this option. Can you please file a bug for this proposal? That we add an option in Options->Delivery format to honor the composition more instead of Autodetect. I think I can implement that easily.

> Then if You add a way to easily switch between HTML and plain, user will be
> always able to send plain or HTML, as he needs. Plus some check when a
> recipient is added, and warn user that some recipients prefers plain
> messages, so user could choose to switch to plain, or send the HTML anyway.
> Let the user decide. Power to the people ;-)

Yes, the user can already choose, there are Send Options in the Preferences(tools->options). We just need to make it work properly in bug 1204379 :)

Comment 148

[Thomas D. (:thomas8)]

There are so many ways of adding formatting even to simple <p> tags. I maintain downgrading HTML to plaintext is NEVER lossless.

What if user references an external style sheet on his company's web server, and adds a rule there which formats all <p> tags without using a class or style attribute? Will we preserve that?

What about <div align=...> is that preserved now (existing bug)?

Comment 149

[:aceman]

(In reply to Thomas D. (currently busy elsewhere; needinfo?me) from comment #148)
> There are so many ways of adding formatting even to simple <p> tags.
Like what?

I asked in bug 1204379 if we can agree to consider <p> convertible, but <p anything else> inconvertible. What ways do you see to add style to simple <p>? Outside of <style> block which is the other question.

> I maintain downgrading HTML to plaintext is NEVER lossless.
Maybe we come to that conclusion, but I am not yet convinced. Depends on how many moles we need to whack to cover all the corner cases :)

> What if user references an external style sheet on his company's web server,
> and adds a rule there which formats all <p> tags without using a class or
> style attribute? Will we preserve that?
Maybe. How exactly do you reference an external style sheet? Using <meta> in <head>? Is that actually accessible in TB?

> What about <div align=...> is that preserved now (existing bug)?
Yes. On a <div> or <span> I allow only our internal classes. Any other attribute or class or style causes inconvertibility.

I have asked for a try build for you. Please try it out and experiment with your example html.

Comment 150

[Ronald Killmer]

An additional method of styling messages is placing the CSS within the <head></head> tags where there scope is message global.  Inline tags are are a local scope effecting only the text wrapped by a tag pair.

Users who are serious about HTML composition are using HTML editors which expose the head for inclusion of CSS.

Comment 151

[rsx11m]

(In reply to Arivald from comment #146)
> Instead of this guess work, there should be simple rule: You edit in HTML -
> send in HTML, You edit in plain, send plain. Simple, no guessing, no
> confusion, no silent loss.

Adding an option to switch off auto-detect entirely is bug 136502.

Comment 152

[Axel Grude [:realRaven]]

(In reply to Ronald Killmer from comment #150)
> An additional method of styling messages is placing the CSS within the
> <head></head> tags where there scope is message global.  Inline tags are are
> a local scope effecting only the text wrapped by a tag pair.
> 
> Users who are serious about HTML composition are using HTML editors which
> expose the head for inclusion of CSS.

True, but using the head tag can be unreliable especially if you are adding rules scoped to body > *; some mail clients will drop these but worst case can be that you are affecting the font of the replying person. I had a lengthy support thread with a SmartTemplate4 user who wanted his replies to be blue (can't find it right now) and it was very hard to make this robust without using inline styles and affecting the next reply.

Also, to support html, Thunderbird composer really needs an integrated "edit source code" tab (the one from Stationery is faulty and breaks when there are images)

Comment 153

[Ronald Killmer]

The editor I have used is EditHTML by Kaosmos. It is a decent source code editor with a structured layout, not the run together lines of the tiny TB Insert HTML dialog. Makes use of the full size of the compose window with a click of a toolbar button on the compose toolbar. 

The purpose of my comment was to ensure that any effect of style in a head block would be evaluated when testing patches.

Comment 154

[:aceman]

(In reply to Ronald Killmer from comment #153)
> The purpose of my comment was to ensure that any effect of style in a head
> block would be evaluated when testing patches.

Sure, a <style> block is obviously the way I have in mind and should be covered already (even without my patch). I ask if there are any special options we did not cover yet.

Comment 155

[:aceman]

Thomas, WADA, please try the build at http://ftp.mozilla.org/pub/mozilla.org/thunderbird/try-builds/[e-mail]-dea5a2d67bb8/ . Only use on test profiles as there are some crashes in it when managing accounts.

Comment 156

[:aceman]

Has anybody tried the build yet?

Comment 157

[Thomas D. (:thomas8)]

Sorry, not yet tried the build.

Comment 158

[Thomas D. (:thomas8)]

(In reply to :aceman from comment #155)
> Thomas, WADA, please try the build at
> http://ftp.mozilla.org/pub/mozilla.org/thunderbird/try-builds/
> [e-mail]-dea5a2d67bb8/ . Only use on test profiles as
> there are some crashes in it when managing accounts.

I now have the try build up and running.

Comment 159

[WADA:World Anti-bad-Duping Agency]

Style case looks separated to bug 674184. i.e. Independent bug. See bug 674184 comment #4.
In which bug is Style case is processd?

Anyway, setting dependency to bug 674184 for ease of tracking/search.

Comment 160

[WADA:World Anti-bad-Duping Agency]

(In reply to :aceman from comment #140)
> WIP patch v3

To aceman.
(Q1) No change for "simple link" in "with <tt> for fixed width sections or with simple links" of bug summary?

"simple" in the "simple link" looks following.
  (i)  simple     : <a> with Link Text == Link URL (created by Paste after Copy of a link in application)
  (ii) not simple : <a> with Link Text != Link URL (created by Insert Link of Composer of Tb).
In "not simple" case of bove, it seems that "sent as text/plain" doesn't occur. 

According to Comment #25 on 2010-06-26, following case seems design.
  link that is equal to the link text
Term of "simple link" was introduced to Bug Summary of this bug on 2012-06-14.

(Q2) I think change for Style is better done in bug 674184 for style.
  Please see bug 674184 comment #4 and Change history of this bug for Bug Summary of this bug.
  Obviously, separation of bug for "<tt> and friends" and bug for "style" is intentionaal.

Comment 161

[WADA:World Anti-bad-Duping Agency]

(Off-Topic)

Comment #121 on 2012-06-24.
> No to both. CSS styles are *not* part of this bug.
> This bug is specifically only about fixed width vs. variable width.

What is actual problem outstanding in this bug? (except isssues written in bug summary)
  - with <tt> for "fixed width" sections, get sent as plain text
  - with simple links, get sent as plain text  
Shall I open bug like next?
  Need option to choose one of following :
    (a) "link that is equal to the link text" is always sent as text/plain
    (b) "link that is equal to the link text" is always sent as text/html

According to coment in this bug, "Link of Link Text==Link URL is Convertible" is current design, as PRE case is current design. (I don't know about P only case).
If "Link of Link Text==Link URL is Convertible" is current design(I belieave it's reasonable, changeable though),
and if some users were confused by it,
there are next 2ways only.
 (i)  Change design of Convertible check for "Link of Link Text==Link URL"
 (ii) Find a good way for user to reduce user's confusion or good way to send as text/html based on user's want.

I don't think that "process or try to change all cases in only one bug merely by problem is relevant to one Convertible check function" is good practice.
If outstanding cases, which is relevant to tag's characteristics itself(excluding style tag), are two cases in current bug summary only, I think following is better.
  1. Close this bug as FIXED/VERIFED, because many cases ware fixed by this bug.
     This bug is already in hard-to-read, hard-to-understand state.
  2. Open separate bug for each specific case, in order to get back Readability,
     in order to make bug easy-to-understand, easy-to-follow state.

Comment 162

[WADA:World Anti-bad-Duping Agency]

FYI.
According to History of this bug, it look for me;
 1. Bug was opened(2008-01-27).
 2. Closed as WONTFIX(2010-06-26), because of current design.
 3. As <tt> for Fixed Width case remained, re-opened(2010-06-26), and accepted(because not closed again).
 4. After it, term of "<tt>" and term of "simple link" were added to Bug Summary"(2012-06-14)
 5. Many comments, many attachmens were added to this bug.
 6. Recently, patch for "<tt>" and "style attribute of tags" were attached to this bug(latest is on 2015-09-17).

Comment 163

[WADA:World Anti-bad-Duping Agency]

attachment 8670368 [details], which is attached to Bug 1210244, is an HTML example based on bug summary of this bug(<tt> and simple link).
Phenomenon is pretty simple in <tt> case of this bug.
  If HTML, by mailer of recipient, Fixed Width Font is used for <tt>, <pre> etc..
  If Text, in mailer of recipient, used font depends on setting in the mailer,
           and "Variable Width Font for P, Fixed Width Font for <PRE> etc. in HTML" doesn't work. 
If HTML tag is not transfered by Tb upon mail send, formatting of HTML by mailer of recipient is not done.
This is a matter of course. 
If current problem in this bug is relevant to "<tt> and simple link", I can't understand reason why many attachments and comments are needed in this bug. I can't understand reason why repeated comments of "format loss or dataloss" were posted in some other bugs such as bug 1204379.

As for <TT>, there is not such tag in HTML 5.
http://www.w3schools.com/tags/tag_tt.asp
A big fault in Tb :
- Even though Tb already switched to HTML 5 in Parser and message display,
  Composer still generates tag which doesn't exist in HTML 5.

I believe proper solution in Composer of Tb for issue like "<tt> relevant issue" is:
  Utilize CSS in Composer of Tb, instead of using obsoleted tag or <FONT> etc.
  Support Style(both tag and attribute) in Convertible check logic in Tb. (Bug 674184)

Comment 164

[WADA:World Anti-bad-Duping Agency]

Bug 389417 is for "<P> and <PRE>" case.
IIUC, "<P> and <PRE> is in same group as Bod/y Text, <html>, <head>" is current design and it'll not be changed.
If so, this bug is not relevant to that bug unless "<P> and <PRE>" case will be resolved by this bug.
Why Bug 389417 was put in Blocks: field of this bug?
"<P> and <PRE>" case will be resolved by this bug?

Comment 165

[WADA:World Anti-bad-Duping Agency]

Quickly read thru this bug.
In 2019, WONTFIX'ed, and quckly re-opened. 
In 2010, <tt> for Fixed Width(<tt> is used to use Fixed Width Font) only was known.
Some comments on <pre> are seen.
In 2012, "link of link text==link URL" case(simple link) was added.

To all problem reporters in this bug since 2012 :
   Is you report/comment for other case than "<tt> for Fixed Width and/or simple link" case?
   Or report/comment on "<tt> for Fixed Width and/or simple link" case?
   For style, it should be processed in Bug 674184. So exclude, please.

Comment 166

[WADA:World Anti-bad-Duping Agency]

Bug 584313, which had been closed as dup of this bug, was report of CSS Style case.
Bug 584313 looks better place than Bug 674184. Chaanging dependency.

Comment 167

[Ben Bucksch (:BenB)]

Comment on attachment 8662548 [details] [diff] [review]
WIP patch v3

Review of attachment 8662548 [details] [diff] [review]:
-----------------------------------------------------------------

+      if (NS_SUCCEEDED(domElement->GetAttribute(NS_LITERAL_STRING("style"), styleValue)) &&
+          !styleValue.IsEmpty())
+      {
+        *_retval = nsIMsgCompConvertible::No;

This is OK in theory, but only if
* we ignore it in quotes, in this algo. We don't want other clients force us here.
* we never do that by default, nor encourage that, in TB

> -              element.LowerCaseEqualsLiteral("tt") ||
>        *_retval = nsIMsgCompConvertible::Plain;

As I wrote before here, I think it's extremely silly to say that "fixed width fonts" cannot be converted to plain text, given that plain text is traditionally rendered in exactly such plain text.

However, if I stand alone on that opinion, and all the other developers disagree with me, I'll concede.

-        if (NS_SUCCEEDED(domElement->GetAttribute(NS_LITERAL_STRING("class"), classValue)) &&
-            StringBeginsWith(classValue, NS_LITERAL_STRING("moz-txt"), nsCaseInsensitiveStringComparator()))
+        if (NS_SUCCEEDED(domElement->GetAttribute(NS_LITERAL_STRING("class"), classValue)))
         {
-          *_retval = nsIMsgCompConvertible::Plain;
-          return rv;  // Inconsistent :-(

I don't know why this code change. And - if I read the change right, which I'm not sure of - it's wrong.

moz-txt links must be treated as nsIMsgCompConvertible::Plain. These are the links we generate ourselves from plain text. They *are* plain text, and must be treated as such.

Please remove all these hunks from the patch.

Comment 168

[WADA:World Anti-bad-Duping Agency]

(In reply to Ben Bucksch (:BenB) from comment #167)
> As I wrote before here, I think it's extremely silly to say that
> "fixed width fonts" cannot be converted to plain text, (snip)

Why "format loss on <tt>" is repeated may be confusion like "Variable Width/Fixed Width is Variable Width Section/Fixed Width Section". If someone says "somthing lost in <tt>", it should be said on <pre> or <p> too, because Fixed Width Font is used for text from <p> if recipient views text/plain with Fixed Width Font, and Variable Width Font is used for text from PRE if recipient views text/plain with Variable Width Font, but no such complant on <pre> or <p>...

Comment 169

[WADA:World Anti-bad-Duping Agency]

Following is Comment 121 by Ben Bucksch (:BenB) on 2012-06-24.
> No to both. CSS styles are *not* part of this bug.
> This bug is specifically only about fixed width vs. variable width.
("simple link" case was added after it)
And, separate bug report for CSS Style case surely exits.
(I think bug 584313, Reported on 2010-08-04, is currently best place for CSS Style case)

Even though it, why many CSS Style cases were injected to this bug?
Why patch for CSS Style case is not proposed to such bug and is not processed in such bug?

Comment 170

[Luke Plant]

(In reply to Ben Bucksch (:BenB) from comment #167)

> As I wrote before here, I think it's extremely silly to say that "fixed
> width fonts" cannot be converted to plain text, given that plain text is
> traditionally rendered in exactly such plain text.

From what I was told in this/another thread, TB sends these "auto-downgraded to plain text" emails (and other plain text emails) as "text/plain, format=flowed".

Such emails are typically *not* rendered as fixed width - one of the main points of the format is to fix the appearance of plain text emails on devices that render them with a proportional font:

https://www.ietf.org/rfc/rfc3676.txt - section 3.1

My experience on phones is that always render this type of email with a proportional font by default.

Comment 171

[Ian Scott]

(In reply to Ben Bucksch (:BenB) from comment #167)
> As I wrote before here, I think it's extremely silly to say that "fixed
> width fonts" cannot be converted to plain text, given that plain text is
> traditionally rendered in exactly such plain text.

The presence of "fixed width fonts" in an email implies the rest of the email is not fixed-width, meaning it cannot be rendered in plain text.

Comment 172

[:aceman]

(In reply to WADA from comment #169)
> Following is Comment 121 by Ben Bucksch (:BenB) on 2012-06-24.
> > No to both. CSS styles are *not* part of this bug.
> > This bug is specifically only about fixed width vs. variable width.
> ("simple link" case was added after it)
> And, separate bug report for CSS Style case surely exits.
> (I think bug 584313, Reported on 2010-08-04, is currently best place for CSS
> Style case)
> 
> Even though it, why many CSS Style cases were injected to this bug?
> Why patch for CSS Style case is not proposed to such bug and is not
> processed in such bug?

I see, thanks. I'll split the support for style (element or attribute) to bug 584313, hopefully that part is less controversial and can be accepted quickly.

Comment 173

[WADA:World Anti-bad-Duping Agency]

(In reply to :aceman from comment #172)
> hopefully that part is less controversial and can be accepted quickly.

Who separarated CSS Style case from this bug(for <tt> only initially, and simple link case was added) was Ben.
And, CSS Style case support will be mandatory(especially attribute of tag case), if Tb's composer will start to utilize CSS Style, instead of using depriciated <FONT>, and removed <tt> in HTML 5.
 
And, please resolve mystery for me of next code in that bug.
  If a div/span/a has class="moz-txt",
  it looks for me that further check for CSS Style attribute, which looks involved in current logic,
  seems not invoked.

http://mxr.mozilla.org/comm-central/source/mailnews/compose/src/nsMsgCompose.cpp#5124
> 5124     else if (
> 5125               element.LowerCaseEqualsLiteral("div") ||
> 5126               element.LowerCaseEqualsLiteral("span") ||
> 5127               element.LowerCaseEqualsLiteral("a")
> 5128             )
> 5129     {
> 5130       /* Do some special checks for these tags. They are inside this |else if|
> 5131          for performance reasons */
> 5132       nsCOMPtr<nsIDOMElement> domElement = do_QueryInterface(node);
> 5133       if (domElement)
> 5134       {
> 5135         nsString classValue;
> 5136         if (NS_SUCCEEDED(domElement->GetAttribute(NS_LITERAL_STRING("class"), classValue)) &&
> 5137             StringBeginsWith(classValue, NS_LITERAL_STRING("moz-txt"), nsCaseInsensitiveStringComparator()))
> 5138         {
> 5139           *_retval = nsIMsgCompConvertible::Plain;
> 5140           return rv;  // Inconsistent :-(
> 5141         }
> 5142       }

Comment 174

[:aceman]

(In reply to WADA from comment #173)
> And, please resolve mystery for me of next code in that bug.
>   If a div/span/a has class="moz-txt",
>   it looks for me that further check for CSS Style attribute, which looks
> involved in current logic,
>   seems not invoked.

Yes, if there are further attributes, that could affect display, they are not checked and the element is deemed convertible.

I am trying to fix exactly this problem, see the patch starting at:
+      // If there are any other attributes on the element, it is not convertible.
+      if (domElement)

Comment 175

[Ben Bucksch (:BenB)]

> If there are any other attributes on the element, it is not convertible.

Yeah, I don't agree with that general assessment. It's too dangerous, it might break the whole algorithm.
E.g. our editor might decide to add <li list-style="circle"> instead of <li>, even though the user didn't specifically select a style. Or whatever other thing like that.

----

FYI, <a class="moz-txt"> elements are links that our own TXT->HTML generator created. If you find this class, then the whole document you see was plaintext to start with, i.e. it *is* plaintext.

Comment 176

[:aceman]

(In reply to Ben Bucksch (:BenB) from comment #175)
> E.g. our editor might decide to add <li list-style="circle"> instead of
> <li>, even though the user didn't specifically select a style. Or whatever
> other thing like that.

But then, is list-style="circle" convertible?

I think even if I remove that block (that now is not only for div/span) the result of the whole function will be convertible::no (the default at the top of it) so nothing is changed.

Please also see bug 584313 where I split part of the changes here. Maybe you spot some problems there.

Comment 177

[WADA:World Anti-bad-Duping Agency]

To Thomas D.
This bug is for "<tt> case(and sinple link case is added)" only. CSS Style case is processed in separated bug.
Please hide(check obsolete flag) all your CSS Style related attachments of this bug ehich is for "<tt> and simple link" only, in order to avoid bug reader's confusion.
If required, please attach most important one only to the separated bug.

Comment 178

[WADA:World Anti-bad-Duping Agency]

(In reply to Ben Bucksch (:BenB) from comment #175)
> > If there are any other attributes on the element, it is not convertible.
> 
> Yeah, I don't agree with that general assessment. It's too dangerous, it might break the whole algorithm.
> E.g. our editor might decide to add <li list-style="circle"> instead of
> <li>, even though the user didn't specifically select a style. Or whatever other thing like that.

It looks for me that what is checked after it is not "attribute which contains string of 'style'". It looks for me "style attribute" because code is domElement->GetAttribute(NS_LITERAL_STRING("style"). IIUC, GetAttribute of DOM returns style attribute only and list-style like one is not returned. So, if the domElement->GetAttribute is Tb's unique code, I think it's similar implementation.

Comment 179

[WADA:World Anti-bad-Duping Agency]

Another question to Ben and aceman.
Even when tag of "moz-txt" is not used, and even though current code has style attribute check logic, Convertible=Yes is returned to following case.
  <p style="border: solid 2px #FF0000;">
style attribute check is not invoked for <p>, <pre> etc. in current code?

Comment 180

[:aceman]

(In reply to WADA from comment #179)
> Another question to Ben and aceman.
> Even when tag of "moz-txt" is not used, and even though current code has
> style attribute check logic, Convertible=Yes is returned to following case.
>   <p style="border: solid 2px #FF0000;">
> style attribute check is not invoked for <p>, <pre> etc. in current code?

I think it is because "current" code only checks "style" on <div> and <span> elements. Please test with bug 584313 applied. In that one all style should be detected.

Comment 181

[WADA:World Anti-bad-Duping Agency]

Change "Fixed Width" , "simple link" in bug summary to "Fixed Width Font" , "<a> && link text=link url" respectively, for ease of understanding this bug.

Comment 182

[WADA:World Anti-bad-Duping Agency]

(In reply to Ben Bucksch (:BenB) from comment #37)
> I think comment 31 makes a good argument.

If so, I can't understand reason why "almost all peoples who complaint on <tt>" don't complain on <pre>.
Why "Proportinal Font for text from <pre>" is acceptable for a user but "Proportinal Font for text from <tt>" can not be acceptable for the user?
Recipent may set Fixed Width Font for text/plain mail even if maany users use Proportional Font for it. In such case, "Proportional Font for text from Body Text and <p>" is lost by "send as text/plain". Why such people won't claim "Body Text/<p> should be Not-Convertible"?

"<tt> for Fixed Width" is "Fixed Width section or block in HTML" for some kind of peoples?

Comment 183

[WADA:World Anti-bad-Duping Agency]

FYI.
Bug summary before I touched is '<tt> for "fixed width" sections'...

Comment 184

[Thomas D. (:thomas8)]

(In reply to WADA from comment #182)
> (In reply to Ben Bucksch (:BenB) from comment #37)
> > I think comment 31 makes a good argument.
> 
> If so, I can't understand reason why "almost all peoples who complaint on
> <tt>" don't complain on <pre>.

We DID complain on <pre>, and <pre> was in the summary, but it was outright rejected by Ben's wrong claims that this would necessarily break the whole algorithm. <pre> is just a bit more tricky because we use it internally with -moz-flags for things which might still accept convertible::plain.
There's a separate bug for <pre> I think, also pointing out that long preformatted lines get lost by forced downgrading.
Needless to say, <pre> is exactly same case as <tt>, wrt loss of formatting style.

Comment 185

[Thomas D. (:thomas8)]

(In reply to WADA from comment #177)
> To Thomas D.
> This bug is for "<tt> case(and sinple link case is added)" only. CSS Style
> case is processed in separated bug.
> Please hide(check obsolete flag) all your CSS Style related attachments of
> this bug ehich is for "<tt> and simple link" only, in order to avoid bug
> reader's confusion.
> If required, please attach most important one only to the separated bug.

There's no need to do that, and it's too late because all the documentation and comments etc. are here.
Until any of these bugs have actually been fixed and landed, it's a question of viewpoint which bug the <tt style=...> problem belongs to. Removing <tt> from the convertible::plain list (this bug) solves the <tt style...> issue, too. Solving the general styles issue in the other bug also solves the styles part of the <tt styles issue>.

Comment 186

[WADA:World Anti-bad-Duping Agency]

(In reply to Thomas D. (currently busy elsewhere; needinfo?me) from comment #184)

Thanks for brief explanation on PRE.
I wasn't ware of "internally used PRE".

(In reply to Thomas D. (currently busy elsewhere; needinfo?me) from comment #185)
> > Please hide(check obsolete flag) all your CSS Style related attachments of
> > this bug which is for "<tt> and simple link" only, in order to avoid bug
> > reader's confusion.
> > If required, please attach most important one only to the separated bug.
> 
> There's no need to do that, and it's too late because all the documentation　and comments etc. are here.

I believe that "comments which is irrelevant to issue of a bug merely by Auto-Detect is relevant" is merely a wrong comment in wrong bug. Even if such "wrong comment" is valuable and important for you, it's useless, rather annoyance, for bug readers, developers, QA peoples.
Please note that here is B.M.O for developers to fix bug. Here is never plsce for complaining to developers, for example, support forum.

Such wrong comments can not be removed at B.M.O. by comment posters, but attachments to wrong place can be hidden at B.M.O.
If you actually/really want to help resolving "CSS Style case" which is processed in bug 584313, please attach good/minimum/essential CSS Style sample and STR to bug 584313 with crisp description, please.

Comment 187

[WADA:World Anti-bad-Duping Agency]

(In reply to Thomas D. (currently busy elsewhere; needinfo?me) from comment #184)
> Needless to say, <pre> is exactly same case as <tt>, wrt loss of formatting style.

Question from me to you.
"a part is shown in Proportional Font, other part is shown in Fixed Width Font" is always so important for mail *RECIPIENTS* in communication between human beings using text in an email?
"a part is shown in Proportional Font, other part is shown in Fixed Width Font" is a kind of styling or layouting for you?
"a part is shown in Proportional Font, other part is shown in Fixed Width Font" is similar to <b>, <i> etc. for you?
Even if so, this is *email* for communication between peoples and this is not document represented by HTML...

Comment 188

[Thomas D. (:thomas8)]

(In reply to WADA from comment #187)
> (In reply to Thomas D. (currently busy elsewhere; needinfo?me) from comment
> #184)
> > Needless to say, <pre> is exactly same case as <tt>, wrt loss of formatting style.
> 
> Question from me to you.
> "a part is shown in Proportional Font, other part is shown in Fixed Width
> Font" is always so important for mail *RECIPIENTS* in communication between
> human beings using text in an email?

If the sender (TB user) thinks it's important (which is why he deliberately used/kept such formatting), then we must send what the sender (TB user) wants to send, and not what TB might want to send. The user is king, never TB.

> "a part is shown in Proportional Font, other part is shown in Fixed Width
> Font" is a kind of styling or layouting for you?

Of course!! No matter why or what for, this is MY deliberate formatting. And all users on this bug concur.

Imagine someone like WADA who loves to do even all sorts of tables in fixed width font rows with spacing. Plus some descriptive variable width text before and after the table. Removing the difference will render that table as garbage on many mailers including gmail on mobile, because they render plaintext as variable width. <tt>,<pre> clearly instruct the rending agent to use fixed-width. If recipient's reader doesn't comply, that's not our problem. We must send the correct thing.

> "a part is shown in Proportional Font, other part is shown in Fixed Width
> Font" is similar to <b>, <i> etc. for you?

Indeed. It's really almost exactly same case as <i>: Remember in the old days, typewriters did not have italics nor bold so they used  s p a c e d  o u t  t e x t  for special emphasis.
Just like spacing, fixed-width vs. variable-width in same message, and even fixed-width only is definitely formatting. Important formatting or not is user's decision, not ours.

> Even if so, this is *email* for communication between peoples and this is
> not document represented by HTML...

?? This is *email deliberately composed in HTML*, and we have no way of telling if fixed width formatting is important for the message content or not. Only user knows, so the user must decide, and we must err on the safe side and respect/preserve such formatting by default. Auto-Detect can only be our default delivery format if it is 100% safe/predictable and error-preventing.

Comment 189

[Thomas D. (:thomas8)]

(In reply to WADA from comment #186)
> (In reply to Thomas D. (currently busy elsewhere; needinfo?me) from comment
> #184)
> 
> Thanks for brief explanation on PRE.
> I wasn't ware of "internally used PRE".
> 
> (In reply to Thomas D. (currently busy elsewhere; needinfo?me) from comment
> #185)
> > > Please hide(check obsolete flag) all your CSS Style related attachments of
> > > this bug which is for "<tt> and simple link" only, in order to avoid bug
> > > reader's confusion.
> > > If required, please attach most important one only to the separated bug.
> > 
> > There's no need to do that, and it's too late because all the documentation　and comments etc. are here.
> 
> I believe that "comments which is irrelevant to issue of a bug merely by
> Auto-Detect is relevant" is merely a wrong comment in wrong bug. Even if
> such "wrong comment" is valuable and important for you, it's useless, rather
> annoyance, for bug readers, developers, QA peoples.

WADA, STOP blaming me. As you know, even for yourself the whole Auto-Downgrade and Auto-Detect conglomerate was pretty much a mystery and it took long to understand all the twists in the behaviour and the code. I correctly attached <tt style...> cases here, because this this bug has always been about <tt>, which gets dumped including inline styles. I correctly attached <a href= style=...> cases here, because this bug has always been about "simple links". Simple links get dumped, including their inline styles. Not dumping these tags in the first place would preserve the styles. Global protection for styles in the other bug is superiour solution for styles - so let's do it. It's never fixed until it's actually fixed. <tt>,<pre>, or any styles, it's all about the same algorithm by same known coder, who was unwilling to listen, cooperate, fix bugs in any way. So presenting problems more clearly in single bug appeared necessary at the time, to refute stubborn uninformed claims that dumping <tt> is always inconsequential formatting.

> Please note that here is B.M.O for developers to fix bug. Here is never
> plsce for complaining to developers, for example, support forum.

I'm here for QA and UX, and if I complain/analyse/focus problems on behalf of users in that role, that's a necessary and helpful thing to do; plus I really know what I'm doing. And for some strange reason, users are virtually always on my side. Not all bugs can just be fixed implemented without discussion on BMO. Plus, I'm also a developer (latest patch: attachment 8674446 [details] [diff] [review] in Bug 118624), so once again, please stop implying otherwise.

> Such wrong comments can not be removed at B.M.O. by comment posters, but
> attachments to wrong place can be hidden at B.M.O.
> If you actually/really want to help resolving "CSS Style case" which is
> processed in bug 584313, please attach good/minimum/essential CSS Style
> sample and STR to bug 584313 with crisp description, please.

I think the problem of bug 584313 is sufficiently obvious without further input. Styles can add a wide range of visible formatting, and any visible formatting cannot be losslessly converted. If we can't convert losslessly as promised by Auto-Downgrade, then we can't auto-downgrade. Full stop. If user still wants to downgrade his own formatting, there are plenty ways of opting into that on a per-message basis. I still assume most users want their visible formatting preserved (ux-wysiwyg, and yes I know it's not strictly, but this is about message integrity and not recipient-side distortions). So we must err in their favor.

Feel free to copy any of my test cases over to the other bug, or just to reference them from there.
I'm not willing to waste my time on that. Remember I'm also a volunteer without ANY obligations.

Comment 190

[WADA:World Anti-bad-Duping Agency]

(In reply to Thomas D. (currently busy elsewhere; needinfo?me) from comment #188)
> > "a part is shown in Proportional Font, other part is shown in Fixed Width
> > Font" is a kind of styling or layouting for you?
> 
> Of course!! No matter why or what for, this is MY deliberate formatting.

Gotha! Aha!

> And all users on this bug concur.

Where is evidence of it? Some of users, isn't it?

Comment 191

[Thomas D. (:thomas8)]

WADA, FYI: As you are well aware, in current TB:

(Auto-Detect)==(Auto-Downgrade+Auto-Detect conglomerate algorithm).

Until recently when we found the exact spot in code where Auto-Downgrade is located, and how easily it could be separated from Auto-Detect and disabled by new user pref, I was not aware of exact difference between Auto-Downgrade (message-centric) vs. Auto-Detect (recipient centric). Intertwining the two without giving the user a transparent choice was a design shortcoming to begin with, because message-centric Auto-Downgrade (when applicable) de facto bypasses recipient-centric Auto-Detect (for some, not all cases, inconsistent again). Hence all the surprises, mysteries, etc. when Auto-Detect silently ignored all user settings in favor of HTML, and dumped users formatting (worst for style cases, but also for simple cases like align=center). So I was only aware of [Auto-Detect big single mystery algorithm] as culprit, so I correctly pointed out on respective bugs that Auto-Detect can dump formatting and unilaterally decide to send plaintext to your "prefers-unknown" friends who can happily receive HTML, even when other recipients of the same message explicitly prefer HTML. I'm a human being so I do come with learning curves. Fully understanding Auto-Downgrade+Auto-Detect intertwined algorithm is nothing short of science, so the learning curve was steep. Thanks to your unrelenting efforts, too, we are now more knowledgeable about the whole thing. You and I more than anyone else.

Comment 192

[WADA:World Anti-bad-Duping Agency]

(In reply to Thomas D. (currently busy elsewhere; needinfo?me) from comment #189)
> I'm not willing to waste my time on that. Remember I'm also a volunteer without ANY obligations.

I'm also a volunteer, and I'm also not willing to waste my time, so I stopped to touch Bug 1215791 any more, which is also relevnt to Auto-Detect, as I stopped to touch Bug 1202227 and Bug 1202276 any more, because I dislike flood of complaints in a bug at B.M.O.

Bye.

Comment 193

[Thomas D. (:thomas8)]

(In reply to WADA from comment #190)
> (In reply to Thomas D. (currently busy elsewhere; needinfo?me) from comment
> #188)
> > > "a part is shown in Proportional Font, other part is shown in Fixed Width
> > > Font" is a kind of styling or layouting for you?
> > 
> > Of course!! No matter why or what for, this is MY deliberate formatting.
> 
> Gotha! Aha!
> 
> > And all users on this bug concur.
> 
> Where is evidence of it? Some of users, isn't it?

Are you kidding?
Random sample: 

* Santiago Romero (Comment 93)
* Arivald (e.g. Comment 116, speaking for many more, and ten thousands who downloaded his Always HTML addon)
* Charles (e.g. Comment 64, Comment 94)
* Luke Plant (e.g. Comment 170)
* Ian Scott (e.g. Comment 171)
* ...others from 4 duplicate bugs...
* ...others from getsatisfaction: http://gsfn.us/t/16j03 (per comment 24)
* rsx11m (e.g. Comment 24)
* Phil Ringnalda (dev, e.g. Comment 12)

You want more?

So who has actually argued *in favor* of dumping <tt> and <pre>, as a sane DEFAULT behaviour for everyone, AND giving understandable REASONS why we should violate ux-error-prevention by default instead of letting those users opt-in by choosing delivery format plaintext, or just compose plaintext to begin with?

Is this SATIRE or what? I thought we are a serious application based on UX-principles. Maybe not.
The good news perhaps, WADA sometimes plays the role of "advocatus diaboli", so some of his comments need to be taken with a pinch of salt. We're actually working well together towards sustainable solutions for everyone, including the dinosaurs...

Btw, note that Magnus, one of our key developers, actually started to wonder about the net benefit of having AUTO-DETECT at all, or if we could just rip that out completely (but at the same time he's in favor of preserving Auto-Downgrade, fair enough if it's optional and transparent).

Comment 194

[WADA:World Anti-bad-Duping Agency]

(In reply to Thomas D. (currently busy elsewhere; needinfo?me) from comment #193)
> Are you kidding?

No.

Following is comment of Bug 766860 comment #4 by bug opener of that bug when he got easy/simple workaround in his bug.
> The Stationay add-on works perfectly. I only had to install the add-on and all mails are send in HTML format by default.
> This means for me, our company disclaimer is added to the sending email.
> Thanks alott!!
> Btw, I'll still hope Mozilla will add this by default in their future release to be able to send HTML by default.

It looks for me that such comment is comment from many ordinal peoples in B.M.O.
It looks for me that who produces flood of complaints in B.M.O is !(ordinal people).

Tohmas D., please stop such comment at B.M.O where is for developer to fix bug.
Please read Bug 1210244 Comment #178 and Bug 1210244 Comment #179.
It's not my joke. I understood your frustrain by your two bugs. Bug 1215791 is an answer to you from me.

Comment 195

[WADA:World Anti-bad-Duping Agency]

Oh, sorry, typo. My right hand frequently works quicker than left hand.
(I type H by left finger. I can't do blind typing...)
  Tohmas D. => Thomas D.

Comment 196

[Jorg K (CEST = GMT+2)]

No progress here? Are we planning to include this in TB 45?

Comment 197

[:aceman]

Jorg, do you know what needs to be done here? AFAIK this bug can now be reduced to the <tt> case. I need to check what is the problem with <a> (reported in the summary).
My patch here is actually obsolete, as the style handling has moved to bug 584313.
And I do not want to work here right now as any patch would clash with the one in 584313.

Comment 198

[Jorg K (CEST = GMT+2)]

Understood. If you ever get bug 584313 landed, one of us can continue here. I put it onto my list.
In the global scheme of things, this is not the most important bug, it's just another: Oops, it got the auto-detect wrong (but IMHO it's not so bad for <tt> and <a>).

Comment 199

[Jorg K (CEST = GMT+2)]

What are we going to do with this one?

Comment 200

[:aceman]

(In reply to Ben Bucksch (:BenB) from comment #167)
> > -              element.LowerCaseEqualsLiteral("tt") ||
> >        *_retval = nsIMsgCompConvertible::Plain;
> 
> As I wrote before here, I think it's extremely silly to say that "fixed
> width fonts" cannot be converted to plain text, given that plain text is
> traditionally rendered in exactly such plain text.
> 
> However, if I stand alone on that opinion, and all the other developers
> disagree with me, I'll concede.

(In reply to Jorg K (PTO during summer, NI me) from comment #199)
> What are we going to do with this one?

Can you collect enough votes to overturn the decision above?
The patch is reduced to moving that single line after bug 584313 was done.

Comment 201

[Jorg K (CEST = GMT+2)]

Ben is right. I mostly compose HTML with fixed font <tt> and I want to have the mail downgraded to plain text.

This bug has become way to confused, so I'm closing this. May people open another bug if they have a specific complaint.

I assume the "link text==link url" issue has been addressed elsewhere?

Comment 202

[:aceman]

(In reply to Jorg K (PTO during summer, NI me) from comment #201)
> I assume the "link text==link url" issue has been addressed elsewhere?

I haven't looked at that one, but in comment 165 WADA summarizes that that problem was added to this bug outside of the original report. So if that one is still a problem it should be filed separately to not drown in the mess here. I think the comments mention the special classes moz-* and we whitelist those to be convertible.

Comment 203

[edifydog.com]

Hi bloggers Please the following link, it really give you additional info about 2017 Digital Marketing Techniques:

http://edifydog.com/digital-marketing-trends-2017-and-importance-of-digital-marketing.html

Comment 204

[edifydog.com]

Edifydog Services:

We Build SEO Friendly Responsive Website for your business and we strictly follow the guidelines of digital marketing. For more info :
http://edifydog.com/web-designing-and-development-services-company.html

we do digital media promotions for your business. For more Info:
http://edifydog.com/social-media-marketing-service.html

In addition we provide online training for 1. Digital Marketing  2.Search Engine Marketing.
http://edifydog.com/social-media-marketing-service.html

We Also give services to SEO & Digital Marketing
http://edifydog.com/search-engine-optimization-service.html
http://edifydog.com/digital-marketing-service.html

thank You
edifydog