31906 - Make plain text msgs styleable and recognize nested quotes

Status: VERIFIED FIXED

(MailNews Core :: MIME, enhancement, P3)

Description

[Ben Bucksch (:BenB)]

Spin-off from bug #31047. See longer description there.

Goal:
1. Recognize individual quotes (i.e. also quotes within quotes) in non-flowed
plain text, so we *can* make them look like HTML quotes (with a bar instead of
"> "s). The latter should be controlled by a pref.
2. Clean code up to make look configurable by stylesheets.

Preferably, 2. also works for the recipient of quoted plain text msgs in HTML
msgs, i.e. I always see quotes of plain text msgs the way I want, even if
another Mozilla Mailnews user replied to them via HTML and sent the result to
me.

I have this already mostly working in my tree. I spent 1-2 hours browsing my
mail archive and the recognition works fine.

Maybe I'll attach some screenshots.

---

What I'm not sure about is, what we should recognize as quote tags.
Currently, I recognize x spaces, directly followed by y upper case chars,
directly followed by ">" or "]", optionally directly followed by a single space.
That is , because some mailers use something freaky as quote tags, e.g. VM
(EMacs):

>>>>>  BB = Ben Bucksch wrote:
  BB> bla blabla
  BB> blabla

I do recognize that, but
1. it is rare
2. it is not completely correctly recognzed in this case. The "xxx wrote:" line
is recognited as a nested quote of level 5, i.e. would get 5 bars.
3. it increases the likelyness of failure. I did not see such a case in
practice, but should it happen, it would be bad, because I remove the plain text
quote tags in favor of the bar.

4.x seems to only recognize "> ", which is the wastly most common case, but the
old 5.0 code (before my rewrite of nsMimeURLUtils) seemed to work the way my new
code works. Not sure, why this difference.

What should I do. Constrain the code to the "> " case or try to recognize
unusual cases?

Comment 1

[Ben Bucksch (:BenB)]

accept

Comment 2

[rhp (gone)]

If you have to lean one way or another, I would be less agressive than 
more. I'd rather have cases slip through the cracks instead of things that 
aren't quotes get "Quote-a-fied"

- rhp

Comment 3

[Phil Peterson]

Ben, you asked me to post a decision in this bug, but I don't understand what
decision you're looking for. Could you clarify?

Comment 4

[Ben Bucksch (:BenB)]

Sure.

A more or less side-effect of customizability by

stylesheets is the possibility to replace text quote tags (e.g. "> ") by

graphical ones (e.g. the blue bar). Of course, this will be a pref. The bars can

look differently depending on the type of the msg, and propably will do so by

default, if no one objects.



My questions to the Mailnews team:

- Will I get UI for the pref? It would read something like "Show graphical

instead of text quote tags (e.g. "> ") for normal plain text msgs" and be showed

below the current style prefs for quoted text.

- What should be the default?

Comment 5

[Ben Bucksch (:BenB)]

Daniel, as you worked on Mailnews, too, I'm also interested in your opinion.

Comment 6

[Phil Peterson]

My preference, as I've expressed in other bugs, is to have quoted material in
text/plain parts shown using '>' rather than <blockquote>

However, if you would like to contribute this feature, you're welcome to do it
-- open source and all. If you do, I'd prefer for the default to be '>'. But
even that I could live without since we could always change the default value
for the Netscape commercial build.

Comment 7

[Daniel Bratell]

I have a question. Is this only about what's displayed to the left of quoted 
text? For me, as a person who wants to make format=flowed work as good as 
possible, it's important that the logical quoting information is preserved so I 
assume that no '>' is physically inserted in the text since that would destroy 
that information. 

More than that, I don't know what's the best look is. I personally prefer a 
vertical bar as in html mail. If it works in html mail, why not use it for other 
mails?

Comment 8

[Ben Bucksch (:BenB)]

Daniel, this part of the bug, the improved recognition of quotes, is only for

normal plain text mails. If they are displayed, I /remove/ the plain text quote

tags (i.e. "> ") in favor of a bar like in html mails, but not physically, but

with some cool CSS features ("display: none" in particular). In fact, all the

preference "use graphcial quote tags" does, is to add a new attribute

("graphical-quote=true"), so I can choose the correct style *in the stylesheet*.

I.e. if I'm insane, I can change the stylesheet in a way, that it does exactly

the opposite of what the preference says :-).

I'll attach some pictures, which should make clear, what this feature does.



BTW: I will surely not mess up flowed quotes.

Comment 9

[Ben Bucksch (:BenB)]

Attachment: Msg 1, Graphical

Comment 10

[Ben Bucksch (:BenB)]

Attachment: Msg 1, Plain italic

Comment 11

[Ben Bucksch (:BenB)]

Attachment: Msg 2, Graphical

Comment 12

[Ben Bucksch (:BenB)]

Attachment: Msg 2, Plain italic

Comment 13

[Ben Bucksch (:BenB)]

Attachment: Msg 3, Graphcial

Comment 14

[Ben Bucksch (:BenB)]

Attachment: Msg 3, Plain italic

Comment 15

[Ben Bucksch (:BenB)]

Attachment: Msg 3, Plain, fixed width font

Comment 16

[Ben Bucksch (:BenB)]

Attachment: Msg 3, Graphcial Italic Serif

Comment 17

[Daniel Bratell]

I think I understand what you want to do. Are you embedding the quoted text in 
<blockquote ...> to do it? If that's possible, it would be nice to use the same 
style for both plain plain text and flowed plain text.

Comment 18

[Ben Bucksch (:BenB)]

Msg 1 and 2 are normal msgs. I'm using "Sans serif" as preferred variable width

font family in my preferences, and this is now also honored in this case

(previously, "serif" was hard-coded). "Italic" shows, that rhp's recent changes

are working together with mime.



Msg 3 shows the unusual quoting I was speaking about above. Currently, the

recognition of them is still enabled, so you can see the result.



The last two show alternative font settings, the first "prefer fixed width fonts

for plain text msgs" and the second shows the result, if the global variable

width font family preference is set to "serif" (and also rhp's change together

with graphcial quote tags).

Comment 19

[Ben Bucksch (:BenB)]

Daniel, yes I use "<blockquote type=cite>". But I use different classes in the

<div>, that encloses the whole msg, for plain text and flowed ("<div

class=text-plain" and "<div class=text-flowed"), because I want to have visual

feedback, if the msg is flowed or plain text. Currently, they are also

technically needed, because I don't use &nspr;s and <br>s, but a "white-space:

-moz-pre-wrap" style. The different style for the quotes is reached with

contextual selectors ("div.text-plain blockquote...", i.e. the <blockquote>s

inside <div class=text-plain>s). I'll attach the current stylesheet for

illustration.



Do you think, this is a problem?

Comment 20

[Ben Bucksch (:BenB)]

Attachment: Current stylesheet

Comment 21

[Daniel Bratell]

If it was possible to display flowed and non-flowed the same, I would prefer 
that since for the normal user (95% or more I guess) it's only confusing that 
mail sometimes looks one way and sometimes another way. 

It might help debugging though to be able to seperate the two kinds easily.

I can't view the attachment in Netscape 4.72. I'll try some other method later.

Comment 22

[Ben Bucksch (:BenB)]

Daniel, I just noticed, that we can't use the same class for plain text and
flowed, unless you want to bars disappear for flowed msgs, if the user chooses
to disable them for plain text msgs. The current behaviour of the tip would be
impossible.

Comment 23

[Ben Bucksch (:BenB)]

Daniel, plain text and flowed are two different things, just as plain text and
html are. They behave differently, so they should look differently. However, we
have freedom, how large the difference is. With the attached stylesheet, the
difference is just a thin (for flowed) vs. medium (for plain text) bar, both in
black, while bars for html are blue, but you can give all of them different
colors, if you want.

You can get to the attached stylesheet with "Save Link As...".

Comment 24

[Ben Bucksch (:BenB)]

Attachment: Current state of work, |cd mailnews/mime; cvs diff -u .|

Comment 25

[Ben Bucksch (:BenB)]

> for the normal user (95% or more I guess) it's only confusing that mail
> sometimes looks one way and sometimes another way.

Not, that I'm misunderstood: I think they should be displayed slightly
differently, because they behave slightly differently, but, of course, I think,
it's a good idea to display plain text quotes with bars, because it's not *that*
different from the other mail formats.

Phil,
we have 2 votes (me and Daniel) for bars and 1 (you) against them, can you arise
the issue in the UI meeting tomorrow (*after* you looked at the screenshots and
saw how well it works :-) ) please and ask, what the other members think, what
the default should be?

Comment 26

[Daniel Bratell]

One more thing regarding the bars. Even though I like them I think we should 
only display them if we are absolutely sure that it represents a quote. With 
format=flowed and html mail we can be sure. It's harder to analyze normal plain 
text mails and we would upset people if we got it wrong.

Comment 27

[Ben Bucksch (:BenB)]

If graphical bars were the default, I would be very conservative, i.e. only "> "

at the beginning of a line counts as quote tag. I think, that is reasonable

save, do you agree? (Sorry, if I misunderstood you, that you are for graphcial

quote tags.)

Comment 28

[Phil Peterson]

Yes, we can cover this in the UI meeting.

Comment 29

[Phil Peterson]

Here's the thinking from today's UI meeting:

0. We are unconvinced that we can explain the difference between format=flowed
messages and regular text/plain messages to users in a way that they will
understand.
1. Since format=flowed messages will be (more) common after mozilla-based
products are released, it would be more consistent to use <blockquote> than to
use '>'
2. So let's put the code in, put it on a Javascript pref, and not expose the
confusing pref in the UI. The default pref value should be to use <blockquote>
for consistency reasons.
3. This means that mozilla users will only see '>' when reading quoting
generated by MUAs not capable of format=flowed. We're worried that using a
quoting style which looks like HTML will be confusing and/or aggravating to
users accustomed to '>'. If this turns out to be a big problem, we can figure
out later how to expose UI for the pref, and potentially change the default.

Make sense?

Comment 30

[Ben Bucksch (:BenB)]

about 1. First of all, I'm happy, that I have "green" for my feature being default!



about 0. I don't know, if we can shield our users from the difference (see above

about differences). However, I agree, that it is very hard to explain. Maybe we

can just refer to plain text msgs as "messages most other mail programs use" and

to flowed msgs as "text messages Mozilla Mailnews (or Netscape Messenger

respectively) uses"? Some users might want to know that in more detail, and we

could explain format=flowed in the manual for them, possibly making use of

hyperlinks in the prefs.



about 2. I have a pretty bad feeling with that. From my experience with the

converter, I'm worried, that we will create hostile reactions, if we don't

provide an *easy* way to turn that feature off. What about "Show graphical (e.g.

"| quote") instead of text (e.g. "> quote") quote tags for text msgs most other

mail programs use"? (The "|" will be a real graphical bar.)



about 3)

> This means that mozilla users will only see '>' when reading quoting

> generated by MUAs not capable of format=flowed.



I don't understand that sentence. The feature is *only* about reading quoting

generated by MUAs not using format=flowed. None of the sample msgs attached are

generated by Mozilla.

With this feature, the only cases, where users might see text quoting tags are:

1. Quoting tags, which are not recognized, are used. (See also discussion in the

description of this bug.)

2. Quoting normal plain text in HTML.

Is it that, what you meant?



I *might* even succeed to make the latter quotes, if generated by Mozilla, to

display the way the pref is sat, i.e. differently depending on the reader, but

that's very hard without editing stylesheets.

Comment 31

[Ben Bucksch (:BenB)]

I'll attach my current state of work, *maybe* it fixes some other wierd
problems. It has not been fully tested with the composer (quoting) - no
garantees what-so-ever. It reintroduces bug #32100. Good luck :-).

Comment 32

[Ben Bucksch (:BenB)]

Attachment: Current state of work; |cd mailnews/mime;cvs diff -u .|

Comment 33

[Ben Bucksch (:BenB)]

This bug will also fix bug #29699 (has been marked WONTFIX because of that).

Comment 34

[John Moreno]

about 3)

> This means that mozilla users will only see '>' when reading quoting

> generated by MUAs not capable of format=flowed.



If you don't mean "only see by default", I would like to refer you to  

comp.mail.eudora.mac -- where a very frequent faq is "how do I turn of the 

excerpt bars".  People love the concept of f=f, they just want to see the ">" 

as they are used to.

Comment 35

[Ben Bucksch (:BenB)]

planb, this is bug #29557.

I wrote:
> With this feature, the only cases, where users might see text quoting tags
> are:
[...]
> 2. Quoting normal plain text in HTML.

I have code working to turn even this off (controlled by a hidden pref), i.e.
nested quotes in plain text msgs are recognized and inserted as blockquote into
the html editor.

I'm not sure, what should be the default - it has large advantages, but it is a
bit
risky at this point of the project.

3. And in the plain text composer, of course.

Comment 36

[Ben Bucksch (:BenB)]

Attachment: Graphical, colored quotes. Note, that this is a plain text msg and still virtually the same code. All just CSS.

Comment 37

[Karl Ove Hufthammer]

| Graphical, colored quotes. Note, that this is a plain
| text msg and still virtually the same code. All just CSS.

That looks *great*!

Comment 38

[Ben Bucksch (:BenB)]

ccing I18N people, so they can help testing.

Comment 39

[Ben Bucksch (:BenB)]

I'll attach my current state of work. It's bascially done, but no garantees. I'm
too tired now to create perfect patches. |cvs diff -u| in the respective dirs.

But feel free to look over and review, that's why I attach them.

Comment 40

[Ben Bucksch (:BenB)]

Attachment: Current state of work - Default prefs

Comment 41

[Ben Bucksch (:BenB)]

Attachment: Current state of work - Stylesheet

Comment 42

[Ben Bucksch (:BenB)]

Attachment: Current state of work - MIME code

Comment 43

[Ben Bucksch (:BenB)]

BTW: The patches contain a lot of cleanup and other bugfixes (for bugs not
filed, e.g. normal text after quotes in flowed msgs displayed in italic).

Comment 44

[Ben Bucksch (:BenB)]

Teh following patches are generated by |POSIXLY_CORRECT=1; cd mozilla; cvs -Q
diff -p .|. You may also want the latest patch for bug 32420; it makes qoute
recognition less agressive (only ">" and "> " arbitarily mixed at the very
beginning of the line are recognized); see above for a discussion.

I hope, this is the last version. I.e. rhp, would you finally review and checkin
them, please? I'm sorry that I'm so late.

I spent propably a full workweek (40 hs) on testing this stuff. Found and fixed
many bugs in the existing (pre-31906) code on the way. General cleanup. I think,
you really want this stuff for M16 (and so in NS6).

We should rename mailheader.css into message.css sometime, as it now also holds
styles for the msg body.

New prefs

          Key          Default   UI             Description

 mail.quoted_graphical true    none  For viewing, format plain text
                                     quotes as blockquote
                                     For quoting (in composer), remove
 mail.quoteasblock     false1  none  plain text quote tags and wrap
                                     quotes in blockquote + <pre>

1Bug 39372


Changes in old behaviour

   * The new features, of course
   * Viewer
        o Quotes are not italic by default, but regular.
        o Quotes are not black by default, but have no color set.
        o Less quotes are recognized.
   * Quoting
        o Quoted plain text is always wrapped (<pre wrap>; see bug
          31200), not only if enabled for viewing


Known bugs

   * There is no way with the UI to reset the color to none once you
     sat a color. (Bug 39230)
   * Emacs VM (?) attribution line ">>>>> "BB" = "Ben Bucksch" wrote:"
     is recognized as quote. This is broken behaviour on the sender's
     side. WONTFIX.

Comment 45

[Ben Bucksch (:BenB)]

Attachment: Final patch (hopefully)

Comment 46

[Ben Bucksch (:BenB)]

I'll write some documentation on my website and post to .mailnews about it
anytime soon. I'll attach a sample user.css (to be located in
<userprofiledir>/chrome/user.css) for the adventurous, who want to play with it.
Show me some cool ideas! :)

Comment 47

[Ben Bucksch (:BenB)]

Attachment: Sample user.css

Comment 48

[lchiang]

cc:pmock

Comment 49

[Daniel Bratell]

I finally got to build and tested your code. 

I discovered an artifact with quotes. With an ordinary one level quote á la 
format=flowed I get a added ">  at the beginning of the paragraph

Source (raw mail):
text text text text text text text text text text text text text text text text 
text text text text text text text text text 

> text text text text text text text text text text text 
> text text text text text text text text text text text 
...

Result:

text text text text text text text text text text text text text text text text 
text text text text text text text text text 

| ">text text text text text text text text text text text 
| text text text text text text text text text text text 
...

Comment 50

[Ben Bucksch (:BenB)]

BTW: The patch also fixes bug 9202, bug 37365 and *maybe* (nsbeta2+) bug 33305.


Daniel, you're speaking about a non-flowed mail, right? I can't reproduce this.
Do you have a recent tree (from May 17)? If not, please update
|netwerk/streamconv/converters/mozTXTToHTMLConv.*|.

Comment 51

[Daniel Bratell]

No, it was a format=flowed mail sent by Mozilla from today (I dared to pull a 
tree despite the checkin rush). I'll attach the exact mail that caused this. 
It's in (gibberish) swedish but I hope you can see the technical malfunction 
anyway.

Comment 52

[Daniel Bratell]

Attachment: Mail causing display trouble

Comment 53

[Ben Bucksch (:BenB)]

Attachment: testmail worksforme. (The thin bar shows, that it ran through mimetpfl.cpp.)

Comment 54

[Ben Bucksch (:BenB)]

I wrote some docs, to be found at
<http://www.bucksch.org/1/projects/mozilla/31906>. Tell me, if you don't
understand them.

Comment 55

[Ben Bucksch (:BenB)]

moving to M17 :(.

Comment 56

[Ben Bucksch (:BenB)]

Attachment: Latest patch (already sent to rhp and Daniel Bratell)

Comment 57

[Ben Bucksch (:BenB)]

BTW: This was version 3.

ccing alecf, because he said, he also wanted to look over the patch.

Comment 58

[Ben Bucksch (:BenB)]

Fix checked in by rhp.

Comment 59

[Ben Bucksch (:BenB)]

Note, that bug 41637 hides this feature and even makes plain text quotes look
bad (graphcial *and* text quote tags) now :-(. Hopefully will be fixed soon.

Comment 60

[pmock]

Wow... long bug.

Using the following builds
 win32 commercial seamonkey build 2000-092909-mn6 installed on P500 Win98
 linux commercial seamonkey build 2000-092909-mn6 installed on P200 RedHat 6.2
 macos commercial seamonkey build 2000-092911-mn6 installed on G3/400 OS 9.04

I verified the new viewer behavior  
        o Quotes are not italic by default, but regular.
        o Quotes are not black by default, but have no color set.
        o Less quotes are recognized.

The user pref " mail.quoted_graphical" works as descibe.

but ...
I am having difficulty verifying the behavior of the user pref 
"mail.quoteasblock".  I set the pref to true and false but I am not seeing any 
effect when viewing in the plain text compose window.  The nested quotes are 
quoted with ">".  

Should I write up a separate bug Ben or reopen?  Or should I be expecting a 
different behavior.  Please advise.  Thank you. /Peter

Comment 61

[Ben Bucksch (:BenB)]

Peter,
thanks for verifying. I'm sure, it was not easy :).

Bug 39372 is specifically about (enabling) mail.quoteasblock (per default). It
should change the quote in the *HTML* composer, not the plaintext composer,
after replying to a real plaintext msg.

BTW: You use real plaintext msgs (those without "format=flowed" in the
Content-Type header) to verify this bug, right?

Comment 62

[pmock]

Verifying bug as fixed.  Thanks for the clarification.

Yes, I did use real plain text (without format=flowed)