222344 - Messages with quoted text are incorrectly copied to clipboard (doubled quote symbols when pasting)

Status: NEW

(MailNews Core :: MIME, defect)

Description

[Casper Gebuijs]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.6a) Gecko/20031012
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.6a) Gecko/20031012

When copying the text from a mail which contains quoted text (a reply as obvious
example) to the clipboard the quotation-characters are doubled and above and
below the quoted text an empty line is inserted.

Reproducible: Always

Steps to Reproduce:
1. Open an email (a reply) from your inbox which contains quoted text and is NOT
send with mozilla! (due to another bug, will file that one later)
2. Select all the text
3. Copy everything to the clipboard
4. View the clipboard
5. Or paste it in an editor
6. Compare the pasted text with the message-source

Actual Results:  
The quotation characters are doubled and a blank line is inserted above and
below the quoted text.

example:

original mail:
Hi, blablabla

you wrote:
> Hi, chatchatchat
> bladiebladiebla

greets


after copy/pasting:
Hi, blablabla

you wrote:

>> Hi, chatchatchat
>> bladiebladiebla


greets

Note the extra ">" before each quoted line and the blank lines that are added
above and below the quoted text.

Expected Results:  
Mozilla should retain the original email-source and shouldn't add extra
quotation characters and blank lines when copy/pasting text from the message
screen to other applications.
See the Additional Information why this is important.
There is an reletively easy workaround for this problem and this involves
editing of the copyed text until all the changes Mozilla made are undone.
Verification is then possible

This message alteration after copying a mail becomes critical when a program
like PGP is used. Verification of PGP sigantures depends on the unaltered
original message. And becouse PGP isn't yet integrated in mozilla, you have to
use copy/paste to verify (and/or decrypt) messages. 

I found this bug in the Mozilla 1.4.1 suite and the latest nightly build of the
Mozilla suite (12 oct 2003).

Comment 1

[Stefan Borggraefe]

*** Bug 240003 has been marked as a duplicate of this bug. ***

Comment 2

[Boris 'pi' Piwinger]

This is a major problem since it makes the copy&paste action almost unusable
(typically used for third party quotes). This must not happen in a release,
asking for blocking.

pi

Comment 3

[Boris 'pi' Piwinger]

Changing summary to get in keyword paste which made me not find this bug.

pi

Comment 4

[Karsten Düsterloh]

I've made a small analysis of the > problem (the empty lines are are actually
another bug) under Win2k.

This short mail body:

Someone wrote:
> there be
>> Dragons!

will be rendered in the messagepane (essentially) by this HTML
(when not using f=f):

    <DIV graphical-quote="false" wrap="true" class="moz-text-plain">
      <PRE wrap="">
Someone wrote:
      </PRE>
      <BLOCKQUOTE type="cite">
        <PRE wrap="">
          <SPAN class="moz-txt-citetags">
&gt;           </SPAN>
there be
        </PRE>
        <BLOCKQUOTE type="cite">
          <PRE wrap="">
            <SPAN class="moz-txt-citetags">
&gt;&gt;             </SPAN>
Dragons!
          </PRE>
        </BLOCKQUOTE>
      </BLOCKQUOTE>
    </DIV>

Depending upon the user_pref "mail.quoted_graphical", the SPAN.moz-txt-citetags
may have the style display:none - the > signs are *always* there, but not always
visible!

Selecting and copying this mail will put this HTML upon the clipboard.

Pasting the clipboard data as plain text will now result in
nsPlainTextSerializer::Write being called - and itself calling
nsPlainTextSerializer::OutputQuotesAndIndent, putting all > in the text a second
time...

Comment 5

[Asa Dotzler [:asa]]

This seems bad that we don't handle pasting qouted text from other mail readers.
I can reproduce in Thunderbird and SeaMonkey on windows.

Comment 6

[Mike Kaply [:mkaply]]

Can you please forward an email that will cause this to [e-mail]?

I'm having trouble reproducing.

Thanks

Comment 7

[Mike Kaply [:mkaply]]

The problem here is not mail specific. See bug 39098.

One could argue that mailnews could use a different technique to fix this
problem - that would be an RFE.

*** This bug has been marked as a duplicate of 39098 ***

Comment 8

[Boris 'pi' Piwinger]

Reopening:

a) Asa just set this to blocking 1.7+. So we have to get it fixed here.

b) This bug is much newer than the bug it was duped to. So I doubt it is really
the same thing.

pi

Comment 9

[Boris 'pi' Piwinger]

The blocking flag was lost. It was originally set by Asa (2004-05-11 13:18 PDT),
see the bug activity.

pi

Comment 10

[Boris 'pi' Piwinger]

My apologies, Asa removed the blocking. I misread it.

But what do we do now? There was the decision to block 1.7 by Asa, now that was
removed (because of the dupe, I guess, but that was not marked to block).

pi

Comment 11

[Stefan Borggraefe]

(In reply to comment #10)
> But what do we do now? There was the decision to block 1.7 by Asa, now that was
> removed (because of the dupe, I guess, but that was not marked to block).

I guess the blocking flag was removed (respectively not transferred to bug
39098) because the bug is very old actually and there weren't many complaints
about this issue even though it was present in all the previous releases.

Why should it block even the next *alpha* release now?

You write you have "doubts" this bug is a dupe. Have you actually tested whether
this bug is newer than the one it was duped against? Has this worked correctly
in a previous release? Reopening a bug just because of doubts and no further
evidence is foolish. I don't say it's impossible mkaply is wrong, but since he
is a experienced Mozilla developer I would trust him more than your doubts. So
please first check your facts before producing this amount of bugspam. Do so now
at least. Thanks!

Comment 12

[Boris 'pi' Piwinger]

Yes it dit work before, I used the functionality before. I cannot say when the
problem occured, though.

pi

Comment 13

[Christoph Schulz]

IMHO I would like to say that the effects of this bug should be considered more
important as it is done now. For example, the bug prohibits the correct use of
cryptographic software like PGP, because all E-Mails with ">" markers that have
been signed will be deemed as altered when opened by the receiver. Why? Because
PGP first copies the E-Mail contents into the clipboard and then tries to verify
the signature... There might exist similar problems with other software, too.

Only my two cents...

Comment 14

[Karsten Düsterloh]

Re comment #13:
> the bug prohibits the correct use of cryptographic software like PGP,
> because all E-Mails with ">" markers that have been signed will be
> deemed as altered when opened by the receiver. Why? Because PGP first
> copies the E-Mail contents into the clipboard and then tries to
> verify the signature...

But that would be clearly a fault on the side of Enigmail (I guess that's what
you mean by PGP here). The messagepane contains a *view* of the message data and
is *not* equivalent to the message data itself!

Comment 15

[Stefan Borggraefe]

(In reply to comment #12)
> Yes it dit work before, I used the functionality before. I cannot say when the
> problem occured, though.

I just did some testing (on Linux). This broke between Mozilla 1.1 and Mozilla 1.2a.

mkaply, do you still think this is a dupe?

Comment 16

[Christoph Schulz]

(In reply to comment #14)
> > the bug prohibits the correct use of cryptographic software like PGP,
> > because all E-Mails with ">" markers that have been signed will be
> > deemed as altered when opened by the receiver. Why? Because PGP first
> > copies the E-Mail contents into the clipboard and then tries to
> > verify the signature...
> 
> But that would be clearly a fault on the side of Enigmail (I guess that's
> what you mean by PGP here). The messagepane contains a *view* of the message
> data and is *not* equivalent to the message data itself!

a) The user must be offered a means to retrieve the message data itself after
all encoding/decoding issues related to E-Mail message protocols etc. have been
resolved. Viewing the message source is not always appropriate (think of
quoted-printable replacements which are not undone in the message source view).
If copying into the clipboard from the message pane is not that very method
please describe how to achieve this.

b) It's more than confusing when copying the message via the clipboard interface
results in a message text that is neither what the sender typed nor what the
receiver sees (nor what has been transmitted over the network). Because of this
the behaviour described above is definitely a bug IMHO.

Comment 17

[Karsten Düsterloh]

> If copying into the clipboard from the message pane is not that very
> method please describe how to achieve this.

As soon as the messagepane display does some fancy magic like quote bars or
smiley substitution, you simply can't use the clipboard data for such tasks like
PGP validation.
Copy should always only copy visible content (and that is indeed bug 39098). But
with quote bars shown instead of >s, what do you expect to be copied? Or with
smileys? Or Umlauts and such, depending upon encoding?

> b) It's more than confusing when copying the message via the 
> clipboard interface results in a message text that is neither what
> the sender typed nor what the receiver sees (nor what has been
> transmitted over the network). Because of this the behaviour
> described above is definitely a bug IMHO.

Agreed.

Comment 18

[Mike Kaply [:mkaply]]

Attachment: Potential fix

OK, so the problem here is that the plain text serializer puts in quoting (>).
but the mail already has quoting (>) that is hidden and when it gets copied to
the clipboard, it's going to put the hidden ones in.

So my fix is not to have the serializer do any substituion - it shouldn't need
to.

Comment 19

[Mike Kaply [:mkaply]]

Attachment: Diff with more context so it is clearer

Comment 20

[Karsten Düsterloh]

Attachment: checking for > at start of line

Sorry, but patch 148324 is breaking the plain text editor if mail.quoteasblock
is true (and that's the default).

If you reply to a message with several quote levels as in comment #4, and
you're using the *plain text editor*, all quote levels are dropped except for
the new one. The plain text editor seems to rely upon the > put into the stream
by the serializer... :(
(If you're using the *HTML editor*, everything works fine.)

I thought about checking the first character of a line not be a > before adding
them (see attached patch), and this does indeed avoid the doubling of >s. 
But other problems will still be there: with mail.quoteasblock == true, the
plain text editor doesn't get the space character behind the >s (but this
actually isn't in the scope of this bug); and the empty lines mentioned in
comment #0 will remain, too...

This is all pretty awful - I still feel that the plain text serializer
shouldn't know about such things as 'quote characters' at all - that should be
handled somewhere inside mailnews. :-/

Comment 21

[Mike Kaply [:mkaply]]

> This is all pretty awful - I still feel that the plain text serializer
> shouldn't know about such things as 'quote characters' at all - that should be
> handled somewhere inside mailnews. :-/

I definitely agree, and I don't know why it does.

Mailnews has the quote characters already in there. The serializer shouldn't be
trying to create quotes.

I don't have the expertise in this code certainly. I'm not sure where to go with
this.

Comment 22

[Mike Kaply [:mkaply]]

There serializer should already be ignoring blockquote type=cite.

So now I am totally confused.

Here's a comment from akk on this:

First, why did mailnews add those hidden > signs?  Where are they
used?  It seems like those are what caused the regression.

It's easy enough to make the serializers ignore certain tags: look at
what they already do with <blockquote type="cite"> or with
<br type="_moz">.  If mail insists on double quoting everything,
once with <SPAN class="moz-txt-citetags">&gt;</SPAN> and once with
<BLOCKQUOTE type=cite>, then I suppose hacking the serializer to
ignore the hidden > isn't adding that much more ugliness.
(BTW, why are these mail tags done in upper case?  I thought
lower case was considered to be better style, and also easier
to compress over transports like modems?)

I would hope that anyone implementing such a thing would also add
a test case to the serializer tests (TestOutSinks, run as part of
the tinderbox tests and described in
http://www.mozilla.org/editor/serializer-tests.html).

Comment 23

[Mike Kaply [:mkaply]]

Attachment: OK, I think I might understand things now - breaks mail.quoted_graphical

Can you try this one on for size?

Seems to fix the problem, and quoting is working fine...

Comment 24

[Karsten Düsterloh]

Huh?
With patch 148444 applied, all >s in the plain text view
(user_pref("mail.quoted_graphical", false)) in the messagepane are gone...

Comment 25

[Mike Kaply [:mkaply]]

How would a user have known about that preference?

That being said, I give up.

I can make it work perfectly until you throw that preference in.

I believe that at its core, this is a MIME Problem.

Comment 26

[Asa Dotzler [:asa]]

we'd consider a reviewd patch but it doesn't look like this is happening for 1.7

Comment 27

[Karsten Düsterloh]

*** Bug 284480 has been marked as a duplicate of this bug. ***

Comment 28

[Mike Cowperthwaite]

(In reply to comment #22)
> [The] serializer should already be ignoring blockquote type=cite.

One thing unremarked so far in this bug: the problem is not with "messages 
from other clients" so much as with "messages that are format=fixed" [i.e. NOT format=flowed].  Copying quoted text in a f=f message (displayed *as* f=f 
[i.e, mailnews.display.disable_format_flowed_support=false]) results in the expected text in the clipboard buffer.  

The DOM for the quoted f=f text is something like:
  <blockquote type=cite>quoted text</blockquote>
with no embedded '>' signs in sight.  So, the <blockquote> in *this* case, at least, must be getting processed to add the quote-prefixes, presumably in the serializer.


It ought to be possible to handle the generation of the '>' at rendering time instead of the border, when quoted_graphical is false.  (It could be done 
using CSS's 'content' for the single-level quote case, altho the generated 
DOM would need to change; I'm not so sure about multiple-level quotes.)  If 
this were implemented (with or without CSS), then the generation of those 
moz-txt-citetag <span>s could be eliminated.


OT: note that TB's support for turning off quoted_graphical is broken: it 
shows the border alongside the '>' marks; bug 289205.

Comment 29

[(not reading, please use [e-mail] instead)]

sorry for the spam.  making bugzilla reflect reality as I'm not working on these bugs.  filter on FOOBARCHEESE to remove these in bulk.

Comment 30

[rsx11m]

Still reproducible with Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1pre) Gecko/2008070102 SeaMonkey/2.0a1pre nightly trunk build.

Copy-pasting from plain text results in either "> > " (Windows, to Wordpad),
or a combination of tab and ">" (Linux with SM 1.1.9, to OpenOffice).

Comment 31

[Ilja Sekler]

Still there in Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.4pre) Gecko/20090915 Lightning/1.0pre Thunderbird/3.0b4pre

Sorry for the spam, but while the release of Thunderbird 3 comes closer and a lot of effort is put into great new features, this old skeleton in the closet is still thirsting for a little love.