Text doesn't wrap when replying to non-f=f emails with all-in-one-line paragraphs (option or threshold to assume f=f for quote)

RESOLVED FIXED in Firefox 48

Status

()

Core
Editor
--
enhancement
RESOLVED FIXED
10 years ago
7 months ago

People

(Reporter: Doug Schepers, Assigned: Jorg K (GMT+2))

Tracking

Trunk
mozilla48
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox48 fixed)

Details

Attachments

(1 attachment)

(Reporter)

Description

10 years ago
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.4) Gecko/20070515 Firefox/2.0.0.4
Build Identifier: version 2.0.0.4 (20070604)

Sometimes when replying to an email, the quoted text doesn't wrap, but is quoted in one long line, with a single ">" in front.  This doesn't happen with all emails, but it is consistently repeatable with those emails.  I believe the common thread is that they have content-type "text/plain; charset=us-ascii".  Selecting "Edit->Rewrap" corrects the wrapping of the quoted text in that reply, breaking it into multiple lines with an intial ">".

Reproducible: Always

Steps to Reproduce:
1. Reply to an email with content-type "text/plain; charset=us-ascii"
2. Note that longer lines are not wrapped
3. Select "Edit->Rewrap" to correct email
Actual Results:  
Each paragraph is a single long line of quoted text 

Expected Results:  
Long lines should be broken down into multiple lines with an initial quote indicator automatically, without having to select "Edit->Rewrap" after selecting "Reply"

I made sure that all the possible options for word wrapping were selected, and even tried toggling those.  I have all replies set to "convert to plain text", so this is not an HTML email issue.

Comment 1

10 years ago
You are composing in plain text mode, right? I see this a lot (typically from outlook users), the charset doesn't matter. The quote doesn't get wrapped because it wasn't wrapped and wasn't format flowed to start with. 

So technically it might not be a bug, but still what's displayed on compose isn't what you'd expect to see.

Comment 2

9 years ago
This is usually seen with quoted-printable messages that do not follow RFC3676 with using format=flowed (as Magnus mentions, Outlook but also Lotus Notes tend to format outgoing e-mails that badly). The problem is that those e-mails are encoded using a plain "=" at the end of a line rather than "=20" indicating a soft line break for wrapping. So, strictly speaking, without format=flowed set, the entire paragraph is encoded as a single line, and since Thunderbird is instructed not to rewrap the message (as it can do in flowed formatting), it ends up being a single line in the reply.

The question would be, if this case is supposed to be handled, how to detect an "overlength" line and how to decide whether or not Thunderbird is allowed to rewrap the line even if format=flowed is not present.

If these assumptions are correct, this is likely a special case of bug 20240.

Comment 3

9 years ago
Bryan, I'd like to get your opinion on this. Assuming that the previous comments are correct and it's a message-encoding issue, with some vendors simply ignoring the flowed standard and making up their own versions based on quoted-printable, this is still an annoyance in terms of user experience. Anything that could/should be done about it?

One could just say that Thunderbird is handling the message according to the standard, hence do nothing, dupe this against bug 32100, and we are done here. But, it intuitively doesn't make much sense to unwrap a paragraph of several hundred characters into a single line for the quote. The Edit > Rewrap option is unsatisfactory as it would also rewrap hard line breaks and breaks flowed format in the process (there are other bugs on that). I could think of either a "consider this message flowed even if it is not" option could be present in the composition window, or some logic based on unlikely line lengths and other formatting clues.
Difficult situation.  To help the most amount of people we need to make things wrap "correctly" automatically.  By "correctly" I mean correctly for the person and not for the standards.  Essentially this "correct" wrap is probably the size of the message window.

Then we could offer an option for people to "undo" our automatic rewrap in case it acted on something a person didn't want wrapped.  And also a more advanced way to turn off our auto-wrap system completely.

Comment 5

9 years ago
Thanks, confirming as RFE based on comment #4 then. I realize that some people may object against weakening behavior according to the standard, but we are wrapping such encoded-as-one-line paragraphs for display already even without f=f set, and it would be consistent to do so also when quoting when it becomes apparent that this "line" was not intended to be preformatted.
Severity: normal → enhancement
Status: UNCONFIRMED → NEW
Component: Message Compose Window → Composition
Ever confirmed: true
OS: Windows XP → All
Product: Thunderbird → MailNews Core
QA Contact: message-compose → composition
Hardware: PC → All
Summary: Text doesn't wrap when replying to emails with content-type "text/plain; charset=us-ascii" → Text doesn't wrap when replying to non-f=f emails with all-in-one-line paragraphs (option or threshold to assume f=f for quote)
Version: unspecified → Trunk

Comment 6

8 years ago
All very technical and academic - but it is now 2010 and three years nothing has changed.
On my Intel iMac I have both Thunderbird  2.0.0.23 AND Apple Mail.
It is all very well to blame the sender's alleged flawed encodings - but the fact remains that the same emails as received into Apple Mail do not spread over several screenwidths when quoted back.
So it absolutely IS a Thunbderbird issue.

Such a pity.  I prefer the Thunderbird Application but its use in context of the subject bug has become a time-waster and quite intolerable.
Duplicate of this bug: 542681

Comment 8

8 years ago
ethos2, this is a mayor annoyance, I agree. It may sound "academic" to discuss standards, but if everything would be done by intuition, we'd end up with even more interoperability issues between e-mail clients. Following the general rule of being strict with the standard when sending out a message but flexible when it comes to interpreting messages written by other clients, the question is how to determine the "intention" of the sender, and how to get a trade-off between those senders who intended a fixed format and those who where just using their e-mail program's defaults.

> (In reply to comment #4) Essentially this "correct" wrap is probably the
> size of the message window.

This sounds right and would provide the user with a means to easily adjust the wrapping. From the implementation point of view, it would imply though that the wrapping (of both the quote and potentially the new text written) has to be adjusted on the fly when the user changes the window width during composition.

The easier way may be to use a limit being a factor of the wrapping size mailnews.wraplength, defaulting to 72 (i.e., using a factor of 1.5 would set the threshold at 108), then flip the f=f flag for that message to "true" and allow it to wrap to 72 when quoted.

Here comes the bad news, the f=f encoding for quotes in replies is currently broken in 3.x (bug 456053, depending on Core bug 448198 as a regression of
bug 215068). Thus, following either of the suggested approaches by a fixed or window width would introduce hard line breaks rather than flowed ones. Again, it's a trade-off which issue is considered more and which is less acceptable.

As for the bug's duration, there are many bugs around for years until they eventually get resolved, so let's stay optimistic.

Comment 9

8 years ago
Thankyou <rsx11m>,   I respect Mozilla and your time in kindly presenting the gentle logic of your opening paragraph.    But even Microslop's notoriously buggy Outlook Express  does not it this context misbehave as does Thunderbird.   I mention too that your above answer as emailed to me and received in Thunderbird does not flow if my screen window is narrowed.   Well ... I guess I should say that it DOES flow but corrupts the line endings which appear as Hard Returns. and thus is barely presentable and a bit of a sod to read..

I am truly surprised that the Mozilla team just can't seem to get Thunderbird right.
The Email module of the old Netscape Communicator walks all over Thunderbird. IMHO.

Comment 10

8 years ago
(In reply to comment #9)
> Thankyou <rsx11m>,   I respect Mozilla and your time in kindly presenting the
> gentle logic of your opening paragraph.    But even Microslop's notoriously
> buggy Outlook Express  does not it this context misbehave as does Thunderbird.

Given that Microsoft and Lotus came up with this non-standard representation of flowed-format text, I'm not surprised that Outlook can handle it "correctly". For the record, I'm not affiliated with Mozilla or Mozilla Messaging, just another user doing some QA and occasionally contributing a fix.

>  I mention too that your above answer as emailed to me and received in
> Thunderbird does not flow if my screen window is narrowed.   Well ... I guess I
> should say that it DOES flow but corrupts the line endings which appear as Hard
> Returns. and thus is barely presentable and a bit of a sod to read..

Indeed, they are hard line breaks. This is how bugzilla represents comments (all of the comments here have the same wrapping regardless of window or font size), so that would be something that the bugzilla developers may have to fix (plus the other quoting regressions I've referred to above).

In general, judging from comment #4, the intention is to cover that specific encoding in a reasonable way since it's frequently met. The question is how exactly to do that, which scheme with which trade-offs to apply, and someone eventually will have to come up with a patch.

Comment 11

6 years ago
Forcing flowing on a single unflowed line seems a no brainer.  There would be no wrapping issues, you are not mixing soft and hard breaks. 

So the simple heuristic could be: when quoting a single line (per depth) from an unflowed message, flow it (consequently it will be soft wrapped per mailnews.wraplength)

This should work for any quote depth as long it's just a single line, per level, otherwise you leave it alone.

long line one 
> long line two
>> long line three

would be flowed and soft wrapped into

> long
> line
> one
>> long
>> line
>> two
>>> long
>>> line
>>> three

Updated

6 years ago
Duplicate of this bug: 720183

Comment 13

6 years ago
(In reply to al_9x from comment #11)
> So the simple heuristic could be: when quoting a single line (per depth)
> from an unflowed message, flow it (consequently it will be soft wrapped per
> mailnews.wraplength)

The "when quoting a single line (per depth)" seems to be the key here. Most malformed messages I receive have a single line (q-p'ed or not) per paragraph and then an empty line, so this would be matched and wrapped accordingly. I think I've received some messages which start a new paragraph with an empty line, but this seems to be the exception.

The "single line" qualifier should also take care of not breaking patches in a reply, given that those usually span more than one line and have "hard" breaks.
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 173918

Comment 15

5 years ago
Ludo, I don't think that this is a duplicate of bug 173918; that bug is about re-wrapping incoming messages created with f=f whereas the issue here occurs with replying to messages which don't have f=f applied but where it was emulated by incorrect usage of the quoted-printable line-wrapping mechanism. Agreed?
Status: RESOLVED → REOPENED
Resolution: DUPLICATE → ---
Agreed.
Status: REOPENED → NEW
See Also: → bug 878616
Removing myslef on all the bugs I'm cced on. Please NI me if you need something on MailNews Core bugs from me.
(Assignee)

Comment 18

a year ago
I started looking at this since fixing this bug is a condition for fixing bug 233705.

I'm testing with the message from attachment 8736335 [details].
This message has this header:
Content-Type: text/plain; charset=utf-8

Looking at the HTML to be converted to plain text here:
https://dxr.mozilla.org/comm-central/source/mailnews/compose/src/nsMsgCompose.cpp#2407
I see:
<pre wrap>
Hi,

this is a mail meant to simulate obnoxious mails ...

Changing the sample e-mail to be flowed, that is,
Content-Type: text/plain; charset=utf-8; format=flowed
I see this to be converted to plaintext:
Hi,
<br>
<br>this is a mail meant to simulate obnoxious mails

Obviously the conversion to plaintext respects the <pre wrap> and doesn't wrap the long line, but I have no idea why <pre wrap> was inserted in the first place.

I'm tipping that it's inserted here:
https://dxr.mozilla.org/comm-central/source/mailnews/mime/src/mimetpla.cpp#372

Does anyone know why?
Blocks: 233705
Flags: needinfo?(ben.bucksch)

Comment 19

a year ago
AFAIK, the <pre> is used for all replies to text/plain messages without f=f to suggest a fixed font (e.g., to ensure proper alignment for patches, etc.). With f=f text can be rewrapped, thus a fixed alignment isn't anticipated and no <pre> environment given.

So, <pre wrap> without a parameter or <pre wrap=""> means "don't wrap" in this case as there is no boundary specified? Making that <pre wrap="120"> might be a (very!) crude workaround then...
Jorg,

plaintext is converted to <pre>, because that HTML tags was basically invented for exactly that use case: To represent plaintext in an HTML document. That's precisely what we're doing here, so arguably, that's the only reasonable conversion. Plaintext not only includes fixed width fonts, which are actually optional and also bad to read, but more importantly changes the space handling, e.g. consecutive spaces are not collapsed by the HTML parser. We added the wrap attribute, because we don't want obnoxiously long lines in the source email to cause vertical scrollbars, which make things unreadable (and can even effectively hide content).

format=flowed is a totally different ballgame. It's deliberately designed to have a similar semantics to structured, flowing text (like HTML, and exactly NOT like plaintext), but with a wire representation to still be readable in plaintext-only readers (unlike HTML). The semantics of f=f are actually like <p>. (I don't remember why we convert to <br> and not <p>, there must be a reason. The code or at worst bugzilla ticket of the implementation would probably tell why, try CVS history. But first, take a look at the f=f spec, it's well-written and also tells about the Why and the idea, which should clear things up a lot.
Flags: needinfo?(ben.bucksch)
I was a little confused what the bug here is, because it's working fine for me.

Reproduction:
1. Load the sample message from bug 233705 by clicking on https://bugzilla.mozilla.org/attachment.cgi?id=8736335 .
2. Click Reply. If you have default settings, you will get the HTML composer. There, all will look fine. If you send it as plaintext, it will be fine in the plaintext on the wire as well, and in the reader. No bug there, as far as I could see.
3. (Still on the original mail) Click Reply while holding Shift key on your keyboard. You will get the plaintext editor. That's a hidden feature that switches to the plaintext editor or vise versa just for this one email.
4. In the plaintext editor, you see:
Foo wrote:
> Blabla bla bla ..... bla
whereas the quote is a very long line, exceeding 80 characters. That's the bug here, I think.

So, this bug applies only when
a) the sender sent a malformed non-f=f plaintext message with excessively long lines. Apple Mail used to do that.
AND
b) our user uses the plaintext editor. He had to manually switch to that, it's not the default (other than for NNTP maybe).

That's why this bug isn't very prominent. It's still good to fix it.
Our quote handling hasn't been the best. At one point, we would convert a plaintext message to HTML and then back to plaintext. I hope we fixed that a decade ago, but not sure. If we did, I think that's what introduced this bug.
In the code path than converts plaintext email to a plaintext quote (in the plaintext editor), we need to enforce the 80 character limited (wrap_width pref, actually 78 characters or so) and break long lines, and add > before any linebreak we insert, so that the whole block is > marked. I think the hardest part is finding the correct code place to fix.
> we need to enforce the 80 character limited (wrap_width pref, actually 78 characters or so)
> and break long lines

Actually, that's wrong. If the quoted line is one that's already been quoted several times, we could introduce the "comb effect". We'd do:
> > > ablabalbla abl blablabl blalballa blabalbla abl blablabl blalballa blabalbla abl
>  ablabalbla abl
> > > blabalbla abl blablabl blalballa blabalbla abl blablabl blalballa blabalbla abl
>  blabalbla
> > > cblabalbla abl blablabl blalballa blabalbla abl blablabl blalballa blabalbla abl
>  cblabalbla abl
> > > dblabalbla abl blablabl blalballa blabalbla abl blablabl blalballa blabalbla abl
>  dblabalbla

which is horrible to read. MS Outlook Express used to have this bug, and we all hated it.
So, you see, nothing about plaintext handling is obvious. There are footangles everywhere!

I don't know how to fix it. When deciding whether to break the line, you could not count quote markers and spaces at the beginning of the line, and additionally leave some leeway of 30 chars or so (in addition to wrap_width pref) before you break the line. Once you do break the line, I guess it should be at wrap_width. But you're running high of introducing the comb effect.
I wrote:
> ... HTML composer. There, all will look fine. If you send it as plaintext, it will be fine in the
> plaintext on the wire as well, and in the reader. No bug there, as far as I could see.

So sorry! That's wrong, too. It looks all fine in TB, in HTML editor and also in our mail viewer, but we send it as long line (which is bad), in plaintext with f=f (!) (which is good). Any f=f reader will break the line when displaying it, because the spec imposes that.

Same is actually true with the plaintext editor. It displays wrong in the editor, but then is also sent as f=f, which displays fine.

A reply to the reply then works fine in both editors, because the source was f=f, which we can break as needed, per spec.

We still can't easily do that with the first non-f=f email quote. So, I guess the problem remains with the comb effect, and related problems, like breaking ASCII art and similar. If I had to fix it at gunpoint, I'd try the algo from the last comment 23.
This is why I was afraid to start looking at this bug myself. Composition code is complicated!

Anyway, I hope this is leading us on the right path to fixing bug 233705 without regressing users who've flipped that pref. If you (Jorg or :BenB) like, I can try to read up on this to help make a decision here, but I don't have too much experience with the correct ways to generate MIME bodies. Otherwise, I'll just watch from the sidelines.
(Assignee)

Comment 26

a year ago
(In reply to Ben Bucksch (:BenB) from comment #21)
> 4. In the plaintext editor, you see:
> Foo wrote:
> > Blabla bla bla ..... bla
> whereas the quote is a very long line, exceeding 80 characters. That's the
> bug here, I think.
Yes.

The problem is a "misunderstanding" between MIME code and serialiser.

The MIME code inserts "pre wrap" for the reasons explained above. The serialiser than converts this to "white-space: pre;" losing the "wrap". Just use the DOM inspector to see it. If I change the "white-space: pre;" to "white-space: pre-wrap;" using the DOM inspector, I get (obviously) what is shown in attachment 8736591 [details].

My fix will be to "educate" the serialiser to respect the input data better. As per bug 233705 comment #29, the fix will be around here:
https://dxr.mozilla.org/mozilla-central/source/editor/libeditor/nsHTMLDataTransfer.cpp#1798

I repeat: The result will look like in attachment 8736591 [details]. It will *NOT* look like in attachment 8736422 [details] (left side).

This way we avoid the comb effect and give the user what they get today with wrap_to_window_width set, but without all the nasty side effects.

Do I have agreement on this?

Jim, I think it's in safe hands and you can just watch ;-) Thanks for the interest.
(Assignee)

Comment 27

a year ago
Safe hands, haha. Sadly, my analysis was wrong.

The MIME code inserts <pre wrap> here, that was correct:
https://dxr.mozilla.org/comm-central/source/mailnews/mime/src/mimetpla.cpp#199

Next, the "<pre wrap>Long line etc." is converted to plain text here:
https://dxr.mozilla.org/comm-central/source/mailnews/compose/src/nsMsgCompose.cpp#2407
resulting in "Long line etc.", so the serialiser obeys the pre-formatting. Of course it loses the HTML tags.

Next, this is placed into a "white-space: pre;" here:
https://dxr.mozilla.org/mozilla-central/source/editor/libeditor/nsHTMLDataTransfer.cpp#1798

So my idea to pass the "pre wrap" info on doesn't fly since right in between the serialiser loses that information.

I'll have to look for a new idea. Of course, hard-coding "white-space: pre-wrap;" as per bug 233705 comment #29 will fix it.
(Assignee)

Comment 28

a year ago
Sorry, I've changed my mind 100%.

This discussion shows that we can't rewrap the long line, all we can do is wrap it to the screen. That however will destroy ASCII art, etc. So if we wrap to the screen, we need to control it via a preference. This preference already exists, wrap_to_window_width, and we have bug 233705 to make it work.

So I suggest now that this bug here is a WONTFIX, we'll tell people so set the preference. We might even expose the preference in the user interface.
JorgK, I think we concluded (and unanimously agreed) that the pref is misguided and should be removed. Very few people use it.

The correct behavior is either leave long lines alone and quote them *exactly* as in the source email (which is what we do today, i.e. WONTFIX this bug), or to wrap the quote, without destroying ASCII art and avoiding the comb effect. I think the algo in comment 23 is fair enough to do that. Maybe some of you bright minds can even improve it.
(Assignee)

Comment 30

a year ago
(In reply to Ben Bucksch (:BenB) from comment #29)
> JorgK, I think we concluded (and unanimously agreed) that the pref is
> misguided and should be removed. Very few people use it.
Sorry, changed my mind. Let me attach my new patch to bug 233705.

> The correct behavior is either leave long lines alone and quote them
> *exactly* as in the source email (which is what we do today, i.e. WONTFIX
> this bug),
My new suggestion is: leave them alone if wrap_to_window_width is not set or ...

> or to wrap the quote, ...
but to the screen only, with wrap_to_window_width set. Then it depends on the width of the screen if ASCII art is badly wrapped.
squib wrote in bug 233705 comment 32:
> it could cause problems for code patches sent inline in bodies (as the Linux kernel devs do).

<rant>
Urgs, do they still do that, more than a decade later? That's so wrong, diffs should be inline attachments.

meh, yes they do: https://www.kernel.org/doc/Documentation/SubmittingPatches Section 6.
Can somebody donate them a Phabricator or something? :-)

A simple Ctrl-A, Reply would allow them to quote inline attachments, but only in Thunderbird...

The problem with plaintext mail is that so many people developed habits based on it, and every single one fights tooth and nails to keep the habit "because that's how it's supposed to work" and "it has always worked", no matter how broken it is. And then you add broken implementations like Apple Mail with overly long lines to the mix, and it gets unworkable.

That's why I'm so afraid to touch anything. It's all so fragile and carefully balanced out.
</rant>

So essentially, I'd propose:
1) we fix up the display in our plaintext composer
2) remove the pref, and
3) otherwise leave things as-is.
(Assignee)

Updated

a year ago
Component: Composition → Editor
Product: MailNews Core → Core
(Assignee)

Comment 32

a year ago
Created attachment 8738048 [details] [diff] [review]
Proposed solution.

Masayuki-san, I carried over one hunk from bug 233705 since Ehsan wanted to separate the issues.

You said in bug 233705 comment #53 (quote):
Although, this seems a risky change but I support this change because I don't like the non-wrapped text.

So I hope for your approval.

I repeat what I said in bug 233705 comment #47:

I've tested these cases:
mailnews.wraplength = 72 (default):
The plaintext body has "white-space: pre-wrap; width: 72ch;".
Any quote sitting inside has "white-space: pre-wrap;" and will also be wrapped to 72.

mailnews.wraplength = 0:
The plaintext body has "white-space: pre-wrap;" and will be wrapped to the screen.
Any quote sitting inside has "white-space: pre-wrap;" and will also be wrapped to the screen.
Assignee: nobody → mozilla
Status: NEW → ASSIGNED
Attachment #8738048 - Flags: review?(masayuki)
Comment on attachment 8738048 [details] [diff] [review]
Proposed solution.

Looks good to me.

There is random thought: If the width is too narrow, pre-wrap causes very tall text rendering. I guess that blockquote(?) should have min-width especially when a lot of blockquote elements are nested due to a deep thread. Of course, not scope of this bug, and up to you what should we do for this issue.
Attachment #8738048 - Flags: review?(masayuki) → review+
(Assignee)

Comment 34

a year ago
Thanks for the review.

Yes, I've thought about what would happen if someone set the width to - say - 5. But then, there is no protection against all sorts of user-caused issues in life ;-)
Keywords: checkin-needed

Comment 35

a year ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/192a5ffbf501
Keywords: checkin-needed

Comment 36

a year ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/192a5ffbf501
Status: ASSIGNED → RESOLVED
Last Resolved: 5 years agoa year ago
status-firefox48: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla48

Comment 37

11 months ago
I am not sue whether I understand all details here ...
With  en-US SeaMonkey 2.40 final Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) from official download page,  Gecko/20100101  Firefox/ 43.0  Build 20160120202951, (Default Classic Theme) on German WIN7 64bit reply-quotes for particular e-mails were a single line with all text in 1 line.

With 2.45 (fix for this one) quote looks as you see here in the upper half of the picture: <http://shortlinks.de/c54n>
The quote has wraps as expected, but the leading blue ">" of the following lines are missing. Editing of quotes in response-emails becomes tricky, if you add own text into new lines within the quote you will have to copy a leading ">" to the rest of the quote following the own text.

I think a next fix-step will be required to get a more satisfying result.

Updated

11 months ago
See Also: → bug 1288911

Updated

7 months ago
Depends on: 1288911
You need to log in before you can comment on or make changes to this bug.