212554 - Saving a Web Page as Text Yields Unwanted "Enhancements" for Styles and Special Characters

Status: RESOLVED INVALID

(Core :: DOM: Serializers, defect)

Description

[David E. Ross]

User-Agent:       Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.4) Gecko/20030624
Build Identifier: Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.4) Gecko/20030624

When I open an HTML file in the browser and then save it as text, special
characters are saved as question marks, Italic text is bracketed with virgules,
bold text is bracketed with asterisks, etc.  The result is not merely a text
file that contains what was seen in the browser window; instead, it has
unwarnted "enhancements".  

Reproducible: Always

Steps to Reproduce:
1.  Display the Web page at the cited URL.  
2.  From the menu bar, select File > Save Page As
3.  In the Save As window, select Text Files for Save as type.  
4.  In the Save As window, change the .html extension to .txt (another problem)
5.  In the Save As window, select the Save button.  
Actual Results:  
The file was saved with "enhancements" as described above.  Note that em-dashes
(—) were saved as question marks.  (Ellipses (…) on a different page
were also saved as question marks.)  

Expected Results:  
I expected a file that showed running text as it would be seen if all stylistic
markups had been suppressed but with the hex bytes for special characters
preserved (so that they could be viewed via Wordpad).  That is, I expected that
saving the page as text would give results similar to Netscape 4.79.   

This is a rewrite of bug #212547 with those portions that duplicated bug #135239
eliminated and with some clarification added.  

Workaround: Display the page.  Select the entire page.  Copy it and then paste
it into a Notepad or Wordpad window.  This workaround, however, shows the
effects of bug #99159.

Comment 1

[Dan Tobias]

One thing you should note:  — is not an em-dash, and … is not an
ellipsis, regardless of what the original reporter said.  Numeric character
references, according to the HTML standards, are with respect to the Unicode
character code positions, not any particular character encoding.  While the
proprietary Windows-1252 encoding does indeed have an em-dash at position 151
(decimal) and an ellipsis at position 133, the Unicode character set has control
characters in these positions (as does ISO-8859-1).  Since this range of control
characters is specifically disallowed in HTML according to the standards, their
meaning in an HTML document is undefined (and a validator will reject a page
that contains them).

It's an interesting question just what the "save as plain text" function ought
to do when it encounters non-ASCII characters (whether invalid ones like noted
above, or valid ones in the Unicode repertoire outside the US-ASCII range); that
would depend on in what character encoding the output text were to be deemed to
be.  The full Unicode repertoire would require an encoding such as UTF-8 that
could potentially take more than one byte per character, which would cause the
output to look funny in text viewers accustomed to single-byte ASCII,
ISO-8859-1, or Windows-1252 characters.

Comment 2

[David E. Ross]

When I look at ISO 8859-1 at <http://www.bbsinc.com/iso8859.html>, I see &#133;
= &hellip; = "low horizontal ellipsis" and &#151; = &mdash; = "em dash".  ISO
8859-1 allows for both numeric character and symbolic entity references. 
(Control characters use character references &#000; through &#031;.)  When I
look at the HTML 4.01 Specification at <http://www.w3.org/TR/REC-html40/>, I
also see that numeric character references are supported.  

I use entity references in preference to character references where the former
display appropriately by Netscape 4.79.  I logged the user agents in visits to
my pages and found that a significant number of users still use Netscape 4.79. 
Where Netscape 4.79 does not display entity references appropriately, I resort
to character references in order to ensure readability by a broad audience.  

In my Web pages, I had been using an implied, default reference to ISO 8859-1. 
However, while researching this bug, I discovered a warning against that
practice.  I will now be updating my pages to include an explicit reference to
ISO 8859-1.  I had also been putting a leading zero in the character references,
a result of the need for a leading zero when coding the characters as escape
sequences under Windows.  As I update my pages, I will eliminate the leading
zeros.  I have made both corrections to my home Web page, but the problem with
characters changing to question marks remains.  

NOTE:  I first noticed the problems described in this bug report while browsing
pages not in my Web site.  The characters that get changed to question marks in
those other pages include "smart quotes" (which I abandoned a few years ago),
which are quite prevalent.  I can deal with them but only if they remain
distinct and not changed.  I only cited my own Web page in this bug report
because it's a relatively simple page that will persist long enough to use it in
a test.  

Comment 3

[Dan Tobias]

The page you cite at <http://www.bbsinc.com/iso8859.html> notes that the
characters in green (including the two you mention) are extended Windows
characters, *not* part of the ISO-8859-1 set.  That page is mistaken when it
cites them via numeric references in the range from 128 to 159, as those
numbers, in the Unicode standard, are control characters (different ones from #0
through #31, but still control characters anyway... see
<http://www.unicode.org/charts/PDF/U0080.pdf>.)

I didn't mean that numeric character references (with or without a leading zero,
which shouldn't matter) are invalid in general, just ones in numeric ranges that
are disallowed in HTML by the standards, as #128 through #159 are.  There are
proper numeric codes for so-called "smart quotes", em dashes, and ellipses, but
they are much higher; for instance. #8220 is a left curly quote.

For more information, here are a couple of relevant pages in my own sites:
http://webtips.dan.info/char.html
http://mailformat.dan.info/body/charsets.html

Comment 4

[David E. Ross]

First of all, I forgot that I cited my own <http://www.rossde.com/> for this
bug.  I cleaned it up not long ago, and it is no longer a suitable test page.  

More important, however, is the fact that there are many, many pages on the Web
that use escaped characters (e.g., Alt-0133 for horizontal ellipses) or
character references (e.g., &#151; for em-dash).  These display on a Mozilla
window as the authors intended.  The problem I described for this bug occurs
when the display is saved to a file.  Indeed, I originally discovered this
problem with someone else's Web page and not my own, cited page.  I cited my own
page only because I used my page to test the repeatability of the problem.  

The primary issue is one of WYSIWYG.  If I see it on my monitor screen, I should
get it in a file.  

Comment 5

[Dan Tobias]

It might be considered a "bug", or at least a "misfeature", that Mozilla
"displays what the author intended" when given —, given that this is a
bogus reference which should not refer to any printable character; the fact that
it's excreted by brain-dead software (mostly from Microsoft) doesn't make it right.

Comment 6

[David E. Ross]

I forgot to include the following in my prior comment #4.  The problem seems to
go away if I declare charset=WINDOWS-1252.  This works for both escaped
characters and character references.  For me, of course, this workaround is
effective only for my own HTML files and not for the many others already on the
Web.  

This raises a question regarding comment #5.  Is it now proposed to disable
recognition of charset=WINDOWS-1252 in Mozilla?  

Comment 7

[Dan Tobias]

If charset=windows-1252 is set, then 8-bit-coded characters in that set
(including the so-called "smart quotes", etc.) are indeed valid.  windows-1252
is itself a proprietary, vendor-specific character set and is not desirable to
use in preference to a more standardized set such as iso-8859-1, but the
windows-* sets have been registered with the standards bodies now so that it's
not invalid to use them (just not a good idea when less vendor-particular sets
are available).  There's nothing wrong with Mozilla supporting it.  However,
numeric character references such as — are, by the standards, always
rendered with respect to the Unicode character set, and shouldn't be affected by
the choice of character encoding for the particular document.  € through
Ÿ always refer to control characters which are not valid for use in HTML
documents, no matter what the "charset" parameter sent by the HTTP server (or,
inferiorly, faked by META tags).

Comment 8

[David E. Ross]

My cited page (URL updated in this report) now validates as "HTML 4.01
Transitional" with the ISO-8859-1 character set.  The problem still appears. 
Now, instead of the &151; character reference, the — entity reference
translates into ?.  On another page of mine that also validates as "HTML 4.01
Transitional" with the ISO-8859-1 character set, the … entity reference
also translates into ?.  

If some users do indeed want these "enhancements", they should be optional.  In
the meantime, saving text with markups not seen in the browser window is an error.  

Comment 9

[Dan Tobias]

Even with the character references done in a valid manner (as you now do)
instead of an invalid one (as you used to do), you still have the problem that
you've got non-ASCII characters that you're trying to save to a flat text file.
 This may not be possible, depending on the character encoding being used for
saving.  If Mozilla's save routine uses "us-ascii", then nothing above character
#127 is representable.  If it's using ISO-8859-1, then the characters in that
set (including some accented letters) would be representable, but not such
things as "em dashes" or "curly quotes", which are not part of that character
set.  In the proprietary Windows-1252 character set, they do exist.  This
probably explains why you got different results depending on what character
encoding was declared, because Mozilla probably uses that as the encoding to
save under.

If the document were to use other Unicode characters, such as Hebrew or Chinese,
then it probably wouldn't be saveable as plain text unless in an encoding like
UTF-8 that supports it.

Comment 10

[Ben Bucksch (:BenB)]

Thanks for the rational discussion.

Following the comments of Dan Tobias, this bug looks INVALID to me, though. If
you use the wrong charset or include invalid entites, of course you get
undefined results. The "?" is Mozilla's placeholder for "I can't display that".

> If I see it on my monitor screen, I should get it in a file.

Not possible. Mozilla takes *great pains* to display Unicode characters. There
is just no way to represent them in plaintext in us-ascii or iso-8859-1 (which
many apps assume).

You could argue that Mozilla should convert to the closest repalcement char that
is available, e.g. mdash -> "-", but that would be an RFE, and it may actually
work already, I didn't check it. Maybe you could reopen this bug and turn it
into this RFE, but you definitely can't represent the whole Unicode character
range this way.

As for /italics/ and *bold*, that's an entirely different matter and would
probably be part of bug 135329.

Comment 11

[Ben Bucksch (:BenB)]

> would probably be part of bug 135329

I meant bug 135239

Comment 12

[Simon Montagu :smontagu]

(In reply to comment #10)

> As for /italics/ and *bold*, that's an entirely different matter and would
> probably be part of bug 135329.

There is already a hidden pref to turn this kind of formatting off:
converter.html2txt.structs