205518 - failed to sniff (undeclared, apparent) ISO-8859-1 encoding

Status: RESOLVED DUPLICATE of bug 181344

(Core :: Internationalization, defect)

Description

[Roland Turner]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.3) Gecko/20030327 Debian/1.3-4
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.3) Gecko/20030327 Debian/1.3-4

Neither the page nor the HTTP header specifies a charset, however the page does
use 8859-1's 0xA3 for the Pound symbol. With default settings, Mozilla replaces
both the pound symbol and the digits following it with a single small square
that appears to contain the characters "FFFD" or "FFF0".

Selecting "View" -> "Character Coding" -> "Western (ISO-8859-1)" causes the page
to re-render with the author's intended contents; the pound symbols and the
digits following them become visible.

This bug report is offered in the hope that its analysis may be of use, but if
the issue of default charsets has already been done to death (or more compelling
arguments than those that I've considered have already been made) then
presumably this bug will be closed as WONTFIX.

http://www.w3.org/TR/html4/charset.html 5.2.2 contains the following non-sequiter:

    The HTTP protocol ([RFC2616], section 3.7.1) mentions
    ISO-8859-1 as a default character encoding when the
    "charset" parameter is absent from the "Content-Type"
    header field. In practice, this recommendation has
    proved useless because some servers don't allow a
    "charset" parameter to be sent, and others may not be
    configured to send the parameter. Therefore, user agents
    must not assume any default value for the "charset"
    parameter.

The relevant text from RFC2616 3.7.1 is:

    The "charset" parameter is used with some media types to define the
    character set (section 3.4) of the data. When no explicit charset
    parameter is provided by the sender, media subtypes of the "text"
    type are defined to have a default charset value of "ISO-8859-1" when
    received via HTTP. Data in character sets other than "ISO-8859-1" or
    its subsets MUST be labeled with an appropriate charset value.

W3's argument contains several flaws:

- That some servers send an incorrect charset parameter has no bearing on
selection of a default charset for the case where no charset is specified by the
server; regardless of any default in the protocol specification, the user agent
can reasonably be expected to use the charset (possibly incorrectly) specified
by the server.

- That some servers will not send a charset parameter when the author would wish
one to be sent is also not an argument for not having a default charset; indeed,
it is a very strong argument for having one, given the very wide prevalence of
documents encoded in 8859-1 (or compatible encodings such as US-ASCII), in part
becaue earlier versions of HTML (e.g.
http://www.w3.org/MarkUp/html-spec/html-spec_6.html#SEC6) assumed ISO-8859-1.
Purists may object to arguments drawn from widespread practice; this is also
fine, such people will also ignore the "lots of servers get it wrong" argument
for the same reasons.

- RFC2616 doesn't "recommend" ISO-8859-1 as a default charset (specifically, it
does not use the word "RECOMMENDED"), it requires it by defining the absence of
a charset parameter to mean ISO-8859-1.

- As one of the many media types that HTTP delivers, HTML has no business
attempting to redefine the behaviour of HTTP.

Whilst I do not ordinarily advocate picking and choosing elements of a standard
to implement, the decision by the authors of the HTML4 specification to include
a requirement that directly contradicts a requirement already present in the
HTTP1.1 specification forces this choice upon user-agent authors, such as the
Mozilla organisation. This alone may be reason enough for some to simply
disregard this part of the HTML4 specification; the flaws listed above provide
an even stronger basis for doing so.

All of that said, a counter-argument may have been made along the lines that
having a default charset would work against the use of sniffing in the first
place; not having seen this argument when it occurred, I'm not able to argue
either way in terms that address that argument, except to say that a document
containing a large number of characters in some encoding other than 8859-1 (e.g.
one of the Chinese encodings) might reasonably be expected to contain one or
more bytes in the range 0x80-0x9F inclusive, which documents encoded in 8859-1
cannot contain. Consequently it may make sense to behave in an RFC2616-compliant
fashion when possible (i.e. when a document is received without a specified
charset and whose encoded form does not contain bytes in the range 0x80-0x9F),
but to perform more elaborate sniffing when not possible (i.e. when no charset
is specified, but bytes in that range do appear).

Reproducible: Always

Steps to Reproduce:
Open http://spicetv.co.uk/bookit/event004893.htm
Actual Results:  
Mozilla replaces both the pound symbol and the digits following it with a single
small square that appears to contain the characters "FFFD" or "FFF0".

Expected Results:  
Mozilla should have displayed a pound symbol followed by some digits.

Comment 1

[Boris Zbarsky [:bzbarsky]]

intl.  You may want to turn on charset auto-detection if you want it to happen.
Failing that, we use the default encoding based on your system locale settings
and your default encoding setting in Navigator preferences.

As for the W3C, this is an instance of a higher-level protocol explicitly
overriding a transport-level protocol -- something that must be allowed to
happen and is explicitly allowed in the HTTP specification....

Comment 2

[Simon Montagu :smontagu]

Roland, did you have charset auto-detection turned on? I found that with
Universal auto-detection on the page was detected as Simplified Chinese
(GB18030) but with auto-detection turned off it was rendered in the default
encoding, which for me is ISO-8859-1.

This looks like a dupe of bug 181344.

Comment 3

[Robert Pollak]

It seems the testpage has disappeared.

Comment 4

[Jörg Hartmann]

This is _not_ related to bug 181344 (but, among others, to bug 210289).

That said, this is _not_ about what Mozilla does when set to "Auto-detect: 
Universal" since this is not Mozilla's default setting. Instead this bug is 
about Mozilla's default behaviour (auto-detection of charsets switched off) 
which is simply strange since Mozilla _does_ chose charsets somehow 
"automatically" (among others, dependent of the charsets used in pages viewed 
before). But it should simply use the default which is ISO-8859-1 (see HTTP's 
spec).

And to be precise: No, it should _not_ display in, e.g., Chinese encodings even 
when range 0x80-0x9F is used when auto-detect is _switched off_. ...

Comment 5

[Roland Turner]

{{ Clearing up stuff that I've not touched for a while, 8 months in this case... }}

My intent in raising this bug was to make an argument that appeared to have been
ignored (or not made) in the hope that doing so might help improve the approach
that Mozilla takes to handling non-US-ASCII characters on the web. I realise on
re-reading that I perhaps did not draw attention to the defect in the HTML spec
clearly enough.

A non-sequiter is generally a false statement of the form "A therefore B" where
A does not actually imply B. In this case, the fact that some servers supress
charset reporting does not mean that user-agents should not have some default
charset (or means of selecting a charset in the default case, e.g.
autodetection) in the same sense that a small child's failure to wait until
you've driven past before stepping onto the street doesn't mean that you should
accelerate; "common" sense dictates that you should make a best effort to
compensate for less-than-perfect behaviour of those around you. From this
standpoint, the HTML spec is simply wrong in that in specifies a behaviour based
upon a false premise (the non-sequiter that wrong behaviour by some web servers
means that all user agents should refuse to compensate). Additionally, and
perhaps related, it incorrectly identifies HTTP's requirement that unspecified
character encodings as a recommendation rather than a requirement.

Settings to make the bug appear: as the original page has dissappeared (and as
newer pages on that site do specify their charset in the HTML header) I am no
longer able to see the bug, however over the last several months I have seen
many 8859-1 pages which contained the British Pound symbol (0xA3) make this same
mistake (the pound and the first few digits following it are eaten by the
unknown-character symbol) it is only from fiddling over the last hour that I've
realised that auto-detection's being set to Universal was not Mozilla's default.
I've switched it to (Off) and will no doubt notice if I continue to get wrong
auto-detections over the next couple of months.

(N.B. Other bug reports on auto-detection suggest that the auto-detection
carries some context between successive pages. If this is so, then bug
reproduction has been made far more difficult.)

There's a related usability bug here. The View menu contains an undifferentiated
mix of transient and persistent configuration changes, I had no idea that some
changes made in this menu (particularly to the Character Coding submenu) were
persistent. When I reported this bug in the first place, I thought that I was
reporting a bug in Mozilla's default behaviour, it now appears that the bug is
in the behaviour of the Universal auto-detection which had been unintentionally
turned on at some stage, perhaps several years ago in some earlier version of
Mozilla.

Boris wrote:

> As for the W3C, this is an instance of a higher-level protocol
> explicitly overriding a transport-level protocol -- something
> that must be allowed to happen and is explicitly allowed in
> the HTTP specification....

Well, this looks to me like an example of a higher-level protocol violating a
lower-level protocol, something which most people would consider unacceptable.
Would you seriously advocate e.g. HTTP mandating a non-compatible interpretation
of TCP sequence numbers? I suspect that most people would consider such a
requirement to be unreasonable. N.B. This is different to a higher-level
protocol _extending_ (in a compatible fashion) an extensible lower-level
protocol. In this instance, the HTML protocol violates the HTTP protocol, and
does so in the context both of a false statement about the violated requirement
and of a non-sequiter to justify it.

Comment 6

[Mike Cowperthwaite]

Since comment 5 indicates that AutoDetect *was* turned on at the time the pages 
failed, and comment 2 indicates that the resultant language setting was GB18030, 
this very likely *is* a dupe of bug 181344.  Comment 4 assumed, apparently 
incorrectly, that AutoDetect was off.

Roland Turner, if you disagree, please reopen and supply a new URL to a page 
that exhibits the same rendering problem but a different setting on the 
View|Encoding menu.  Bug 177505 contains a mail message with a number of pound 
signs in it that exhibits the problem (except that Mail/News fails to update the 
Encoding menu properly) -- and also a comment explaining exactly why the 
Universal Autodetector makes the mistake.

*** This bug has been marked as a duplicate of 181344 ***