1497037 - Default Firefox encoding should be Unicode (UTF-8) and not Western

Status: RESOLVED WONTFIX

(Core :: Internationalization, defect, P5)

Description

[Sorin Sbarnea]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:62.0) Gecko/20100101 Firefox/62.0

Steps to reproduce:

Open any plain text file having UTF-8 characters file in Firefox and you will observe that Firefox will not display it correctly unless you change default encoding to Unicode.

We are 2018, not 2008... and default browser settings should be one that suits most users and that is not biased against a specific demography. 

Example file with UTF-8 content (look for npm section): http://logs.openstack.org/04/607504/14/check/openstack-tox-linters-npm/df6f3e8/job-output.txt.gz

Funny this has nothing to do with non-English content, as you can see that it affects even pure english content.


Actual results:

Garbage characters are displayed.


Expected results:

Have Unicode (UTF-8) default encoding on Firefox.

Comment 1

[Makoto Kato [:m_kato]]

I guess that this depends on settings for of auto encoding detection.  When using my environment, it can show correctly with auto detection.

When using Edge or Chrome, encoding detection is failed, so character corruption occurs.

Comment 2

[Sorin Sbarnea]

The only user configurable option related to Encodings does not allow picking Unicode as it only allows "locale" selection. Check the screenshot I added.

Now regardin environment, in my case, this is a MacOS 10.14 (Mojave) using English (UK) locale settings. Still, Firefox decides to display the above file as US-ASCII because server does not return any encoding the the message headers.

This is visible in Inspect console as this message appears:

The character encoding of the HTML document was not declared. The document will render with garbled text in some browser configurations if the document contains characters from outside the US-ASCII range. The character encoding of the page must be declared in the document or in the transfer protocol. job-output.txt.gz

If the default behaviour would be to use UTF-8, this issue could have being avoided.

UTF-8 is a superset of US-ASCII (7bit) which means that any US-ASCII test is rendered corrected if loaded as UTF-7, making it as a poor default value.

Comment 3

[Sorin Sbarnea]

Attachment: firefox-encoding-preferences.png

The locale dropdown has few options but there is no "Unicode" one among them, making it useless for changing browser config.

Comment 4

[Sorin Sbarnea]

Maybe it would also worth mentioning the popularity graph of different encodings from https://en.wikipedia.org/wiki/UTF-8 -- Based on this I really see no reason for not using UTF-8 as fallback encoding.

Comment 5

[RH]

I have also experienced that a text/plain file without encoding is by default always viewed as "Western" (Firefox 78.0.2), although UTF-8 is a much better default choice nowadays.

Please change the default to UTF-8.

Comment 6

[Henri Sivonen (:hsivonen)]

We no longer have the concept of configurable fallback encoding. See also https://hsivonen.fi/utf-8-detection/