Switch from windows-1251 to UTF-8 for default Ukrainian encoding
Categories
(Core :: Internationalization, enhancement, P3)
Tracking
()
People
(Reporter: serhiy.int, Unassigned)
Details
User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0
Steps to reproduce:
Opening any static text file with non-English texts.
Actual results:
They default to windows-1251 encoding but all of the ones I've seen in years are in UTF-8 and I have to change encoding manually to make them readable. I haven't seen a site that haven't switched to UTF-8 yet in a really long time.
Expected results:
Default to UTF-8, it's the new norm.
|
||
Comment 1•6 years ago
•
|
||
Moving back, this has nothing to do with localization in a specific language (Mozilla Localizations is a product to track translation issues).
|
|
||
Comment 3•6 years ago
|
||
Interesting. There is indeed one key that is localizable, but it's already set to the correct value (ukprob).
# LOCALIZATION NOTE (intl.charset.detector):
# This preference controls the initial setting for the character encoding
# detector. Valid values are ja_parallel_state_machine for Japanese, ruprob
# for Russian and ukprob for Ukrainian and the empty string to turn detection
# off. The value must be empty for locales other than Japanese, Russian and
# Ukrainian.
intl.charset.detector=
intl.charset.default was removed in bug 910192. As far as I understand it was replaced by this file, but I have no clue of the impact of changing the default there.
Trying with Core::Internationalization, feel free to move if there's a better place for this.
So I have to change the HTML standard itself to change default encoding. Ok. Is there a way to override this? I tried clearing intl.charset.detector, setting intl.charset.fallback.override to "UTF-8" and clearing intl.fallbackCharsetList.ISO-8859-1 but it didn't help.
|
||
Updated•6 years ago
|
(In reply to Сергій from comment #0)
Opening any static text file with non-English texts.
If you open a local (file: URL) UTF-8 file that lacks an encoding declaration, Firefox opens it as UTF-8. (Please let me know if not.) For non-file: URLs, UTF-8 needs to be declared. In the non-file: case, the whole file isn't available when we want to start processing it, so we can't do whole-file detection up front and partial-content detection of UTF-8 would make the platform more brittle when not only considering Ukrainian but also languages like English, Dutch, and even German. Using UTF-8 unconditionally would break legacy content (which exists)--just like using the standards mode unconditionally would. Therefore, both use opt-in mechanisms.
I'll write a blog post explaining the issues in more detail.
(In reply to Сергій from comment #4)
Is there a way to override this? I tried clearing intl.charset.detector, setting intl.charset.fallback.override to "UTF-8" and clearing intl.fallbackCharsetList.ISO-8859-1 but it didn't help.
These prefs are no longer used as of Firefox 73. Removing these from the codebase is bug 1603712.
Furthermore, back when intl.charset.fallback.override still was used, it deliberately ignored the value UTF-8 to prevent people from setting it to UTF-8, relying on it when creating HTML, and publishing content broken for everyone else.
(In reply to Henri Sivonen (:hsivonen) from comment #5)
I'll write a blog post explaining the issues in more detail.
I don't really care about autodetection, what I'd like is a way to opt-in for UTF-8 being applied everywhere. For me, the sites that do not use UTF-8 are irrelevant and I don't think I use any of them. But because of this legacy (that should be deprecated at least to not be default IMO) I'm frustrated every time I have to open a UTF-8 text file. Like patches on drupal.org.
https://www.drupal.org/project/drupal/issues/3169212
It makes UX of using Firefox miserable and I can't even tune it.
(In reply to Serhiy from comment #7)
I don't really care about autodetection, what I'd like is a way to opt-in for UTF-8 being applied everywhere. For me, the sites that do not use UTF-8 are irrelevant and I don't think I use any of them. But because of this legacy (that should be deprecated at least to not be default IMO) I'm frustrated every time I have to open a UTF-8 text file.
The default is not changing, but I intend to re-evaluate whether to enable UTF-8 autodetection for text/plain in April.
Like patches on drupal.org.
https://www.drupal.org/project/drupal/issues/3169212
It makes UX of using Firefox miserable and I can't even tune it.
AFAICT, the patches look equally bad in Chrome. Please file a bug on drupal.org to ask them to set ; charset=utf-8
Thank you! Found the issue as already reported here https://www.drupal.org/project/infrastructure/issues/2922638 and followed it.
(In reply to Henri Sivonen (:hsivonen) from comment #8)
The default is not changing, but I intend to re-evaluate whether to enable UTF-8 autodetection for
text/plainin April.
Unfortunately, the telemetry results were unexpected to the point of not making sense, so I adjusted the telemetry collection in bug 1702248 and it will be another four weeks to get results from that. This in now tracked as bug 1706861.
Description
•