1335761 - [prod] Studio component does not parse UTF encoding correctly

Status: RESOLVED WORKSFORME

(support.mozilla.org :: Lithium Migration, task)

Description

[vesper]

Attachment: Community_Settings_-_Mozilla_Support_Community_-_2017-02-01_17.04.23.png

(See attached screenshot)

When I try to add the strings below through Studio (formatted in UTF-8), they get incorrectly parsed, resulting in "?" instead of correct characters. This means that adding strings with more "exotic" character sets may not work properly - or may not be showing properly on Studio's side.

Is there a quick workaround for this or does that require changes to Studio's code?

profile.language.ar.title                          = عربي
profile.language.bg.title                          = Български
profile.language.ca.title                          = català
profile.language.cs.title                          = Čeština
profile.language.da.title                          = Dansk
profile.language.de.title                          = Deutsch
profile.language.el.title                          = Ελληνικά
profile.language.en-gb.title                       = English (UK)
profile.language.en.title                          = English (US)
profile.language.es-mx.title                       = Español (419)
profile.language.es.title                          = Español
profile.language.fi.title                          = suomi
profile.language.fr.title                          = Français
profile.language.hr.title                          = Hrvatski
profile.language.hu.title                          = Magyar
profile.language.in.title                          = Bahasa Indonesia
profile.language.it.title                          = Italiano
profile.language.iw.title                          = Hebrew
profile.language.ja.title                          = 日本語
profile.language.ko.title                          = 한국어
profile.language.lv.title                          = Latvian
profile.language.ms.title                          = Malay
profile.language.nl.title                          = Nederlands
profile.language.no.title                          = Norsk
profile.language.pl.title                          = polski
profile.language.pt-br.title                       = Português (do Brasil)
profile.language.pt.title                          = Português (Europeu)
profile.language.ro.title                          = română
profile.language.ru.title                          = Русский
profile.language.sk.title                          = slovenčina
profile.language.sq.title                          = Shqip
profile.language.sv.title                          = Svenska
profile.language.th.title                          = ไทย
profile.language.tr.title                          = Türkçe
profile.language.uk.title                          = Українська
profile.language.vi.title                          = Tiếng Việt
profile.language.zh-CN.title                       = 中文 (简体)
profile.language.zh-TW.title                       = 正體中文 (繁體)

Comment 1

[scott.riley]

Chavi and Tyson both tried this out and it looks to be working w/straight copy and paste.  Working also w/ISO 8859-1 as well.  Is it possible that the source content is not UTF?

Comment 2

[vesper]

Update: tried again on my side, and it would not work. I keep getting "???"s for the likes of 正體中文 (繁體) or slovenčina.

Any chance you could take the list of the articles from the top message here and propagate it to all installed locale packages?

If not, I may need help in getting the right formatting sorted out.

Comment 3

[Cor vL (cor-el)]

Hi Michał

I see all character in your above posts.
Do you see them in your post here on Bugzilla as well?

You probably have a problem with font support for those languages or with the font that is used.
You would have to check that.
ja: 日本語: x65e5;x672c;x8a9e;
ko: 한국어: xd55c;xad6d;xc5b4;
sk: č: x10d;
zh-CN: 中文 (简体): x4e2d;x6587;x20;x28;x7b80;x4f53;x29;
zh-TW: 正體中文 (繁體): x6b63;x9ad4;x4e2d;x6587;x20;x28;x7e41;x9ad4;x29;

https://en.wikipedia.org/wiki/Plane_%28Unicode%29

You can check the font used for selected text in the Font tab in the right pane of the Inspector.
Set devtools.fontinspector.enable to true on the about:config page to enabled the Font tab.
https://developer.mozilla.org/Tools/Page_Inspector/How_to/View_fonts

Comment 4

[vesper]

I can see the text here correctly as well (displaying all special characters).

On Bugzilla, the font shown by the Inspector in the site's CSS is defined as: "Droid Sans Mono", Menlo, Monaco, "Courier New", Courier, monospace;

On production, for the list elements in the Other languages component it's defined as: "Open Sans",Arial,Helvetica,sans-serif;

If I am the only one seeing this, we can probably disregard this bug. I do have quite a few add-ons that could affect site display.

Comment 5

[vesper]

More info on this (while I poke around): German seems to be showing (and parsing) it OK: 

https://hwsfp35778.lithium.com/t5/Einstellungen-und-Add-ons/Die-Sucheinstellungen-von-Firefox-anpassen/ta-p/39227

I checked in the Text Editor in Studio and the German strings look OK.

Comment 6

[madalina]

We're going to hide it for now.